{"assignment":{"_schema_version":2,"course_id":37,"date_created":"2025-08-17T17:34:15.065132+00:00","date_modified":"2026-02-06T19:37:20.204495+00:00","extra_instructor_files":"","extra_starting_files":"","forked_id":null,"forked_version":null,"hidden":false,"id":2469,"instructions":"## More Testing\n\n-   Find problems earlier.\n-   Make it easier to change things later.\n-   Have more confidence in gluing together code.\n\nTesting is a crucial part of software development.\nThey help you identify errors in code, improve code quality, and ensure that your code behaves as expected as you make changes around the code.\nOften, you should write tests before you even begin writing a function, to make sure that you understand what the function should do.\nOf course, the reality is that many programmers skip testing entirely, leading to bugs and issues down the line.\nMost legacy codebases are not very well tested, making it difficult to change or refactor the code without introducing new bugs.\n\nThey say that the best time to plant a tree is twenty years ago. The second best time is now.\nWriting tests for existing code is a worthwhile process, and will help us identify bugs and make it easier to make changes down the line.\n\n## Testing for Coverage\n\n```python\nfrom bakery import assert_equal\n\ndef reveal_letter(secret: str, guesses: list[str], index: int) -> str:\n    if secret[index] in guesses:\n        return secret[index]\n    return \"_\"\n\n# The first character IS in the guesses\nassert_equal(reveal_letter(\"heat\", [\"h\"], 0), \"h\")\n# The second character is NOT in the guesses\nassert_equal(reveal_letter(\"heat\", [\"h\"], 1), \"_\")\n```\n\nA good starting point for writing tests is to at least make sure that your code has full code coverage.\nRecall that code coverage is a measure of how much of your code is executed while running your tests. The goal is to have as much of your code covered by tests as possible, ideally 100%.\nThis does not ensure that your tests are thoroughly validating the behavior of your code, but it is a good first step.\n\nIn the `reveal_letter` function shown here, the `if` statement has two paths: one for when the letter is in the guesses, and one for when it is not.\nTherefore, at the minimum, we can write a test for each path.\nMore thorough tests would check for situations like having an empty list of guesses, having multiple letters in the guesses, and for different possible indices.\n\nWe'd also want to consider whether it is worth testing the behavior when the index is out of bounds.\nDepending on how the function is used and documented, those inputs might be considered invalid, and the function could be modified to handle them more gracefully.\n\n## Branch Coverage\n\n```python\nfrom bakery import assert_equal\n\ndef is_loss(wrong: int) -> bool:\n    return wrong >= 6\n\nassert_equal(is_loss(5), False)\nassert_equal(is_loss(6), True)\nassert_equal(is_loss(7), True)\n```\n\nA more subtle consideration of coverage is **branch coverage**, which ensures that all branches of conditional statements are tested.\nIn the `is_loss` function, there are two branches: one for when the wrong count is less than 6, and one for when it is 6 or greater.\nTo achieve branch coverage, we need to test both branches.\nThis example is meant to highlight that code branches because of conditional expressions, not because of `if` statements (although `if` statements are a common way to introduce conditional expressions, and allow more complex branching).\n\n## Valid Tests\n\n```python check-if-valid-happy-paths\nfrom bakery import assert_equal\nfrom string import ascii_letters\n\ndef check_if_valid_guess(current_guess: str, past_guesses: list[str]) -> bool:\n    is_single_letter = not len(current_guess) > 1\n    is_letter = current_guess in ascii_letters\n    is_not_repeated = current_guess not in past_guesses\n    return is_single_letter and is_letter and is_not_repeated\n\nassert_equal(check_if_valid_guess(\"h\", []), True)\nassert_equal(check_if_valid_guess(\"H\", []), False)\nassert_equal(check_if_valid_guess(\"1\", []), False)\nassert_equal(check_if_valid_guess(\"hello\", []), False)\nassert_equal(check_if_valid_guess(\"oH\", []), False)\nassert_equal(check_if_valid_guess(\"!\", []), False)\nassert_equal(check_if_valid_guess(\"h\", [\"h\"]), False)\nassert_equal(check_if_valid_guess(\"h\", [\"x\", \"y\", \"h\"]), False)\nassert_equal(check_if_valid_guess(\"z\", [\"x\", \"y\", \"h\"]), True)\n```\n\nA more complicated example of non-`if` branching is in the `check_if_valid_guess` function, which has three conditional expressions combined with `and`.\nThere are actually `2^3 = 8` possible outcomes for the combination of these conditions: all `True`, all `False`, the cases where only one of them is `True`, and the cases where two of them are `True`.\nHowever, there are no `if` statements.\n\nThis leaves us with a few different ways of thinking about code coverage.\nA single test would technically execute every single code pathway, achieving 100% code coverage.\nWe would only need two tests to achieve every possible outcome.\nA minimum of 8 tests would be needed to cover all the possible value combinations of the conditions.\nBut all of that would still ignore that the individual conditional expressions all have their own interesting cases.\n\nHere we have written 9 tests, sampling a large number of different scenarios that might occur in the game.\nYou should be able to run them yourself.\nAll of these tests are valid, but are they thorough enough?\n\n## Edge Cases\n\n```python check-if-valid-happy-paths\nfrom bakery import assert_equal\nfrom string import ascii_letters\n\ndef check_if_valid_guess(current_guess: str, past_guesses: list[str]) -> bool:\n    is_single_letter = not len(current_guess) > 1\n    is_letter = current_guess in ascii_letters\n    is_not_repeated = current_guess not in past_guesses\n    return is_single_letter and is_letter and is_not_repeated\n\n# Edge case: empty guess\nassert_equal(check_if_valid_guess(\"\", []), False)\n\n# ... copy over existing tests too!\n```\n\nWhen writing tests, it's important to consider edge cases - scenarios that are not typical but could cause the application to behave unexpectedly.\nFor example, what happens if a user submits an empty guess?\nThe empty string is a valid input to the `check_if_valid_guess` function, even though it is not a valid guess.\nIf you run this implementation of the function, a very subtle bug is exposed: the empty string returns `True`!\nThe reason is that the empty string is not greater than a single character (it is zero characters), it is in the string of ASCII letters (because the empty string is in all strings), and it is not in the list of past guesses (because the list is empty).\nThe logic for `is_single_letter` needs to be modified, so that it explicitly checks that the string is just one character long, no more or less. An easy fix is to change the condition to `len(current_guess) == 1`.\n\nNote that we did not copy over the existing tests. This does not mean that you would delete the other tests.\nAs long as tests are valid, there is no reason to discard them.\nAlways look towards adding new additional tests, not replacing existing ones.\n\n## Testing More Complex State\n\n```python\nfrom drafter import *\nfrom bakery import assert_equal\n\n# ...\n\ninitial_state = State(\"\", [], 0, 0, -1, 0)\n\nafter_index = index(initial_state).state\nassert_equal(after_index, State(\"\", [], 0, 0, -1, 0))\n\nlevel_1 = next_level(after_index).state\nassert_equal(level_1, State(\"heat\", [], 0, 0, 0, 0))\n\nfirst_guess = submit_guess(level_1, \"x\").state\nassert_equal(first_guess, State(\"heat\", [\"x\"], 1, 1, 0, 0))\n```\n\nWriting tests for pieces of the codebase will help debug smaller parts, but what about testing the entire application?\nThis is where **Integration Testing** comes in.\nIntegration testing focuses on verifying the interactions between different components of the application to ensure they work together as expected.\n\nWe can use our route functions to simulate actually playing the game.\nAt each step, we capture the `state` instead of the entire `Page` content by assigning the result of the route functions to temporary variables, along with some simple attribute access.\nThis allows us to give symbolic names to the different states in the game, and also allows us to reuse them between the `assert_equal` tests and the subsequent route function call.\nWe begin with the same initial state passed into the `start_server` call.\nThen, we call the `index`, `next_level`, and `submit_guess` functions, testing the state at each step.\nYou can see from the second argument of `assert_equal` how the state evolves over time as we play the game.\nAlso note how we pass a second argument to the `submit_guess` function to simulate the user's guess.\n\n## Testing Critical Points\n\n```python\nclose_to_heat = State(\"heat\", [\"h\", \"e\", \"a\"], 3, 0, 0, 0)\n\nwon_level = submit_guess(close_to_heat, \"t\").state\n# Score increases too\nassert_equal(won_level, State(\"heat\", [\"h\", \"e\", \"a\", \"t\"], 4, 0, 1, 1))\n\nlevel_2 = next_level(won_level).state\nassert_equal(level_2, State(\"dogs\", [], 4, 0, 0, 1))\n# FAILURE, predicted answer was\n#    State(\"dogs\", [], 4, 0, 1, 1)\n# Computed answer was\n#    State(\"dogs\", [\"h\", \"e\", \"a\", \"t\"], 4, 0, 1, 1)\n```\n\nIf you try to exhaustively model every possible state of the application, you may quickly find that the number of possible states is too large to test all of them.\nIn these cases, it's important to identify and test the most critical points in the application - the states that are most likely to cause issues or that are most important to the application's functionality.\n\nIn this example, we show a test for the transition from level 1 to level 2.\nThis exposes a mistake in the game's `next_level` logic, which is not properly resetting the guessed letters.\nThe correct test demonstrates that the guessed letters list should be empty, but if you ran our application at this point, you would see the shown message that the old letters were retained.\n\n## Drafter Tests\n\n```python drafter-tests\nassert_equal(submit_guess(State('heat', [], 0, 0, 0), 'h'),\n             Page(State('heat', ['h'], 0, 0, 0), [\n                Header('Level 0'),\n                \"Word: h___\",\n                \"Guesses so far: \",\n                BulletedList(['h']),\n                \"Guess a letter:\",\n                TextBox('guess'),\n                Button('Submit', 'submit_guess')\n             ]))\n\n# This also works!\n\nguess_page = submit_guess(State('heat', [], 0, 0, 0), 'h')\nassert_equal(guess_page.content[0], Header('Level 0'))\nassert_equal(guess_page.content[1], \"Word: h___\")\nassert_equal(guess_page.content[2], \"Guesses so far: \")\nassert_equal(guess_page.content[3], BulletedList(['h']))\nassert_equal(guess_page.content[4], \"Guess a letter:\")\nassert_equal(guess_page.content[5], TextBox('guess'))\nassert_equal(guess_page.content[6], Button('Submit', 'submit_guess'))\n```\n\nDrafter doesn't just support testing state, you can also test the entire `Page` object, including its content.\nWe saw previously how this could be a little tedious, since you end up writing a lot of boilerplate code to set up the expected `Page` content for each test.\nYou can apply a similar trick to accessing the `Page` content in your tests that we saw with our integration tests.\nIn the alternate version of the tests shown below, we captured the result of `submit_guess` and then check each element of the `Page` content against the expected values.\nIf you were not interested in testing certain visual aspects of the `Page`, you could simply omit those checks.\n\nWriting tests for your Drafter routes is a great way to ensure that your application behaves as expected, without having to manually run your application every time you make a change.\nAlthough this can be a bit painful up front as you have to figure out what the expected State and Page Content should be for each route, the benefits in terms of catching bugs and ensuring correctness are well worth the effort.\n\n## Automatic Drafter Tests\n\n![A screenshot of the Drafter debug area, demonstrating how you can access automatically generated tests](bakery_projects_testing_drafter_tests.png)\n\nA very nifty feature of Drafter is that interacting with the application generates tests automatically.\nIf you scroll down to the `Page Load History` section of its debug area, then you will see all of the routes you have visited in the application so far (starting with `index`).\nAs you click new pages, the list of visited routes will update to reflect your navigation.\nEven better, if you expand the `Page Content` section, then it will show you the exact content that was rendered for each route as a prewritten `assert_equal` test.\nIf you believe that the behavior is correct, then you can copy and paste this test into your test suite.\nOr, if it is at least close, then you can use it as a starting point for your own tests.\n\nA major downside of these automatically generated tests is that Drafter will be more verbose and include more implementation details than you might want in your test suite.\nMost of these details can be easily ignored, and you will eventually learn what is necessary to include in your tests and what can be left out.\nFor now, you can mostly just ignore the extra output, and focus on the recognizable parts of the test code.\n\n## Brittle Tests\n\n```python brittle-test\nassert_equal(submit_guess(State('heat', [], 0, 0, 0), 'h'),\n             Page(State('heat', ['h'], 0, 0, 0), [\n                Header('Level 0'),\n                \"Word: h___\",\n                # What if you removed the colon?\n                # All your tests would be invalid!\n                \"Guesses so far: \",\n                BulletedList(['h']),\n                \"Guess a letter:\",\n                TextBox('guess'),\n                Button('Submit', 'submit_guess')\n             ]))\n```\n\nAutomatic tests generated by Drafter can be a great starting point, but they can also be brittle.\nIf you make changes to your application that affect the `Page` content, then these tests may break, even if the overall behavior of the application is still correct.\nThis is because the tests are checking for specific content that may have changed, rather than checking for the overall functionality of the application.\nWhen you have a test that is too brittle, it can be more of a hindrance than a help, since it will require you to update the test every time you make a change to the application, even if the change is not related to the functionality being tested.\n\nIdeally, you want to write tests that are robust and flexible enough to accommodate changes in the application's implementation, while still ensuring that the core functionality is correct.\nThis often involves finding the right balance between testing specific details and testing overall behavior, and may require some trial and error to find the right level of abstraction for your tests.\n\n## Summary\n\n-   At the minimum, make sure you are achieving 100% code coverage.\n-   When dealing with conditions, consider all the possible outcomes.\n-   Never delete valid tests, just focus on adding more thorough tests.\n-   Test both the happy path and edge cases.\n-   Think critically about key parts of your application that might need more thorough testing.\n-   Drafter provides a powerful way to test your application's state and behavior.\n-   Integration testing is where you test how different parts of your application work together.\n-   You can write tests for the entire `Page` object, or for individual `state` and `content` elements.\n-   Drafter's automatic test generation can save time by creating tests from your interactions, but may include unnecessary details.\n-   Be cautious of brittle tests that may break with minor changes to the application's content, and strive for a balance between specificity and flexibility in your tests.","ip_ranges":"","name":"5B3) More Testing","on_change":"","on_eval":"","on_run":"","owner_id":1,"owner_id__email":"acbart@udel.edu","points":1,"public":true,"reviewed":false,"sample_submissions":[],"settings":"{\n  \"small_layout\": true,\n  \"header\": \"More Testing\"\n}","starting_code":"","subordinate":false,"tags":[],"type":"reading","url":"bakery_projects_more_testing_read","version":4},"ip":"216.73.216.157","submission":{"_schema_version":3,"assignment_id":2469,"assignment_version":4,"attempts":0,"code":"","correct":false,"course_id":37,"date_created":"2026-05-20T12:38:52.597931+00:00","date_due":"","date_graded":"","date_locked":"","date_modified":"2026-05-20T12:38:52.597931+00:00","date_started":"","date_submitted":"","endpoint":"","extra_files":"","feedback":"","grading_status":"NotReady","id":2036727,"score":0.0,"submission_status":"Started","time_limit":"","url":"submission_url-adbdf830-0deb-4c95-9ab6-899256c9480a","user_id":2044658,"user_id__email":"","version":0},"success":true}