7.4 Testing code

Code that performs a program's intended task is application code. For example, the functions in the time-calculation module calculate travel time and convert its units. Application code used in a system by its intended users is often called production code.

Test code calls that code with selected inputs and compares the observed results with the required results.

You have already traced algorithms and compared expected and observed results. An automated test records that comparison in executable code.

For the time calculator, util.calculate_time_h(100, 50) should return 2.0 hours. pytest finds and runs test functions so that you can repeat such checks after changing a module.

Tests are an essential part of software development. They help ensure that your code works as expected and that new changes do not introduce bugs or regressions. In this tutorial, you will learn how to write and run tests in Python using the pytest testing framework.

The claim "it works on my computer" leaves the tested behavior unspecified. A test records an input and its expected result so another person can repeat the check.

With pytest, you can run those checks automatically and inspect failures. Passing tests provide evidence for the cases checked.

A virtual environment keeps a project's installed packages separate.

Sharing the code, tests, Python version, and package versions lets a teammate recreate the setup and repeat the checks on their computer. The pytest getting-started guide shows how assertions specify expected results and how the runner reports failures.

7.4.1 Why write tests?

Writing tests is an important practice in software development for several reasons:

Quality assurance: Tests help ensure that your code works as expected and that new changes do not introduce defects or regressions. They provide a safety net that allows you to make changes with confidence.
Documentation: Tests serve as documentation for your code by specifying the expected behavior of your functions and modules. They help you understand how your code should work and how it should be used.
Refactoring: Tests let you carry out refactoring with confidence, by holding the external behavior fixed while the structure changes. They allow you to make changes to your code without worrying about introducing bugs.
Debugging: Tests help you identify and diagnose issues in your code by providing a way to reproduce and isolate problems. They help you narrow down the cause of the issue and fix it quickly.
Collaboration: Tests help improve collaboration among team members by providing a common language to discuss the expected behavior of the code. They help ensure that everyone is on the same page and understands how the code should work.
Continuous integration: Tests are an essential part of continuous integration and continuous deployment (CI/CD) pipelines. They help automate the testing process and ensure that your code is tested automatically before deployment.
Confidence: Tests give you confidence in your code by providing a safety net that allows you to make changes without fear of breaking existing functionality. They help you write better code and improve the quality of your software.

Tests are meant to be run frequently during development to ensure that your code works as expected. They help you catch bugs early and fix them before they become more difficult to diagnose and resolve. By writing tests, you can improve the quality of your code and build more reliable software.

Figure 1 illustrates the uncertainty of relying on an unchecked program.

7.4.2 Boundary-value testing

Sometimes, testing the boundaries of your code is as important as testing the code itself. Boundary-value analysis is a testing technique that focuses on the boundaries of input values.

The idea is to test the minimum and maximum values of the input range, as well as the values just below and above the boundaries.

This helps identify issues related to boundary conditions and ensures that the code handles edge cases correctly. For example:

If a function accepts a range of values from 1 to 10, you would test the values 0, 1, 10, and 11 to ensure that the function handles the boundaries correctly.
If a function accepts a list of values, you would test an empty list, a list with one element, and a list with multiple elements to ensure that the function handles different input sizes. If the specification sets a maximum list length, test that boundary too.
If a function accepts a string, you would test an empty string, a string with one character, and a string with multiple characters to ensure that the function handles different string lengths.

7.4.3 Test-driven development

Test-driven development (TDD) is a technique that involves writing tests before writing the actual code.

The TDD cycle consists of three steps: writing a failing test, writing the code to make the test pass, and refactoring the code.

TDD helps you write better code by focusing on the expected behavior of your code and ensuring that it works as expected.

7.4.4 Black-box, functional, and unit testing

There are several types of testing that you can perform on your code to ensure its quality and reliability. We focus on tests at the intersection of these three types:

Black-box testing: Black-box testing is a testing technique that focuses on testing the functionality of the software without looking at the internal code. The idea is to test the software based on its input and output without knowing how it works internally. Black-box testing helps you identify issues related to the functionality of the software and ensures that it behaves as expected.
Functional testing: Functional testing is a testing technique that focuses on testing the functionality of the software by providing input and checking the output. The goal of functional testing is to verify that the software works as expected and that it meets the requirements. Functional testing describes what is checked: the specified behavior. It can be applied to an individual function or to a complete system.
Unit testing: Unit testing focuses on individual units or components of your code in isolation. Each check is one test case. A collection of executable tests is a test suite; a test plan describes what should be checked and how. The goal of unit testing is to verify that each unit of your code works correctly and that it behaves as expected. Unit tests are typically small, focused tests that test a single function or method and its behavior. They help you identify issues in your code early and ensure that each unit of your code works as expected.

7.4.5 Alpha and beta testing

Acceptance testing checks whether people can use a complete system for its intended purpose. Alpha and beta testing are forms of acceptance testing in the ISTQB testing syllabus.

Alpha testing takes place in the developer's test environment with potential users or independent testers. Beta testing takes place with potential or existing users in their own settings. The aim includes finding problems that appear during everyday use. Beta testing may follow alpha testing or take place without an earlier alpha test, as described in the ISTQB definitions of alpha and beta testing.

Software projects also use alpha and beta as prerelease labels. For Python packages, 1.0a1 identifies an alpha release and 1.0b1 a beta release in the Python packaging version scheme. A version label identifies the release category. Test records show which behavior was checked and what happened.

Automated tests can be repeated during development and when evaluating a release.

7.4.6 Check inputs and calculated results

Input validation checks whether supplied data meets the rules for a calculation. In the completion records, ordered_units is a positive order quantity. A known completed_units value must be between zero and that quantity.

The value None records an unknown completion quantity.

Result checking compares the program's output with an independently calculated expectation.

For a completion fraction, identify the eligible orders and calculate the completed-unit and ordered-unit totals before running the code.

The completion report's rule includes known zeros and excludes unknown quantities from both totals. Compare the fraction with those totals.

A test checks the implementation of the agreed rule; interpreting the result also requires checking that the rule suits the question being answered.

7.4.7 Writing tests in Python

Python provides several tools and libraries for writing tests, including the built-in assert statement and the pytest testing framework. You can use these tools to write tests for your Python code and ensure its quality and reliability.

Writing tests with pytest

You can use the pytest testing framework to test your Python code. pytest is a popular testing framework for Python that makes it easy to write simple tests. You can find more information about pytest in the official documentation.

pytest runs test functions that use assert to compare observed and expected results. You can write tests as regular Python functions and use assertions to check the expected results. To write tests with pytest, follow these steps:

Create a new Python file for your tests. You can name the file test_<module>.py, where <module> is the name of the module you want to test. For example, if you want to test a module called basic_math.py, you can name the test file test_basic_math.py. Alternatively, you can create a tests directory and place your test files inside it (e.g., tests/test_basic_math.py). A separate tests directory does not require an __init__.py file for ordinary pytest discovery.
Write test functions inside the test file. Test functions should start with test_ to be recognized by pytest as test functions. You can write multiple test functions to test different parts of your code. It is important to use test_ so that pytest can automatically discover and run the test functions. You can use separate test functions for distinct cases of the same function so that each result is reported independently.
Use the assert statement inside the test functions to check the expected results of your code. If the assertion fails, pytest will raise an AssertionError and mark the test as failed.
Run the tests using the pytest command in the terminal. pytest will automatically discover and run the test functions in the test file and display the results.

For example, Listing 1 is a small module basic_math.py holding two functions, add_two and divide.

Listing 1: The module basic_math.py, holding the two functions under test, add_two and divide.

# basic_math.py

def add_two(a, b):
    return a + b

def divide(a, b):
    return a / b

The tests for them go in test_basic_math.py, shown in Listing 2.

Listing 2: Example of a test file test_basic_math.py that contains test functions for the add_two and divide functions in the basic_math.py module.

# test_basic_math.py

import basic_math

def test_add_two():
    assert basic_math.add_two(1, 2) == 3

def test_divide():
    assert basic_math.divide(10, 2) == 5
    assert basic_math.divide(10, -2) == -5

In this example, we have two test functions: test_add_two and test_divide.

The test_add_two function checks the expected result of the add_two function, and the test_divide function checks the expected results of the divide function.

If the expected result does not match the actual result, pytest will raise an AssertionError and mark the test as failed.

A worked example: classify

A test function that you call yourself works, but you have to remember to call it, and it stops at the first assertion that fails.

A test runner does both jobs for you: it finds the tests, runs all of them, and reports every failure at once. This is how testing is done outside a course.

pytest is a Python test runner. It asks for almost nothing: put the tests in a file whose name starts with test_, name each test function test_..., and keep using the assert statement you already know.

The files

Split the work across two files in the same folder. The function under test goes in one, its tests in the other, and pytest is what connects them.

`classify.py`

The function on its own, with no test code and no call at the bottom.

See Listing 3.

Listing 3: The classify function returns Positive, Negative, or Zero for a number.

def classify(value: float) -> str:
    if value > 0:
        return "Positive"
    if value < 0:
        return "Negative"
    return "Zero"

`test_classify.py`

One test function per case, rather than one function holding every case. When a test fails, its name is what pytest reports, so the name should say what broke.

See Listing 4.

Listing 4: Three separate tests check positive, negative, and zero inputs.

from classify import classify


def test_positive():
    assert classify(5) == "Positive"


def test_negative():
    assert classify(-3) == "Negative"


def test_zero():
    assert classify(0) == "Zero"

The from classify import classify line is what makes the test file a separate program that uses your code, exactly as another part of a real project would. It also explains the naming: classify.py is the module, classify inside it is the function.

Terminal

Install the runner once, then run it from the folder holding both files. pytest discovers test_classify.py by its name, collects the three test_ functions, and runs them.

See Listing 5.

Listing 5: Install pytest and run the tests from the folder containing both Python files.

pip install pytest
pytest

Reading the result

A passing run reports the collected tests and the final count, as in Listing 6. Each dot is one test.

Listing 6: A passing pytest run. The three dots are the three test functions, and the header reports the Python and pytest versions the run used.

============================= test session starts =============================
platform win32 -- Python 3.14.6, pytest-8.4.2, pluggy-1.6.0
rootdir: C:\projects\classify
collected 3 items

test_classify.py ...                                                     [100%]

============================== 3 passed in 0.03s ==============================

Now suppose the boundary is wrong again and classify uses value <= 0 for its second condition. Listing 7 shows what pytest reports.

Listing 7: A failing pytest run. The failing test is named, the expected and actual values are shown as a difference, and the other two tests still ran.

collected 3 items

test_classify.py ..F                                                     [100%]

================================== FAILURES ===================================
__________________________________ test_zero __________________________________

    def test_zero():
>       assert classify(0) == "Zero"
E       AssertionError: assert 'Negative' == 'Zero'
E         
E         - Zero
E         + Negative

test_classify.py:13: AssertionError
=========================== short test summary info ===========================
FAILED test_classify.py::test_zero - AssertionError: assert 'Negative' == 'Zero'
========================= 1 failed, 2 passed in 0.10s =========================

Three things here are worth more than the convenience. The report names test_zero, so you know which case broke without reading the code.

pytest rewrites the assertion to show both sides as a difference, - Zero against + Negative, without you having to write a failure message at all.

And the run did not stop: the two passing tests still ran, so you see the full state of the code rather than only its first problem.

Other runners, and other kinds of testing

pytest is a choice, not the only one. Python ships with unittest in its standard library, which needs no installation and organizes tests into classes. Every language has its own: JUnit for Java, Jest for JavaScript, RSpec for Ruby. The idea travels; only the syntax changes.

The tests in this chapter were written after the code. Some developers prefer test-driven development, where the test is written first and fails, and the code is then written to make it pass.

Working that way forces you to decide what the function should do before deciding how it will do it.

What this chapter covers is the smallest and most common kind of test, a unit test, which checks one function in isolation.

Real projects also test how parts work together, how a system behaves under load, and whether it still does what the people who asked for it wanted.

If this interests you, the pytest documentation is a practical next step, and the course materials listed in the materials library go further into testing than this chapter does.

How to run tests with pytest

Use the Python environment selected for the course. If pip or pytest refers to another environment, use python -m pip install pytest and python -m pytest to run both tools with the same interpreter.

To run the tests, follow these steps:

Open the terminal in Visual Studio Code by clicking Terminal > New Terminal or using the keyboard shortcut Ctrl + `.
Run the following command to install the pytest package:

See Listing 8.
Listing 8: Install pytest in the selected project environment.
pip install pytest
Run the tests from the folder containing your test files with Listing 9.
Listing 9: Run the tests from the project's test folder.
pytest
Verify that the tests pass. A . (dot) marks a passed test, and an F marks a failed test.
If a test fails, review the error message to identify the issue in your code. The error message will provide information about the failed test and the expected result. For example, if the test expects the result to be True but the actual result is False, or if the test expects the result to be 5 but the actual result is 10, you will need to review your code to identify the issue.
Make the necessary changes to your code to fix the issue.
Run the tests again to verify that the issue has been resolved. Repeating the failed test after the correction is a confirmation test; running the ones that already passed checks for a regression in those cases.

How to inspect pytest results

When you run pytest, you will see a summary of the test results at the end of the output. The summary shows how many tests passed, failed, or were skipped.

The per-file lines report test verdicts and progress; failure details compare a test case's observed result with its expected result.

First, you will see a summary of the test results:

See Listing 10.

Listing 10: pytest progress for four tests across three files; F marks a failure and a dot marks a pass.

============================= test session starts ==============================
platform linux -- Python 3.13.0, pytest-7.0.1, pluggy-1.0.0 -- /usr/bin/python3

collected 4 items

tests/test_exercise_1.py FF [50%]
tests/test_exercise_2.py .  [75%]
tests/test_exercise_3.py F  [100%]

=================================== FAILURES ===================================

In this case, FF means that the file test_exercise_1.py has two failing tests, . means that the file test_exercise_2.py has one passing test, and F means that the file test_exercise_3.py has one failing test.

The percentages show cumulative progress through the collected tests, regardless of whether they pass or fail.

In this case, 4 tests were run (2 inside test_exercise_1.py, 1 inside test_exercise_2.py, and 1 inside test_exercise_3.py), and 1 test passed (the test inside test_exercise_2.py).

Therefore, 50%, 75%, and 100% indicate that the first file has 50% of tests (i.e., 2/4), the first two files have 75% of tests (i.e., 3/4), and all files have 100% of tests (i.e., 4/4).

After the summary, you will see the detailed output of the tests. Search for the FAILURES section to see which tests failed and why they failed.

The output will show you the name of the failed test, the line of code that caused the failure, and the reason for the failure (e.g., an assertion error).

For example, the following output shows that the test test_exercise_1.py::test_add_two failed because the expected result was 3 but the actual result was None:

See Listing 11.

Listing 11: A failed assertion compares the observed None result with the expected value 3.

=================================== FAILURES ===================================
______________________________ test_add_two ______________________________

    def test_add_two():
>       assert basic_math.add_two(1, 2) == 3
E       assert None == 3
E        +  where None = <function add_two at 0x7f8b3b3b6c10>(1, 2)
E        +    where <function add_two at 0x7f8b3b3b6c10> = basic_math.add_two

In this case, the test failed because the function add_two returned None instead of the expected result 3. You must review the code in the basic_math.py file to identify and fix the issue.

You do not need to understand all the details of the output, but you should be able to identify the failed test and its cause (check the E lines for more information).

At the end of the file, you will see a summary of the test results, including the number of tests that passed, the number of tests that failed, and the number of tests that were skipped:

See Listing 12.

Listing 12: The short summary identifies the three failing tests and their failed comparisons.

=========================== short test summary info ============================
FAILED tests/test_exercise_1.py::test_add_two - assert None == 3
FAILED tests/test_exercise_1.py::test_multiply_two - assert None == 2
FAILED tests/test_exercise_3.py::test_hello_world_output - AssertionError: assert '' == 'Hello, world!'

In this case, the summary shows that three tests failed (test_add_two, test_multiply_two, and test_hello_world_output) and one test passed. The summary also briefly describes the reason for the failure (e.g., assert None == 3).

Finally, the output will show the total number of tests run and the time it took to run the tests:

See Listing 13.

Listing 13: The final count reports three failures, one pass, and the elapsed time.

========================= 3 failed, 1 passed in 0.10s ==========================

In this case, the output shows that 3 tests failed, 1 test passed, and the tests took 0.10 seconds to run.

Tuning the verbosity of pytest

pytest provides different verbosity levels to control the output displayed when running tests. By default, pytest will display a summary of the test results, showing the number of tests that passed, the number of tests that failed, and the number of tests that were skipped.

According to the pytest documentation, you can control the verbosity level of the output by using the following flags:

See Listing 14.

Listing 14: Commands select quiet, verbose, or more verbose pytest output.

pytest -q               # quiet - less verbose
pytest -v               # increase verbosity, display individual test names
pytest -vv              # more verbose, display more details from the test output

The examples below compare the output at each verbosity level.

Less verbose output (`pytest -q`)

If you prefer a less verbose output, you can use the -q flag to run the tests in "quiet" mode:

See Listing 15.

Listing 15: Run pytest in quiet mode.

pytest -q

The -q flag stands for "quiet" and provides a more concise output for each test. The details of the session (e.g., the platform, Python version, and pytest version) will not be displayed, and instead of the per-file summary, you will see something like this:

See Listing 16.

Listing 16: Quiet-mode progress reports four tests, with only the third passing.

FF.F     [100%]

In this case, FF.F means that we have four tests, and the first, second, and fourth tests failed, while the third test passed. The rest of the output will be the same as before, showing the detailed output of the tests and the summary of the test results.

Verbose output (`pytest -v`)

If you want to see more detailed output, you can use the -v flag:

See Listing 17.

Listing 17: Run pytest in verbose mode.

pytest -v

The -v flag stands for "verbose" and provides a more detailed output for each test. In this mode, you will see the name of each test and its verdict (PASSED or FAILED), one test per line, so a failing test is named rather than reduced to a letter:

See Listing 18.

Listing 18: Verbose progress identifies each test and its verdict.

tests/test_exercise_1.py::test_add_two FAILED                 [25%]
tests/test_exercise_1.py::test_multiply_two FAILED            [50%]
tests/test_exercise_2.py::test_greeting_output PASSED         [75%]
tests/test_exercise_3.py::test_hello_world_output FAILED      [100%]

More verbose output (`pytest -vv`)

You can use the -vv flag if you want even more detailed output. This will increase the verbosity of the "short test summary" at the end of the output:

See Listing 19.

Listing 19: More verbose output provides additional details for each failed assertion.

=========================== short test summary info ============================
FAILED tests/test_exercise_1.py::test_add_two - assert None == 3
 + where None = <function add_two at 0x7f8b3b3b6c10>(1, 2)
    +   where <function add_two at 0x7f8b3b3b6c10> = basic_math.add_two
FAILED tests/test_exercise_1.py::test_multiply_two - assert None == 2
    + where None = <function multiply_two at 0x7f8b3b3b6d30>(1, 2)
        +   where <function multiply_two at 0x7f8b3b3b6d30> = basic_math.multiply_two
FAILED tests/test_exercise_3.py::test_hello_world_output - AssertionError: assert '' == 'Hello, world!'

   - Hello, world!
========================= 3 failed, 1 passed in 0.10s ==========================

In this case, the output shows the same information as before but with more details about the failed tests. For example, it shows the expected result (3) and the actual result (None) for the test test_add_two.

7.4.8 Testing in VS Code

Visual Studio Code runs and debugs tests from the editor, without the terminal. Its testing support covers several frameworks, pytest among them, and is documented in the Python testing guide.

To use pytest in VS Code, follow these steps:

Install the Python extension if you have not already. Test discovery is part of it, so there is no separate testing extension to install.
Open the Testing view by clicking the flask icon in the Activity Bar. The first time, VS Code asks which framework you use and which folder holds the tests; choose pytest and the folder with your test files.
VS Code writes that choice to .vscode/settings.json, so it is remembered for the project.

The extension provides several features to help you write and run tests in VS Code:

Test Explorer: The Test Explorer displays a list of tests in your workspace and allows you to run and debug them. You can view the test results, navigate to the test files, and filter the tests based on their status (e.g., passed, failed, skipped).
Run Tests: You can run tests directly from the editor by clicking the Run Test button next to the test function or file (a green play button). This will run the test and display the results in the Test Explorer.
Debug Tests: You can debug tests directly from the editor by clicking the Debug Test button next to the test function or file (a bug icon). This will run the test in debug mode and allow you to step through the code, inspect variables, and identify issues.
View Test Output: You can view the output of the tests in the Test Output panel. This panel displays the detailed output of the tests, including the test results, error messages, and stack traces.