CoverageTestRunner ================== The Python standard library module `unittest` aids in implementing and running unit tests. The `coverage.py` module (not in the standard library) measures which parts of a program are executed. A proper unit test suite will execute all parts of a program. `CoverageTestRunner` uses `coverage.py` to run test suites implemented with `unittest`, and fails the tests if they do not test the entire program. `CoverageTestRunner` makes some assumptions, the most important is that the test suite is written so that each module corresponds to a test module, and that test module should execute everything in the module. If there is a code module `foo.py`, then it must be fully tested by `foo_tests.py`. Ideally, we would test each class and its corresponding test suite, but unfortunately `coverage.py` only allows module granularity. It also only measures at the statement level. This is, however, typically better than can be achieved without any measurement tools. Example ------- Assume a Python module in `foo.py`, and its test suite in `foo_tests.py` in the same directory. Then running the test suite using `CoverageTestRunner` can be very simple: python -m CoverageTestRunner `CoverageTestRunner` scans the current directory and its subdirectories for test modules with corresponding code modules, based on filenames, and then executes them pairwise. If you have a more complicated setup, you can pair the code and test modules manually, using a wrapper script such as follows: from CoverageTestRunner import CoverageTestRunner r = CoverageTestRunner() r.add_pair("code/foo.py", "tests/foo.py") r.run() Output ------ `CoverageTestRunner` writes its output to the standard output, and for a successful run it looks like this: Running test 2/2: testTrue (foo_tests.FooTests) OK Time: 0.0 s An unsuccessful run might look like this (shortened for space reasons): Running test 226/226: test (configTests.WriteDefaultConfigTests) FAILED ERROR: testEmptyFile (ioTests.FileContentsTests) Traceback (most recent call last): File "unittests/ioTests.py", line 228, in testEmptyFile id = obnam.io.create_file_contents_object(self.context, filename) File "/home/liw/Braawi/obnam/new-storage/obnam/io.py", line 240, in create_file_contents_object stdout=subprocess.PIPE) File "subprocess.py", line 593, in __init__ errread, errwrite) File "subprocess.py", line 1135, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory Statements missed by per-module tests: Module Missed statements /home/liw/Braawi/obnam/new-storage/obnam/cfgfile.py 70, 303 /home/liw/Braawi/obnam/new-storage/obnam/cmp.py 155 /home/liw/Braawi/obnam/new-storage/obnam/config.py 234-237 /home/liw/Braawi/obnam/new-storage/obnam/filelist.py 88 /home/liw/Braawi/obnam/new-storage/obnam/format.py 74, 97 1 failures, 3 errors Time: 5.4 s In this case, after the line with `FAILED` comes a list of all the failed tests, with some indication of why they failed, complete with Python stack traces. Additionally, if some parts of the code were not executed during the tests, a list of files and statement numbers is included in the output. Excluding some statements from the coverage analysis ---------------------------------------------------- Use `# pragma: no cover` to exclude some statements from being included in the coverage analysis. The line with that comment is excluded. If the line is an `if`, `while`, `def`, or similar, with a body, the body is also excluded. For example: if always_true_during_testing: print foo else: # pragma: no cover print bar The last two lines of codes are excluded from coverage analysis even if the test suite does not execute them. You should only use this after thorough consideration. Excluding whole modules ----------------------- Sometimes there are code modules that should not have test modules. For example, `setup.py`. You can use the `--ignore-missing` option to ignore all such problem, or `--ignore-missing-from=FILE` to list excluded modules explicitly. By default, the `setup.py` module is automatically excluded. However, if you list any modules explicitly, you should also list `setup.py`. Your tests are too slow ----------------------- `CoverageTestRunner` keeps track of how long each test takes. If the total run time of the tests is more than ten seconds, even if all tests pass, it then prints out a list of the slowest tests. This is useful for projects whose test suites should run quickly so that developers don't mind running them all the time. This is obviously a bad thing for projects with large test suites. If it bothers you, please ask me, and I'll provide a way to disable it, or to set the time limit. Contacts -------- Python's `unittest` is part of the standard library, and was (originally?) written by Steve Purcell. `coverage.py` was originally written by Gareth Rees and is now maintained by Net Batchelder. Its home page is at http://www.nedbatchelder.com/code/modules/coverage.html CoverageTestRunner was written by Lars Wirzenius (`liw@liw.fi`). The home page is at http://liw.fi/coverage-test-runner/ I welcome any feedback you may have. License -------- Copyright 2007-2013 Lars Wirzenius This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . =*= License: GPL-3+ =*=