CoverageTestRunner
==================

The Python standard library unittest aids in implementing and running
unit tests. The coverage.py module (not in the standard library) measures
which parts of a program are executed. A proper unit test suite will
execute all parts of a program. CoverageTestRunner uses coverage.py to
run test suites implemented with unittest, and fails the tests if they
do not test the entire program.

CoverageTestRunner makes some assumptions, the most important of which is
that the test suite is written so that each module corresponds to a test
module, and that test module should execute everything in the module.

Ideally, we would test each class and its corresponding test suite, but
unfortunately coverage.py only allows module granularity. It also only
measures at the statement level. This is, however, typically better
than can be achieved without any measurement tools.


Example
=======

Assume a Python module in foo.py, and its test suite in foo_tests.py.
Then running the test suite using CoverageTestRunner can be very simple:

    python -m CoverageTestRunner

This will invoke the module's main program, which scans the current 
directory (and its subdirectories) for test modules with corresponding
code modules, based on filenames, and then executes them pairwise.

If you have a more complicated setup, you can pair the code and test
modules manually, using a wrapper script such as follows:

    from CoverageTestRunner import CoverageTestRunner
    
    r = CoverageTestRunner()
    r.add_pair("code/foo.py", "tests/foo.py")
    r.run()


Output
======

CoverageTestRunner writes its output to the standard output, and for
a successful run it looks like this:

    Running test 2/2: testTrue (foo_tests.FooTests) 
    
    OK
    Time: 0.0 s

An unsuccessful run might look like this (shortened for space reasons):

    Running test 226/226: test (configTests.WriteDefaultConfigTests)        
    
    FAILED
    
    ERROR: testEmptyFile (ioTests.FileContentsTests)
    Traceback (most recent call last):
      File "unittests/ioTests.py", line 228, in testEmptyFile
        id = obnam.io.create_file_contents_object(self.context, filename)
      File "/home/liw/Braawi/obnam/new-storage/obnam/io.py", line 240, in create_file_contents_object
        stdout=subprocess.PIPE)
      File "subprocess.py", line 593, in __init__
        errread, errwrite)
      File "subprocess.py", line 1135, in _execute_child
        raise child_exception
    OSError: [Errno 2] No such file or directory

    Statements missed by per-module tests:
      Module                                                 Missed statements
      /home/liw/Braawi/obnam/new-storage/obnam/cfgfile.py    70, 303
      /home/liw/Braawi/obnam/new-storage/obnam/cmp.py        155
      /home/liw/Braawi/obnam/new-storage/obnam/config.py     234-237
      /home/liw/Braawi/obnam/new-storage/obnam/filelist.py   88
      /home/liw/Braawi/obnam/new-storage/obnam/format.py     74, 97
    
    1 failures, 3 errors
    Time: 5.4 s

In this case, after the line with "FAILED" comes a list of all the
failed tests, with some indication of why they failed, complete with
Python stack traces. Additionally, if some parts of the code were not
executed during the tests, a list of files and statement numbers is
included in the output.


Your tests are too slow
=======================

CoverageTestRunner keeps track of how long each test takes. If the total
run time of the tests is more than ten seconds, even if all tests pass,
it then prints out a list of the slowest tests. This is useful for projects
whose test suites should run quickly so that developers don't mind running
them all the time.

This is obviously a bad thing for projects with large test suites. If it
bothers you, please ask me, and I'll provide a way to disable it, or to
set the time limit.


Contacts
========

Python's unittest is part of the standard library, and was (originally?)
written by Steve Purcell. coverage.py was originally written by Gareth
Rees and is now maintained by Net Batchelder. Its home page is at

    http://www.nedbatchelder.com/code/modules/coverage.html

CoverageTestRunner was written by Lars Wirzenius (liw@iki.fi), and is
licensed under the GNU General Public License version 2 or at your option
any later version. The home page is at

    http://braawi.org/coveragetestrunner.html

I welcome any feedback you may have.


License
=======

Copyright (C) 2007  Lars Wirzenius <liw@iki.fi>

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.