README for debug logging
========================

Log files can be used for at least two purposes: for the program user
(or sysadmin) to have a rough idea of what is happening and for the
programmer to get a very detailed idea of what is happening, in order
to find and fix problems.

For the debugging programmer, especially, it would be helpful if
the log file was in a machine parseable format. `debuglog` is a
little helper class to make such log files. They're not intended
to be useful for non-programmers.

Some assumptions:

* It is not necessary to log everything during a program run. Instead,
  saving the last N log messages and writing those when the software
  crashes is enough.
* Log levels are not fine grained enough. It's better to have a
  machine parseable format that lends itself to more detailed filtering.
  Therefore, save everything and filter afterwards.

`debuglog`  consists of two parts:

1. A Python module, `debuglog`, with a class that collects structured
   log messages, and writes them (when requested) to a file as JSON.
2. A script, `debuglogtool`, that reads the JSON file and filters it
   in some ways.

The structured log messages are key/value pairs. For example:

    dlog = debuglog.DebugLog()
    ...
    dlog.log(msg='Opening input file', filename=filename, cwd=os.getcwd())

The `log` method gets any number of keyword arguments. These are saved
as a JSON record, with all values converted to strings.

The JSON log file can, of course, be filtered by any code that understands
JSON, if what `debuglogtool` provides is not sufficiently powerful.

Example:

    debuglogtool -e msg='Opening input file' -e 'filename~/\.JPG$/' *.debuglog

The above would output all JSON records that have a field called `msg`
with a value that is exactly the string "Opening input file", and
additionally have a field called `filename` which matches the regular
expression given. All such records are written out entirely, as a JSON
list. If there are no matches, nothing is output, and the exit code is
non-zero.

A single expression can use the following operators:

* `=` for exact string equality.
* `!=` for string inequality.
* `~` for regexp matching (regexp to be surrounded by slashes).
* `!~` for regexp NOT matching.

Regular expressions are PCRE.

Each expression is given with the `-e` option (`--expression`).
All expressions must match. There is no way to express full Boolean
logic, at this time.