Start rewrite using Python 3, apifw, slog

apifw and slog are two libraries I've written for work. They make
writing RESTful HTTP JSON APIs easier.

5 files changed, 0 insertions, 551 deletions

diff --git a/yarns/000.yarn b/yarns/000.yarn
deleted file mode 100644
index 7a19824..0000000
--- a/yarns/000.yarn
+++ /dev/null
@@ -1,24 +0,0 @@
----
-title: Ick2 controller yarns
-...
-
-
-Introduction
-=============================================================================
-
-Ick2 will be a continuous integration system. Its core component is
-the **controller**, which does nothing, except decides what workers should
-do. It knows of the projects that can be built, and keeps track what
-step is being run on each project, and collects build output from the
-workers.
-
-This document specifies the yarn test scenarios for a minimal viable
-version of the controller: there will be a small number of project,
-each project builds a web site from source in git, using ikiwiki, and
-publishes the website on a server using rsync.
-
-The controller provides an HTTP API for controlling a build. The API
-is used by an external entity (such as the git server) to trigger a
-build, and by workers or worker proxies to request something to do,
-and to report results. The API may also eventually be used by end
-users to query status and results.
diff --git a/yarns/100-hello.yarn b/yarns/100-hello.yarn
deleted file mode 100644
index d719497..0000000
--- a/yarns/100-hello.yarn
+++ /dev/null
@@ -1,15 +0,0 @@
-# Controller smoke test
-
-This scenario is just for making sure we can, in our tests, start and
-stop the controller, and make requests to it.
-
-    SCENARIO controller smoke test
-    GIVEN a running controller instance
-
-    WHEN user calls GET /version
-    THEN response has status 200, and JSON body "{ "version": "1.0" }"
-
-    WHEN user calls GET /blatherskite
-    THEN response has status 404
-
-    FINALLY stop controller instance
diff --git a/yarns/200-build.yarn b/yarns/200-build.yarn
deleted file mode 100644
index 3465f50..0000000
--- a/yarns/200-build.yarn
+++ /dev/null
@@ -1,305 +0,0 @@
-Building a project
-=============================================================================
-
-One build, one worker
------------------------------------------------------------------------------
-
-This section uses the controller to walk through all the steps for a
-build. We start with some setup, defining a git repo and an ick
-project and starting the controller.
-
-    SCENARIO run a build with one worker
-
-    GIVEN a git repo foo.git with file index.mdwn containing
-    ... "hello, world\n"
-    AND a project foo, using foo.git, publishing to foo-web
-    AND a running controller instance
-
-Ensure controller knows of the project.
-
-    WHEN user calls GET /projects
-    THEN response has status 200, 
-    ... and JSON body "{ "projects": [ "foo" ] }"
-
-There is no job running now, so if the worker manager asks for work,
-it gets nothing.
-
-    WHEN worker manager calls GET /worker/bar
-    THEN response has status 200, and an empty body
-
-Trigger a new build. There is now work to do.
-
-    WHEN git server calls GET /projects/foo/+trigger
-    THEN response has status 200
-
-    WHEN worker manager calls GET /worker/bar
-    THEN response has status 200, and JSON body "{
-    ...     "project": "foo",
-    ...     "git": "foo.git",
-    ...     "shell": "ikiwiki --build"
-    ... }"
-
-Pretend a job is running, and send output to the controller. Don't send
-an exit code, since the pretend job hasn't finished. Check that the
-pretend output we sent ends up in the current build log.
-
-    WHEN worker manager calls POST /worker/bar/snippet,
-    ... with JSON body '{
-    ...     "project": "foo",
-    ...     "stdout": "ikiwiki build output",
-    ...     "stderr": "",
-    ...     "exit-code": null
-    ... }'
-    AND user calls GET /projects/foo/logs/current
-    THEN response has status 200, and text body "ikiwiki build output"
-
-The current build step hasn't changed.
-
-    WHEN worker manager calls GET /worker/bar
-    THEN response has status 200,
-    ... and JSON body "{
-    ...     "project": "foo",
-    ...     "git": "foo.git",
-    ...     "shell": "ikiwiki --build"
-    ... }"
-
-Pretend current command finishes. Make sure current log updates, and
-that we get a new thing to run.
-
-    WHEN worker manager calls POST /worker/bar/snippet,
-    ... with JSON body '{
-    ...     "project": "foo",
-    ...     "stdout": "|more output",
-    ...     "stderr": "",
-    ...     "exit-code": 0
-    ... }'
-    AND user calls GET /projects/foo/logs/current
-    THEN response has status 200, and an empty body
-
-    WHEN user calls GET /projects/foo/logs/previous
-    THEN response has status 200, 
-    ... and text body "ikiwiki build output|more output"
-
-    WHEN worker manager calls GET /worker/bar
-    THEN response has status 200,
-    ... and JSON body "{
-    ...     "project": "foo",
-    ...     "git": "foo.git",
-    ...     "shell": "rsync"
-    ... }"
-
-Tell worker the rsync command also finishes. After that, there should
-be nothing more to do. The current log should become empty, the
-previous log will contain the previously current log.
-
-    WHEN worker manager calls POST /worker/bar/snippet,
-    ... with JSON body '{
-    ...     "project": "foo",
-    ...     "stdout": "rsync output",
-    ...     "stderr": "",
-    ...     "exit-code": 0
-    ... }'
-
-    WHEN user calls GET /projects/foo/logs/current
-    THEN response has status 200, and an empty body
-
-    WHEN user calls GET /projects/foo/logs/previous
-    THEN response has status 200,  and text body "rsync output"
-
-    WHEN user calls GET /worker/bar
-    THEN response has status 200, and an empty body
-
-And we're done.
-
-    FINALLY stop controller instance
-
-
-Two builds, two workers
------------------------------------------------------------------------------
-
-This section runs two builds using two workers. The primary goal here
-is to make sure each worker gets consecutive steps for its own build,
-so that it runs all the steps in the same workspace.
-
-We start with some setup, defining a git repo and an ick project and
-starting the controller.
-
-    SCENARIO run two builds with two workers
-
-    GIVEN a git repo foo.git with file index.mdwn containing
-    ... "hello, world\n"
-    AND a git repo bar.git with file index.mdwn containing
-    ... "hello, bar\n"
-    AND a project foo, using foo.git, publishing to foo-web
-    AND a project bar, using bar.git, publishing to bar-web
-    AND a running controller instance
-
-Ensure controller knows of both projects. The list of project names
-should be sorted alphabetically.
-
-    WHEN user calls GET /projects
-    THEN response has status 200, 
-    ... and JSON body "{ "projects": [ "bar", "foo" ] }"
-
-There is no job running now, so if the worker manager asks for work,
-it gets nothing.
-
-    WHEN worker manager calls GET /worker/one
-    THEN response has status 200, and an empty body
-
-Trigger new builds on both projects. There is now work to do.
-
-    WHEN git server calls GET /projects/foo/+trigger
-    THEN response has status 200
-
-    WHEN worker manager calls GET /worker/one
-    THEN response has status 200, and JSON body "{
-    ...     "project": "foo",
-    ...     "git": "foo.git",
-    ...     "shell": "ikiwiki --build"
-    ... }"
-
-The second worker should still get nothing to do.
-
-    WHEN worker manager calls GET /worker/two
-    THEN response has status 200, and an empty body
-
-Trigger the other project, and the second worker gets something to do.
-
-    WHEN git server calls GET /projects/bar/+trigger
-    THEN response has status 200
-
-    WHEN worker manager calls GET /worker/two
-    THEN response has status 200, and JSON body "{
-    ...     "project": "bar",
-    ...     "git": "bar.git",
-    ...     "shell": "ikiwiki --build"
-    ... }"
-
-Pretend the build step for the first project is running, and send
-output to the controller. Don't send an exit code, since the pretend
-step hasn't finished. Check that the pretend output we sent ends up in
-the current build log.
-
-    WHEN worker manager calls POST /worker/one/snippet,
-    ... with JSON body '{
-    ...     "project": "foo",
-    ...     "stdout": "ikiwiki build output",
-    ...     "stderr": "",
-    ...     "exit-code": null
-    ... }'
-    AND user calls GET /projects/foo/logs/current
-    THEN response has status 200, and text body "ikiwiki build output"
-
-The current build step hasn't changed.
-
-    WHEN worker manager calls GET /worker/one
-    THEN response has status 200,
-    ... and JSON body "{
-    ...     "project": "foo",
-    ...     "git": "foo.git",
-    ...     "shell": "ikiwiki --build"
-    ... }"
-
-Pretend the build step finishes. Make sure current log updates, and
-that we get a new thing to run.
-
-    WHEN worker manager calls POST /worker/one/snippet,
-    ... with JSON body '{
-    ...     "project": "foo",
-    ...     "stdout": "|more output",
-    ...     "stderr": "",
-    ...     "exit-code": 0
-    ... }'
-    AND user calls GET /projects/foo/logs/current
-    THEN response has status 200, and an empty body
-
-    WHEN user calls GET /projects/foo/logs/previous
-    THEN response has status 200, 
-    ... and text body "ikiwiki build output|more output"
-
-    WHEN worker manager calls GET /worker/one
-    THEN response has status 200,
-    ... and JSON body "{
-    ...     "project": "foo",
-    ...     "git": "foo.git",
-    ...     "shell": "rsync"
-    ... }"
-
-The other worker is still running its step, and if it asks, it gets
-the same build step to run.
-
-    WHEN worker manager calls GET /worker/two
-    THEN response has status 200, and JSON body "{
-    ...     "project": "bar",
-    ...     "git": "bar.git",
-    ...     "shell": "ikiwiki --build"
-    ... }"
-
-Tell controller the rsync command of the first project also finishes.
-After that, there should be nothing more to do. The current log should
-become empty, the previous log will contain the previously current
-log.
-
-    WHEN worker manager calls POST /worker/one/snippet,
-    ... with JSON body '{
-    ...     "project": "foo",
-    ...     "stdout": "rsync output",
-    ...     "stderr": "",
-    ...     "exit-code": 0
-    ... }'
-
-    WHEN user calls GET /projects/foo/logs/current
-    THEN response has status 200, and an empty body
-
-    WHEN user calls GET /projects/foo/logs/previous
-    THEN response has status 200,  and text body "rsync output"
-
-    WHEN user calls GET /worker/one
-    THEN response has status 200, and an empty body
-
-Finish the other project build.
-
-    WHEN worker manager calls POST /worker/two/snippet,
-    ... with JSON body '{
-    ...     "project": "bar",
-    ...     "stdout": "ikiwiki output",
-    ...     "stderr": "",
-    ...     "exit-code": 0
-    ... }'
-    AND user calls GET /projects/bar/logs/current
-    THEN response has status 200, and an empty body
-
-    WHEN user calls GET /projects/bar/logs/previous
-    THEN response has status 200, 
-    ... and text body "ikiwiki output"
-
-    WHEN worker manager calls GET /worker/two
-    THEN response has status 200,
-    ... and JSON body "{
-    ...     "project": "bar",
-    ...     "git": "bar.git",
-    ...     "shell": "rsync"
-    ... }"
-
-    WHEN worker manager calls POST /worker/two/snippet,
-    ... with JSON body '{
-    ...     "project": "bar",
-    ...     "stdout": "second worker rsync output",
-    ...     "stderr": "",
-    ...     "exit-code": 0
-    ... }'
-
-    WHEN user calls GET /projects/bar/logs/current
-    THEN response has status 200, and an empty body
-
-    WHEN user calls GET /projects/bar/logs/previous
-    THEN response has status 200,  and text body 
-    ... "second worker rsync output"
-
-    WHEN user calls GET /worker/two
-    THEN response has status 200, and an empty body
-
-And we're done.
-
-    FINALLY stop controller instance
diff --git a/yarns/900.yarn b/yarns/900.yarn
deleted file mode 100644
index 8ee31f3..0000000
--- a/yarns/900.yarn
+++ /dev/null
@@ -1,96 +0,0 @@
-# Scenario step implementations
-
-# Manage controller instance
-
-    IMPLEMENTS GIVEN a running controller instance
-    controller = os.path.join(srcdir, 'controller')
-    cliapp.runcmd(
-        ['/usr/sbin/daemonize', '-c.',
-         controller, '--debug', '--pid-file=pid',
-         '--port-file=port', '--projects=ick.ick', '--log=log'])
-    vars['pid'] = cat('pid').strip()
-    vars['port'] = cat('port').strip()
-
-    IMPLEMENTS FINALLY stop controller instance
-    import signal
-    print 'killing process', repr(vars['pid'])
-    os.kill(int(vars['pid']), signal.SIGTERM)
-
-
-## Git repositories
-
-    IMPLEMENTS GIVEN a git repo (\S+) with file (\S+) containing "(.+)"
-    repo = yarnutils.get_next_match()
-    filename = yarnutils.get_next_match()
-    content = yarnutils.get_next_match()
-    pathname = os.path.join(repo, filename)
-    os.mkdir(repo)
-    git(repo, 'init', '.')
-    git(repo, 'config', 'user.email', 'user@example.com')
-    git(repo, 'config', 'user.name', 'J. Random User')
-    write(pathname, unescape(content))
-    git(repo, 'add', '.')
-    git(repo, 'commit', '-minitial')
-
-## Controller configuration
-
-    IMPLEMENTS GIVEN a project (\S+), using (\S+), publishing to (\S+)
-    project = yarnutils.get_next_match()
-    repo = yarnutils.get_next_match()
-
-    if os.path.exists('ick.ick'):
-        config = yaml.safe_load(open('ick.ick'))
-    else:
-        config = {
-            'projects': {}
-        }
-    config['projects'][project] = {
-        'git': repo,
-        'shell_steps': [
-            'ikiwiki --build',
-            'rsync',
-        ],
-    }
-    write('ick.ick', yaml.safe_dump(config))
-
-## API use
-
-    IMPLEMENTS WHEN (user|worker manager|git server) calls (\S+) (\S+)
-    who = yarnutils.get_next_match()
-    method = yarnutils.get_next_match()
-    path = yarnutils.get_next_match()
-    url = controller_url(vars['port'], path)
-    vars['status'], vars['body'] = request(method, url)
-
-    IMPLEMENTS WHEN worker manager calls (\S+) (\S+), with JSON body '(.+)'
-    method = yarnutils.get_next_match()
-    path = yarnutils.get_next_match()
-    body_text = yarnutils.get_next_match()
-    url = controller_url(vars['port'], path)
-    body = parse_json(body_text)
-    vars['status'], vars['body'] = request(method, url, body=body_text)
-
-    IMPLEMENTS THEN response has status (\d+), and an empty body
-    status = yarnutils.get_next_match()
-    yarnutils.assertEqual(int(status), int(vars['status']))
-    yarnutils.assertEqual(vars['body'], '')
-
-    IMPLEMENTS THEN response has status (\d+), and text body "(.+)"
-    status = yarnutils.get_next_match()
-    bodypat = yarnutils.get_next_match()
-    yarnutils.assertEqual(int(status), int(vars['status']))
-    yarnutils.assertEqual(vars['body'], unescape(bodypat) )
-
-    IMPLEMENTS THEN response has status (\d+), and JSON body "(.+)"
-    status = yarnutils.get_next_match()
-    bodytext = yarnutils.get_next_match()
-    print 'varsbody:', repr(vars['body'])
-    print 'bodytext:', repr(bodytext)
-    bodyjson = parse_json(bodytext)
-    print 'bodyjson:', repr(bodyjson)
-    yarnutils.assertEqual(int(status), int(vars['status']))
-    yarnutils.assertEqual(parse_json(vars['body']), parse_json(bodytext) )
-
-    IMPLEMENTS THEN response has status (\d+)
-    status = yarnutils.get_next_match()
-    yarnutils.assertEqual(int(status), int(vars['status']))
diff --git a/yarns/lib.py b/yarns/lib.py
deleted file mode 100644
index fc4acfc..0000000
--- a/yarns/lib.py
+++ /dev/null
@@ -1,111 +0,0 @@
-import errno
-import json
-import os
-import StringIO
-import time
-
-import cliapp
-import requests
-import yaml
-import yarnutils
-
-
-datadir = os.environ['DATADIR']
-srcdir = os.environ['SRCDIR']
-
-vars = yarnutils.Variables(datadir)
-
-
-MAX_CAT_TIME = 5 # seconds
-def cat(filename):
-    start = time.time()
-    while time.time() - start < MAX_CAT_TIME:
-        try:
-            with open(filename) as f:
-                data = f.read()
-                if len(data) == 0:
-                    continue
-                return data
-        except (IOError, OSError) as e:
-            if e.errno == errno.ENOENT:
-                continue
-            raise
-    raise Exception("cat took more then %s seconds" % MAX_CAT_TIME)
-
-
-def write(filename, content):
-    with open(filename, 'w') as f:
-        f.write(content)
-
-
-def git(repo, *argv):
-    return cliapp.runcmd(['git'] + list(argv), cwd=repo)
-
-
-def controller_url(port, path):
-    return 'http://localhost:{}{}'.format(port, path)
-
-
-def request(method, url, body=None):
-    funcs = {
-        'POST': requests.post,
-        'PUT': requests.put,
-        'GET': requests.get,
-        'DELETE': requests.delete,
-    }
-
-    headers = {
-        'Content-Type': 'application/json',
-    }
-
-    response = funcs[method](
-        url,
-        headers=headers,
-        data=body,
-    )
-
-    return response.status_code, response.text
-
-
-def parse_json(text):
-    return json.loads(text, object_pairs_hook=dictify)
-
-
-
-def dictify(pairs):
-    return {
-        stringify(key): stringify(value)
-        for key, value in pairs
-    }
-
-
-def stringify(x):
-    if isinstance(x, unicode):
-        return str(x)
-    if isinstance(x, list):
-        return [stringify(y) for y in x]
-    if isinstance(x, dict):
-        return {
-            stringify(key): stringify(value)
-            for key, value in pairs
-        }
-    return x
-
-
-def parse_yaml(text):
-    f = StringIO.StringIO(text)
-    return yaml.safe_load(stream=f)
-
-
-def unescape(text):
-    def helper(text):
-        while text:
-            if text.startswith('\\n'):
-                skip = 2
-                answer = '\n'
-            else:
-                skip = 1
-                answer = text[0]
-            text = text[skip:]
-            yield answer
-    return ''.join(helper(text))