Add: some preliminary design for controller resources

1 files changed, 178 insertions, 0 deletions

diff --git a/arch/ick2-arch.mdwn b/arch/ick2-arch.mdwn
index 5ff6bf0..525f0e1 100644
--- a/arch/ick2-arch.mdwn
+++ b/arch/ick2-arch.mdwn
@@ -553,6 +553,184 @@ end note
 
 The website is now built and published.
 
+Ick APIs
+=============================================================================
+
+APIs follow the RESTful style
+-----------------------------------------------------------------------------
+
+All the Ick APIs aRE [RESTful][]. Server-side state is represented by
+a set of "resources". These data objects that can be addressed using
+URLs and they are manipulated using HTTP methods (verbs): GET, POST,
+PUT, DELETE. There can be many instances of a type of resource. These
+are handled as a collection. Example: given a resource type for
+projects Ick should build, the API would have the following calls:
+
+    POST /projects -- create a new project, giving it an ID
+    GET /projects -- get list of all project ids
+    GET /projects/ID -- get info on project ID
+    PUT /projects/ID -- update project ID
+    DELETE /projects/ID -- remove a project
+
+[RESTful]: https://en.wikipedia.org/wiki/Representational_state_transfer
+
+Resources are all handled the same way, regardless of the type of the
+resource. This gives a consistency that makes it easier to use the
+APIs.
+
+Note that the server doesn't store any client-side state at all. There
+are sessions, no logins, etc. Authentication is handled by attaching
+(in the `Authorization` header) a token to each request. An Identity
+Provider gives out the tokens to API clients, on request.
+
+Note also the API doesn't have RPC style calls. The server end may
+decide to do some action as a side effect of a resource being created
+or updated, but the API client can't invoke the action directly. Thus,
+there's no way to "run this pipeline"; instead, there's a resource
+showing the state of a pipeline, and changing that resource to say
+state is "triggered" instead of "idle" is how an API client tells the
+server to run a pipeline.
+
+
+Ick controller resources and API
+-----------------------------------------------------------------------------
+
+A project consists of a workspace specification, and an ordered list
+of pipelines. Additionally the project has a list of builds, and for
+each build a build log, and metadata (time and duration of build, what
+triggered it, whether it was successful or not). Also, a current state
+of the workspace.
+
+A project resource:
+
+    {
+        "project": "liw.fi",
+        "parameters": {
+            "rsync_target": "www-data@www.example.com/srv/http/liw.fi"
+        },
+        "workspace": {
+            "gits": [
+                {
+                    "git": "ssh://git@git.liw.fi/liw.fi",
+                    "branch": "master",
+                    "dir": "src"
+                }
+            ]
+        },
+        "pipelines": [
+            {
+                "name": "workspace-setup",
+                "actions": [
+                    { "name": "clone-gits" },
+                ]
+            },
+            {
+                "name": "ikiwiki-config",
+                "actions": [
+                    { "shell": "cat src/ikiwiki.setup.template > ikiwiki.setup" },
+                    { "shell": "echo \"destdir: {{ workspace }}/html\" >> ikiwiki.setup" },
+                    { "name": "mkdir", "dirname": "html" }
+                ]
+            },
+            {
+                "name": "ikiwiki-run",
+                "actions": [
+                    { "shell": "ikiwiki --setup ikiwiki.setup" }
+                ]
+            }
+            {
+                "name": "rsync",
+                "actions": [
+                    { "shell": "rsync -a --delete html/. \"{{ rsync_target }}/.\" }
+                ]
+            }
+        ]
+    }
+
+Here:
+
+- the pipeline consists of a sequence of steps
+- each step is a shell snippet (expanded with jinja2) or a built-in
+  operation implemented by the worker-manager directly
+- project parameters are used by steps
+
+A pipeline status resource at
+`/projects/PROJECTNAME/pipelines/PIPELINENAME`, created automatically
+when a project resource is updated to include the pipeline:
+
+    {
+        "status": "idle/triggered/running/paused"
+    }
+
+To trigger a pipelie, PUT a pipeline resource with a status field of
+"triggered". It is an error to do that when current status is not
+idle.
+
+A build resource is created automatically, at
+/projects/PROJECTNAME/builds, when a pipeline actually starts (not
+when it's triggered). It can't be changed via the API.
+
+    {
+        "build": "12765",
+        "project": "liw.fi",
+        "pipeline": "ikiwiki-run",
+        "worker": "bartholomew",
+        "status": "running/success/failure",
+        "started": "TIMESTAMP",
+        "ended": "TIMESTAMP",
+        "triggerer": "WHO/WHAT",
+        "trigger": "WHY"
+    }
+
+A build log is stored at `/projects/liw.fi/builds/12765/log` as a
+blob. The build log is appended to by the worker-manager by reporting
+output.
+
+Workers are registered to the controller by creating a worker
+resource. Later on, we can add useful metadata to the resource, but
+for now we'll have just the name.
+
+    {
+        "worker": "bartholomew"
+    }
+
+A work resource resource tells a worker what to do next:
+
+    {
+        "project": "liw.fi",
+        "pipeline": "ikiwiki-run",
+        "step": {
+            "shell": "ikiwiki --setup ikiwiki.setup"
+        },
+        "parameters": {
+            "rsync-target": "..."
+        }
+    }
+
+The controller provides a simple API to give work to each worker:
+
+    GET /work/bartholomew
+    PUT /work/bartholomew
+
+The controller keeps track of which worker is currently running which
+pipeline
+
+Work output resource:
+
+    {
+        "worker": "bartholomew",
+        "project": "liw.fi",
+        "pipeline": "ikiwiki-run",
+        "exít_code": null,
+        "stdout": "...",
+        "stderr": "...",
+        "timestamp": "..."
+    }
+
+When `exit_code` is non-null, the step has finished, and the
+controller knows it should schedule the next step in the pipeline.
+
+
 Known problems
 =============================================================================