docs: bring architecture document up to date

Various updates to bring the text of the architecture document up to
date with what has been implemented. Most of the speculative "we will
probably do it like this" parts have been dropped. Various fixes have
been made.

Signed-off-by: Lars Wirzenius <liw@liw.fi>

8 files changed, 134 insertions, 77 deletions

diff --git a/doc/Makefile b/doc/Makefile
index f5a3aa8..a635d69 100644
--- a/doc/Makefile
+++ b/doc/Makefile
@@ -1,7 +1,11 @@
-.SUFFIXES: .uml .svg .pik .md .html
+.SUFFIXES: .uml .svg .dot .pik .md .html
 
 .md.html:
-	pandoc --toc --standalone --self-contained $< -o $@
+	pandoc -V date="Version: $$(git describe --long --dirty --all)" --toc --number-sections --standalone --self-contained $< -o $@
+
+.dot.svg:
+	dot -Tsvg $< > $@.tmp
+	mv $@.tmp $@
 
 .uml.svg:
 	plantuml -tsvg --output=. $<
@@ -12,4 +16,4 @@
 
 all: architecture.html
 
-architecture.html: architecture.svg architecture-ext.svg components.svg components-ext.svg
+architecture.html: architecture.svg architecture-ext.svg comp.svg comp-ext.svg Makefile
diff --git a/doc/architecture-ext.uml b/doc/architecture-ext.uml
index cb0450f..24ded70 100644
--- a/doc/architecture-ext.uml
+++ b/doc/architecture-ext.uml
@@ -7,9 +7,7 @@ engine -> worker : start run
 note over worker : perform the run
 engine -> adapter : response with run id
 adapter -> broker : response: run id
-broker -> node : create CI COB
 worker -> engine : run finished
-engine -> adapter : web hook
+engine -> adapter : web hook?
 adapter -> broker : response: result
-broker -> node : update CI COB
 @enduml
diff --git a/doc/architecture.md b/doc/architecture.md
index aae14d9..f778770 100644
--- a/doc/architecture.md
+++ b/doc/architecture.md
@@ -1,9 +1,33 @@
 ---
 title: Radicle CI architecture
+author: The Radicle Team
 ...
 
 # Overview
 
+[Radicle](https://radicle.xyz/) is a peer-to-peer collaboration system
+built on top of the git version control system. Radicle has support
+for integrating with continuous integration (CI) systems, using an
+architecture where a "broker" listens to events about changes to
+repositories stored in a node, and launching the appropriate "adapter"
+for each change, according to its configuration.
+
+This means each node can opt into running CI for what projects and
+changes according to the interests of the person whose node it is.
+
+* The delegates for a repository might run CI on all patches to make
+  merge decisions with more confidences.
+* Someone whose contributing to a project might only care about
+  patches they themselves created, and only run CI for those.
+* A third party might run CI for projects they use, to know if it's OK
+  to deploy to production.
+
+Radicle provides its own, very simple "native CI" solution. It's just
+good enough for the Radicle project to use itself. In addition, there
+are adapters that integrate with external CI systems.
+
+## Components in native CI
+
 CI support in Radicle consists of several components. For native CI
 they are:
 
@@ -13,6 +37,12 @@ they are:
 
 These all run on the same host.
 
+![Components for native CI](comp.svg)
+
+![Sequence diagram for native CI](architecture.svg)
+
+## Components when integrating an external CI system
+
 For external CI, the components are:
 
 * the Radicle node
@@ -21,44 +51,39 @@ For external CI, the components are:
 * the external CI instance
 
 The first three of these run on the same host, but the external CI
-instance can run anywhere.
+instance can run anywhere. The adapter talks to the CI instance using
+whatever protocol the CI instance supports, such as HTTP.
 
-The child process is called "the CI adapter" in this document.
-
-CI works like this:
+External CI integration works like this:
 
 * a repository known to the node changes
   - a git ref is updated
   - the ref can be a branch, tag, or something else, such as a Radicle
     COB
   - the node emits an event describing the change
-* the CI broker listens to node event
-  - the broker subscribers to node events via the node control socket,
+* the CI broker listens to node events
+  - the broker subscribes to node events via the node control socket,
     which is a Unix domain socket
 * the CI broker filters events, based on its configuration, and the
   configuration for the repository involved
 * for an event that passes its filters, the CI broker spawns the
-  adapter process
-  - the adapter process is either the native CI engine, or an
-    integration with an external CI engine
-  - the CI broker uses the executable for native CI by default, but
-    this can be overridden by the broker configuration file, and that
-    can be overridden by the `.radicle/ci/config.yaml` file in the repository
-    the event is for
+  appropriate adapter process
+  - there are different adapter for different CI implementations
 * the broker sends a request object to the adapter as a child process,
   via the child's stdin, and reads any responses from the child's
   stdout
   - the request is JSON
   - the responses are in the form of JSON Lines: a JSON object per
     line serialized to not contain newline characters
+* the adapter communicates with the external CI instance in whatever
+  way is suitable for that instance
+  - this is usually over HTTP
+  - it may involve the CI instance making a web hook request back to
+    the adapter
 
-## Native CI
-
-![Sequence diagram for native CI](architecture.svg)
+![Components for external CI](comp-ext.svg)
 
-## External CI
-
-![Sequence diagram for external CI](architecture.svg)
+![Sequence diagram for external CI](architecture-ext.svg)
 
 # The adapter
 
@@ -66,9 +91,6 @@ The adapter process reads the request to perform a CI run on a
 specific commit in a repository, and responds with the id of the run,
 then later with the status of the finished run.
 
-Note: this is for the first version. This will be expanded later with
-other requests and responses, as needed.
-
 For native CI, the adapter actually is the CI engine and performs the
 CI run itself. For external CI, the adapter process does whatever it
 needs to do to get the external CI engine instance to perform the CI
@@ -79,8 +101,11 @@ it.
 External CI engines allow complex pipelines to be written and support 
 a variety of workflows. Different jobs or tasks can be triggered
 based on different events (e.g. `push`, `patch created`, `patch 
-updated`, etc.), to satisfy different workflow needs. Some examples 
-of real-world use-cases:
+updated`, etc.), to satisfy different workflow needs. (Only a few of
+these are actually implemented yet, but the scaffolding to support
+more is there.)
+
+Some examples of real-world use-cases:
 
 - trigger "fast" tests on every push to any branch
 - trigger "relatively fast" tests only when a Patch is created / 
@@ -98,6 +123,26 @@ adapters, so they can pass it on to external CI systems, as appropriate.
 Note: the JSON objects below are formatted on multiple lines to make
 them easier to read. The actual wire format is one line per message.
 
+Run the `broker-messages` binary to get actual examples produced by
+code.
+
+~~~{.sh .numberLines}
+$ cargo run -q --bin broker-messages
+Trigger request:
+{"request":"trigger","event_type":"push","repository":{"id":"rad:zwTxygwuz5LDGBq255RA2CbNGrz8","name":"radicle-ci-broker","description":"Radicle CI broker","private":false,"default_branch":"main","delegates":["did:key:z6MkgEMYod7Hxfy9qCvDv5hYHkZ4ciWmLFgfvm3Wn1b2w2FV"]},"pusher":{"id":"did:key:z6MkgEMYod7Hxfy9qCvDv5hYHkZ4ciWmLFgfvm3Wn1b2w2FV","alias":"liw"},"before":"b4fb1e347be7db19f0859717062f94116b5bec9f","after":"b4fb1e347be7db19f0859717062f94116b5bec9f","branch":"patches/8d8232ddcb217fa1402eec4d955e227ef3bb5881","commits":[]}
+
+Triggered response:
+{"response":"triggered","run_id":{"id":"any-string-works-as-run-id"}}
+
+Successful response:
+{"response":"finished","result":"success"}
+
+Failure response:
+{"response":"finished","result":"failure"}
+
+Error response:
+{"response":"finished","result":{"error":"error message\nsecond line"}}
+~~~
 
 ## Push Event Request
 
@@ -113,7 +158,7 @@ An example request that the broker sends looks like this:
     },
     "before": "<BEFORE_COMMIT>",
     "after": "<AFTER_COMMIT>",
-    "branch": "<BRANCH_NAME>"
+    "branch": "<BRANCH_NAME>",
     "commits": [
         "<SOME_OTHER_COMMIT_BEING_PUSHED>",
         "<AFTER_COMMIT>"
@@ -138,10 +183,9 @@ where:
   - `<RID>` is the repository ID, in its `rad:` URN format,
   - `<BRANCH_NAME>` is the branch name where the push occurred,
   - `<AFTER_COMMIT>` is the commit id of the last commit being pushed,
-  - `<BEFORE_COMMIT>` is the commit id of the **parent** of the first 
-
-commit being pushed (i.e. ` <SOME_OTHER_COMMIT_BEING_PUSHED>`),
-(the SHA checksum). 
+  - `<BEFORE_COMMIT>` is the commit id of the **parent** of the first
+     commit being pushed (i.e. ` <SOME_OTHER_COMMIT_BEING_PUSHED>`),
+     (the SHA checksum).
 
 The `request` fields allows us to extend this in the future.
 
@@ -219,10 +263,9 @@ where:
 
   - `<RID>` is the reposiatory ID, in its `rad:` URN format,
   - `<AFTER_COMMIT>` is the commit id of the last commit being pushed,
-  - `<BEFORE_COMMIT>` is the commit id of the **parent** of the first 
-
-commit being pushed (i.e. ` <SOME_OTHER_COMMIT_BEING_PUSHED>`),
-(the SHA checksum). 
+  - `<BEFORE_COMMIT>` is the commit id of the **parent** of the first
+    commit being pushed (i.e. ` <SOME_OTHER_COMMIT_BEING_PUSHED>`),
+    (the SHA checksum).
 
 The `request` fields allows us to extend this in the future.
 
@@ -249,4 +292,30 @@ The second response from the adapter looks like this:
 ~~~
 
 where `<STATUS>` is either the string `success` or `failure`. Note
-that the run id is not repeated as the context makes this clear.
+that the run id is not repeated as the context makes this clear: the
+response comes from the same process, via the same stdout pipe, as the
+previous message.
+
+# Report generation
+
+The CI broker has an SQLite database file for persistent storage of
+information of the CI runs it triggers. This is used to generate
+report pages.
+
+The report pages are HTML, generated from the information in the
+database. The broker loads information for all runs when it starts,
+from the database, and then pushed information about new runs when
+they happens. This avoids the broker having to read everything every
+time it updates the report pages.
+
+The report page generation is done in its own thread, separate from
+the main thread of the CI broker. This allows the reporting to happens
+independently from what the main thread is doing. In particular, it
+means the main thread does not need to do anything to trigger reports
+from being updated.
+
+When it comes to per-run logs, for external CI these are kept by the
+external CI instance and the broker never sees them. For native CI,
+the native CI adapter writes them to the report directory, as
+`$RUNID/log.html`, and the broker generates report pages that link to
+those files.
diff --git a/doc/architecture.uml b/doc/architecture.uml
index 4b53913..85027c4 100644
--- a/doc/architecture.uml
+++ b/doc/architecture.uml
@@ -4,7 +4,5 @@ broker -> adapter : spawn
 broker -> adapter : send request
 adapter -> broker : response: run id
 note over adapter : perform the run
-broker -> node : create CI COB
 adapter -> broker : response: result
-broker -> node : update CI COB
 @enduml
diff --git a/doc/comp-ext.dot b/doc/comp-ext.dot
new file mode 100644
index 0000000..246db7a
--- /dev/null
+++ b/doc/comp-ext.dot
@@ -0,0 +1,12 @@
+digraph "" {
+   radicle_node [label="Radicle node"];
+   broker [label="CI broker"];
+   adapter [label="Adapter"];
+   engine [label="External CI system"];
+
+   radicle_node -> broker [label="change event"];
+   broker -> adapter [label="invoke"];
+   adapter -> engine [label="run"];
+   engine -> adapter [label="web hook?"];
+   adapter -> broker [label="result"];
+}
\ No newline at end of file
diff --git a/doc/comp.dot b/doc/comp.dot
new file mode 100644
index 0000000..f6d34d8
--- /dev/null
+++ b/doc/comp.dot
@@ -0,0 +1,11 @@
+digraph "" {
+   radicle_node [label="Radicle node"];
+   broker [label="CI broker"];
+   native [label="Native CI"];
+   commands [label="Shell commands"];
+
+   radicle_node -> broker [label="change event"];
+   broker -> native [label="invoke"];
+   native -> commands [label="run"];
+   native -> broker [label="result"];
+}
\ No newline at end of file
diff --git a/doc/components-ext.pik b/doc/components-ext.pik
deleted file mode 100644
index 7b73dc0..0000000
--- a/doc/components-ext.pik
+++ /dev/null
@@ -1,24 +0,0 @@
-down
-Node: cylinder "node"
-move
-Broker: box "CI broker"
-move
-Adapter: box "external" "adapter"
-right
-move
-
-Ext: [
-    Engine: box "External" "CI engine"
-    move
-    Worker: box "CI worker"
-    arrow from Engine.e to Worker.w "control" aligned above
-]
-Border: box thin width Ext.width+5mm height Ext.height+5mm at Ext.center
-Caption: text "External host" with .n at 5mm below Ext.s
-
-arrow from Node.s to Broker.n "event" aligned above
-arrow from Broker.s to Adapter.n "spawn" aligned above
-arrow from Adapter.e to Ext.w "trigger" aligned above
-arc from Ext.nw to Adapter.ne ->
-move from last arc.n up 0.0
-text "webhook"
diff --git a/doc/components.pik b/doc/components.pik
deleted file mode 100644
index 694e255..0000000
--- a/doc/components.pik
+++ /dev/null
@@ -1,11 +0,0 @@
-down
-Node: cylinder "node"
-move
-Broker: box "CI broker"
-move
-Adapter: box "native" "adapter"
-
-arrow thin from Node.s to Broker.n "event" aligned above
-arc thin from Broker.sw to Adapter.nw -> "spawn" aligned above
-arrow thin from Broker.s to Adapter.n "request" aligned above
-arc thin from Adapter.ne to Broker.se -> "responses" aligned
\ No newline at end of file