diff options
authordistix ticketing system <>2019-02-08 21:43:11 +0000
committerdistix ticketing system <>2019-02-08 21:43:11 +0000
commitea478846a93bceb1c0d1f623299d21371bce1125 (patch)
parent2b1d101a89a9f6aea428df6dbde340f170c97671 (diff)
imported mails
5 files changed, 219 insertions, 0 deletions
diff --git a/tickets/7503b896b90049d3a89ad5a7f4ed021d/Maildir/cur/.this-dir-not-empty/.empty/empty-file b/tickets/7503b896b90049d3a89ad5a7f4ed021d/Maildir/cur/.this-dir-not-empty/.empty/empty-file
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tickets/7503b896b90049d3a89ad5a7f4ed021d/Maildir/cur/.this-dir-not-empty/.empty/empty-file
diff --git a/tickets/7503b896b90049d3a89ad5a7f4ed021d/Maildir/new/.this-dir-not-empty/.empty/empty-file b/tickets/7503b896b90049d3a89ad5a7f4ed021d/Maildir/new/.this-dir-not-empty/.empty/empty-file
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tickets/7503b896b90049d3a89ad5a7f4ed021d/Maildir/new/.this-dir-not-empty/.empty/empty-file
diff --git a/tickets/7503b896b90049d3a89ad5a7f4ed021d/Maildir/new/1549662191.M217124P14948Q1.koom b/tickets/7503b896b90049d3a89ad5a7f4ed021d/Maildir/new/1549662191.M217124P14948Q1.koom
new file mode 100644
index 0000000..95dd600
--- /dev/null
+++ b/tickets/7503b896b90049d3a89ad5a7f4ed021d/Maildir/new/1549662191.M217124P14948Q1.koom
@@ -0,0 +1,213 @@
+Return-Path: <>
+Received: from ( [])
+ by (Postfix) with ESMTPS id 43452431FA
+ for <>; Fri, 8 Feb 2019 21:42:34 +0000 (UTC)
+Received: from (unknown [])
+ by (Postfix) with ESMTP id 1491A41318
+ for <>; Fri, 8 Feb 2019 21:42:34 +0000 (GMT)
+Received: from ip6-localhost.nat ([::1]
+ by with esmtp (Exim 4.80 #2 (Debian))
+ id 1gsDur-0006qU-Vo; Fri, 08 Feb 2019 21:42:33 +0000
+Received: from ([]
+ by with esmtpsa (Exim 4.80 #2 (Debian))
+ id 1gsDur-0006qF-0d
+ for <>; Fri, 08 Feb 2019 21:42:33 +0000
+Received: from exolobe4 ( [])
+ by (Postfix) with ESMTPSA id 44190431FA
+ for <>; Fri, 8 Feb 2019 21:42:30 +0000 (UTC)
+Message-ID: <>
+From: Lars Wirzenius <>
+Date: Fri, 08 Feb 2019 13:42:10 -0800
+User-Agent: Evolution 3.30.4-1
+Mime-Version: 1.0
+X-Pepperfish-Transaction: d5be-f6af-3d3b-972a
+X-Pepperfish-Transaction-By: platypus
+Subject: Plan for using Muck for the Ick controller
+X-Mailman-Version: 2.1.5
+Precedence: list
+List-Id: discussions about the ick CI system <>
+List-Unsubscribe: <>,
+ <>
+List-Archive: <>
+List-Post: <>
+List-Help: <>
+List-Subscribe: <>,
+ <>
+Content-Type: multipart/mixed; boundary="===============6608641942915556769=="
+Mime-version: 1.0
+Content-Type: multipart/signed; micalg="pgp-sha512";
+ protocol="application/pgp-signature"; boundary="=-2hGZcifWjHnZE7/eshss"
+Content-Type: text/plain; charset="UTF-8"
+Content-Transfer-Encoding: quoted-printable
+TL;DR: The Ick controller can be changed in a straightforward manner
+to use Muck for persistent data storage, except for log files. It
+doesn't seem worthwhile to support an in-place conversion. Instead,
+the few people affected can re-create their projects and pipelines on
+a new, fresh demo Ick instance, which will replace the current one.
+TS;RF (too short; rambling follows):
+Currently, the Ick controller stores various resources directly in its
+local filesystem. I wish to change the controller to use Muck instead.
+The main motivation for this is to have better access control: users
+of the same controller shouldn't see or be able to change projects or
+pipelines for other users.
+Muck is a JSON data store with access control. Part of the access
+control is that every JSON object ("resource") is owned by a specific
+user. Every user can only access their own objects, for now (although
+this will become more flexible later).
+Where the controller currently creates, say, a resource for the
+project foo, by storing it as YAML in the file
+"/var/lib/ick/state/projects/foo", I plan to change that so that it
+creates a JSON resource in Muck like this:
+ {
+ "_type": "project",
+ "_name": "foo",
+ ... # all other fields as in the current YAML file
+ }
+Muck invents a unique identifier for each object, and guarantees no
+other object has that identifier. The controller will not use this,
+and will instead use a search on the "_type" and "_name" fields to
+find the right object. This is so that users may refer to projects and
+pipelines using more humane names: "" instead of
+"48053f4f-71d9-42a1-b3ca-8574cbb788aa" for example.
+Muck allows arbitrary JSON objects to be stored. For the Ick
+controller, the following approach seems like a reasonable first
+* Each object will have a "_type" field, which specifies the type of
+ the object: project, pipeline, build, log, or worker. This is needed
+ so that one can search for a "project foo", as opposed to "pipeline
+ foo".
+* Each object will have a user-assigned "_name" field, which the
+ controller makes sure is unique and prefixed with the object owner's
+ username.
+ Muck does not have transactions that span multiple HTTP requests.
+ The controller will do its best to ensure a name is unique, but it
+ can't guarantee that. However, if it notices a name clash later, it
+ will treat that as an error. For example, if there are two projects
+ named "liw/", the controller will refuse to trigger
+ either. (This is a limitation in Muck and will be fixed later,
+ possibly by teaching Muck about user-defined names for resources,
+ and having it make sure they're unique. But that will have to wait
+ for a later version of Muck.)
+* Depending on the type of the object, it may contain other fields as
+ well, see below.
+* The controller creates objects in Muck based on API calls to the
+ controller, and passes on the access token it gets. Muck uses the
+ access token's "sub" field to assign ownership of the object; some
+ access tokens do not have such a field, and can thus not be
+ accressed by any user; this will be the case for workers that
+ register themselves.
+The following object types will be supported initially:
+* project - projects defined by the user
+* pipeline - pipelines defined by the user
+* build - a build that's been triggered, is running, or is finished
+* log - a build log
+* worker - a worker
+The contents of these object types are as they are in the controller
+now. Switching to Muck does not change that, except for logs, which
+need special handling to avoid very bad performance.
+Muck stores all incoming resources to a "change log". A build log may
+get thousands of updates: each line of output may become an update.
+Doing a new update of the log object would result in thousands of
+nearly identical copies in the change log, which is quite wasteful:
+each new version of the log resource would be identical to the
+previous one, except with a line of new text. When a build produces a
+thousand lines of output, Muck would store a thousand copies of the
+log object in its change log.
+To avoid this waste, the controller will be changed to store the build
+logs as follows:
+* The worker-manager will send each build log snippet as a separate
+ update, as before, but it will also add a sequence number to the
+ updates.
+* The controller will create a separate log object for reach update in
+ Muck. This object will contain only the new snippet. The new log
+ object will have the log snippet, plus a reference to the build for
+ which it is part of, and the sequence number.
+* The controller will reconstruct the whole build log when it's
+ requested for a complete log, by fetching all log objects that refer
+ to the specified build, and catenating the log output snippets in
+ order of the sequence number.
+I think this will work, but I haven't written any code yet. Comments?
+I plan on starting work on this next week, when I get back from gallivantin=
+in SF.
+I don't plan on converting existing projects and pipeline on the demo Ick
+instance. That seems like work that would be useful, but it's also work
+that's maybe not worthwhile yet, given the demo Ick instance only has a few
+users. I'm lazy, sorry.
+Content-Type: application/pgp-signature; name="signature.asc"
+Content-Description: This is a digitally signed message part
+Content-Transfer-Encoding: 7bit
+Content-Type: text/plain; charset="us-ascii"
+MIME-Version: 1.0
+Content-Transfer-Encoding: 7bit
+Content-Disposition: inline
+ick-discuss mailing list
diff --git a/tickets/7503b896b90049d3a89ad5a7f4ed021d/Maildir/tmp/.this-dir-not-empty/.empty/empty-file b/tickets/7503b896b90049d3a89ad5a7f4ed021d/Maildir/tmp/.this-dir-not-empty/.empty/empty-file
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tickets/7503b896b90049d3a89ad5a7f4ed021d/Maildir/tmp/.this-dir-not-empty/.empty/empty-file
diff --git a/tickets/7503b896b90049d3a89ad5a7f4ed021d/ticket.yaml b/tickets/7503b896b90049d3a89ad5a7f4ed021d/ticket.yaml
new file mode 100644
index 0000000..4f49bc0
--- /dev/null
+++ b/tickets/7503b896b90049d3a89ad5a7f4ed021d/ticket.yaml
@@ -0,0 +1,6 @@
+- open
+- 7503b896b90049d3a89ad5a7f4ed021d
+- Plan for using Muck for the Ick controller