Message-ID: <5e2b278847b76dd0311d5050b3455b12b4dd3077.camel@liw.fi>
From: Lars Wirzenius <liw@liw.fi>
To: ick-discuss@ick.liw.fi
Date: Fri, 08 Feb 2019 13:42:10 -0800
User-Agent: Evolution 3.30.4-1 
Mime-Version: 1.0
Subject: Plan for using Muck for the Ick controller
Precedence: list
Content-Type: multipart/mixed; boundary="===============6608641942915556769=="
Mime-version: 1.0
Sender: ick-discuss-bounces@ick.liw.fi
Errors-To: ick-discuss-bounces@ick.liw.fi


--===============6608641942915556769==
Content-Type: multipart/signed; micalg="pgp-sha512";
 protocol="application/pgp-signature"; boundary="=-2hGZcifWjHnZE7/eshss"


--=-2hGZcifWjHnZE7/eshss
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

TL;DR: The Ick controller can be changed in a straightforward manner
to use Muck for persistent data storage, except for log files. It
doesn't seem worthwhile to support an in-place conversion. Instead,
the few people affected can re-create their projects and pipelines on
a new, fresh demo Ick instance, which will replace the current one.

TS;RF (too short; rambling follows):

Currently, the Ick controller stores various resources directly in its
local filesystem. I wish to change the controller to use Muck instead.
The main motivation for this is to have better access control: users
of the same controller shouldn't see or be able to change projects or
pipelines for other users.

Muck is a JSON data store with access control. Part of the access
control is that every JSON object ("resource") is owned by a specific
user. Every user can only access their own objects, for now (although
this will become more flexible later).

Where the controller currently creates, say, a resource for the
project foo, by storing it as YAML in the file
"/var/lib/ick/state/projects/foo", I plan to change that so that it
creates a JSON resource in Muck like this:

    {
        "_type": "project",
        "_name": "foo",
        ... # all other fields as in the current YAML file
    }

Muck invents a unique identifier for each object, and guarantees no
other object has that identifier. The controller will not use this,
and will instead use a search on the "_type" and "_name" fields to
find the right object. This is so that users may refer to projects and
pipelines using more humane names: "ick.liw.fi" instead of
"48053f4f-71d9-42a1-b3ca-8574cbb788aa" for example.

Muck allows arbitrary JSON objects to be stored. For the Ick
controller, the following approach seems like a reasonable first
attempt:

* Each object will have a "_type" field, which specifies the type of
  the object: project, pipeline, build, log, or worker. This is needed
  so that one can search for a "project foo", as opposed to "pipeline
  foo".

* Each object will have a user-assigned "_name" field, which the
  controller makes sure is unique and prefixed with the object owner's
  username.

  Muck does not have transactions that span multiple HTTP requests.
  The controller will do its best to ensure a name is unique, but it
  can't guarantee that. However, if it notices a name clash later, it
  will treat that as an error. For example, if there are two projects
  named "liw/ick.liw.fi", the controller will refuse to trigger
  either. (This is a limitation in Muck and will be fixed later,
  possibly by teaching Muck about user-defined names for resources,
  and having it make sure they're unique. But that will have to wait
  for a later version of Muck.)

* Depending on the type of the object, it may contain other fields as
  well, see below.

* The controller creates objects in Muck based on API calls to the
  controller, and passes on the access token it gets. Muck uses the
  access token's "sub" field to assign ownership of the object; some
  access tokens do not have such a field, and can thus not be
  accressed by any user; this will be the case for workers that
  register themselves.

The following object types will be supported initially:

* project - projects defined by the user
* pipeline - pipelines defined by the user
* build - a build that's been triggered, is running, or is finished
* log - a build log
* worker - a worker

The contents of these object types are as they are in the controller
now. Switching to Muck does not change that, except for logs, which
need special handling to avoid very bad performance.

Muck stores all incoming resources to a "change log". A build log may
get thousands of updates: each line of output may become an update.
Doing a new update of the log object would result in thousands of
nearly identical copies in the change log, which is quite wasteful:
each new version of the log resource would be identical to the
previous one, except with a line of new text. When a build produces a
thousand lines of output, Muck would store a thousand copies of the
log object in its change log.

To avoid this waste, the controller will be changed to store the build
logs as follows:

* The worker-manager will send each build log snippet as a separate
  update, as before, but it will also add a sequence number to the
  updates.

* The controller will create a separate log object for reach update in
  Muck. This object will contain only the new snippet. The new log
  object will have the log snippet, plus a reference to the build for
  which it is part of, and the sequence number.

* The controller will reconstruct the whole build log when it's
  requested for a complete log, by fetching all log objects that refer
  to the specified build, and catenating the log output snippets in
  order of the sequence number.

I think this will work, but I haven't written any code yet. Comments?
I plan on starting work on this next week, when I get back from gallivantin=
g
in SF.

I don't plan on converting existing projects and pipeline on the demo Ick
instance. That seems like work that would be useful, but it's also work
that's maybe not worthwhile yet, given the demo Ick instance only has a few
users. I'm lazy, sorry.


--=-2hGZcifWjHnZE7/eshss
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part
Content-Transfer-Encoding: 7bit

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEETNTnrewG6wEE1EJ3bC+mFux6IDEFAlxd97IACgkQbC+mFux6
IDE+uA//ciOWY53Sv6SW4p5fOZPZ0SGtOJEP2+Y36Bx+b9s0v4/RpW1b00Zs6alT
E50Y4MoFBD2IR6S/jPs2kDlve4CSrIUiI0gfMt5iBWrcJK3GlzqvOK33RY+ofQFW
OCdA5o1U/YFdSZzdJqJJqx5hg0BKrR887WXKwQOX/K9eq8hCSgtrtKNho2TgPO/s
8s3hveugFXJND8AtEuhuTyuh3e3PhKluiZYt7gTjCFyYMi83I/0k1i0SX1xlUisl
IV83ZbnAfQWC9f9q6b4Qmq/rmEEMJEchwGKKlrbbQx1A2H8yAo1Rq+TKonygXCSA
58WVTBpkREh42X9AzLsk53hQmGtrKZ4GTr/tilr8rStxwfWGO07v19mtdTLuAhYx
xKR621JcGv06TYE6BUmCpW45fKyGkfWFOzfyQrMK47Of/Ke0/oFspWkswzHZInvk
O/cGPjjjS7/2+ouXiiXNy45te26uyWU1uIy6bwYCgqZqgQsT59pgOb8UvqinG6rW
qz6Bhqj2AkQCHHa1TJHABV7vgC5YxUMy9l4UnmlfJbTk/vBPMTBk1KXjvltfHPlL
3EvC/o2zU096Q6IurxnYcdbeO1+zP3tv9T3tCxrQ9KTgfTvB2WFQ4xXAl6EXS8sp
zp92RajksrkbunjGwB6DBxQqKn2NbpvioIljZUWlMn0Nk5IkVpU=
=hHLc
-----END PGP SIGNATURE-----

--=-2hGZcifWjHnZE7/eshss--


--===============6608641942915556769==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
ick-discuss mailing list
ick-discuss@ick.liw.fi
https://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/ick-discuss-ick.liw.fi

--===============6608641942915556769==--