From 53900f1562bb49bdd10e45c6fba2208cdf8d7fd3 Mon Sep 17 00:00:00 2001 From: distix ticketing system Date: Wed, 4 Jul 2018 07:30:07 +0000 Subject: imported mails --- .../Maildir/new/1530689407.M245846P12632Q1.koom | 340 +++++++++++++++++++++ 1 file changed, 340 insertions(+) create mode 100644 tickets/d20aa3eb7bb042c2ba3912f8086deef7/Maildir/new/1530689407.M245846P12632Q1.koom diff --git a/tickets/d20aa3eb7bb042c2ba3912f8086deef7/Maildir/new/1530689407.M245846P12632Q1.koom b/tickets/d20aa3eb7bb042c2ba3912f8086deef7/Maildir/new/1530689407.M245846P12632Q1.koom new file mode 100644 index 0000000..b75c95f --- /dev/null +++ b/tickets/d20aa3eb7bb042c2ba3912f8086deef7/Maildir/new/1530689407.M245846P12632Q1.koom @@ -0,0 +1,340 @@ +Return-Path: +X-Original-To: distix@pieni.net +Delivered-To: distix@pieni.net +Received: from yaffle.pepperfish.net (yaffle.pepperfish.net [88.99.213.221]) + by pieni.net (Postfix) with ESMTPS id 5C3CC41271 + for ; Wed, 4 Jul 2018 07:29:17 +0000 (UTC) +Received: from platypus.pepperfish.net (unknown [10.112.101.20]) + by yaffle.pepperfish.net (Postfix) with ESMTP id 35BC3415BE + for ; Wed, 4 Jul 2018 08:29:17 +0100 (BST) +Received: from ip6-localhost.nat ([::1] helo=platypus.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1facE1-0002fK-5n; Wed, 04 Jul 2018 08:29:17 +0100 +Received: from koom.pieni.net ([88.99.190.206] helo=pieni.net) + by platypus.pepperfish.net with esmtpsa (Exim 4.80 #2 (Debian)) + id 1facE0-0002f8-Df + for ; Wed, 04 Jul 2018 08:29:16 +0100 +Received: from exolobe3.liw.fi (62-78-212-250.bb.dnainternet.fi + [62.78.212.250]) by pieni.net (Postfix) with ESMTPSA id 0DE4641271 + for ; Wed, 4 Jul 2018 07:29:16 +0000 (UTC) +Received: from exolobe3.liw.fi (localhost [127.0.0.1]) + by exolobe3.liw.fi (Postfix) with ESMTPS id 238388811CB + for ; Wed, 4 Jul 2018 10:29:15 +0300 (EEST) +Date: Wed, 4 Jul 2018 10:29:14 +0300 +From: Lars Wirzenius +To: ick-discuss@ick.liw.fi +Message-ID: <20180704072913.GA7501@exolobe3.liw.fi> +References: <20180702152447.GA2363@exolobe3.liw.fi> + <20180703122826.siolafamphg7mpfo@somnambulist.local> +MIME-Version: 1.0 +In-Reply-To: <20180703122826.siolafamphg7mpfo@somnambulist.local> +User-Agent: Mutt/1.10.0 (2018-05-17) +X-Pepperfish-Transaction: 8fe6-8500-7903-5a56 +X-Pepperfish-Transaction-By: platypus +Subject: Re: Making the "git" action in ick more useful +X-BeenThere: ick-discuss@ick.liw.fi +X-Mailman-Version: 2.1.5 +Precedence: list +List-Id: discussions about the ick CI system +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +Content-Type: multipart/mixed; boundary="===============0125566059845942189==" +Mime-version: 1.0 +Sender: ick-discuss-bounces@ick.liw.fi +Errors-To: ick-discuss-bounces@ick.liw.fi + + +--===============0125566059845942189== +Content-Type: multipart/signed; micalg=pgp-sha512; + protocol="application/pgp-signature"; boundary="dDRMvlgZJXvWKvBx" +Content-Disposition: inline + + +--dDRMvlgZJXvWKvBx +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +Content-Transfer-Encoding: quoted-printable + +On Tue, Jul 03, 2018 at 01:28:26PM +0100, Daniel Silverstone wrote: +> On Mon, Jul 02, 2018 at 18:24:48 +0300, Lars Wirzenius wrote: +> > However, the above design isn't flexible enough for Daniel, who needs +> > to access multiple git repositories, not just once, and build a tree +> > of clones. I'll let Daniel tell us what his use-case is like. (I think +> > I know, but it's better if he tells it himself. I can be a bit of a +> > broken telephone.) +> +> [description of use cases omitted] + +> I have many projects, luxio, lua-scrypt, clod, gall, supple, lace, tongue, +> and gitano, all of which can be built with essentially the same pipeline +> stages, so long as they are parameterised by the project name. As such, +> my projects have a reponame parameter. Given that, if we assumed that +> the git action could be parameterised in the pipelines: +>=20 +> - action: git +> repo: "git://git.gitano.org.uk/{{ reponame }}.git" +> ref: "{{ refname }}" +> path: "{{ reponame }}" +> - action: git +> repo: "git://git.gitano.org.uk/{{ reponame }}/debian.git" +> ref: "{% if debianrefname %}{{ debianrefname }}{% else %}{{ refname }}{= +% endif %}" +> path: "{{ reponame }}/debian" +> - action: git +> repo: git://git.gitano.org.uk/gp-packaging-tools.git +> ref: master +> path: gp-packaging-tools +>=20 +> would be approximately what I'd need for my git checkout pipeline. +>=20 +> If "optional" parameters is not a thing, then the debianrefname parameter= + would +> just have to be provided at all times, defaulting to master as refname wo= +uld. +>=20 +> If, on the other hand, the ref was irrelevant because it was expected that +> there'd be a stage doing `git checkout ${refname}` or similar later, then= + I can +> handle the case of debianrefname being empty/not-provided in shell. +>=20 +> My "out-there" use-case is what I call, in Gitano, system branches. That= + works +> by having branches of the same name in any/all of the above repos, and th= +en +> they're checked out together (along with master for any repo without the +> branch) and all built together for test purposes. This project would not +> produce any debs because it purely exists to prove out the test suite on a +> system other than my development system, and I have tooling to produce the +> checkouts and build the code, providing that it can run somewhere with ne= +twork +> access. We'd need something like: +>=20 +> - action: git +> repo: "git://git.gitano.org.uk/gitano.git" +> ref: "{{ sysbranchref }}" +> fallback-ref: master +> path: gitano +>=20 +> Where fallback-ref is checked out if ref isn't available in the remote. +>=20 +> I don't know how much of this you, or others, would want in Ick, but ther= +e you +> have it. +>=20 +> For me, the obviously desirable option is (a) that the git action can take +> repo, ref, and path directly in the pipeline stage, and (b) that we can u= +se +> some kind of syntax to interpolate parameters into those strings (using j= +inja2 +> seems obvious, though requiring that at the ick controller end might be a +> smidge iffy for maintaining the possibility of replacing the python +> implementation in the future). +>=20 +> I look forward to your take on all the above, +>=20 +> D. +>=20 +> --=20 +> Daniel Silverstone http://www.digital-scurf.org/ +> PGP mail accepted and encouraged. Key Id: 3CCE BABE 206C 3B69 +>=20 +> _______________________________________________ +> ick-discuss mailing list +> ick-discuss@ick.liw.fi +> https://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/ick-discuss-ic= +k.liw.fi +> + +Thanks, Daniel. That's illuminating and interesting. + +I'm going to rephrase what you wrote, to show how I understood it, and +to suggest something. + +First, a comment on adding a templating language on top of YAML. I +first encountered this with Ansible, and I've since used in my own +vmdb2 project. It's a powerful concept, but I find it to sometimes be +problematic: the input files aren't actually YAML anymore, and they +become harder to understand and harder to debug. Sometimes that's OK, +the added power is worth it. However, I'd like to avoid that for ick, +for the time being. + +(Also, as you say, using jinja2 for templating might become +problematic if and when ick gets rewritten in another language.) + +Thus, I'll write out your examples without templating. They'll be more +verbose, and more repetitive, but should be more obvious as to +meaning. + +That also means I'm, at this time, not ready to have action attributes +that vary from project to project. Perhaps that'd be a useful addition +later on, but it seems to require some form of (templating) language +on top of YAML. + +Your first use case seems to me to be like this: + +* all your projects live on the same git server, next to each other +* your main repository for a component is foo.git, branch master +* its Debian packaging is in foo-debian.git, branch master for unstable +* additionally, you have some common build tooling in +* gp-packaging-tools.git + +I note that this has a fairly tight coupling between the repositories +for the main code and the Debian packaging. For ick, I'd prefer to +avoid mandating that: one use case might be that I want to package +your code and I put the Debian packaging on my server, while your code +remains on yours. + +My suggestion would be to change the git action to look for a project +parameter "git" (or "sources"?), which would be a *list* of +url/ref/dir tuples. For you, each project would have a parameter like +this: + + - project: foo + parameters: + git: + - url: git://git.gitano.org.uk/foo.git + ref: master + dir: src + - url: git://git.gitano.org.uk/foo-debian.git + ref: debian/unstable + dir: src/debian + - url: git://git.gitano.org.uk/gp-packaging-tools.git + ref: master + dir: gp-packaging-tools + +Which would result in a directory tree like this: + + /workspace + /workspace/src + /workspace/src/.git + /workspace/src/debian + /workspace/src/debian/.git + /workspace/gp-packaging-tools + /workspace/gp-packaging-tools/.git + +I admit the git parameter gets a bit repetitive. If that's enough of a +problem, I'd be willing to make the git action understand the optional +parameter "git_server" (or "git_base_url"): + + - project: foo + parameters: + git_server: git:///git.gitano.org.uk + git: + - repo: foo.git + ref: master + dir: src + - repo: foo-debian.git + ref: debian/unstable + dir: src/debian + - repo: gp-packaging-tools.git + ref: master + dir: gp-packaging-tools + +In the tuples in the git parameter, url would still work and would be +the full URL to the repository. Otherwise, the repo field would be +appended to the git_server value. + +Also, we could make "master" be the default branch. Actually, that +suggests to me that a mechanism for providing defaults might be +useful. Maybe something like this: + + - project: foo + parameters: + git_defaults: + server: git:///git.gitano.org.uk + ref: master + git: + - repo: foo.git + dir: src + - repo: foo-debian.git + ref: debian/unstable + dir: src/debian + - repo: gp-packaging-tools.git + dir: gp-packaging-tools + - server: git://git.liw.fi + repo: liw-stuff.git + +In thie example, the url/ref/dir tuple is replaced with a +server/repo/ref/dir tuple and the full repo URL is built by combining +server and repo fields. (Though it might be helpful to provide a url +field that can be used instead of server+repo.) See below for +overriding values in triggers. + +Handling a list of dicts in shell with jq is not necessarily the kind +of thing I enjoy doing. I'd do it in Python instead, where it's not as +awkward. + +Your system branch use case is interesting. My initial suggestion for +that would be change ick to allow a trigger call, which starts a new +build, to override project parameters for a specific build. Currnetly +it's just a simple GET: + + GET /projects/foo/+trigger + +We could add another way to do triggers: + + POST /projects/foo/+trigger + Content-Type: application/json + + { + "parameters": { + "git_defaults": { + "ref": "liw/fix" + } + } + } + +This would set the "ref" field in the "git_defaults" parameter to +"liw/fix". The "server" field would not be changed. The +foo-debian.git still use the debian/unstable branch. + +While this would not be as powerful as full jinja2 support, it would, +I think, be a useful feature, entirely separate from templating. + +What I suggest here isn't really what you suggested, but what do you +think? Would this be workable? + +--=20 +I want to build worthwhile things that might last. --joeyh + +--dDRMvlgZJXvWKvBx +Content-Type: application/pgp-signature; name="signature.asc" + +-----BEGIN PGP SIGNATURE----- + +iQIzBAABCgAdFiEETNTnrewG6wEE1EJ3bC+mFux6IDEFAls8d0kACgkQbC+mFux6 +IDFK2w//fIADlC7DdPcm8qORw0V8Cxr0L3XojFWVxeSdCV2aZdc6SXSraxczDzix +LP4911zPdi/l6qalFQNlsWBwjWkF3AEvKvwfrc5bMduI351VOm8bs/px9qjPY+6e +phDkmN7UdkGmvBArlFmDgfi+U3wYI8nhkMCSMh4vz+GfLWkugBmCcyTJzBvWa034 +Q/0xYqewOgSXbrSMV6U7lXlqZDL6im4/Nbz0KpLZI0KmQZFOuL+Is7DQLIJnV3m9 +a/cBGcb0frJQf2kb9da/knSPt3GcZFG3DjNdROsBzJvnQIOQkNIxeF2ituWPCTiv +mSc4ApudXWVUKVGVrRQA1t/m0rAqItXznewSHp15uBu9PS6HZ1AtDNavbPXuIFC9 +YkPsb9PauTQnsKirjFFOBHSo1AmMK+lDXd7xi5Y0IEU+rxiL63dhpQYRbg9MLx+O +jaqFPIy/2oay1kcQgtEvAodS38KFft93ayUlqLZfNBAJyM+aMSO3kNvisECyxNj+ +pGcqeE9zOdCcQMb95kgEhbq0RERnQoYp/XNKPwTyhYFVTykXuSqtM7rWSfE+UQmp +Dqi7kAJOVnvRLTm1FSY6lQ/rIyUny5+IbubLUVmknvJa38Tu+UlBj/Xiw9hbHtHP +KgoUYcXpZkLdryJ0AcFx8j7Lemh8AP6qsDrYpPfmWuZa5sVIIfA= +=32jP +-----END PGP SIGNATURE----- + +--dDRMvlgZJXvWKvBx-- + + +--===============0125566059845942189== +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit +Content-Disposition: inline + +_______________________________________________ +ick-discuss mailing list +ick-discuss@ick.liw.fi +https://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/ick-discuss-ick.liw.fi + +--===============0125566059845942189==-- + -- cgit v1.2.1