Date: Thu, 5 Jul 2018 11:35:31 +0100
From: Daniel Silverstone <dsilvers@digital-scurf.org>
To: ick-discuss@ick.liw.fi
Message-ID: <20180705103531.v2jauqhcgcbhs7oo@somnambulist.local>
References: <20180702152447.GA2363@exolobe3.liw.fi>
 <20180703122826.siolafamphg7mpfo@somnambulist.local>
 <20180704072913.GA7501@exolobe3.liw.fi>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20180704072913.GA7501@exolobe3.liw.fi>
User-Agent: NeoMutt/20170113 (1.7.2)
Subject: Re: Making the "git" action in ick more useful
Precedence: list
Sender: ick-discuss-bounces@ick.liw.fi
Errors-To: ick-discuss-bounces@ick.liw.fi

On Wed, Jul 04, 2018 at 10:29:14 +0300, Lars Wirzenius wrote:
> Thanks, Daniel. That's illuminating and interesting.

I'm glad :-)

> I'm going to rephrase what you wrote, to show how I understood it, and
> to suggest something.

Fair, I'll respond inline here :-)

> First, a comment on adding a templating language on top of YAML. I
> first encountered this with Ansible, and I've since used in my own
> vmdb2 project. It's a powerful concept, but I find it to sometimes be
> problematic: the input files aren't actually YAML anymore, and they
> become harder to understand and harder to debug. Sometimes that's OK,
> the added power is worth it. However, I'd like to avoid that for ick,
> for the time being.
> 
> (Also, as you say, using jinja2 for templating might become
> problematic if and when ick gets rewritten in another language.)

Yes, absolutely.  At the "worst" we can come up with a convention by which
icktool expands things before it sends them to the controller, but let's avoid
templating languages for now.

> Your first use case seems to me to be like this:
> 
> * all your projects live on the same git server, next to each other
> * your main repository for a component is foo.git, branch master
> * its Debian packaging is in foo-debian.git, branch master for unstable
> * additionally, you have some common build tooling in
> * gp-packaging-tools.git

Yep (though foo/debian.git) :-)

> I note that this has a fairly tight coupling between the repositories
> for the main code and the Debian packaging. For ick, I'd prefer to
> avoid mandating that: one use case might be that I want to package
> your code and I put the Debian packaging on my server, while your code
> remains on yours.

Yes, the coupling is tight for the packages I have.

> My suggestion would be to change the git action to look for a project
> parameter "git" (or "sources"?), which would be a *list* of
> url/ref/dir tuples. For you, each project would have a parameter like
> this:
> 
>     - project: foo
>       parameters:
>         git:
>         - url: git://git.gitano.org.uk/foo.git
>           ref: master
>           dir: src
>         - url: git://git.gitano.org.uk/foo-debian.git
>           ref: debian/unstable
>           dir: src/debian
>         - url: git://git.gitano.org.uk/gp-packaging-tools.git
>           ref: master
>           dir: gp-packaging-tools
> 
> Which would result in a directory tree like this:
> 
>     /workspace
>     /workspace/src
>     /workspace/src/.git
>     /workspace/src/debian
>     /workspace/src/debian/.git
>     /workspace/gp-packaging-tools
>     /workspace/gp-packaging-tools/.git

That looks acceptable (presumably there'd need to be an action:git pipeline
step somewhere too?)

> I admit the git parameter gets a bit repetitive. If that's enough of a
> problem, I'd be willing to make the git action understand the optional
> parameter "git_server" (or "git_base_url"):
> 
>     - project: foo
>       parameters:
>         git_server: git:///git.gitano.org.uk
>         git:
>         - repo: foo.git
>           ref: master
>           dir: src
>         - repo: foo-debian.git
>           ref: debian/unstable
>           dir: src/debian
>         - repo: gp-packaging-tools.git
>           ref: master
>           dir: gp-packaging-tools

I'd call it repo_base and expect that repos which aren't absolute are urljoin'd
onto it as you suggest.

> Also, we could make "master" be the default branch. Actually, that
> suggests to me that a mechanism for providing defaults might be
> useful. Maybe something like this:
[snip]

Sounds good, though again I'd call it default_ref so that it's clear what its
purpose is.

> In thie example, the url/ref/dir tuple is replaced with a
> server/repo/ref/dir tuple and the full repo URL is built by combining
> server and repo fields. (Though it might be helpful to provide a url
> field that can be used instead of server+repo.) See below for
> overriding values in triggers.

As I suggested, I'd just go with "if repo is absolute, use it, otherwise
urljoin" as a non-surprising behaviour.

> Handling a list of dicts in shell with jq is not necessarily the kind
> of thing I enjoy doing. I'd do it in Python instead, where it's not as
> awkward.

Fair.

> Your system branch use case is interesting. My initial suggestion for
> that would be change ick to allow a trigger call, which starts a new
> build, to override project parameters for a specific build. Currnetly
> it's just a simple GET:
> 
>     GET /projects/foo/+trigger
> 
> We could add another way to do triggers:
> 
>     POST /projects/foo/+trigger
>     Content-Type: application/json
> 
>     {
>         "parameters": {
>             "git_defaults": {
>                 "ref": "liw/fix"
>             }
>         }
>     }
> 
> This would set the "ref" field in the "git_defaults" parameter to
> "liw/fix". The "server" field would not be changed. The
> foo-debian.git still use the debian/unstable branch.

So long as you have a mechanism for removing values as well, that'd work quite
well for the most part.  It still lacks the fallback for repos which lack the
ref, sadly, but perhaps that can't be usefully managed generically?

> While this would not be as powerful as full jinja2 support, it would,
> I think, be a useful feature, entirely separate from templating.
> 
> What I suggest here isn't really what you suggested, but what do you
> think? Would this be workable?

For my primary use-case this would be perfectly fine.  For the system branch
use-case it's still not quite enough to prevent a need for an additional
pipeline step to handle the checkouts.

I think, for now, we should go with the primary-use case stuff, and worry about
the complex trigger solution in another iteration in the future.  Do you want
to combine all of the stuff into a specification and then I'll review that so
we can close this work item off?

D.

-- 
Daniel Silverstone                         http://www.digital-scurf.org/
PGP mail accepted and encouraged.            Key Id: 3CCE BABE 206C 3B69

_______________________________________________
ick-discuss mailing list
ick-discuss@ick.liw.fi
https://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/ick-discuss-ick.liw.fi