Date: Wed, 4 Jul 2018 10:29:14 +0300
From: Lars Wirzenius <liw@liw.fi>
To: ick-discuss@ick.liw.fi
Message-ID: <20180704072913.GA7501@exolobe3.liw.fi>
References: <20180702152447.GA2363@exolobe3.liw.fi>
 <20180703122826.siolafamphg7mpfo@somnambulist.local>
MIME-Version: 1.0
In-Reply-To: <20180703122826.siolafamphg7mpfo@somnambulist.local>
User-Agent: Mutt/1.10.0 (2018-05-17)
Subject: Re: Making the "git" action in ick more useful
Precedence: list
Content-Type: multipart/mixed; boundary="===============0125566059845942189=="
Mime-version: 1.0
Sender: ick-discuss-bounces@ick.liw.fi
Errors-To: ick-discuss-bounces@ick.liw.fi


--===============0125566059845942189==
Content-Type: multipart/signed; micalg=pgp-sha512;
 protocol="application/pgp-signature"; boundary="dDRMvlgZJXvWKvBx"
Content-Disposition: inline


--dDRMvlgZJXvWKvBx
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Jul 03, 2018 at 01:28:26PM +0100, Daniel Silverstone wrote:
> On Mon, Jul 02, 2018 at 18:24:48 +0300, Lars Wirzenius wrote:
> > However, the above design isn't flexible enough for Daniel, who needs
> > to access multiple git repositories, not just once, and build a tree
> > of clones. I'll let Daniel tell us what his use-case is like. (I think
> > I know, but it's better if he tells it himself. I can be a bit of a
> > broken telephone.)
>
> [description of use cases omitted]

> I have many projects, luxio, lua-scrypt, clod, gall, supple, lace, tongue,
> and gitano, all of which can be built with essentially the same pipeline
> stages, so long as they are parameterised by the project name.  As such,
> my projects have a reponame parameter.  Given that, if we assumed that
> the git action could be parameterised in the pipelines:
>=20
> - action: git
>   repo: "git://git.gitano.org.uk/{{ reponame }}.git"
>   ref: "{{ refname }}"
>   path: "{{ reponame }}"
> - action: git
>   repo: "git://git.gitano.org.uk/{{ reponame }}/debian.git"
>   ref: "{% if debianrefname %}{{ debianrefname }}{% else %}{{ refname }}{=
% endif %}"
>   path: "{{ reponame }}/debian"
> - action: git
>   repo: git://git.gitano.org.uk/gp-packaging-tools.git
>   ref: master
>   path: gp-packaging-tools
>=20
> would be approximately what I'd need for my git checkout pipeline.
>=20
> If "optional" parameters is not a thing, then the debianrefname parameter=
 would
> just have to be provided at all times, defaulting to master as refname wo=
uld.
>=20
> If, on the other hand, the ref was irrelevant because it was expected that
> there'd be a stage doing `git checkout ${refname}` or similar later, then=
 I can
> handle the case of debianrefname being empty/not-provided in shell.
>=20
> My "out-there" use-case is what I call, in Gitano, system branches.  That=
 works
> by having branches of the same name in any/all of the above repos, and th=
en
> they're checked out together (along with master for any repo without the
> branch) and all built together for test purposes.  This project would not
> produce any debs because it purely exists to prove out the test suite on a
> system other than my development system, and I have tooling to produce the
> checkouts and build the code, providing that it can run somewhere with ne=
twork
> access.  We'd need something like:
>=20
> - action: git
>   repo: "git://git.gitano.org.uk/gitano.git"
>   ref: "{{ sysbranchref }}"
>   fallback-ref: master
>   path: gitano
>=20
> Where fallback-ref is checked out if ref isn't available in the remote.
>=20
> I don't know how much of this you, or others, would want in Ick, but ther=
e you
> have it.
>=20
> For me, the obviously desirable option is (a) that the git action can take
> repo, ref, and path directly in the pipeline stage, and (b) that we can u=
se
> some kind of syntax to interpolate parameters into those strings (using j=
inja2
> seems obvious, though requiring that at the ick controller end might be a
> smidge iffy for maintaining the possibility of replacing the python
> implementation in the future).
>=20
> I look forward to your take on all the above,
>=20
> D.
>=20
> --=20
> Daniel Silverstone                         http://www.digital-scurf.org/
> PGP mail accepted and encouraged.            Key Id: 3CCE BABE 206C 3B69
>=20
> _______________________________________________
> ick-discuss mailing list
> ick-discuss@ick.liw.fi
> https://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/ick-discuss-ic=
k.liw.fi
>

Thanks, Daniel. That's illuminating and interesting.

I'm going to rephrase what you wrote, to show how I understood it, and
to suggest something.

First, a comment on adding a templating language on top of YAML. I
first encountered this with Ansible, and I've since used in my own
vmdb2 project. It's a powerful concept, but I find it to sometimes be
problematic: the input files aren't actually YAML anymore, and they
become harder to understand and harder to debug. Sometimes that's OK,
the added power is worth it. However, I'd like to avoid that for ick,
for the time being.

(Also, as you say, using jinja2 for templating might become
problematic if and when ick gets rewritten in another language.)

Thus, I'll write out your examples without templating. They'll be more
verbose, and more repetitive, but should be more obvious as to
meaning.

That also means I'm, at this time, not ready to have action attributes
that vary from project to project. Perhaps that'd be a useful addition
later on, but it seems to require some form of (templating) language
on top of YAML.

Your first use case seems to me to be like this:

* all your projects live on the same git server, next to each other
* your main repository for a component is foo.git, branch master
* its Debian packaging is in foo-debian.git, branch master for unstable
* additionally, you have some common build tooling in
* gp-packaging-tools.git

I note that this has a fairly tight coupling between the repositories
for the main code and the Debian packaging. For ick, I'd prefer to
avoid mandating that: one use case might be that I want to package
your code and I put the Debian packaging on my server, while your code
remains on yours.

My suggestion would be to change the git action to look for a project
parameter "git" (or "sources"?), which would be a *list* of
url/ref/dir tuples. For you, each project would have a parameter like
this:

    - project: foo
      parameters:
        git:
        - url: git://git.gitano.org.uk/foo.git
          ref: master
          dir: src
        - url: git://git.gitano.org.uk/foo-debian.git
          ref: debian/unstable
          dir: src/debian
        - url: git://git.gitano.org.uk/gp-packaging-tools.git
          ref: master
          dir: gp-packaging-tools

Which would result in a directory tree like this:

    /workspace
    /workspace/src
    /workspace/src/.git
    /workspace/src/debian
    /workspace/src/debian/.git
    /workspace/gp-packaging-tools
    /workspace/gp-packaging-tools/.git

I admit the git parameter gets a bit repetitive. If that's enough of a
problem, I'd be willing to make the git action understand the optional
parameter "git_server" (or "git_base_url"):

    - project: foo
      parameters:
        git_server: git:///git.gitano.org.uk
        git:
        - repo: foo.git
          ref: master
          dir: src
        - repo: foo-debian.git
          ref: debian/unstable
          dir: src/debian
        - repo: gp-packaging-tools.git
          ref: master
          dir: gp-packaging-tools

In the tuples in the git parameter, url would still work and would be
the full URL to the repository. Otherwise, the repo field would be
appended to the git_server value.

Also, we could make "master" be the default branch. Actually, that
suggests to me that a mechanism for providing defaults might be
useful. Maybe something like this:

    - project: foo
      parameters:
        git_defaults:
          server: git:///git.gitano.org.uk
          ref: master
        git:
        - repo: foo.git
          dir: src
        - repo: foo-debian.git
          ref: debian/unstable
          dir: src/debian
        - repo: gp-packaging-tools.git
          dir: gp-packaging-tools
        - server: git://git.liw.fi
          repo: liw-stuff.git

In thie example, the url/ref/dir tuple is replaced with a
server/repo/ref/dir tuple and the full repo URL is built by combining
server and repo fields. (Though it might be helpful to provide a url
field that can be used instead of server+repo.) See below for
overriding values in triggers.

Handling a list of dicts in shell with jq is not necessarily the kind
of thing I enjoy doing. I'd do it in Python instead, where it's not as
awkward.

Your system branch use case is interesting. My initial suggestion for
that would be change ick to allow a trigger call, which starts a new
build, to override project parameters for a specific build. Currnetly
it's just a simple GET:

    GET /projects/foo/+trigger

We could add another way to do triggers:

    POST /projects/foo/+trigger
    Content-Type: application/json

    {
        "parameters": {
            "git_defaults": {
                "ref": "liw/fix"
            }
        }
    }

This would set the "ref" field in the "git_defaults" parameter to
"liw/fix". The "server" field would not be changed. The
foo-debian.git still use the debian/unstable branch.

While this would not be as powerful as full jinja2 support, it would,
I think, be a useful feature, entirely separate from templating.

What I suggest here isn't really what you suggested, but what do you
think? Would this be workable?

--=20
I want to build worthwhile things that might last. --joeyh

--dDRMvlgZJXvWKvBx
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEETNTnrewG6wEE1EJ3bC+mFux6IDEFAls8d0kACgkQbC+mFux6
IDFK2w//fIADlC7DdPcm8qORw0V8Cxr0L3XojFWVxeSdCV2aZdc6SXSraxczDzix
LP4911zPdi/l6qalFQNlsWBwjWkF3AEvKvwfrc5bMduI351VOm8bs/px9qjPY+6e
phDkmN7UdkGmvBArlFmDgfi+U3wYI8nhkMCSMh4vz+GfLWkugBmCcyTJzBvWa034
Q/0xYqewOgSXbrSMV6U7lXlqZDL6im4/Nbz0KpLZI0KmQZFOuL+Is7DQLIJnV3m9
a/cBGcb0frJQf2kb9da/knSPt3GcZFG3DjNdROsBzJvnQIOQkNIxeF2ituWPCTiv
mSc4ApudXWVUKVGVrRQA1t/m0rAqItXznewSHp15uBu9PS6HZ1AtDNavbPXuIFC9
YkPsb9PauTQnsKirjFFOBHSo1AmMK+lDXd7xi5Y0IEU+rxiL63dhpQYRbg9MLx+O
jaqFPIy/2oay1kcQgtEvAodS38KFft93ayUlqLZfNBAJyM+aMSO3kNvisECyxNj+
pGcqeE9zOdCcQMb95kgEhbq0RERnQoYp/XNKPwTyhYFVTykXuSqtM7rWSfE+UQmp
Dqi7kAJOVnvRLTm1FSY6lQ/rIyUny5+IbubLUVmknvJa38Tu+UlBj/Xiw9hbHtHP
KgoUYcXpZkLdryJ0AcFx8j7Lemh8AP6qsDrYpPfmWuZa5sVIIfA=
=32jP
-----END PGP SIGNATURE-----

--dDRMvlgZJXvWKvBx--


--===============0125566059845942189==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
ick-discuss mailing list
ick-discuss@ick.liw.fi
https://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/ick-discuss-ick.liw.fi

--===============0125566059845942189==--