1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
|
Return-Path: <ick-discuss-bounces@ick.liw.fi>
X-Original-To: distix@pieni.net
Delivered-To: distix@pieni.net
Received: from yaffle.pepperfish.net (yaffle.pepperfish.net [88.99.213.221])
by pieni.net (Postfix) with ESMTPS id 78237433A3
for <distix@pieni.net>; Thu, 5 Jul 2018 10:35:34 +0000 (UTC)
Received: from platypus.pepperfish.net (unknown [10.112.101.20])
by yaffle.pepperfish.net (Postfix) with ESMTP id 1FA29424AA
for <distix@pieni.net>; Thu, 5 Jul 2018 11:35:34 +0100 (BST)
Received: from ip6-localhost.nat ([::1] helo=platypus.pepperfish.net)
by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian))
id 1fb1bq-0001mG-0v; Thu, 05 Jul 2018 11:35:34 +0100
Received: from [148.252.241.226] (helo=somnambulist.local)
by platypus.pepperfish.net with esmtpsa (Exim 4.80 #2 (Debian))
id 1fb1bo-0001lf-Lg
for <ick-discuss@ick.liw.fi>; Thu, 05 Jul 2018 11:35:32 +0100
Date: Thu, 5 Jul 2018 11:35:31 +0100
From: Daniel Silverstone <dsilvers@digital-scurf.org>
To: ick-discuss@ick.liw.fi
Message-ID: <20180705103531.v2jauqhcgcbhs7oo@somnambulist.local>
References: <20180702152447.GA2363@exolobe3.liw.fi>
<20180703122826.siolafamphg7mpfo@somnambulist.local>
<20180704072913.GA7501@exolobe3.liw.fi>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20180704072913.GA7501@exolobe3.liw.fi>
User-Agent: NeoMutt/20170113 (1.7.2)
X-Pepperfish-Transaction: e55a-5c23-b40b-adb7
X-Pepperfish-Transaction-By: platypus
Subject: Re: Making the "git" action in ick more useful
X-BeenThere: ick-discuss@ick.liw.fi
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: discussions about the ick CI system <ick-discuss-ick.liw.fi>
List-Unsubscribe: <https://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/ick-discuss-ick.liw.fi>,
<mailto:ick-discuss-request@ick.liw.fi?subject=unsubscribe>
List-Archive: <http://listmaster.pepperfish.net/pipermail/ick-discuss-ick.liw.fi>
List-Post: <mailto:ick-discuss@ick.liw.fi>
List-Help: <mailto:ick-discuss-request@ick.liw.fi?subject=help>
List-Subscribe: <https://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/ick-discuss-ick.liw.fi>,
<mailto:ick-discuss-request@ick.liw.fi?subject=subscribe>
Sender: ick-discuss-bounces@ick.liw.fi
Errors-To: ick-discuss-bounces@ick.liw.fi
On Wed, Jul 04, 2018 at 10:29:14 +0300, Lars Wirzenius wrote:
> Thanks, Daniel. That's illuminating and interesting.
I'm glad :-)
> I'm going to rephrase what you wrote, to show how I understood it, and
> to suggest something.
Fair, I'll respond inline here :-)
> First, a comment on adding a templating language on top of YAML. I
> first encountered this with Ansible, and I've since used in my own
> vmdb2 project. It's a powerful concept, but I find it to sometimes be
> problematic: the input files aren't actually YAML anymore, and they
> become harder to understand and harder to debug. Sometimes that's OK,
> the added power is worth it. However, I'd like to avoid that for ick,
> for the time being.
>
> (Also, as you say, using jinja2 for templating might become
> problematic if and when ick gets rewritten in another language.)
Yes, absolutely. At the "worst" we can come up with a convention by which
icktool expands things before it sends them to the controller, but let's avoid
templating languages for now.
> Your first use case seems to me to be like this:
>
> * all your projects live on the same git server, next to each other
> * your main repository for a component is foo.git, branch master
> * its Debian packaging is in foo-debian.git, branch master for unstable
> * additionally, you have some common build tooling in
> * gp-packaging-tools.git
Yep (though foo/debian.git) :-)
> I note that this has a fairly tight coupling between the repositories
> for the main code and the Debian packaging. For ick, I'd prefer to
> avoid mandating that: one use case might be that I want to package
> your code and I put the Debian packaging on my server, while your code
> remains on yours.
Yes, the coupling is tight for the packages I have.
> My suggestion would be to change the git action to look for a project
> parameter "git" (or "sources"?), which would be a *list* of
> url/ref/dir tuples. For you, each project would have a parameter like
> this:
>
> - project: foo
> parameters:
> git:
> - url: git://git.gitano.org.uk/foo.git
> ref: master
> dir: src
> - url: git://git.gitano.org.uk/foo-debian.git
> ref: debian/unstable
> dir: src/debian
> - url: git://git.gitano.org.uk/gp-packaging-tools.git
> ref: master
> dir: gp-packaging-tools
>
> Which would result in a directory tree like this:
>
> /workspace
> /workspace/src
> /workspace/src/.git
> /workspace/src/debian
> /workspace/src/debian/.git
> /workspace/gp-packaging-tools
> /workspace/gp-packaging-tools/.git
That looks acceptable (presumably there'd need to be an action:git pipeline
step somewhere too?)
> I admit the git parameter gets a bit repetitive. If that's enough of a
> problem, I'd be willing to make the git action understand the optional
> parameter "git_server" (or "git_base_url"):
>
> - project: foo
> parameters:
> git_server: git:///git.gitano.org.uk
> git:
> - repo: foo.git
> ref: master
> dir: src
> - repo: foo-debian.git
> ref: debian/unstable
> dir: src/debian
> - repo: gp-packaging-tools.git
> ref: master
> dir: gp-packaging-tools
I'd call it repo_base and expect that repos which aren't absolute are urljoin'd
onto it as you suggest.
> Also, we could make "master" be the default branch. Actually, that
> suggests to me that a mechanism for providing defaults might be
> useful. Maybe something like this:
[snip]
Sounds good, though again I'd call it default_ref so that it's clear what its
purpose is.
> In thie example, the url/ref/dir tuple is replaced with a
> server/repo/ref/dir tuple and the full repo URL is built by combining
> server and repo fields. (Though it might be helpful to provide a url
> field that can be used instead of server+repo.) See below for
> overriding values in triggers.
As I suggested, I'd just go with "if repo is absolute, use it, otherwise
urljoin" as a non-surprising behaviour.
> Handling a list of dicts in shell with jq is not necessarily the kind
> of thing I enjoy doing. I'd do it in Python instead, where it's not as
> awkward.
Fair.
> Your system branch use case is interesting. My initial suggestion for
> that would be change ick to allow a trigger call, which starts a new
> build, to override project parameters for a specific build. Currnetly
> it's just a simple GET:
>
> GET /projects/foo/+trigger
>
> We could add another way to do triggers:
>
> POST /projects/foo/+trigger
> Content-Type: application/json
>
> {
> "parameters": {
> "git_defaults": {
> "ref": "liw/fix"
> }
> }
> }
>
> This would set the "ref" field in the "git_defaults" parameter to
> "liw/fix". The "server" field would not be changed. The
> foo-debian.git still use the debian/unstable branch.
So long as you have a mechanism for removing values as well, that'd work quite
well for the most part. It still lacks the fallback for repos which lack the
ref, sadly, but perhaps that can't be usefully managed generically?
> While this would not be as powerful as full jinja2 support, it would,
> I think, be a useful feature, entirely separate from templating.
>
> What I suggest here isn't really what you suggested, but what do you
> think? Would this be workable?
For my primary use-case this would be perfectly fine. For the system branch
use-case it's still not quite enough to prevent a need for an additional
pipeline step to handle the checkouts.
I think, for now, we should go with the primary-use case stuff, and worry about
the complex trigger solution in another iteration in the future. Do you want
to combine all of the stuff into a specification and then I'll review that so
we can close this work item off?
D.
--
Daniel Silverstone http://www.digital-scurf.org/
PGP mail accepted and encouraged. Key Id: 3CCE BABE 206C 3B69
_______________________________________________
ick-discuss mailing list
ick-discuss@ick.liw.fi
https://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/ick-discuss-ick.liw.fi
|
