1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
|
# Continuous fun
Ick aims to make CI and CD fun and easy.
Lars Wirzenius (liw@liw.fi)
WIP
---
CI? CD?
=============================================================================
* Continuous integration and delivery
* Merge into master frequently, incrementally, in small steps.
* Deploy on every merge.
vvv
CI/CD strategies
=============================================================================
* Avoid long-lived branches. Avoid massive, fragile merges.
Avoid conflicts. Avoid most integration mistakes.
* Automatically build and test software, triggered by version control
system.
vvv
* If it passess automated tests, deploy into production, possibly via
a staging environment.
* Sometimes continuous delivery is all that's possible. E.g., to
an APT repository.
* Very easy to get changes from developer into the hands of users.
* Don't forget about code review.
---
CI/CD - other view
=============================================================================
* Automate the running of automated tests, because developers are too
careless to run them manually. Also, all manual steps are tedious.
* Automate the deployment of software, because developers take too
many shortcuts and sysadmins are expensive.
---
Ick
=============================================================================
* There are a number of existing tools and services for CI/CD:
Jenkins, Buildbot, Travis, Circle-CI, Go CD, ...
* They tend to be implemented in Java, and be ugly. Also heavy.
* I wrote my own. Because I dislike all the ones I've looked at.
vvv
Ick
=============================================================================
* Currently at ALPHA level. CD for ick itself. Builds all my personal
projects, and websites. Now also in use for Qvisqve at QvarnLabs: CD
to qvisqve-demo.h.qvarnlabs.eu.
* Free software. Might some day become a hosted service for paying
customers. Will still be free software.
---
Headline features
=============================================================================
* Builds from source in git.
* Can build .deb packages, upload them to a dedicated APT repository.
* Can build in containers (systemd-nspawn).
* Builds the container itself, as directed by the user.
vvv
* Can deploy to a production server (ick, Qvisve).
* Can send notification emails when builds end.
* "Everything" is user-configurable.
* APIs, not plugins.
---
Ick concepts
=============================================================================
* A **project** consists of some **parameters** and a list of
**pipelines**.
* A **pipeline** is a sequence of **actions** to achieve some goal
- get source code from git
- build Debian package
- an action may be a shell snippet that gets executed on the build
host.
vvv
* A **worker** does the actual work of building, testing, etc, a
project, by executing actions.
* A **controller** keeps track of what projects and pipelines exist,
and which builds are running, and tells each worker what their next
action should be.
* A **workspace** is where the build of a project happens. Each build
has its own workspace, which gets populated by the source code,
compiles binaries, etc.
vvv
* A **container** is an isolated build environment, combining a
**system tree** (or **systree**) and the workspace. What happens in
the container, stays in the container. Think "chroot" except better
isolated.
* A build **artifact** is a blob of some data produced by building a
project, or some part of a project.
---
Ick architecture
=============================================================================
<img src="ick-arch.png" />
vvv
* Several co-operating micro-services.
* The controller keeps track of all projects and project builds, and
which parts of the build have been done and what the next step is.
* A worker asks the controller for an action to execute. It deals with
one action at a time. It reports the output and exit code of the
action to the controller.
* Other services support the core: store artifacts, send
notifications, ...
---
UX
=============================================================================
* No web UI at this time. Will happen later, but requires end-user
authentication in Qvisqve.
* Command line tool, icktool, uses controller API to make things
happen.
---
.ick file
=============================================================================
projects:
- project: stretch_systree
pipelines:
- build_debian_systree
parameters:
debian_codename: stretch
packages:
- build-essential
- git
- jq
artifact_name: stretch_systree
vvv
## .ick file
pipelines:
- pipeline: build_debian_systree
parameters:
- debian_codename
- packages
- artifact_name
actions:
...
vvv
## .ick file
actions:
- debootstrap: auto
mirror: http://deb.debian.org/debian
where: host
- shell: |
apt-get install -y python3
where: chroot
vvv
## .ick file
- python: |
import os, subprocess
def runcmd(argv, **kwargs):
subprocess.check_call(argv, **kwargs)
runcmd(['apt-get', 'install', '-y'] +
params['packages'])
where: chroot
- archive: workspace
where: host
vvv
## .ick file
- project: hello
pipelines:
- prepare_container
- hello
parameters:
systree_name: stretch_systree
vvv
## .ick file
- pipeline: prepare_container
actions:
- action: populate_systree
where: host
- pipeline: hello
actions:
- shell: |
echo hello, world
pwd
where: container
---
Someday / maybe
=============================================================================
* Continue building ick into a hosted service
* keep self-hosted as an option
* Add a web UI. Maybe a mobile app.
* Build for multiple targets: operating systems,
CPU architectures, toolchains, configurations, ...
vvv
* Build concurrently, when possible: for amd64, arm64, and mipsel at
the same time.
* More notification methods: IRC, Matrix, SMS, ...
* More notification points: start/end of action, end of pipeline, ...
* Speed. Scalability. Robustness.
vvv
* Ease of use. Fix any rough corners. Remove any tedious parts.
* Ick does the merging: if merge fails or tests fail afterwards,
discard changes.
* More CD features: controlled roll-out via staging, roll-back in case
of problems, ...
vvv
* Distributed or federated CI? "Listed on" git servers without git
servers having to know about each listener.
* Maybe via Mastodon?
---
# Business maybe?
* Provide ick as a hosted service
* For paying customers, no ads
* Cost less than doing it yourself, pay for worker time
* Slow-growth
* Enought to pay costs, give a bit of pocket money
* Doesn't eat too much into my free time
---
# Thank you
|
