From 1aeedefe755c292960dfe1f4c83f3aa32bbffc2c Mon Sep 17 00:00:00 2001 From: Lars Wirzenius Date: Sun, 1 Jul 2018 15:51:00 +0300 Subject: Add: ick slides --- slides.md | 310 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 310 insertions(+) create mode 100644 slides.md (limited to 'slides.md') diff --git a/slides.md b/slides.md new file mode 100644 index 0000000..1017453 --- /dev/null +++ b/slides.md @@ -0,0 +1,310 @@ +# Continuous fun + +Lars Wirzenius (liw@liw.fi) + +WIP + +--- + +CI? CD? +============================================================================= + +* Continuous integration and delivery + +* Merge into master frequently, incrementally, in small steps. + +* Deploy on every merge. + +vvv + +CI/CD strategies +============================================================================= + +* Avoid long-lived branches. Avoid massive, fragile merges. + Avoid conflicts. Avoid most integration mistakes. + +* Automatically build and test software, triggered by version control + system. + +vvv + +* If it passess automated tests, deploy into production, possibly via + a staging environment. + + * Sometimes continuous delivery is all that's possible. E.g., to + an APT repository. + +* Very easy to get changes from developer into the hands of users. + +* Don't forget about code review. + +--- + +CI/CD - other view +============================================================================= + +* Automate the running of automated tests, because developers are too + careless to run them manually. Also, all manual steps are tedious. + +* Automate the deployment of software, because developers take too + many shortcuts and sysadmins are expensive. + +--- + +Ick +============================================================================= + +* There are a number of existing tools and services for CI/CD: + Jenkins, Buildbot, Travis, Circle-CI, Go CD, ... + +* They tend to be implemented in Java, and be ugly. Also heavy. + +* I wrote my own. Because I dislike all the ones I've looked at. + +vvv + +Ick +============================================================================= + +* Currently at ALPHA level. CD for ick itself. Builds all my personal + projects, and websites. Now also in use for Qvisqve at QvarnLabs: CD + to qvisqve-demo.h.qvarnlabs.eu. + +* Free software. Might some day become a hosted service for paying + customers. Will still be free software. + +--- + +Headline features +============================================================================= + +* Builds from source in git. + +* Can build .deb packages, upload them to a dedicated APT repository. + +* Can build in containers (systemd-nspawn). + +* Builds the container itself, as directed by the user. + +vvv + +* Can deploy to a production server (ick, Qvisve). + +* Can send notification emails when builds end. + +* "Everything" is user-configurable. + +* APIs, not plugins. + +--- + +Ick concepts +============================================================================= + +* A **project** consists of some **parameters** and a list of + **pipelines**. + +* A **pipeline** is a sequence of **actions** to achieve some goal + - get source code from git + - build Debian package + - an action may be a shell snippet that gets executed on the build + host. + +vvv + +* A **worker** does the actual work of building, testing, etc, a + project, by executing actions. + +* A **controller** keeps track of what projects and pipelines exist, + and which builds are running, and tells each worker what their next + action should be. + +* A **workspace** is where the build of a project happens. Each build + has its own workspace, which gets populated by the source code, + compiles binaries, etc. + +vvv + +* A **container** is an isolated build environment, combining a + **system tree** (or **systree**) and the workspace. What happens in + the container, stays in the container. Think "chroot" except better + isolated. + +* A build **artifact** is a blob of some data produced by building a + project, or some part of a project. + +--- + +Ick architecture +============================================================================= + + + +vvv + +* Several co-operating micro-services. + +* The controller keeps track of all projects and project builds, and + which parts of the build have been done and what the next step is. + +* A worker asks the controller for an action to execute. It deals with + one action at a time. It reports the output and exit code of the + action to the controller. + +* Other services support the core: store artifacts, send + notifications, ... + +--- + +UX +============================================================================= + +* No web UI at this time. Will happen later, but requires end-user + authentication in Qvisqve. + +* Command line tool, icktool, uses controller API to make things + happen. + +--- + +.ick file +============================================================================= + + projects: + + - project: stretch_systree + pipelines: + - build_debian_systree + parameters: + debian_codename: stretch + packages: + - build-essential + - git + - jq + artifact_name: stretch_systree + +vvv + +## .ick file + + pipelines: + + - pipeline: build_debian_systree + parameters: + - debian_codename + - packages + - artifact_name + actions: + ... + +vvv + +## .ick file + + actions: + - debootstrap: auto + mirror: http://deb.debian.org/debian + where: host + + - shell: | + apt-get install -y python3 + where: chroot + +vvv + +## .ick file + + + - python: | + import os, subprocess + def runcmd(argv, **kwargs): + subprocess.check_call(argv, **kwargs) + runcmd(['apt-get', 'install', '-y'] + + params['packages']) + where: chroot + + - archive: workspace + where: host + +vvv + +## .ick file + + - project: hello + pipelines: + - prepare_container + - hello + parameters: + systree_name: stretch_systree + +vvv + +## .ick file + + - pipeline: prepare_container + actions: + - action: populate_systree + where: host + + - pipeline: hello + actions: + - shell: | + echo hello, world + pwd + where: container + +--- + +Coming soon +============================================================================= + +* Continue building ick into a hosted service + * keep self-hosted as an option + +* Add a web UI. Maybe a mobile app. + +* Build for multiple targets: operating systems, + CPU architectures, toolchains, configurations, ... + +vvv + +* Build concurrently, when possible: for amd64, arm64, and mipsel at + the same time. + +* More notification methods: IRC, Matrix, SMS, ... + +* More notification points: start/end of action, end of pipeline, ... + +* Speed. Scalability. Robustness. + +vvv + +* Ease of use. Fix any rough corners. Remove any tedious parts. + +* Ick does the merging: if merge fails or tests fail afterwards, + discard changes. + +* More CD features: controlled roll-out via staging, ... + +vvv + +* Distributed or federated CI? "Listed on" git servers without git + servers having to know about each listener. + + * Maybe via Mastodon? + +--- + +# Business? + +* Provide ick as a hosted service + +* For paying customers, no ads + +* Cost less than doing it yourself, pay for worker time + +* Slow-growth + +--- + +# Thank you -- cgit v1.2.1