From 6a6912a7ab1954fca167deebadbbe5f781330a88 Mon Sep 17 00:00:00 2001 From: Lars Wirzenius Date: Fri, 1 Mar 2019 10:14:44 +0200 Subject: Change: rewrite the presentation --- index.html | 2 +- slides.md | 349 +++++++++++++------------------------------------------------ 2 files changed, 75 insertions(+), 276 deletions(-) diff --git a/index.html b/index.html index 5f32a04..7227fc2 100644 --- a/index.html +++ b/index.html @@ -7,7 +7,7 @@ ick - continuous fun - + diff --git a/slides.md b/slides.md index 5a34411..6080533 100644 --- a/slides.md +++ b/slides.md @@ -1,321 +1,120 @@ Continuous fun ----------------------------------------------------------------------------- -Ick aims to make CI and CD fun and easy. - -Lars Wirzenius (liw@liw.fi) - -WIP +https://ick.liw.fi --- -CI? CD? ------------------------------------------------------------------------------ - -* Continuous integration and delivery and deployment - -* Merge into master frequently, incrementally, in small steps. - -* Deploy on every merge. - -vvv - -CI/CD strategies +Ick? ----------------------------------------------------------------------------- -* Avoid long-lived branches. Avoid massive, fragile merges. - Avoid conflicts. Avoid most integration mistakes. +* Continuous integration, delivery, and deployment + * See "**Continuous Delivery**", by Humble, Farley +* Free software, of course (AGPL3+) +* Aims to be nice to install, run, admin, use, ... -* Automatically build and test and publish and deploy software, - triggered by version control system. +Note: +* you push changes to git server +* this triggers ick +* ick gets code from git server, builds, tests, deploys vvv -* If it passess automated tests, deploy into production, possibly - piecemeal to allow graceful retreat in case of problems. (Staging, - canaries, rollback.) - - * Sometimes continuous delivery is all that's possible. E.g., - publish to an APT repository. - -* Very easy to get changes from developer into the hands of users. +* My current hobby project. One of many. + * a couple of contributors, aiming to grow +* "I used a Jenkins, and I didn't like it." + * implementation irritants + * architecture annoyances. +* "NIH is strong with this one." -* Don't forget about code review. +Note: +* I like to write my own crappy code, not configure other people's + silly stuff. +* Turns out there's many options now, and I can afford to pay for + cloudy stuff. Might not start Ick now. --- -CI/CD - other view +Highlights ----------------------------------------------------------------------------- -* Automate the running of automated tests, because developers are too - careless to run them manually. Also, all manual steps are tedious. +* ALPHA quality software + * slow, lacks features, buggy, ... +* In my own active use + * builds, tests my software + * builds, publishes .deb packages of my software + * builds my websites from source in git +* A demo instance with restricted access -* Automate the deployment of software, because developers take too - many shortcuts and sysadmins are expensive. +Note: +* still feeling my way forward to what a good solution should be like --- -Ick ------------------------------------------------------------------------------ - -* There are a number of existing tools and services for CI/CD: - Jenkins, Buildbot, Travis, Circle-CI, Go CD, ... - -* They tend to be implemented in Java, and be ugly. Also heavy. - -* I wrote my own. Because I dislike all the ones I've looked at. - -vvv - -Ick +Some day, hopefully ----------------------------------------------------------------------------- -* Currently at ALPHA level. CD for Ick itself. Builds all my personal - projects, and websites. - -* Free software. Might some day become a hosted service for paying - customers. Will still be free software. +* **Hostable**: secure, safe, reliable +* **Distributed, federated**: a healthy eco system of providers of + components +* **Scalable**: large projects (Debian), many projects ("the world's CI") +* **Fast**: minor overhead compared to local build+test +* **Fun**: to use, not just to develop --- -Headline features ------------------------------------------------------------------------------ - -* Builds from source in git. - -* Can build .deb packages, upload them to a dedicated APT repository. - -* Can build in containers (systemd-nspawn). - -* Builds the container image itself, as directed by the user. - -vvv - -* Can deploy to a production server. - -* Can send notification emails when builds end. - -* "Everything" is user-configurable. - -* APIs, not plugins. - ---- - -Ick concepts ------------------------------------------------------------------------------ - -* A **project** consists of some **parameters** and a list of - **pipelines**. - -* A **pipeline** is a sequence of **actions** to achieve some goal - - get source code from git - - build Debian package - - an action may be a shell snippet that gets executed on the build - host. - -vvv - -* A **worker** does the actual work of building, testing, etc, a - project, by executing actions. - -* A **controller** keeps track of what projects and pipelines exist, - and which builds are running, and tells each worker what their next - action should be. - -* A **workspace** is where the build of a project happens. Each build - has its own workspace, which gets populated by the source code, - compiles binaries, etc. - -vvv - -* A **container** is an isolated build environment, combining a - **system tree** (or **systree**) and the workspace. What happens in - the container, stays in the container. Think "chroot" except better - isolated. - -* A build **artifact** is a blob of some data produced by building a - project, or some part of a project. - ---- - -Ick architecture +Architecture ----------------------------------------------------------------------------- -vvv - -* Several co-operating micro-services. - -* The controller keeps track of all projects and project builds, and - which parts of the build have been done and what the next step is. - -* A worker asks the controller for an action to execute. It deals with - one action at a time. It reports the output and exit code of the - action to the controller. - -* Other services support the core: store artifacts, send - notifications, ... +Note: +* multiple components running on different hosts, communicating over + HTTP using RESTful APIs and JSON +* controller - keeps track what to do, which worker is doing what +* worker and worker-manager - actually do +* qvisqve - authenticates users, API clients (OAuth2, OIDC) +* artifact store - stores blobs, any large files +* apt - .deb packages (automatically uploaded) +* notification service - tells people builds have finished +* muck - stores JSON objects, with ownership and access control --- -UX +Plans ----------------------------------------------------------------------------- -* Extremely ugly, limited web UI at this time. Help wanted. - -* Command line tool, icktool, uses controller API to make things - happen. +* good web UI - react; contributors are starting +* trigger service - tells controller when to start builds +* yuck - replace qvisqve for authn +* rock - new artifact store + +Note: +* web UI: react +* TS: + * security, choosy on git changes, trigger multiple projects on + one change + * trigger builds when dependencies change + * trigger builds on schedule + * other new triggers +* rewrite in Rust +* make things friendly to users, sysadmins +* make things fast --- -.ick file +Demo ----------------------------------------------------------------------------- - projects: - - - project: stretch_systree - pipelines: - - build_debian_systree - parameters: - debian_codename: stretch - packages: - - build-essential - - git - - jq - artifact_name: stretch_systree - -vvv - -## .ick file - - pipelines: - - - pipeline: build_debian_systree - parameters: - - debian_codename - - packages - - artifact_name - actions: - ... - -vvv - -## .ick file - - actions: - - debootstrap: auto - mirror: http://deb.debian.org/debian - where: host - - - shell: | - apt-get install -y python3 - where: chroot - -vvv - -## .ick file - - - - python: | - import os, subprocess - def runcmd(argv, **kwargs): - subprocess.check_call(argv, **kwargs) - runcmd(['apt-get', 'install', '-y'] + - params['packages']) - where: chroot - - - archive: workspace - where: host - -vvv - -## .ick file - - - project: hello - pipelines: - - prepare_container - - hello - parameters: - systree_name: stretch_systree - -vvv - -## .ick file - - - pipeline: prepare_container - actions: - - action: populate_systree - where: host - - - pipeline: hello - actions: - - shell: | - echo hello, world - pwd - where: container +Note: +* use demo instance +* icktool +* ickweb --- -Someday / maybe +SEE ALSO ----------------------------------------------------------------------------- -* Continue building Ick into a hosted service - * keep self-hosted as an option - -* Add a web UI. Maybe a mobile app. - -* Build for multiple targets: operating systems, - CPU architectures, toolchains, configurations, ... - -vvv - -* Build concurrently, when possible: for amd64, arm64, and mipsel at - the same time. - -* More notification methods: IRC, Matrix, SMS, ... - -* More notification points: start/end of action, end of pipeline, ... - -* Speed. Scalability. Robustness. - -vvv - -* Ease of use. Fix any rough corners. Remove any tedious parts. - -* Ick does the merging: if merge fails or tests fail afterwards, - discard changes. - -* More CD features: controlled roll-out via staging, roll-back in case - of problems, ... - -vvv - -* Distributed or federated CI? "Listen to" git servers without git - servers having to know about each listener. - - * Maybe via Mastodon? - -* User can pick and choose between controllers, artifact stores, - authentication services, workers, etc. - -* Those who wish can provide them as well, for themselves or others to - use. - ---- - -## Someday maybe? - -* Provide Ick as a hosted service - -* For paying customers, no ads; possibly donated workers - -* Not too many customers; slow growth - -* Enough to pay costs, plus provide CI services for free software projects - -* Doesn't eat too much into my free time - ---- - -## Thank you +* https://ick.liw.fi/ +* Ask me for access to demo instance -- cgit v1.2.1