class: center, middle, inverse # CI WG phase 2 Let's get real --- class: center, middle, inverse Argo, GitLab CI, Zuul v3—how do we choose? --- # What do we really, really want? * **Faster** cycle times! **Higher** satisfaction! **Stronger** empowerment! All the good things! * Boost productivity and raise developer satisfaction in the movement * Get more bang for the donated buck * Self-serve CI—empower developers, spread the work, remove RelEng from the critical path * Speed—what's the minimal cycle time we can achieve? * New requirement: Deploy CI via itself! No more manual deployments for anything ever! --- # Overall approach * Pick at least one of the candidates: Argo, GitLab CI, Zuul v3 (more if there's time and people) * Set up a CI system in parallel with the existing one that does everything "right", hopefully including deployment to production * Use this "for real", with a subset of the projects in CI, in collaboration with their developers * CI itself * Blubber * Hopefully other projects, too * Gather feedback, make changes, repeat, rinse * Gradually switch everything to new CI, keeping old one running until it's not needed anymore --- Pipeline design ============================================================================= * Does not need to be backwards compatible * Code review on Gerrit. Or try GitLab merge requests? * Pipeline triggered by developer pushing a change; merges still done by CI * Stages: commit, acceptance test, deployment to production --- Commit stage ============================================================================= * build the program; produce binaries and Docker images and other artifacts that will eventually be used in all later stages and in production * run unit tests * run code health checks * commands to build, run unit tests are specified by developer * other commands specified by RelEng, can't be overridden by developer * for anything the developers can't be allowed to decide, for the safety of production * e.g., how to build a Docker image * if commit stage fails, the pipeline stops * commit stage should be fast, less then 300 seconds * developers are expected to wait for commit stage to pass after they push before they move to their next task --- Acceptance stage ============================================================================= * using artifacts from commit stage, deploy a test instance, run automated acceptance tests against test instance * acceptance tests implemented by developers * they know what their requirements are * they will be responsible for production not breaking so they need to be able to add any tests they need to be confident * additional tests specified by RelEng * security checks (port scan for un-acceptable ports?) * more? * if this stage fails, the pipeline stops * this stage may take a while. --- Further things to consider ============================================================================= * Account management? * Security support, upgrades, general longevity? * Security update embargoes? * Transparency into what's happening in CI? * Build triggers: scheduled? manual? dependency project built? * Rollbacks in development? * Traceability from production to the git commit it is built from? * Build in K8s or VMs? Does it matter? * Artifact storage? * Metrics? --- class: center, middle, inverse I have nothing to offer but blood, toil, tears and sweat. And fun. Because if CI isn't fun, we're doing it wrong.