From 167165a8a1495a4ebcc38632f4afb6721529a4c9 Mon Sep 17 00:00:00 2001 From: Lars Wirzenius Date: Mon, 6 May 2019 20:48:18 +0300 Subject: Change: formatting, bullet points to paragraphs, etc --- ci-arch.mdwn | 239 +++++++++++++++++++++++++++++++++-------------------------- 1 file changed, 134 insertions(+), 105 deletions(-) (limited to 'ci-arch.mdwn') diff --git a/ci-arch.mdwn b/ci-arch.mdwn index c89fd96..07d4694 100644 --- a/ci-arch.mdwn +++ b/ci-arch.mdwn @@ -501,114 +501,143 @@ would prefer to avoid having such changes. # Architecture: The WMF development ecosystem -![The WMF development ecosystem, roughly](ecosystem.svg) - -* the above figure is simplistic, but gives the general idea - * developer pushes a change to Gerrit - * CI builds and tests change (commit stage) - * CI deploys to a test environment, runs tests against that - (acceptance test stage) - * CI can deploy to an environment dedicated for manual testing - * after successful code review, CI merges changes to the master - branch, run all automated tests again, and deploys to the - production environment - -* the commit and acceptance stages are triggered as soon as developer - pushes changes to be reviewed; human reviews won't be requested - until the two stages pass, as there's no point in spending human - attention on things that are not going to be candidates for - deployment to production; they may be re-run after code review - accepts the changes, to make sure nothing unforeseen has changed - while review took place - -* other stages may run in parallel with code review, but if they fail - they may nullify candidacy? for example, stages for manual and - capacity testing, and security test/review; depending on the change - and the component in question, some or all of these may be necessary +![The WMF development ecosystem, roughly](ecosystem.svg){ height=25% } + +The figure above is simplistic, but gives the general idea of what +happens when a developer is finished with a change: + +1. developer pushes a change to Gerrit, which trigger CI +1. CI builds and tests change (commit stage) +1. CI deploys to a test environment, runs tests against that + (acceptance test stage); if everythins is OK, Gerrit is notified + and requests code reviews from relevant parties +1. testers can request CI it deploy the change to an environment + dedicated for manual testing +1. after a successful code review, CI merges changes to the master + branch, runs all automated tests again, and deploys to the + production environment + +The commit and acceptance stages are triggered as soon as developer +pushes changes to be reviewed. Human reviews won't be requested until +the two stages pass, as there's no point in spending human attention +on things that are not going to be candidates for deployment to +production. The two stages may be re-run after code review, to make +sure nothing unforeseen has changed while the review took place. + +Other stages may run in parallel with code review, and if they fail +they may nullify the release candidacy of the change. For example, +stages for manual and capacity testing, and security test/review; +depending on the change and the component in question, some or all of +these may be necessary. # The (default?) pipeline -![The default pipeline](pipeline.svg) - -* CI will provide a default pipeline for all projects - - * divided into several stages - - * mandatory stages: commit, acceptance; other stages may be added to - other projects as needed - - * the goal is that if commit + acceptance stages pass, the project - has a candidate that can be deployed to production, unless the - project is such that it needs (say) manual testing or other human - decision for the production deployment decision - - * if commit or acceptance stage fails, there is not production - candidate - -* commit stage - - * builds all the artifacts that will be used by later stages - - * runs unit tests - - * other tests, possibly integration tests - - * code health checks - - * the commit stage is expected to be fast, aiming at less than five - minutes, so that we can expect developers to wait for it to pass - successfully - - * the commands to build (compile) or run automated tests are stored - in the repository, either explicity, or by indicating the type of - build needed; for example, the repository may specify "make" as - the command to run, or it may specify that it's a Go project, and - CI would know how to build a Go project; in the latter case we can - change the commands to build a Go project by changing CI only, - without having to change each git repository with a Go program - - * only the declarative style is possible for building Docker images, - as we want control over how that is done - - * CI may enforce specific additional commands to run, to build or - test further things; this can be used by RelEng to enforce - specific code health checks, for example, or to enable (or - disable) debug symbols in all builds - - * all tests run in an isolated build tree, and may not use anything - outside the tree, including databases or other backing services - - * any build dependencies must be specified explicitly; for example, - which version of Go should be installed in the build environment, - or if a project build-depends on another project, which artifacts - it needs installed from the other project; explicit is more work, - but results in fewer problems due to broken heuristics - -* acceptance tests - - * deploys artifacts from commit stage to containers in special test - environments, runs tests against deployed artifacts - - * possibly run slow tests from the build tree as well, if they don't - fit into the commit stage's time budget - - * this stage can be slower than the commit stage, but should still - pass in, say, an hour, instead of taking days - -* capacity tests - - * these are tests that benchmark the system as deployed into an - environment that's sufficiently production-like also as far as - hardware resources are concerned - -* manual (exploratory) tests - - * testers will have dedicated environments to which they can trigger - deployment of specific builds, and in which they can, for example, - test that specific bugs are fixed - - * this can also be used to demonstrate upcoming features that are - not yet enabled in production +![The default pipeline](pipeline.svg){ height=25% } + +CI will provide a default pipeline for all projects. Projects may use +that or specify another one. + +The pipeline will be divided into several stages. Mandatory stages for +all changes and all projects are commit and acceptance stage. Other +stages may be added to specific changes projects as needed. + +The goal is that if the commit and acceptance stages pass, the change +is a candidate that can be deployed to production, unless the +project is such that it needs (say) manual testing or other human +decision for the production deployment decision. Likewise, if the +component or the change is particularly security or performance +sensitive, stages that check those aspects may be required. CI will +have ways of indicating the required changes per component, and also +per change. (It is unclear how this will be managed.) + +If the commit or acceptance stage fails, there is not production +candidate. The pipeline as a whole fails. Any artifacts built by the +pipeline will not be deployable to production, but they may be +deployable to test environments, or downloaded by developers for +inspection. + +## The commit stage + +The commit stage builds any deployable artifacts, such as executable +binaries, minimized Javascript, translation files, or Docker images. +It is important that artifacts don't get rebuilt by later stages, +because rebuilding does not always result in bitwise identical output. +Instead the goal is to build once, test the artifacts, and deploy the +tested artifacts, instead of rebuilding and maybe deploying something +different than what was tested. + +The commit stage also runs unit tests, and any other tests that can be +run in isolation from other parts of the system, and that also are +quick. The commit stage does not have access to backing services, such +as databases or other components of the overall system. For example, +when the pipeline processes a change to a MediaWiki extenasion, the +commit stage doesn't have access to MediaWiki core or the MariaDB +MediaWiki uses. Integration or system tests should be done in the +acceptance test stage. + +The commit stage also runs code health checks. + +The commit stage is expected to be fast, aiming at less than five +minutes, so that we can expect developers to wait for it to pass +successfully. This will be a new requirement on our developers. + +The commands to build (compile) or run automated tests are stored in +the repository, either explicity, or by indicating the type of build +needed. There might be a `.pipeline/config.yaml` file in the +repository, which specifies that `make` is the command that builds the +artifacts. Otherwise, the file may specify that it's a Go project, and +CI would know how to build a Go project. In this case we can change +the commands to build a Go project by changing CI only, without having +to change each git repository with a Go program. + +Only the declarative style will be possible for building Docker images, +as we want control over how that is done (**SECURE** requirement). + +CI may enforce specific additional commands to run, to build or test +further things; this can be used by RelEng to enforce certain things. +For example, we may enforce code health checks, or to enable (or +disable) debug symbols in all builds. Such enforcement will be done in +collaboration with our developers. + +Any build dependencies needed during the commit stage must be +specified explicitly. For example, the minimum required version of Go +that should be installed in the build environment would be a build +dependency. If a project build-depends on another project, it needs to +specify which project, and which artifacts it needs installed from the +other project. Explicit build-dependencies is more work, but results +in fewer problems due to broken heuristics. + +## The acceptance stage + +During the acceptance stage CI deploys artifacts built in the commit +stage to a production-like system that has the same versions of all +sofware as production, except for the changes being processed by the +pipeline. CI will then run automated acceptance tests, and other +integration and system tests, against the deployed software. The test +environment is clean and empty, and well-known, unless and until the +test suite inserts data or makes changes. + +The acceptance stage can take time. Developers are not expected to +wait until it is finished before they move on to working on something +else. + +## Manual tests + +Testers may instruct CI to deploy any recent built set of artifacts to +a dedicated test environment, and can use the software in that +environment where it is isolated from others, and won't suddenly +change underneath them. The details of how this will be implemented +are to be determined later. + +This feature of the CI can also be used to demonstrate upcoming +features that are not yet ready to be deployed to or enabled in +production. + +## Capacity tests, non-functional requirements + +Capacity tests, and other tests for non-functional requirements, will +also be done in dedicated, isolated production-like environments. +RelEng will work with the performance team to sort out the details. # Architecture: internals -- cgit v1.2.1