From 70deec35b7e9798d3216c350ab1181f2aaa6e278 Mon Sep 17 00:00:00 2001 From: Lars Wirzenius Date: Wed, 22 May 2019 21:15:35 +0300 Subject: Change: move pipeline to before architecture --- ci-arch.mdwn | 234 +++++++++++++++++++++++++++++------------------------------ 1 file changed, 117 insertions(+), 117 deletions(-) diff --git a/ci-arch.mdwn b/ci-arch.mdwn index 56d7501..d3ea8d7 100644 --- a/ci-arch.mdwn +++ b/ci-arch.mdwn @@ -451,7 +451,123 @@ items. patches. This means CI should be able to build and deploy changes that can't be made public yet, for security embargo reasons. -# Architecture +# The CI pipeline + +![The default pipeline](pipeline.svg){ height=50% } + +CI will provide a default pipeline for all projects. Projects may use +that or specify another one. + +The pipeline will be divided into several stages. Mandatory stages for +all changes and all projects are commit, acceptance stage, and +deployment to production. Other stages may be added to specific +changes or projects as needed. + +The goal is that if the commit and acceptance stages pass, the change +is a candidate that can be deployed to production, unless the project +is such that it needs (say) manual testing or other human decision for +the production deployment decision. Likewise, if the component or the +change is particularly security or performance sensitive, stages that +check those aspects may be required. CI will have ways of indicating +the required changes per component, and also per change. (It is +unclear how this will be managed.) + +If the commit or acceptance stage fails, there is not a production +candidate. The pipeline as a whole fails. Any artifacts built by the +pipeline will not be deployable to production, but they may be +deployable to test environments, or downloaded by developers for +inspection. + +## The commit stage + +The commit stage builds any deployable artifacts, such as executable +binaries, minimized JavaScript, translation files, or Docker images. +It is important that artifacts don't get rebuilt by later stages, +because rebuilding does not always result in bitwise identical output. +Instead the goal is to build once, test the artifacts, and deploy the +tested artifacts, instead of rebuilding and maybe deploying something +different than what was tested. + +The commit stage also runs unit tests, and any other tests that can be +run in isolation from other parts of the system, and that are also +quick. The commit stage does not have access to backing services, such +as databases or other components of the overall system. For example, +when the pipeline processes a change to a MediaWiki extension, the +commit stage doesn't have access to MediaWiki core or the MariaDB +MediaWiki uses. Integration or system tests should be done in the +acceptance test stage. + +The commit stage also runs code health checks. + +The commit stage is expected to be fast, aiming at less than five +minutes, so that we can expect developers to wait for it to pass +successfully. This will be a new requirement on our developers. + +The commands to build (compile) or run automated tests are stored in +the repository, either explicitly, or by indicating the type of build +needed. There might be a `.pipeline/config.yaml` file in the +repository, which specifies that `make` is the command that builds the +artifacts. Otherwise, the file may specify that it's a Go project, and +CI would know how to build a Go project. In this case we can change +the commands to build a Go project by changing CI only, without having +to change each git repository with a Go program. + +Only the declarative style will be possible for building Docker images, +as we want control over how that is done (**SECURE** requirement). + +CI may enforce specific additional commands to run, to build or test +further things; this can be used by RelEng to enforce certain things. +For example, we may enforce code health checks, or enable (or +disable) debug symbols in all builds. Such enforcement will be done in +collaboration with our developers. + +Any build dependencies needed during the commit stage must be +specified explicitly. For example, the minimum required version of Go +that should be installed in the build environment would be a build +dependency. If a project build-depends on another project, it needs to +specify which project, and which artifacts it needs installed from the +other project. Explicit build-dependencies are more work, but result +in fewer problems due to broken heuristics. + +## The acceptance stage + +During the acceptance stage CI deploys artifacts built in the commit +stage to a production-like system that has the same versions of all +software as production, except for the changes being processed by the +pipeline. CI will then run automated acceptance tests, and other +integration and system tests, against the deployed software. The test +environment is clean and empty, and well-known, unless and until the +test suite inserts data or makes changes. + +The acceptance stage can take time. Developers are not expected to +wait until it is finished before they move on to working on something +else. + +## Deployment to production + +If prior stages have passed successfully, and manual code review +("Gerrit CR:+2 vote") has approved the change, this stage deploys the +change to production. + +## Manual tests + +Testers may instruct CI to deploy any recently built set of artifacts +to a dedicated test environment, and can use the software in that +environment where it is isolated from others, and won't suddenly +change underneath them. The details of how this will be implemented +are to be determined later. + +This feature of the CI can also be used to demonstrate upcoming +features that are not yet ready to be deployed to or enabled in +production. + +## Capacity tests, non-functional requirements + +Capacity tests, and other tests for non-functional requirements, will +also be done in dedicated, isolated production-like environments. +RelEng will work with the performance team to sort out the details. + +# CI Architecture ## The WMF development ecosystem @@ -705,122 +821,6 @@ credentials it needs for accessing the artifacts and doing the deployment. There may be several deployment services, for deploying to environments with different security needs. -# The CI pipeline - -![The default pipeline](pipeline.svg){ height=50% } - -CI will provide a default pipeline for all projects. Projects may use -that or specify another one. - -The pipeline will be divided into several stages. Mandatory stages for -all changes and all projects are commit, acceptance stage, and -deployment to production. Other stages may be added to specific -changes or projects as needed. - -The goal is that if the commit and acceptance stages pass, the change -is a candidate that can be deployed to production, unless the project -is such that it needs (say) manual testing or other human decision for -the production deployment decision. Likewise, if the component or the -change is particularly security or performance sensitive, stages that -check those aspects may be required. CI will have ways of indicating -the required changes per component, and also per change. (It is -unclear how this will be managed.) - -If the commit or acceptance stage fails, there is not a production -candidate. The pipeline as a whole fails. Any artifacts built by the -pipeline will not be deployable to production, but they may be -deployable to test environments, or downloaded by developers for -inspection. - -## The commit stage - -The commit stage builds any deployable artifacts, such as executable -binaries, minimized JavaScript, translation files, or Docker images. -It is important that artifacts don't get rebuilt by later stages, -because rebuilding does not always result in bitwise identical output. -Instead the goal is to build once, test the artifacts, and deploy the -tested artifacts, instead of rebuilding and maybe deploying something -different than what was tested. - -The commit stage also runs unit tests, and any other tests that can be -run in isolation from other parts of the system, and that are also -quick. The commit stage does not have access to backing services, such -as databases or other components of the overall system. For example, -when the pipeline processes a change to a MediaWiki extension, the -commit stage doesn't have access to MediaWiki core or the MariaDB -MediaWiki uses. Integration or system tests should be done in the -acceptance test stage. - -The commit stage also runs code health checks. - -The commit stage is expected to be fast, aiming at less than five -minutes, so that we can expect developers to wait for it to pass -successfully. This will be a new requirement on our developers. - -The commands to build (compile) or run automated tests are stored in -the repository, either explicitly, or by indicating the type of build -needed. There might be a `.pipeline/config.yaml` file in the -repository, which specifies that `make` is the command that builds the -artifacts. Otherwise, the file may specify that it's a Go project, and -CI would know how to build a Go project. In this case we can change -the commands to build a Go project by changing CI only, without having -to change each git repository with a Go program. - -Only the declarative style will be possible for building Docker images, -as we want control over how that is done (**SECURE** requirement). - -CI may enforce specific additional commands to run, to build or test -further things; this can be used by RelEng to enforce certain things. -For example, we may enforce code health checks, or enable (or -disable) debug symbols in all builds. Such enforcement will be done in -collaboration with our developers. - -Any build dependencies needed during the commit stage must be -specified explicitly. For example, the minimum required version of Go -that should be installed in the build environment would be a build -dependency. If a project build-depends on another project, it needs to -specify which project, and which artifacts it needs installed from the -other project. Explicit build-dependencies are more work, but result -in fewer problems due to broken heuristics. - -## The acceptance stage - -During the acceptance stage CI deploys artifacts built in the commit -stage to a production-like system that has the same versions of all -software as production, except for the changes being processed by the -pipeline. CI will then run automated acceptance tests, and other -integration and system tests, against the deployed software. The test -environment is clean and empty, and well-known, unless and until the -test suite inserts data or makes changes. - -The acceptance stage can take time. Developers are not expected to -wait until it is finished before they move on to working on something -else. - -## Deployment to production - -If prior stages have passed successfully, and manual code review -("Gerrit CR:+2 vote") has approved the change, this stage deploys the -change to production. - -## Manual tests - -Testers may instruct CI to deploy any recently built set of artifacts -to a dedicated test environment, and can use the software in that -environment where it is isolated from others, and won't suddenly -change underneath them. The details of how this will be implemented -are to be determined later. - -This feature of the CI can also be used to demonstrate upcoming -features that are not yet ready to be deployed to or enabled in -production. - -## Capacity tests, non-functional requirements - -Capacity tests, and other tests for non-functional requirements, will -also be done in dedicated, isolated production-like environments. -RelEng will work with the performance team to sort out the details. - # CI implementation FIXME This needs to be written, but it needs a lot of thinking first -- cgit v1.2.1