Change: formatting, bullet points to paragraphs, etc

3 files changed, 172 insertions, 156 deletions

diff --git a/ci-arch.html b/ci-arch.html
index d9b7f82..a29cab4 100644
--- a/ci-arch.html
+++ b/ci-arch.html
@@ -40,7 +40,12 @@
 <li><a href="#interdependent-changes-to-multiple-components">Interdependent changes to multiple components</a></li>
 </ul></li>
 <li><a href="#architecture-the-wmf-development-ecosystem">Architecture: The WMF development ecosystem</a></li>
-<li><a href="#the-default-pipeline">The (default?) pipeline</a></li>
+<li><a href="#the-default-pipeline">The (default?) pipeline</a><ul>
+<li><a href="#the-commit-stage">The commit stage</a></li>
+<li><a href="#the-acceptance-stage">The acceptance stage</a></li>
+<li><a href="#manual-tests">Manual tests</a></li>
+<li><a href="#capacity-tests-non-functional-requirements">Capacity tests, non-functional requirements</a></li>
+</ul></li>
 <li><a href="#architecture-internals">Architecture: internals</a></li>
 <li><a href="#acceptance-criteria">Acceptance criteria</a></li>
 </ul>
@@ -216,61 +221,43 @@
 <p>FIXME All or none of the changes need to merged and deployed. Lars would prefer to avoid having such changes.</p>
 <h1 id="architecture-the-wmf-development-ecosystem">Architecture: The WMF development ecosystem</h1>
 <figure>
-<img src="ecosystem.svg" alt="The WMF development ecosystem, roughly" /><figcaption>The WMF development ecosystem, roughly</figcaption>
+<img src="ecosystem.svg" alt="The WMF development ecosystem, roughly" style="height:25.0%" /><figcaption>The WMF development ecosystem, roughly</figcaption>
 </figure>
-<ul>
-<li>the above figure is simplistic, but gives the general idea
-<ul>
-<li>developer pushes a change to Gerrit</li>
+<p>The figure above is simplistic, but gives the general idea of what happens when a developer is finished with a change:</p>
+<ol type="1">
+<li>developer pushes a change to Gerrit, which trigger CI</li>
 <li>CI builds and tests change (commit stage)</li>
-<li>CI deploys to a test environment, runs tests against that (acceptance test stage)</li>
-<li>CI can deploy to an environment dedicated for manual testing</li>
-<li>after successful code review, CI merges changes to the master branch, run all automated tests again, and deploys to the production environment</li>
-</ul></li>
-<li><p>the commit and acceptance stages are triggered as soon as developer pushes changes to be reviewed; human reviews won’t be requested until the two stages pass, as there’s no point in spending human attention on things that are not going to be candidates for deployment to production; they may be re-run after code review accepts the changes, to make sure nothing unforeseen has changed while review took place</p></li>
-<li><p>other stages may run in parallel with code review, but if they fail they may nullify candidacy? for example, stages for manual and capacity testing, and security test/review; depending on the change and the component in question, some or all of these may be necessary</p></li>
-</ul>
+<li>CI deploys to a test environment, runs tests against that (acceptance test stage); if everythins is OK, Gerrit is notified and requests code reviews from relevant parties</li>
+<li>testers can request CI it deploy the change to an environment dedicated for manual testing</li>
+<li>after a successful code review, CI merges changes to the master branch, runs all automated tests again, and deploys to the production environment</li>
+</ol>
+<p>The commit and acceptance stages are triggered as soon as developer pushes changes to be reviewed. Human reviews won’t be requested until the two stages pass, as there’s no point in spending human attention on things that are not going to be candidates for deployment to production. The two stages may be re-run after code review, to make sure nothing unforeseen has changed while the review took place.</p>
+<p>Other stages may run in parallel with code review, and if they fail they may nullify the release candidacy of the change. For example, stages for manual and capacity testing, and security test/review; depending on the change and the component in question, some or all of these may be necessary.</p>
 <h1 id="the-default-pipeline">The (default?) pipeline</h1>
 <figure>
-<img src="pipeline.svg" alt="The default pipeline" /><figcaption>The default pipeline</figcaption>
+<img src="pipeline.svg" alt="The default pipeline" style="height:25.0%" /><figcaption>The default pipeline</figcaption>
 </figure>
-<ul>
-<li><p>CI will provide a default pipeline for all projects</p>
-<ul>
-<li><p>divided into several stages</p></li>
-<li><p>mandatory stages: commit, acceptance; other stages may be added to other projects as needed</p></li>
-<li><p>the goal is that if commit + acceptance stages pass, the project has a candidate that can be deployed to production, unless the project is such that it needs (say) manual testing or other human decision for the production deployment decision</p></li>
-<li><p>if commit or acceptance stage fails, there is not production candidate</p></li>
-</ul></li>
-<li><p>commit stage</p>
-<ul>
-<li><p>builds all the artifacts that will be used by later stages</p></li>
-<li><p>runs unit tests</p></li>
-<li><p>other tests, possibly integration tests</p></li>
-<li><p>code health checks</p></li>
-<li><p>the commit stage is expected to be fast, aiming at less than five minutes, so that we can expect developers to wait for it to pass successfully</p></li>
-<li><p>the commands to build (compile) or run automated tests are stored in the repository, either explicity, or by indicating the type of build needed; for example, the repository may specify “make” as the command to run, or it may specify that it’s a Go project, and CI would know how to build a Go project; in the latter case we can change the commands to build a Go project by changing CI only, without having to change each git repository with a Go program</p></li>
-<li><p>only the declarative style is possible for building Docker images, as we want control over how that is done</p></li>
-<li><p>CI may enforce specific additional commands to run, to build or test further things; this can be used by RelEng to enforce specific code health checks, for example, or to enable (or disable) debug symbols in all builds</p></li>
-<li><p>all tests run in an isolated build tree, and may not use anything outside the tree, including databases or other backing services</p></li>
-<li><p>any build dependencies must be specified explicitly; for example, which version of Go should be installed in the build environment, or if a project build-depends on another project, which artifacts it needs installed from the other project; explicit is more work, but results in fewer problems due to broken heuristics</p></li>
-</ul></li>
-<li><p>acceptance tests</p>
-<ul>
-<li><p>deploys artifacts from commit stage to containers in special test environments, runs tests against deployed artifacts</p></li>
-<li><p>possibly run slow tests from the build tree as well, if they don’t fit into the commit stage’s time budget</p></li>
-<li><p>this stage can be slower than the commit stage, but should still pass in, say, an hour, instead of taking days</p></li>
-</ul></li>
-<li><p>capacity tests</p>
-<ul>
-<li>these are tests that benchmark the system as deployed into an environment that’s sufficiently production-like also as far as hardware resources are concerned</li>
-</ul></li>
-<li><p>manual (exploratory) tests</p>
-<ul>
-<li><p>testers will have dedicated environments to which they can trigger deployment of specific builds, and in which they can, for example, test that specific bugs are fixed</p></li>
-<li><p>this can also be used to demonstrate upcoming features that are not yet enabled in production</p></li>
-</ul></li>
-</ul>
+<p>CI will provide a default pipeline for all projects. Projects may use that or specify another one.</p>
+<p>The pipeline will be divided into several stages. Mandatory stages for all changes and all projects are commit and acceptance stage. Other stages may be added to specific changes projects as needed.</p>
+<p>The goal is that if the commit and acceptance stages pass, the change is a candidate that can be deployed to production, unless the project is such that it needs (say) manual testing or other human decision for the production deployment decision. Likewise, if the component or the change is particularly security or performance sensitive, stages that check those aspects may be required. CI will have ways of indicating the required changes per component, and also per change. (It is unclear how this will be managed.)</p>
+<p>If the commit or acceptance stage fails, there is not production candidate. The pipeline as a whole fails. Any artifacts built by the pipeline will not be deployable to production, but they may be deployable to test environments, or downloaded by developers for inspection.</p>
+<h2 id="the-commit-stage">The commit stage</h2>
+<p>The commit stage builds any deployable artifacts, such as executable binaries, minimized Javascript, translation files, or Docker images. It is important that artifacts don’t get rebuilt by later stages, because rebuilding does not always result in bitwise identical output. Instead the goal is to build once, test the artifacts, and deploy the tested artifacts, instead of rebuilding and maybe deploying something different than what was tested.</p>
+<p>The commit stage also runs unit tests, and any other tests that can be run in isolation from other parts of the system, and that also are quick. The commit stage does not have access to backing services, such as databases or other components of the overall system. For example, when the pipeline processes a change to a MediaWiki extenasion, the commit stage doesn’t have access to MediaWiki core or the MariaDB MediaWiki uses. Integration or system tests should be done in the acceptance test stage.</p>
+<p>The commit stage also runs code health checks.</p>
+<p>The commit stage is expected to be fast, aiming at less than five minutes, so that we can expect developers to wait for it to pass successfully. This will be a new requirement on our developers.</p>
+<p>The commands to build (compile) or run automated tests are stored in the repository, either explicity, or by indicating the type of build needed. There might be a <code>.pipeline/config.yaml</code> file in the repository, which specifies that <code>make</code> is the command that builds the artifacts. Otherwise, the file may specify that it’s a Go project, and CI would know how to build a Go project. In this case we can change the commands to build a Go project by changing CI only, without having to change each git repository with a Go program.</p>
+<p>Only the declarative style will be possible for building Docker images, as we want control over how that is done (<strong>SECURE</strong> requirement).</p>
+<p>CI may enforce specific additional commands to run, to build or test further things; this can be used by RelEng to enforce certain things. For example, we may enforce code health checks, or to enable (or disable) debug symbols in all builds. Such enforcement will be done in collaboration with our developers.</p>
+<p>Any build dependencies needed during the commit stage must be specified explicitly. For example, the minimum required version of Go that should be installed in the build environment would be a build dependency. If a project build-depends on another project, it needs to specify which project, and which artifacts it needs installed from the other project. Explicit build-dependencies is more work, but results in fewer problems due to broken heuristics.</p>
+<h2 id="the-acceptance-stage">The acceptance stage</h2>
+<p>During the acceptance stage CI deploys artifacts built in the commit stage to a production-like system that has the same versions of all sofware as production, except for the changes being processed by the pipeline. CI will then run automated acceptance tests, and other integration and system tests, against the deployed software. The test environment is clean and empty, and well-known, unless and until the test suite inserts data or makes changes.</p>
+<p>The acceptance stage can take time. Developers are not expected to wait until it is finished before they move on to working on something else.</p>
+<h2 id="manual-tests">Manual tests</h2>
+<p>Testers may instruct CI to deploy any recent built set of artifacts to a dedicated test environment, and can use the software in that environment where it is isolated from others, and won’t suddenly change underneath them. The details of how this will be implemented are to be determined later.</p>
+<p>This feature of the CI can also be used to demonstrate upcoming features that are not yet ready to be deployed to or enabled in production.</p>
+<h2 id="capacity-tests-non-functional-requirements">Capacity tests, non-functional requirements</h2>
+<p>Capacity tests, and other tests for non-functional requirements, will also be done in dedicated, isolated production-like environments. RelEng will work with the performance team to sort out the details.</p>
 <h1 id="architecture-internals">Architecture: internals</h1>
 <p>FIXME This needs to be written, but it needs a lot of thinking first</p>
 <h1 id="acceptance-criteria">Acceptance criteria</h1>
diff --git a/ci-arch.mdwn b/ci-arch.mdwn
index c89fd96..07d4694 100644
--- a/ci-arch.mdwn
+++ b/ci-arch.mdwn
@@ -501,114 +501,143 @@ would prefer to avoid having such changes.
 
 # Architecture: The WMF development ecosystem
 
-![The WMF development ecosystem, roughly](ecosystem.svg)
-
-* the above figure is simplistic, but gives the general idea
-    * developer pushes a change to Gerrit
-    * CI builds and tests change (commit stage)
-    * CI deploys to a test environment, runs tests against that
-      (acceptance test stage)
-    * CI can deploy to an environment dedicated for manual testing
-    * after successful code review, CI merges changes to the master
-      branch, run all automated tests again, and deploys to the
-      production environment
-
-* the commit and acceptance stages are triggered as soon as developer
-  pushes changes to be reviewed; human reviews won't be requested
-  until the two stages pass, as there's no point in spending human
-  attention on things that are not going to be candidates for
-  deployment to production; they may be re-run after code review
-  accepts the changes, to make sure nothing unforeseen has changed
-  while review took place
-
-* other stages may run in parallel with code review, but if they fail
-  they may nullify candidacy? for example, stages for manual and
-  capacity testing, and security test/review; depending on the change
-  and the component in question, some or all of these may be necessary
+![The WMF development ecosystem, roughly](ecosystem.svg){ height=25% }
+
+The figure above is simplistic, but gives the general idea of what
+happens when a developer is finished with a change:
+
+1. developer pushes a change to Gerrit, which trigger CI
+1. CI builds and tests change (commit stage)
+1. CI deploys to a test environment, runs tests against that
+   (acceptance test stage); if everythins is OK, Gerrit is notified
+   and requests code reviews from relevant parties
+1. testers can request CI it deploy the change to an environment
+   dedicated for manual testing
+1. after a successful code review, CI merges changes to the master
+   branch, runs all automated tests again, and deploys to the
+   production environment
+
+The commit and acceptance stages are triggered as soon as developer
+pushes changes to be reviewed. Human reviews won't be requested until
+the two stages pass, as there's no point in spending human attention
+on things that are not going to be candidates for deployment to
+production. The two stages may be re-run after code review, to make
+sure nothing unforeseen has changed while the review took place.
+
+Other stages may run in parallel with code review, and if they fail
+they may nullify the release candidacy of the change. For example,
+stages for manual and capacity testing, and security test/review;
+depending on the change and the component in question, some or all of
+these may be necessary.
 
 # The (default?) pipeline
 
-![The default pipeline](pipeline.svg)
-
-* CI will provide a default pipeline for all projects
-
-  * divided into several stages
-
-  * mandatory stages: commit, acceptance; other stages may be added to
-    other projects as needed
-
-  * the goal is that if commit + acceptance stages pass, the project
-    has a candidate that can be deployed to production, unless the
-    project is such that it needs (say) manual testing or other human
-    decision for the production deployment decision
-
-  * if commit or acceptance stage fails, there is not production
-    candidate
-
-* commit stage
-
-  * builds all the artifacts that will be used by later stages
-
-  * runs unit tests
-
-  * other tests, possibly integration tests
-
-  * code health checks
-
-  * the commit stage is expected to be fast, aiming at less than five
-    minutes, so that we can expect developers to wait for it to pass
-    successfully
-
-  * the commands to build (compile) or run automated tests are stored
-    in the repository, either explicity, or by indicating the type of
-    build needed; for example, the repository may specify "make" as
-    the command to run, or it may specify that it's a Go project, and
-    CI would know how to build a Go project; in the latter case we can
-    change the commands to build a Go project by changing CI only,
-    without having to change each git repository with a Go program
-
-  * only the declarative style is possible for building Docker images,
-    as we want control over how that is done
-
-  * CI may enforce specific additional commands to run, to build or
-    test further things; this can be used by RelEng to enforce
-    specific code health checks, for example, or to enable (or
-    disable) debug symbols in all builds
-
-  * all tests run in an isolated build tree, and may not use anything
-    outside the tree, including databases or other backing services
-
-  * any build dependencies must be specified explicitly; for example,
-    which version of Go should be installed in the build environment,
-    or if a project build-depends on another project, which artifacts
-    it needs installed from the other project; explicit is more work,
-    but results in fewer problems due to broken heuristics
-
-* acceptance tests
-
-  * deploys artifacts from commit stage to containers in special test
-    environments, runs tests against deployed artifacts
-
-  * possibly run slow tests from the build tree as well, if they don't
-    fit into the commit stage's time budget
-
-  * this stage can be slower than the commit stage, but should still
-    pass in, say, an hour, instead of taking days
-
-* capacity tests
-
-  * these are tests that benchmark the system as deployed into an
-    environment that's sufficiently production-like also as far as
-    hardware resources are concerned
-
-* manual (exploratory) tests
-
-  * testers will have dedicated environments to which they can trigger
-    deployment of specific builds, and in which they can, for example,
-    test that specific bugs are fixed
-
-  * this can also be used to demonstrate upcoming features that are
-    not yet enabled in production
+![The default pipeline](pipeline.svg){ height=25% }
+
+CI will provide a default pipeline for all projects. Projects may use
+that or specify another one.
+
+The pipeline will be divided into several stages. Mandatory stages for
+all changes and all projects are commit and acceptance stage. Other
+stages may be added to specific changes projects as needed.
+
+The goal is that if the commit and acceptance stages pass, the change
+is a candidate that can be deployed to production, unless the
+project is such that it needs (say) manual testing or other human
+decision for the production deployment decision. Likewise, if the
+component or the change is particularly security or performance
+sensitive, stages that check those aspects may be required. CI will
+have ways of indicating the required changes per component, and also
+per change. (It is unclear how this will be managed.)
+
+If the commit or acceptance stage fails, there is not production
+candidate. The pipeline as a whole fails. Any artifacts built by the
+pipeline will not be deployable to production, but they may be
+deployable to test environments, or downloaded by developers for
+inspection.
+
+## The commit stage
+
+The commit stage builds any deployable artifacts, such as executable
+binaries, minimized Javascript, translation files, or Docker images.
+It is important that artifacts don't get rebuilt by later stages,
+because rebuilding does not always result in bitwise identical output.
+Instead the goal is to build once, test the artifacts, and deploy the
+tested artifacts, instead of rebuilding and maybe deploying something
+different than what was tested.
+
+The commit stage also runs unit tests, and any other tests that can be
+run in isolation from other parts of the system, and that also are
+quick. The commit stage does not have access to backing services, such
+as databases or other components of the overall system. For example,
+when the pipeline processes a change to a MediaWiki extenasion, the
+commit stage doesn't have access to MediaWiki core or the MariaDB
+MediaWiki uses. Integration or system tests should be done in the
+acceptance test stage.
+
+The commit stage also runs code health checks.
+
+The commit stage is expected to be fast, aiming at less than five
+minutes, so that we can expect developers to wait for it to pass
+successfully. This will be a new requirement on our developers.
+
+The commands to build (compile) or run automated tests are stored in
+the repository, either explicity, or by indicating the type of build
+needed. There might be a `.pipeline/config.yaml` file in the
+repository, which specifies that `make` is the command that builds the
+artifacts. Otherwise, the file may specify that it's a Go project, and
+CI would know how to build a Go project. In this case we can change
+the commands to build a Go project by changing CI only, without having
+to change each git repository with a Go program.
+
+Only the declarative style will be possible for building Docker images,
+as we want control over how that is done (**SECURE** requirement).
+
+CI may enforce specific additional commands to run, to build or test
+further things; this can be used by RelEng to enforce certain things.
+For example, we may enforce code health checks, or to enable (or
+disable) debug symbols in all builds. Such enforcement will be done in
+collaboration with our developers.
+
+Any build dependencies needed during the commit stage must be
+specified explicitly. For example, the minimum required version of Go
+that should be installed in the build environment would be a build
+dependency. If a project build-depends on another project, it needs to
+specify which project, and which artifacts it needs installed from the
+other project. Explicit build-dependencies is more work, but results
+in fewer problems due to broken heuristics.
+
+## The acceptance stage
+
+During the acceptance stage CI deploys artifacts built in the commit
+stage to a production-like system that has the same versions of all
+sofware as production, except for the changes being processed by the
+pipeline. CI will then run automated acceptance tests, and other
+integration and system tests, against the deployed software. The test
+environment is clean and empty, and well-known, unless and until the
+test suite inserts data or makes changes.
+
+The acceptance stage can take time. Developers are not expected to
+wait until it is finished before they move on to working on something
+else.
+
+## Manual tests
+
+Testers may instruct CI to deploy any recent built set of artifacts to
+a dedicated test environment, and can use the software in that
+environment where it is isolated from others, and won't suddenly
+change underneath them. The details of how this will be implemented
+are to be determined later.
+
+This feature of the CI can also be used to demonstrate upcoming
+features that are not yet ready to be deployed to or enabled in
+production.
+
+## Capacity tests, non-functional requirements
+
+Capacity tests, and other tests for non-functional requirements, will
+also be done in dedicated, isolated production-like environments.
+RelEng will work with the performance team to sort out the details.
 
 # Architecture: internals
 
diff --git a/ci-arch.pdf b/ci-arch.pdf
index fbb4910..b7977ad 100644
--- a/ci-arch.pdf
+++ b/ci-arch.pdf