summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorLars Wirzenius <lwirzenius@wikimedia.org>2019-05-06 18:03:45 +0300
committerLars Wirzenius <lwirzenius@wikimedia.org>2019-05-06 18:03:45 +0300
commit5b023bd11d751779006f073f45bf8d862f7ac1a6 (patch)
tree0ecf075b74571d8463e596496bd6049bdbe2b8b3
parentf4863818374398e9c5b2a89f687fd6aedba49e33 (diff)
downloadwmf-ci-arch-5b023bd11d751779006f073f45bf8d862f7ac1a6.tar.gz
Change: format requirements formatting
-rw-r--r--ci-arch.html112
-rw-r--r--ci-arch.mdwn116
-rw-r--r--ci-arch.pdfbin314927 -> 314729 bytes
3 files changed, 114 insertions, 114 deletions
diff --git a/ci-arch.html b/ci-arch.html
index 4a3b77b..6d80ae7 100644
--- a/ci-arch.html
+++ b/ci-arch.html
@@ -58,103 +58,103 @@
<p><strong>FIXME:</strong> We may want to have a way to track which requirements are being fulfilled, or tested by automated acceptance tests. Need to add something for this, maybe a spreadsheet.</p>
<p>These requirements were originally written up in the <a href="https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/CI_Futures_WG/Requirements">WG wiki pages</a> and have been changed a little compared to that (as of the 21 March 2019 version).</p>
<h2 id="very-hard-requirements">Very hard requirements</h2>
+<p>These are non-negotiable requirement that must all be fulfilled by our future CI system.</p>
<ul>
-<li><p>These are non-negotiable requirement that must all be fulfilled by a our future CI system.</p></li>
-<li><p><strong>(SELFHOSTABLE)</strong> Must be hostable by the Foundation. It’s not acceptable for WMF to rely on an outside service for this.</p>
+<li><p><strong>SELFHOSTABLE</strong> Must be hostable by the Foundation. It’s not acceptable for WMF to rely on an outside service for this.</p>
<ul>
-<li><p><strong>(FREESOFTWARE)</strong> Must be free software / open source. “Open core” like GitLab is be good enough, as long as we only need the parts that provide software freedom.</p>
+<li><p><strong>FREESOFTWARE</strong> Must be free software / open source. “Open core” like GitLab is be good enough, as long as we only need the parts that provide software freedom.</p>
<p>This is partly due to the <strong>SELFHOSTABLE</strong> requirement, but also because a WMF value is to prefer open source.</p></li>
</ul></li>
-<li><p><strong>(GITSUPPORT)</strong> Must support git. We’re not switching version control systems for CI.</p></li>
-<li><p><strong>(UNDERSTANDABLE)</strong> Must be understandable without too much effort to our developers so that they can use CI/CD productively.</p></li>
-<li><p><strong>(SELFSERVE)</strong> Must support self-serve CI, meaning we don’t block people if they want CI for a new repo. Due to <strong>PROTECTPRODUCTION</strong>, there will probably to be some human approval requirement for new projects, but as much as possible, people should be allowed to do their work without having to ask permission.</p>
+<li><p><strong>GITSUPPORT</strong> Must support git. We’re not switching version control systems for CI.</p></li>
+<li><p><strong>UNDERSTANDABLE</strong> Must be understandable without too much effort to our developers so that they can use CI/CD productively.</p></li>
+<li><p><strong>SELFSERVE</strong> Must support self-serve CI, meaning we don’t block people if they want CI for a new repo. Due to <strong>PROTECTPRODUCTION</strong>, there will probably to be some human approval requirement for new projects, but as much as possible, people should be allowed to do their work without having to ask permission.</p>
<ul>
-<li><strong>(SELFSERVE2)</strong> Should allow the developers to define or declare at least parts of the pipeline jobs in the repository: what commands to run for building, testing, etc.</li>
+<li><strong>SELFSERVE2</strong> Should allow the developers to define or declare at least parts of the pipeline jobs in the repository: what commands to run for building, testing, etc.</li>
</ul></li>
</ul>
<h2 id="hard-requirements">Hard requirements</h2>
+<p>These are not absolute requirements, and can be negotiated, but only to a minor degree.</p>
<ul>
-<li><p>These are not absolute requirements, and can be negotiated, but only to a minor degree.</p></li>
-<li><p><strong>(FAST)</strong> Must be fast enough that it isn’t perceived as a bottleneck by developers. We will need a metric for this.</p>
+<li><p><strong>FAST</strong> Must be fast enough that it isn’t perceived as a bottleneck by developers. We will need a metric for this.</p>
<ul>
-<li><strong>(SHORTCYCLETIME)</strong> Must enable us to have a short cycle time (from idea to running in production). CI is not the only thing that affects this, but it is an important factor. We probably need a metric for this.</li>
+<li><strong>SHORTCYCLETIME</strong> Must enable us to have a short cycle time (from idea to running in production). CI is not the only thing that affects this, but it is an important factor. We probably need a metric for this.</li>
</ul></li>
-<li><p><strong>(TRANSPARENT)</strong> Must make its status and what-is-going-on visible so that its operation can be monitored and so that our developers can check the status of their builds themselves. Also the overall status of CI, for example, so they can see if their build is blocked by waiting on others.</p></li>
-<li><p><strong>(FEEDBACK)</strong> Must provide feedback to the developers as early as possible for the various stages of a build, especially the early stages (“can get source from git”, “can build”, “can run unit tests”, etc.).</p>
+<li><p><strong>TRANSPARENT</strong> Must make its status and what-is-going-on visible so that its operation can be monitored and so that our developers can check the status of their builds themselves. Also the overall status of CI, for example, so they can see if their build is blocked by waiting on others.</p></li>
+<li><p><strong>FEEDBACK</strong> Must provide feedback to the developers as early as possible for the various stages of a build, especially the early stages (“can get source from git”, “can build”, “can run unit tests”, etc.).</p>
<p>The goal is to give feedback as soon as possible, especially in the case of the build failing.</p>
<ul>
-<li><strong>(FEEDBACK2)</strong> Must support providing feedback via Gerrit, IRC, and Phabricator, at the very least. These are our current main feedback channels.</li>
+<li><strong>FEEDBACK2</strong> Must support providing feedback via Gerrit, IRC, and Phabricator, at the very least. These are our current main feedback channels.</li>
</ul></li>
-<li><p><strong>(SECURE)</strong> Must be secure enough that we can open it to community developers to use without too much supervision.</p></li>
-<li><p><strong>(MAINTAINED)</strong> Must be maintained and supported upstream. The CI system should not require substantial development from the Foundation. Some customization is expected to be necessary.</p></li>
-<li><p><strong>(MANYREPOS)</strong> Must be able to handle the number of repositories, projects, builds, and deployments that we have, and will have in the foreseeable future.</p></li>
-<li><p><strong>(METRICS)</strong> Must enable us to instrument it to get metrics for CI use and effectiveness as we need. Things like cycle times, build times, build failures, etc.</p></li>
-<li><p><strong>(GERRIT)</strong> Must work with Gerrit as well as other self-hostable code-review systems (e.g., GitLab), if we decide to move to that later. This means, code review happens on Gerrit, after building and automated tests pass, and positive code review triggers deployment to production.</p></li>
-<li><p><strong>(NOREBUILDING)</strong> Must promote (copy) Docker images and other build artifacts from “testing” to “staging” to “production”, rather than rebuilding them, since rebuilding takes time and can fail. Once a binary, Docker image, or other build artifact has been built, exactly that artifact should be tested, and eventually deployed to production.</p></li>
-<li><p><strong>(LOCALTESTS)</strong> Must allow developer to replicate locally the tests that CI runs. This is necessary to allow lower friction in development, as well as to aid debugging. For example, if CI builds and tests using Docker container, a developer should be able to download the same image and run the tests locally.</p></li>
-<li><p><strong>(AUTOMATEDEPLOYMENT)</strong> Must allow deployment to be fully automated.</p>
+<li><p><strong>SECURE</strong> Must be secure enough that we can open it to community developers to use without too much supervision.</p></li>
+<li><p><strong>MAINTAINED</strong> Must be maintained and supported upstream. The CI system should not require substantial development from the Foundation. Some customization is expected to be necessary.</p></li>
+<li><p><strong>MANYREPOS</strong> Must be able to handle the number of repositories, projects, builds, and deployments that we have, and will have in the foreseeable future.</p></li>
+<li><p><strong>METRICS</strong> Must enable us to instrument it to get metrics for CI use and effectiveness as we need. Things like cycle times, build times, build failures, etc.</p></li>
+<li><p><strong>GERRIT</strong> Must work with Gerrit as well as other self-hostable code-review systems (e.g., GitLab), if we decide to move to that later. This means, code review happens on Gerrit, after building and automated tests pass, and positive code review triggers deployment to production.</p></li>
+<li><p><strong>NOREBUILDING</strong> Must promote (copy) Docker images and other build artifacts from “testing” to “staging” to “production”, rather than rebuilding them, since rebuilding takes time and can fail. Once a binary, Docker image, or other build artifact has been built, exactly that artifact should be tested, and eventually deployed to production.</p></li>
+<li><p><strong>LOCALTESTS</strong> Must allow developer to replicate locally the tests that CI runs. This is necessary to allow lower friction in development, as well as to aid debugging. For example, if CI builds and tests using Docker container, a developer should be able to download the same image and run the tests locally.</p></li>
+<li><p><strong>AUTOMATEDEPLOYMENT</strong> Must allow deployment to be fully automated.</p>
<ul>
-<li><strong>(AUTOMATEDSELFDEPLOYMENT)</strong> Must be automatically deployable by us or SRE, onto a fresh server.</li>
+<li><strong>AUTOMATEDSELFDEPLOYMENT</strong> Must be automatically deployable by us or SRE, onto a fresh server.</li>
</ul></li>
-<li><p><strong>(HSCALABLE)</strong> Must be horizontally scalable: we need to be able to add more hardware easily to get more capacity. This is particularly important for build workers, which are the mostly likely bottleneck. Also, probably environments used for testing.</p></li>
-<li><p><strong>(PROGLANGS)</strong> Must be able to support all programming languages we currently support or are likely to support in the future. These include, at least, shell, Python, Ruby, Java, PHP, and Go. Some languages may be needed in several versions.</p></li>
-<li><p><strong>(OUTPUTLINKS)</strong> Must support HTTP linking to build results for easier reference and discussion. This way a build log, or a build artifact, can be reference using a simple HTTP (or HTTPS) link.</p></li>
-<li><p><strong>(ARTIFACTARCHIVE)</strong> Should allow archiving build logs, executables, Docker images, and other build artifacts for a long period.</p>
+<li><p><strong>HSCALABLE</strong> Must be horizontally scalable: we need to be able to add more hardware easily to get more capacity. This is particularly important for build workers, which are the mostly likely bottleneck. Also, probably environments used for testing.</p></li>
+<li><p><strong>PROGLANGS</strong> Must be able to support all programming languages we currently support or are likely to support in the future. These include, at least, shell, Python, Ruby, Java, PHP, and Go. Some languages may be needed in several versions.</p></li>
+<li><p><strong>OUTPUTLINKS</strong> Must support HTTP linking to build results for easier reference and discussion. This way a build log, or a build artifact, can be reference using a simple HTTP (or HTTPS) link.</p></li>
+<li><p><strong>ARTIFACTARCHIVE</strong> Should allow archiving build logs, executables, Docker images, and other build artifacts for a long period.</p>
<ul>
-<li><strong>(RETENTION)</strong> The retention period should be configurable based on artifact type, and whether the build ended up being deployed to production.</li>
+<li><strong>RETENTION</strong> The retention period should be configurable based on artifact type, and whether the build ended up being deployed to production.</li>
</ul></li>
-<li><p><strong>(CONFIGVC)</strong> Must keep configuration in version control. This is needed so that we can track changes over time.</p></li>
-<li><p><strong>(GATING)</strong> Must support gating / pre-merge testing. FIXME: This needs to be explained.</p></li>
-<li><p><strong>(PERIODICBUILDS)</strong> Must support periodic / scheduled testing. This is needed so that we can test that changes to the environment haven’t broken anything. An example would be changes to Debian, upon which we base our container images.</p></li>
-<li><p><strong>(POSTMERGETESTS)</strong> Must support post-merge testing. FIXME: This needs to be explained.</p></li>
-<li><p><strong>(CIMERGES)</strong> Must support tooling to do the merging, instead of developers. We don’t want developer merging by hand and pushing the merges. CI should test changes and merge only if tests pass, so that the branches for main lines of development are always releaseble.</p></li>
-<li><p><strong>(TESTVC)</strong> Must support storing tests in version control. This is probably best achieved by having tests be stored in the same git repository where the code is.</p></li>
-<li><p><strong>(BUILDDEPS)</strong> Must have some way to declare dependent repositories / software needed for testing. FIXME: This needs to be explained.</p></li>
-<li><p><strong>(TESTSERVICES)</strong> Must support services for tests — i.e., some PHPUnit tests require MySQL. These are most important for integration tests. Proper unit tests do not depend on any external stuff. However, integration tests may well need MediaWiki, some specific extensions, and backing services, such as databases, “oid” services, and possibly more. CI needs to be able to provide such environments for testing.</p></li>
-<li><p><strong>(OTHERGITORTICKETING)</strong> Must allow changing git repository, code review, and ticketing systems from Gerrit and Phabricator. We are not currently looking at switching away from Gerrit and Phabricator, but the future CI solution should not lock us into specific code review or ticketing solutions.</p></li>
-<li><p><strong>(PROTECTPRODUCTION)</strong> Must protect production by detecting problems before they’re deployed, and must in general support a sensible CI/CD pipeline. This is necessary both for the safety and security of our production systems, a higher speed of development, and higher productivity. The protection brings developer confidence, which tends to bring speed and productivity.</p>
+<li><p><strong>CONFIGVC</strong> Must keep configuration in version control. This is needed so that we can track changes over time.</p></li>
+<li><p><strong>GATING</strong> Must support gating / pre-merge testing. FIXME: This needs to be explained.</p></li>
+<li><p><strong>PERIODICBUILDS</strong> Must support periodic / scheduled testing. This is needed so that we can test that changes to the environment haven’t broken anything. An example would be changes to Debian, upon which we base our container images.</p></li>
+<li><p><strong>POSTMERGETESTS</strong> Must support post-merge testing. FIXME: This needs to be explained.</p></li>
+<li><p><strong>CIMERGES</strong> Must support tooling to do the merging, instead of developers. We don’t want developer merging by hand and pushing the merges. CI should test changes and merge only if tests pass, so that the branches for main lines of development are always releaseble.</p></li>
+<li><p><strong>TESTVC</strong> Must support storing tests in version control. This is probably best achieved by having tests be stored in the same git repository where the code is.</p></li>
+<li><p><strong>BUILDDEPS</strong> Must have some way to declare dependent repositories / software needed for testing. FIXME: This needs to be explained.</p></li>
+<li><p><strong>TESTSERVICES</strong> Must support services for tests — i.e., some PHPUnit tests require MySQL. These are most important for integration tests. Proper unit tests do not depend on any external stuff. However, integration tests may well need MediaWiki, some specific extensions, and backing services, such as databases, “oid” services, and possibly more. CI needs to be able to provide such environments for testing.</p></li>
+<li><p><strong>OTHERGITORTICKETING</strong> Must allow changing git repository, code review, and ticketing systems from Gerrit and Phabricator. We are not currently looking at switching away from Gerrit and Phabricator, but the future CI solution should not lock us into specific code review or ticketing solutions.</p></li>
+<li><p><strong>PROTECTPRODUCTION</strong> Must protect production by detecting problems before they’re deployed, and must in general support a sensible CI/CD pipeline. This is necessary both for the safety and security of our production systems, a higher speed of development, and higher productivity. The protection brings developer confidence, which tends to bring speed and productivity.</p>
<ul>
-<li><strong>(ENFORCETESTS)</strong> Must allow Release Engineering team to enforce tests on top of what a self-serving developer specifies, to allow us to set minimal technical standards.</li>
+<li><strong>ENFORCETESTS</strong> Must allow Release Engineering team to enforce tests on top of what a self-serving developer specifies, to allow us to set minimal technical standards.</li>
</ul></li>
-<li><p><strong>(CACHEDEPS)</strong> Must support dependency caching – we have castor, maybe we could do better? Maybe some CI systems have this figured out? This means, for example, caching npm and PyPI packages so that every build doesn’t need to download them directly from the centralised package repositories. This is needed for speed.</p></li>
+<li><p><strong>CACHEDEPS</strong> Must support dependency caching – we have castor, maybe we could do better? Maybe some CI systems have this figured out? This means, for example, caching npm and PyPI packages so that every build doesn’t need to download them directly from the centralised package repositories. This is needed for speed.</p></li>
</ul>
<h2 id="softer-requirements">Softer requirements</h2>
+<p>These requirements are even more easily negotiated.</p>
<ul>
-<li><p>These requirements are even more easily negotiated.</p></li>
-<li><p><strong>(HA)</strong> Should be highly available - can restart any component without disrupting service.</p></li>
-<li><p><strong>(LIVELOG)</strong> Should have live console output of build.</p></li>
-<li><p><strong>(MAXBUILDTIME)</strong> Should have build timeouts so that a build may fail if it takes too long. Among other reasons, this is useful to automatically work around builds that get “stuck” indefinitely.</p></li>
-<li><p><strong>(CLEANWORKSPACE)</strong> Should provide a clean workspace for each test run - either a clean VM or container.</p></li>
-<li><p><strong>(RATELIMIT)</strong> Should have rate limiting - one user/project can not take over most/all resources.</p></li>
-<li><p><strong>(CHECKSIG)</strong> Should support validation and creation of GPG/PGP-signed git commits</p></li>
-<li><p><strong>(SECRETS)</strong> Should support secure storage of credentials / secrets.</p></li>
+<li><p><strong>HA</strong> Should be highly available - can restart any component without disrupting service.</p></li>
+<li><p><strong>LIVELOG</strong> Should have live console output of build.</p></li>
+<li><p><strong>MAXBUILDTIME</strong> Should have build timeouts so that a build may fail if it takes too long. Among other reasons, this is useful to automatically work around builds that get “stuck” indefinitely.</p></li>
+<li><p><strong>CLEANWORKSPACE</strong> Should provide a clean workspace for each test run - either a clean VM or container.</p></li>
+<li><p><strong>RATELIMIT</strong> Should have rate limiting - one user/project can not take over most/all resources.</p></li>
+<li><p><strong>CHECKSIG</strong> Should support validation and creation of GPG/PGP-signed git commits</p></li>
+<li><p><strong>SECRETS</strong> Should support secure storage of credentials / secrets.</p></li>
</ul>
<h2 id="would-be-nice">Would be nice</h2>
<ul>
<li><p>These are so soft they aren’t even requirements, and more wish list items.</p></li>
-<li><p><strong>(LIMITBOILERPLATE)</strong> Would be nice for test abstractions to limit boiler-plate, i.e., all of our services are tested roughly the same way without having to copy instructions to every repository.</p></li>
-<li><p><strong>(PRIORITIZEJOBS)</strong> Would be nice to prioritize jobs.</p>
+<li><p><strong>LIMITBOILERPLATE</strong> Would be nice for test abstractions to limit boiler-plate, i.e., all of our services are tested roughly the same way without having to copy instructions to every repository.</p></li>
+<li><p><strong>PRIORITIZEJOBS</strong> Would be nice to prioritize jobs.</p>
<ul>
<li><p>Use case: if there is a queue of jobs, there should be some mechanism of jumping that queue for jobs that have a higher priority.</p></li>
<li><p>We currently have a Gating queue that is a higher priority than periodic jobs that calculate Code Coverage.</p></li>
</ul></li>
-<li><p><strong>(ISOLATION)</strong> Would be nice to support isolation / sandboxing.</p>
+<li><p><strong>ISOLATION</strong> Would be nice to support isolation / sandboxing.</p>
<ul>
<li><p>Jobs should be isolated from one another.</p></li>
<li><p>Jobs should be able to install apt-packages without affecting dependencies of other jobs.</p></li>
</ul></li>
-<li><p><strong>(CONTROLAFFINITY)</strong> Would be nice to have configurable job requirements/affinity.</p>
+<li><p><strong>CONTROLAFFINITY</strong> Would be nice to have configurable job requirements/affinity.</p>
<ul>
<li>Be able to schedule a job only on nodes that have at least X available disk space/ram/cpu/whatever OR try to schedule on nodes where a current build of this job isn’t already running.</li>
</ul></li>
-<li><p><strong>(POSTMERGEBISECT)</strong> Would be nice to post-merge git-bisect to find patch that caused a particular problem with a Selenium test.</p></li>
-<li><p><strong>(DEPLOYWHEREVER)</strong> Would be nice to have a mechanism for deployment to staging, production, pypi, packagist, toollabs. We could do with a way to deploy to any of several possible environments, for various use cases, such as bug repoduction, manual exploratory testing, capacity testing, and production. FIXME: what do pypi and packagist do in the list?</p></li>
-<li><p><strong>(MATRIXBUILDS)</strong> Would be nice to have efficient matrix builds.</p>
+<li><p><strong>POSTMERGEBISECT</strong> Would be nice to post-merge git-bisect to find patch that caused a particular problem with a Selenium test.</p></li>
+<li><p><strong>DEPLOYWHEREVER</strong> Would be nice to have a mechanism for deployment to staging, production, pypi, packagist, toollabs. We could do with a way to deploy to any of several possible environments, for various use cases, such as bug repoduction, manual exploratory testing, capacity testing, and production. FIXME: what do pypi and packagist do in the list?</p></li>
+<li><p><strong>MATRIXBUILDS</strong> Would be nice to have efficient matrix builds.</p>
<ul>
<li>E.g., we currently run phpunit tests and browser tests for the Cartesian product of {PHP7 PHP7.1 PHP7.2 HHVM} x {MySQL, SQLite, PostgreSQL} x {Composer, MediaWiki vendor}, but we perform setup/git clone for all of those tests. Doing that in a space and time efficient way would be good.</li>
</ul></li>
-<li><p><strong>(MOBILE)</strong> Would be nice to support building and testing mobile applications (at minimum for iOS and Android).</p></li>
-<li><p><strong>(EMBARGO)</strong> Would be nice to be able to run for secret/security patches. This means CI should be able to build and deploy changes that can’t be made public yet, for security embargo reasons.</p></li>
+<li><p><strong>MOBILE</strong> Would be nice to support building and testing mobile applications (at minimum for iOS and Android).</p></li>
+<li><p><strong>EMBARGO</strong> Would be nice to be able to run for secret/security patches. This means CI should be able to build and deploy changes that can’t be made public yet, for security embargo reasons.</p></li>
</ul>
<h1 id="important-use-cases">Important use cases</h1>
<p>These are some of the important use cases for the CI system, and how we plan CI to implement them.</p>
diff --git a/ci-arch.mdwn b/ci-arch.mdwn
index 5c9e6f4..df00710 100644
--- a/ci-arch.mdwn
+++ b/ci-arch.mdwn
@@ -76,55 +76,55 @@ and have been changed a little compared to that (as of the 21 March
## Very hard requirements
-* These are non-negotiable requirement that must all be fulfilled by a
- our future CI system.
+These are non-negotiable requirement that must all be fulfilled by our
+future CI system.
-* **(SELFHOSTABLE)** Must be hostable by the Foundation. It's not
+* **SELFHOSTABLE** Must be hostable by the Foundation. It's not
acceptable for WMF to rely on an outside service for this.
- * **(FREESOFTWARE)** Must be free software / open source. "Open
+ * **FREESOFTWARE** Must be free software / open source. "Open
core" like GitLab is be good enough, as long as we only need the
parts that provide software freedom.
This is partly due to the **SELFHOSTABLE** requirement, but also
because a WMF value is to prefer open source.
-* **(GITSUPPORT)** Must support git. We're not switching version control
+* **GITSUPPORT** Must support git. We're not switching version control
systems for CI.
-* **(UNDERSTANDABLE)** Must be understandable without too much effort to
+* **UNDERSTANDABLE** Must be understandable without too much effort to
our developers so that they can use CI/CD productively.
-* **(SELFSERVE)** Must support self-serve CI, meaning we don't block
+* **SELFSERVE** Must support self-serve CI, meaning we don't block
people if they want CI for a new repo. Due to **PROTECTPRODUCTION**,
there will probably to be some human approval requirement for new
projects, but as much as possible, people should be allowed to do
their work without having to ask permission.
- * **(SELFSERVE2)** Should allow the developers to define or
+ * **SELFSERVE2** Should allow the developers to define or
declare at least parts of the pipeline jobs in the repository:
what commands to run for building, testing, etc.
## Hard requirements
-* These are not absolute requirements, and can be negotiated, but only
- to a minor degree.
+These are not absolute requirements, and can be negotiated, but only
+to a minor degree.
-* **(FAST)** Must be fast enough that it isn't perceived as a bottleneck
+* **FAST** Must be fast enough that it isn't perceived as a bottleneck
by developers. We will need a metric for this.
- * **(SHORTCYCLETIME)** Must enable us to have a short cycle time
+ * **SHORTCYCLETIME** Must enable us to have a short cycle time
(from idea to running in production). CI is not the only thing
that affects this, but it is an important factor. We probably
need a metric for this.
-* **(TRANSPARENT)** Must make its status and what-is-going-on visible so
+* **TRANSPARENT** Must make its status and what-is-going-on visible so
that its operation can be monitored and so that our developers can
check the status of their builds themselves. Also the overall status
of CI, for example, so they can see if their build is blocked by
waiting on others.
-* **(FEEDBACK)** Must provide feedback to the developers as early as
+* **FEEDBACK** Must provide feedback to the developers as early as
possible for the various stages of a build, especially the early
stages ("can get source from git", "can build", "can run unit
tests", etc.).
@@ -132,98 +132,98 @@ and have been changed a little compared to that (as of the 21 March
The goal is to give feedback as soon as possible, especially in the
case of the build failing.
- * **(FEEDBACK2)** Must support providing feedback via Gerrit, IRC, and
+ * **FEEDBACK2** Must support providing feedback via Gerrit, IRC, and
Phabricator, at the very least. These are our current main feedback
channels.
-* **(SECURE)** Must be secure enough that we can open it to community
+* **SECURE** Must be secure enough that we can open it to community
developers to use without too much supervision.
-* **(MAINTAINED)** Must be maintained and supported upstream. The CI
+* **MAINTAINED** Must be maintained and supported upstream. The CI
system should not require substantial development from the
Foundation. Some customization is expected to be necessary.
-* **(MANYREPOS)** Must be able to handle the number of repositories,
+* **MANYREPOS** Must be able to handle the number of repositories,
projects, builds, and deployments that we have, and will have in the
foreseeable future.
-* **(METRICS)** Must enable us to instrument it to get metrics for CI use
+* **METRICS** Must enable us to instrument it to get metrics for CI use
and effectiveness as we need. Things like cycle times, build times,
build failures, etc.
-* **(GERRIT)** Must work with Gerrit as well as other self-hostable
+* **GERRIT** Must work with Gerrit as well as other self-hostable
code-review systems (e.g., GitLab), if we decide to move to that
later. This means, code review happens on Gerrit, after building and
automated tests pass, and positive code review triggers deployment
to production.
-* **(NOREBUILDING)** Must promote (copy) Docker images and other build
+* **NOREBUILDING** Must promote (copy) Docker images and other build
artifacts from "testing" to "staging" to "production", rather than
rebuilding them, since rebuilding takes time and can fail. Once a
binary, Docker image, or other build artifact has been built,
exactly that artifact should be tested, and eventually deployed to
production.
-* **(LOCALTESTS)** Must allow developer to replicate locally the tests
+* **LOCALTESTS** Must allow developer to replicate locally the tests
that CI runs. This is necessary to allow lower friction in
development, as well as to aid debugging. For example, if CI builds
and tests using Docker container, a developer should be able to
download the same image and run the tests locally.
-* **(AUTOMATEDEPLOYMENT)** Must allow deployment to be fully automated.
+* **AUTOMATEDEPLOYMENT** Must allow deployment to be fully automated.
- * **(AUTOMATEDSELFDEPLOYMENT)** Must be automatically deployable
+ * **AUTOMATEDSELFDEPLOYMENT** Must be automatically deployable
by us or SRE, onto a fresh server.
-* **(HSCALABLE)** Must be horizontally scalable: we need to be able to add
+* **HSCALABLE** Must be horizontally scalable: we need to be able to add
more hardware easily to get more capacity. This is particularly
important for build workers, which are the mostly likely bottleneck.
Also, probably environments used for testing.
-* **(PROGLANGS)** Must be able to support all programming languages we
+* **PROGLANGS** Must be able to support all programming languages we
currently support or are likely to support in the future. These
include, at least, shell, Python, Ruby, Java, PHP, and Go. Some
languages may be needed in several versions.
-* **(OUTPUTLINKS)** Must support HTTP linking to build results for
+* **OUTPUTLINKS** Must support HTTP linking to build results for
easier reference and discussion. This way a build log, or a build
artifact, can be reference using a simple HTTP (or HTTPS) link.
-* **(ARTIFACTARCHIVE)** Should allow archiving build logs,
+* **ARTIFACTARCHIVE** Should allow archiving build logs,
executables, Docker images, and other build artifacts for a long
period.
- * **(RETENTION)** The retention period should be configurable
+ * **RETENTION** The retention period should be configurable
based on artifact type, and whether the build ended up being
deployed to production.
-* **(CONFIGVC)** Must keep configuration in version control. This is
+* **CONFIGVC** Must keep configuration in version control. This is
needed so that we can track changes over time.
-* **(GATING)** Must support gating / pre-merge testing. FIXME: This
+* **GATING** Must support gating / pre-merge testing. FIXME: This
needs to be explained.
-* **(PERIODICBUILDS)** Must support periodic / scheduled testing. This
+* **PERIODICBUILDS** Must support periodic / scheduled testing. This
is needed so that we can test that changes to the environment
haven't broken anything. An example would be changes to Debian, upon
which we base our container images.
-* **(POSTMERGETESTS)** Must support post-merge testing. FIXME: This
+* **POSTMERGETESTS** Must support post-merge testing. FIXME: This
needs to be explained.
-* **(CIMERGES)** Must support tooling to do the merging, instead of
+* **CIMERGES** Must support tooling to do the merging, instead of
developers. We don't want developer merging by hand and pushing the
merges. CI should test changes and merge only if tests pass, so that
the branches for main lines of development are always releaseble.
-* **(TESTVC)** Must support storing tests in version control. This is
+* **TESTVC** Must support storing tests in version control. This is
probably best achieved by having tests be stored in the same git
repository where the code is.
-* **(BUILDDEPS)** Must have some way to declare dependent repositories /
+* **BUILDDEPS** Must have some way to declare dependent repositories /
software needed for testing. FIXME: This needs to be explained.
-* **(TESTSERVICES)** Must support services for tests — i.e., some
+* **TESTSERVICES** Must support services for tests — i.e., some
PHPUnit tests require MySQL. These are most important for
integration tests. Proper unit tests do not depend on any external
stuff. However, integration tests may well need MediaWiki, some
@@ -231,24 +231,24 @@ and have been changed a little compared to that (as of the 21 March
services, and possibly more. CI needs to be able to provide such
environments for testing.
-* **(OTHERGITORTICKETING)** Must allow changing git repository, code
+* **OTHERGITORTICKETING** Must allow changing git repository, code
review, and ticketing systems from Gerrit and Phabricator. We are
not currently looking at switching away from Gerrit and Phabricator,
but the future CI solution should not lock us into specific code
review or ticketing solutions.
-* **(PROTECTPRODUCTION)** Must protect production by detecting problems
+* **PROTECTPRODUCTION** Must protect production by detecting problems
before they're deployed, and must in general support a sensible
CI/CD pipeline. This is necessary both for the safety and security
of our production systems, a higher speed of development, and higher
productivity. The protection brings developer confidence, which
tends to bring speed and productivity.
- * **(ENFORCETESTS)** Must allow Release Engineering team to
+ * **ENFORCETESTS** Must allow Release Engineering team to
enforce tests on top of what a self-serving developer specifies,
to allow us to set minimal technical standards.
-* **(CACHEDEPS)** Must support dependency caching – we have castor, maybe
+* **CACHEDEPS** Must support dependency caching – we have castor, maybe
we could do better? Maybe some CI systems have this figured out?
This means, for example, caching npm and PyPI packages so that every
build doesn't need to download them directly from the centralised
@@ -256,38 +256,38 @@ and have been changed a little compared to that (as of the 21 March
## Softer requirements
-* These requirements are even more easily negotiated.
+These requirements are even more easily negotiated.
-* **(HA)** Should be highly available - can restart any component without
+* **HA** Should be highly available - can restart any component without
disrupting service.
-* **(LIVELOG)** Should have live console output of build.
+* **LIVELOG** Should have live console output of build.
-* **(MAXBUILDTIME)** Should have build timeouts so that a build may
+* **MAXBUILDTIME** Should have build timeouts so that a build may
fail if it takes too long. Among other reasons, this is useful to
automatically work around builds that get "stuck" indefinitely.
-* **(CLEANWORKSPACE)** Should provide a clean workspace for each test
+* **CLEANWORKSPACE** Should provide a clean workspace for each test
run - either a clean VM or container.
-* **(RATELIMIT)** Should have rate limiting - one user/project can not
+* **RATELIMIT** Should have rate limiting - one user/project can not
take over most/all resources.
-* **(CHECKSIG)** Should support validation and creation of GPG/PGP-signed
+* **CHECKSIG** Should support validation and creation of GPG/PGP-signed
git commits
-* **(SECRETS)** Should support secure storage of credentials / secrets.
+* **SECRETS** Should support secure storage of credentials / secrets.
## Would be nice
* These are so soft they aren't even requirements, and more wish list
items.
-* **(LIMITBOILERPLATE)** Would be nice for test abstractions to limit
+* **LIMITBOILERPLATE** Would be nice for test abstractions to limit
boiler-plate, i.e., all of our services are tested roughly the same
way without having to copy instructions to every repository.
-* **(PRIORITIZEJOBS)** Would be nice to prioritize jobs.
+* **PRIORITIZEJOBS** Would be nice to prioritize jobs.
* Use case: if there is a queue of jobs, there should be some
mechanism of jumping that queue for jobs that have a higher
@@ -296,31 +296,31 @@ and have been changed a little compared to that (as of the 21 March
* We currently have a Gating queue that is a higher priority than
periodic jobs that calculate Code Coverage.
-* **(ISOLATION)** Would be nice to support isolation / sandboxing.
+* **ISOLATION** Would be nice to support isolation / sandboxing.
* Jobs should be isolated from one another.
* Jobs should be able to install apt-packages without affecting
dependencies of other jobs.
-* **(CONTROLAFFINITY)** Would be nice to have configurable job
+* **CONTROLAFFINITY** Would be nice to have configurable job
requirements/affinity.
* Be able to schedule a job only on nodes that have at least X
available disk space/ram/cpu/whatever OR try to schedule on
nodes where a current build of this job isn't already running.
-* **(POSTMERGEBISECT)** Would be nice to post-merge git-bisect to find
+* **POSTMERGEBISECT** Would be nice to post-merge git-bisect to find
patch that caused a particular problem with a Selenium test.
-* **(DEPLOYWHEREVER)** Would be nice to have a mechanism for
+* **DEPLOYWHEREVER** Would be nice to have a mechanism for
deployment to staging, production, pypi, packagist, toollabs. We
could do with a way to deploy to any of several possible
environments, for various use cases, such as bug repoduction, manual
exploratory testing, capacity testing, and production. FIXME: what
do pypi and packagist do in the list?
-* **(MATRIXBUILDS)** Would be nice to have efficient matrix builds.
+* **MATRIXBUILDS** Would be nice to have efficient matrix builds.
* E.g., we currently run phpunit tests and browser tests for the
Cartesian product of {PHP7 PHP7.1 PHP7.2 HHVM} x {MySQL,
@@ -328,10 +328,10 @@ and have been changed a little compared to that (as of the 21 March
perform setup/git clone for all of those tests. Doing that in a
space and time efficient way would be good.
-* **(MOBILE)** Would be nice to support building and testing mobile
+* **MOBILE** Would be nice to support building and testing mobile
applications (at minimum for iOS and Android).
-* **(EMBARGO)** Would be nice to be able to run for secret/security
+* **EMBARGO** Would be nice to be able to run for secret/security
patches. This means CI should be able to build and deploy changes
that can't be made public yet, for security embargo reasons.
diff --git a/ci-arch.pdf b/ci-arch.pdf
index 9ac7a26..4e24a46 100644
--- a/ci-arch.pdf
+++ b/ci-arch.pdf
Binary files differ