summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorLars Wirzenius <lwirzenius@wikimedia.org>2019-05-20 19:13:36 +0300
committerLars Wirzenius <lwirzenius@wikimedia.org>2019-05-20 19:13:36 +0300
commit1e4268f293470b8d2dec339ce34ede37a10331be (patch)
tree1ea49b3d691642260c7af9cffe947e5be0ad8d2b
parentc8fa7b686f9eabaa73ae2f1523e806db7105535d (diff)
downloadwmf-ci-arch-1e4268f293470b8d2dec339ce34ede37a10331be.tar.gz
Change: show separate deployment stage for pipeline, drop artifacts
-rw-r--r--ci-arch.html7
-rw-r--r--ci-arch.pdfbin285787 -> 286488 bytes
-rw-r--r--pipeline.dot10
-rw-r--r--pipeline.svg89
4 files changed, 46 insertions, 60 deletions
diff --git a/ci-arch.html b/ci-arch.html
index 71c3195..2153b3c 100644
--- a/ci-arch.html
+++ b/ci-arch.html
@@ -90,11 +90,12 @@
<p>This is partly due to the <strong>SELFHOSTABLE</strong> requirement, but also because a WMF value is to prefer open source.</p></li>
</ul></li>
<li><p><strong>GITSUPPORT</strong> Must support git. We’re not switching version control systems for CI.</p></li>
-<li><p><strong>UNDERSTANDABLE</strong> Must be understandable without too much effort to our developers so that they can use CI/CD productively. Acceptance critera: our developers can use CI productively, after given a short, 1-page tutorial on how to specify build instructions for their software. FIXME: This might not be very clear.</p></li>
+<li><p><strong>UNDERSTANDABLE</strong> Must be understandable without too much effort to our developers so that they can use CI/CD productively. Acceptance critera: our developers can use CI productively, after given a short, 1-page tutorial on how to specify build instructions for their software. FIXME: This might not be very clear; suggetions welcome.</p></li>
<li><p><strong>SELFSERVE</strong> Must support self-serve CI, meaning we don’t block people if they want CI for a new repo. Due to <strong>PROTECTPRODUCTION</strong>, there will probably to be some human approval requirement for new projects, but as much as possible, people should be allowed to do their work without having to ask permission.</p>
<ul>
<li><strong>SELFSERVE2</strong> Should allow the developers to define or declare at least parts of the pipeline jobs in the repository: what commands to run for building, testing, etc.</li>
</ul></li>
+<li><p><strong>POSTMERGETESTS</strong> All automated tests must pass both before and after merging a change. This can be achieved by various means (run tests twice, or make sure target branch doesn’t change in between). The justification is that the way Gerrit and Zuul currently work, the target branch may change while other changes are reviewed and tested, so we run some test after a merge (but possibly before publishing the merge?). FIXME: is this adequate?</p></li>
</ul>
<h2 id="hard-requirements">Hard requirements</h2>
<p>These are not absolute requirements, and can be negotiated, but only to a minor degree.</p>
@@ -132,7 +133,6 @@
<li><p><strong>CONFIGVC</strong> Must keep configuration in version control. This is needed so that we can track changes over time.</p></li>
<li><p><strong>GATING</strong> Must support gating / pre-merge testing. FIXME: This needs to be explained.</p></li>
<li><p><strong>PERIODICBUILDS</strong> Must support periodic / scheduled testing. This is needed so that we can test that changes to the environment haven’t broken anything. An example would be changes to Debian, upon which we base our container images.</p></li>
-<li><p><strong>POSTMERGETESTS</strong> Must support post-merge testing. FIXME: This needs to be explained.</p></li>
<li><p><strong>CIMERGES</strong> Must support tooling to do the merging, instead of developers. We don’t want developer merging by hand and pushing the merges. CI should test changes and merge only if tests pass, so that the branches for main lines of development are always releaseble.</p></li>
<li><p><strong>TESTVC</strong> Must support storing tests in version control. This is probably best achieved by having tests be stored in the same git repository where the code is.</p></li>
<li><p><strong>BUILDDEPS</strong> Must have some way to declare dependent repositories / software needed for testing. FIXME: This needs to be explained.</p></li>
@@ -150,7 +150,8 @@
<li><p><strong>SLA</strong>: RelEng provides a promise, similar to a service-level agreement, for how well CI serves WMF and the movement.</p></li>
</ul></li>
<li><p><strong>HA</strong> Must be highly available - can restart any component without disrupting service.</p></li>
-<li><p><strong>CLEANWORKSPACE</strong> Must provide a clean workspace for each build or test run - either a clean VM or container.</p></li>
+<li><p><strong>CLEANBUILDENV</strong> Must provide a build environment with only explicitly declared build dependencies installed.</p></li>
+<li><p><strong>CLEANWORKSPACE</strong> Must provide a clean workspace for each build or test run - either a clean VM or container. The workspace should have the source code of the project to be built, with the right commit checked out from version control, and anything else explicitly declared, but nothing else.</p></li>
<li><p><strong>SECRETS</strong> Must support secure storage of credentials / secrets.</p></li>
</ul>
<h2 id="softer-requirements">Softer requirements</h2>
diff --git a/ci-arch.pdf b/ci-arch.pdf
index 09edfad..f3a6de8 100644
--- a/ci-arch.pdf
+++ b/ci-arch.pdf
Binary files differ
diff --git a/pipeline.dot b/pipeline.dot
index 0c56148..dd31fa0 100644
--- a/pipeline.dot
+++ b/pipeline.dot
@@ -2,17 +2,15 @@ digraph pipeline {
developer [shape="circle" label="developer"];
commit [shape="rect" label="Commit stage"];
acceptance [shape="rect" label="Acceptance stage"];
- artifacts [shape="cylinder" label="Artifact store"];
testenv [shape="folder" label="A test environment"];
+ deployprod [shape="rect" label="Deploy to production"];
production [shape="folder" label="Production environment"];
developer -> commit;
- commit -> artifacts;
commit -> acceptance;
- acceptance -> testenv;
- artifacts -> testenv;
+ acceptance -> testenv [style=dotted];
- acceptance -> production;
- artifacts -> production;
+ acceptance -> deployprod;
+ deployprod -> production;
}
diff --git a/pipeline.svg b/pipeline.svg
index 7992246..97a3d75 100644
--- a/pipeline.svg
+++ b/pipeline.svg
@@ -4,89 +4,76 @@
<!-- Generated by graphviz version 2.40.1 (20161225.0304)
-->
<!-- Title: pipeline Pages: 1 -->
-<svg width="298pt" height="315pt"
- viewBox="0.00 0.00 297.50 314.99" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
-<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 310.9919)">
+<svg width="290pt" height="387pt"
+ viewBox="0.00 0.00 289.50 386.99" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
+<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 382.9919)">
<title>pipeline</title>
-<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-310.9919 293.5,-310.9919 293.5,4 -4,4"/>
+<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-382.9919 285.5,-382.9919 285.5,4 -4,4"/>
<!-- developer -->
<g id="node1" class="node">
<title>developer</title>
-<ellipse fill="none" stroke="#000000" cx="138" cy="-261.496" rx="45.4919" ry="45.4919"/>
-<text text-anchor="middle" x="138" y="-257.796" font-family="Times,serif" font-size="14.00" fill="#000000">developer</text>
+<ellipse fill="none" stroke="#000000" cx="134" cy="-333.496" rx="45.4919" ry="45.4919"/>
+<text text-anchor="middle" x="134" y="-329.796" font-family="Times,serif" font-size="14.00" fill="#000000">developer</text>
</g>
<!-- commit -->
<g id="node2" class="node">
<title>commit</title>
-<polygon fill="none" stroke="#000000" points="185,-180 91,-180 91,-144 185,-144 185,-180"/>
-<text text-anchor="middle" x="138" y="-158.3" font-family="Times,serif" font-size="14.00" fill="#000000">Commit stage</text>
+<polygon fill="none" stroke="#000000" points="181,-252 87,-252 87,-216 181,-216 181,-252"/>
+<text text-anchor="middle" x="134" y="-230.3" font-family="Times,serif" font-size="14.00" fill="#000000">Commit stage</text>
</g>
<!-- developer&#45;&gt;commit -->
<g id="edge1" class="edge">
<title>developer&#45;&gt;commit</title>
-<path fill="none" stroke="#000000" d="M138,-215.8248C138,-207.1147 138,-198.2368 138,-190.2688"/>
-<polygon fill="#000000" stroke="#000000" points="141.5001,-190.0684 138,-180.0684 134.5001,-190.0685 141.5001,-190.0684"/>
+<path fill="none" stroke="#000000" d="M134,-287.8248C134,-279.1147 134,-270.2368 134,-262.2688"/>
+<polygon fill="#000000" stroke="#000000" points="137.5001,-262.0684 134,-252.0684 130.5001,-262.0685 137.5001,-262.0684"/>
</g>
<!-- acceptance -->
<g id="node3" class="node">
<title>acceptance</title>
-<polygon fill="none" stroke="#000000" points="134.5,-108 23.5,-108 23.5,-72 134.5,-72 134.5,-108"/>
-<text text-anchor="middle" x="79" y="-86.3" font-family="Times,serif" font-size="14.00" fill="#000000">Acceptance stage</text>
+<polygon fill="none" stroke="#000000" points="189.5,-180 78.5,-180 78.5,-144 189.5,-144 189.5,-180"/>
+<text text-anchor="middle" x="134" y="-158.3" font-family="Times,serif" font-size="14.00" fill="#000000">Acceptance stage</text>
</g>
<!-- commit&#45;&gt;acceptance -->
-<g id="edge3" class="edge">
+<g id="edge2" class="edge">
<title>commit&#45;&gt;acceptance</title>
-<path fill="none" stroke="#000000" d="M123.1118,-143.8314C116.247,-135.454 107.9699,-125.3531 100.4489,-116.1749"/>
-<polygon fill="#000000" stroke="#000000" points="103.1341,-113.9297 94.0886,-108.4133 97.7197,-118.3665 103.1341,-113.9297"/>
+<path fill="none" stroke="#000000" d="M134,-215.8314C134,-208.131 134,-198.9743 134,-190.4166"/>
+<polygon fill="#000000" stroke="#000000" points="137.5001,-190.4132 134,-180.4133 130.5001,-190.4133 137.5001,-190.4132"/>
</g>
-<!-- artifacts -->
+<!-- testenv -->
<g id="node4" class="node">
-<title>artifacts</title>
-<path fill="none" stroke="#000000" d="M241.5,-104.7273C241.5,-106.5331 221.5544,-108 197,-108 172.4456,-108 152.5,-106.5331 152.5,-104.7273 152.5,-104.7273 152.5,-75.2727 152.5,-75.2727 152.5,-73.4669 172.4456,-72 197,-72 221.5544,-72 241.5,-73.4669 241.5,-75.2727 241.5,-75.2727 241.5,-104.7273 241.5,-104.7273"/>
-<path fill="none" stroke="#000000" d="M241.5,-104.7273C241.5,-102.9214 221.5544,-101.4545 197,-101.4545 172.4456,-101.4545 152.5,-102.9214 152.5,-104.7273"/>
-<text text-anchor="middle" x="197" y="-86.3" font-family="Times,serif" font-size="14.00" fill="#000000">Artifact store</text>
+<title>testenv</title>
+<polygon fill="none" stroke="#000000" points="122,-108 119,-112 98,-112 95,-108 0,-108 0,-72 122,-72 122,-108"/>
+<text text-anchor="middle" x="61" y="-86.3" font-family="Times,serif" font-size="14.00" fill="#000000">A test environment</text>
</g>
-<!-- commit&#45;&gt;artifacts -->
-<g id="edge2" class="edge">
-<title>commit&#45;&gt;artifacts</title>
-<path fill="none" stroke="#000000" d="M152.8882,-143.8314C159.753,-135.454 168.0301,-125.3531 175.5511,-116.1749"/>
-<polygon fill="#000000" stroke="#000000" points="178.2803,-118.3665 181.9114,-108.4133 172.8659,-113.9297 178.2803,-118.3665"/>
+<!-- acceptance&#45;&gt;testenv -->
+<g id="edge3" class="edge">
+<title>acceptance&#45;&gt;testenv</title>
+<path fill="none" stroke="#000000" stroke-dasharray="1,5" d="M115.579,-143.8314C106.8471,-135.219 96.2683,-124.7851 86.7577,-115.4048"/>
+<polygon fill="#000000" stroke="#000000" points="88.9925,-112.6931 79.4151,-108.1628 84.077,-117.6769 88.9925,-112.6931"/>
</g>
-<!-- testenv -->
+<!-- deployprod -->
<g id="node5" class="node">
-<title>testenv</title>
-<polygon fill="none" stroke="#000000" points="122,-36 119,-40 98,-40 95,-36 0,-36 0,0 122,0 122,-36"/>
-<text text-anchor="middle" x="61" y="-14.3" font-family="Times,serif" font-size="14.00" fill="#000000">A test environment</text>
+<title>deployprod</title>
+<polygon fill="none" stroke="#000000" points="273.5,-108 140.5,-108 140.5,-72 273.5,-72 273.5,-108"/>
+<text text-anchor="middle" x="207" y="-86.3" font-family="Times,serif" font-size="14.00" fill="#000000">Deploy to production</text>
</g>
-<!-- acceptance&#45;&gt;testenv -->
+<!-- acceptance&#45;&gt;deployprod -->
<g id="edge4" class="edge">
-<title>acceptance&#45;&gt;testenv</title>
-<path fill="none" stroke="#000000" d="M74.4578,-71.8314C72.5116,-64.0463 70.1932,-54.7729 68.0337,-46.1347"/>
-<polygon fill="#000000" stroke="#000000" points="71.4242,-45.2658 65.6033,-36.4133 64.6332,-46.9636 71.4242,-45.2658"/>
+<title>acceptance&#45;&gt;deployprod</title>
+<path fill="none" stroke="#000000" d="M152.421,-143.8314C161.1529,-135.219 171.7317,-124.7851 181.2423,-115.4048"/>
+<polygon fill="#000000" stroke="#000000" points="183.923,-117.6769 188.5849,-108.1628 179.0075,-112.6931 183.923,-117.6769"/>
</g>
<!-- production -->
<g id="node6" class="node">
<title>production</title>
-<polygon fill="none" stroke="#000000" points="289.5,-36 286.5,-40 265.5,-40 262.5,-36 140.5,-36 140.5,0 289.5,0 289.5,-36"/>
-<text text-anchor="middle" x="215" y="-14.3" font-family="Times,serif" font-size="14.00" fill="#000000">Production environment</text>
-</g>
-<!-- acceptance&#45;&gt;production -->
-<g id="edge6" class="edge">
-<title>acceptance&#45;&gt;production</title>
-<path fill="none" stroke="#000000" d="M113.3185,-71.8314C131.0359,-62.4516 152.8345,-50.9112 171.6984,-40.9244"/>
-<polygon fill="#000000" stroke="#000000" points="173.4922,-43.935 180.6924,-36.1628 170.2169,-37.7485 173.4922,-43.935"/>
+<polygon fill="none" stroke="#000000" points="281.5,-36 278.5,-40 257.5,-40 254.5,-36 132.5,-36 132.5,0 281.5,0 281.5,-36"/>
+<text text-anchor="middle" x="207" y="-14.3" font-family="Times,serif" font-size="14.00" fill="#000000">Production environment</text>
</g>
-<!-- artifacts&#45;&gt;testenv -->
+<!-- deployprod&#45;&gt;production -->
<g id="edge5" class="edge">
-<title>artifacts&#45;&gt;testenv</title>
-<path fill="none" stroke="#000000" d="M164.0782,-72.5708C146.0262,-63.0139 123.4323,-51.0524 104.0019,-40.7657"/>
-<polygon fill="#000000" stroke="#000000" points="105.5144,-37.6063 95.0389,-36.0206 102.2392,-43.7928 105.5144,-37.6063"/>
-</g>
-<!-- artifacts&#45;&gt;production -->
-<g id="edge7" class="edge">
-<title>artifacts&#45;&gt;production</title>
-<path fill="none" stroke="#000000" d="M201.5422,-71.8314C203.4884,-64.0463 205.8068,-54.7729 207.9663,-46.1347"/>
-<polygon fill="#000000" stroke="#000000" points="211.3668,-46.9636 210.3967,-36.4133 204.5758,-45.2658 211.3668,-46.9636"/>
+<title>deployprod&#45;&gt;production</title>
+<path fill="none" stroke="#000000" d="M207,-71.8314C207,-64.131 207,-54.9743 207,-46.4166"/>
+<polygon fill="#000000" stroke="#000000" points="210.5001,-46.4132 207,-36.4133 203.5001,-46.4133 210.5001,-46.4132"/>
</g>
</g>
</svg>