From f52ce479fbf21f8b9da5f0449d5a7007b98a9cd7 Mon Sep 17 00:00:00 2001 From: Lars Wirzenius Date: Tue, 7 May 2019 16:08:38 +0300 Subject: Change: un-bulletize the credentials management --- ci-arch.mdwn | 108 +++++++++++++++++++++++++++++++---------------------------- 1 file changed, 57 insertions(+), 51 deletions(-) (limited to 'ci-arch.mdwn') diff --git a/ci-arch.mdwn b/ci-arch.mdwn index 590c108..ee32a82 100644 --- a/ci-arch.mdwn +++ b/ci-arch.mdwn @@ -508,60 +508,66 @@ we plan CI to implement them. ## Credentials management and access control -* Credentials and other secrets are used to allow access to servers, - services, and files. They are often highly security sensitive data. - The CI system needs to protect them, but allow controlled use of - them. - -* Example: a CI job needs to deploy a Docker image with a tested and - reviewed change as a container orchestrated by Kubernetes. For this, - it needs to authenticate itself to the Kubernetes API. This is - typically done by a username/password combination. How will the - future CI system handle this? - -* Example: for tests, and in production, a MediaWiki container needs - access to a MariaDB database, and MW needs to authenticate itself to - the database. MW gets the necessary credentials for this from its - configuration, which CI will install during deployment. The - configuration will be specific for what the container is being used: - if it's for testing a change, the configuration only allows access - to a test database, but for production it provides access to the - production database. - * FIXME: This is unclear as yet, the text below is some incoherent preliminary rambling by Lars which needs review and fixing. -* Builds are done in isolated containers. These containers have no - credentials. Build artifacts are extracted from the containers and - stored in an artifact storage system by the CI system, and this is - done in a controlled environment, where only vetted code is run, not - code from the repository being tested. - -* Deployments happen in controlled environments, with access to the - credentials needed for deployment. The deployment retrieve artifacts - from the artifact storage system. The deployments are to containers, - and the deployed continers don't have any credentials, unless CI has - been configured to install them, in which case CI installs the - credentials for the intended use of the container. - -* Tests run against software deployed to containers, and those - containers only have access to the backing services needed for the - test. - -* The CI system needs a way to store the credentials that can only be - accessed by CI itself, when it's deploying a container (Kubernetes - API access) or configuring the container (installing credentials for - the intended use of container). - - This might be, for example, a set of files deployed to the CI host - where container deployment or configuration runs, with access - control provided by Unix permissions. Not sure if this is - sufficiently secure. - -## Interdependent changes to multiple components - -FIXME All or none of the changes need to merged and deployed. Lars -would prefer to avoid having such changes. +Credentials and other secrets are needed to allow access to servers, +services, and files. They are highly security sensitive data. The CI +system needs to protect them, but allow controlled use of them. + +Example: a CI job needs to deploy a Docker image with a tested and +reviewed change as a container orchestrated by production Kubernetes. +For this, it needs to authenticate itself to the Kubernetes API. This +is typically done by a username/password combination, but might be an +API token of some kind (though it doesn't really matter; it's all just +secret bits at some level). How will the future CI system handle this? + +Example: for tests, and in production, a MediaWiki container needs +access to a MariaDB database, and MW needs to authenticate itself to +the database. MW gets the necessary credentials for this from its +configuration, which CI will install during deployment. The +configuration will be specific for what the container is being used: +if it's for testing a change, the configuration only allows access to +a test database, but for production it provides access to the +production database. + +Builds are done in isolated containers. These containers have no +credentials. Build artifacts are extracted from the containers and +stored in an artifact storage system by the CI system, and this +extraction is done in a controlled environment, where only vetted code +is run, not code from the repository being tested. The build +environment can't push artifacts directly to the artifact store. + +Deployments happen in controlled environments, with access to the +credentials needed for deployment. The deployment retrieves artifacts +from the artifact storage system. The deployments are to containers, +and the deployed containers don't have any credentials, unless CI has +been configured to install them, in which case CI installs the +credentials for the intended use of the container. + +Note that credentials should not come directly from the source code of +the deployed program. CI deploys configuration when it deploys the +software. This way, the same software (build artifacts) can be +deployed to different environment. (This may be complicated by the way +MediaWiki is configured, using a PHP file in the source tree. This +will need discussion.) + +Tests run against software deployed to containers, and those +containers only have access to the backing services needed for the +test, and may even be firewalled to not have access to any other +network locations. + +Suggestion: Deployments will be done dedicated deployment +environments, which run a "pingee" service. When a pipeline executes +a deployment stage, deploying to any environment, the stage runs in a +suitable container, but doesn't actually do the deployment itself. +Instead, it "pings" a deployment service, with information of who is +deploying, what, and where, and the deployment service inspects the +change, and if it looks acceptable, does the actual deployment to the +desired environment. The deployment service has access to the +credentials it needs for accessing the artifacts and doing the +deployment. There may be several deployment services, for deploying to +environments with different security needs. # Architecture: The WMF development ecosystem -- cgit v1.2.1