summaryrefslogtreecommitdiff
path: root/docker
AgeCommit message (Collapse)AuthorFilesLines
2018-12-10Use JSON as canonical config formatTyler Cipriani1-4/+4
Uses the github.com/ghodss/yaml library to convert YAML to JSON before unmarshaling for the purposes of supporting YAML and JSON input while converting to only support JSON internally. Bug: T207694 Change-Id: I00668014907e9ea54917f5d5067cac08d0668053
2018-10-01Remove support for `sharedvolume` configurationDan Duvall2-14/+0
Support for mounting a shared volume instead of copying application files was meant to provide an option for development use cases. This functionality has never been thoroughly tested or utilized for any use case. It should be removed for now. Relates tangentially to task T204591 that simplifies node support. Change-Id: Ib70cb7bceb504841897a38d732880ba376fe67c8
2018-08-14Bump Blubber config version to v3Dan Duvall1-4/+4
Change-Id: I91cac046976cf88d513487e27e1b52bf77a6f974
2018-07-31Change import paths from Phabricator to GerritDan Duvall4-9/+9
All import paths and other references to the previously Blubber repo in Phabricator have been changed to point to the new Gerrit project URL. Note that building or testing against this change will require you to move your working directory to `$GOPATH/src/gerrit.wikimedia.org/r/blubber`. Contribution documentation was updated to reflect the changes, and the `.arcvendor` submodule has been removed. Bug: T200452 Change-Id: I3ab23e420b2825e86e2bd7c9b3da9d4de23edaa1
2018-07-26Add slash to directory path if copying > 1 fileTyler Cipriani1-5/+5
Summary: Ran into a Docker error when attempting to build a node project with more than one requirements files: When using COPY with more than one source file, the destination must be a directory and end with a / Reviewers: dduvall, #release-engineering-team Reviewed By: dduvall, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1080
2018-05-20Bump config version to v2Dan Duvall1-4/+4
2018-04-05Introduce strict/versioned config parsingDan Duvall1-0/+4
Summary: Introduced a `version` config field that must be specified and match `config.CurrentVersion`. Changed `config.ReadConfig` to use `yaml.UnmarshalStrict` to ensure that errors are surfaced when unknown/bad fields are present in the given YAML config. A smaller `config.VersionConfig` is now unmarshaled first to prevalidate the new `version` field before the entire config is parsed. Fixes T191460 Test Plan: Run `go test ./...`. Run `blubber` against some configuration containing invalid fields and ensure that it surfaces a YAML error. Reviewers: thcipriani, demon, hashar, mmodell, mobrovac, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Maniphest Tasks: T191460 Differential Revision: https://phabricator.wikimedia.org/D1021
2018-03-22Fix ownership on artifact copiesDan Duvall2-6/+23
Summary: The implementation of D984 did not include enforcing ownership for `build.CopyFrom` instruction and so artifacts copied from one image to another via `copies:` were problematically owned as root. In order to fix this behavior: 1. `config.ArtifactConfig` `build.CopyFrom` instructions are now injected duration `build.PhaseInstall` 2. `config.VariantConfig` calls `build.ApplyUser` for these artifact instructions as well using the `runs.as` user 3. `build.CopyAs` was refactored to wrap any `build.Instruction` which should only really be used with `build.Copy` or `build.CopyFrom`. Test Plan: Run `go test ./...`. Run `blubber` against configuration with a variant that uses `copies` and verify that the `COPY --from` instructions also include a `--chown` flag. Reviewers: thcipriani, mmodell, hashar, #release-engineering-team, demon Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1002
2018-03-05Generalize instructions for entrypoint and working directoryDan Duvall3-17/+29
Summary: Introduce new `build.EntryPoint` and `build.WorkingDirectory` instructions to allow configuration to inject them instead of hard coding their generation in the Docker compiler. Simplified the Docker compiler to simply iterate over build phases as returned by a new function `build.Phases()`. Depends on D990 Test Plan: Run `go test ./...`. Reviewers: thcipriani, demon, hashar, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D991
2018-03-05Simplify Docker instruction compilationDan Duvall2-178/+102
Summary: After adding a number of new instructions, it seemed there was a lot of redundant implementation between concrete types. This refactor uses just one concrete unexported type and a few options for determining the output format instead of one struct type per possible Docker instruction. The exported `docker.Instruction` interface and signature of `docker.NewInstruction` is unchanged with this refactor. Unit tests for Docker instruction compilation were simplified to not make type assertions but were otherwise left to ensure no regression. Depends on D984 Test Plan: Run `go test ./...`. Reviewers: thcipriani, demon, hashar, mmodell, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D990
2018-03-05Fix application files/runtime permissions schemeDan Duvall3-8/+60
Summary: Introduces new `lives` configuration that provides the name/UID/GID of the user that will own application files and installed dependencies. This new configuration is distinct from `runs` in that the former determines application file location ownership and the latter now only determines runtime process ownership. Default configuration has also been introduced for both config sections. In addition to the new configuration, a new `build.CopyAs` instruction has been introduced that ensures correct UID/GID ownership of files copied into the container image, and all unqualified `build.Copy` instructions are wrapped by the new `build.CopyAs` instruction using the UID/GID appropriate for the current build phase. A new `build.User` instruction is also introduced and injected into the build at the start of certain phases to enforce ownership of `build.Run` processes. This effective process/file ownership model is: PhasePrivileged - "root" PhasePrivilegedDropped - lives.as PhasePreInstall - lives.as PhaseInstall - lives.as PhasePostInstall - runs.as Fixes T187372 Test Plan: Run `go test ./...`. Reviewers: thcipriani, hashar, demon, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Subscribers: mmodell Tags: #release-engineering-team Maniphest Tasks: T187372 Differential Revision: https://phabricator.wikimedia.org/D984
2017-11-06Conform to all linter warnings/adviceDan Duvall3-49/+49
Summary: Fixed all linter warnings and advice except for vet's rule about unkeyed composite literals which was disabled via a `-composites=false` flag in `.arclint`. Most unkeyed literals (e.g. `build.Run{"command"}`) in this project just seem too usefully succinct compared to their more verbose keyed counterparts. Depends on D841 Test Plan: Run `arc lint --everything` and verify there are no warnings or advice. Reviewers: thcipriani, hashar, #release-engineering-team Reviewed By: thcipriani, hashar, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D845
2017-11-06Documented all exported types, functions, and interfacesDan Duvall2-3/+40
Summary: Wrote inline documentation for all the things. Fixes T168000 Test Plan: Execute `godoc -http=:6060` and proofread all the things. Reviewers: thcipriani, hashar, #release-engineering-team, demon Reviewed By: thcipriani, #release-engineering-team, demon Tags: #release-engineering-team Maniphest Tasks: T168000 Differential Revision: https://phabricator.wikimedia.org/D841
2017-10-19Include meta data as labels in Dockerfile outputDan Duvall4-19/+75
Summary: Certain meta data including the Blubber version and variant used at invocation may be useful in downstream tracking. Implemented abstract and Docker-specific instructions for labels and modified the Docker compiler to include these instructions at the end of the output for the final stage. Depends on D816 Fixes T178022 Test Plan: Run unit tests. Run `make && bin/blubber blubber.example.yaml production` and verify that both `blubber.version` and `blubber.variant` labels are present and accurate. Reviewers: thcipriani, hashar, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Maniphest Tasks: T178022 Differential Revision: https://phabricator.wikimedia.org/D818
2017-10-10Move error handling to mainTyler Cipriani2-10/+14
Summary: Rather than calling `log.Fatal` in `Compile` bubble the error up to `main` and handle it there. Test Plan: go test ./... Reviewers: dduvall, #release-engineering-team Reviewed By: dduvall, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D784
2017-09-14Switch to github import path for testifyDan Duvall2-2/+2
Summary: Debian only provides a package for the GitHub version so it's either this or we'd have to maintain a patch to change the import paths at packaging, a moving target for sure. Since it's only a development dependency, we're not too worried about versioning it and this is a much more simple solution. Refs T175609 Test Plan: Run `go get ./... && go test ./...`. Reviewers: thcipriani, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Maniphest Tasks: T175609 Differential Revision: https://phabricator.wikimedia.org/D782
2017-09-14Use import paths without `.git` suffixDan Duvall4-7/+8
Summary: Now that our Phabricator instance handles `go get` requests, we can use a path without the `.git` suffix as the canonical import path. Test Plan: Run `go test ./...` and `go build`. Reviewers: thcipriani, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D780
2017-09-11Recursive variant expansionTyler Cipriani1-12/+15
Summary: Problem running blubber cfg.yml test with this config: ``` base: nodejs-slim variants: build: base: nodejs-devel development: includes: [build] entrypoint: [npm, start] test: includes: [development] entrypoint: [npm, test] ``` yields a Dockerfile with `FROM nodejs-slim`; however, I expected that the base would be `nodejs-devel` since `test` inherits from `development` which inherits from `build`. In order for this to work as expected we have to recursively expand variants as in this patch. Reviewers: dduvall, #release-engineering-team Reviewed By: dduvall, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D773
2017-09-07Smarter copies/sharedvolume/default behaviorDan Duvall3-5/+25
Summary: Defined new abstract `build.Volume` and corresponding `docker.DockerVolume` instructions. Refactored compilation of main `COPY` or `VOLUME` instruction for application files to use the new instructions and moved injection of these instructions out of the compiler and into `VariantConfig`. The latter can be smarter about the following cases: 1. When `copies` is set, simply depend on artifacts for the application files and do not copy anything from the build host. 2. When `sharedvolume` is `true`, inject a `build.Volume` instruction for the application working directory. 3. When neither of the above are set, copy application files from the host. Fixes T174623 Depends on D768 Test Plan: Run `go test ./...`. Run `blubber blubber.example.yaml production` and ensure: 1. The `prep` stage has a `COPY . .` instruction. 2. The final stage has no `COPY . .` instruction, only `COPY --from=prep` instructions. Reviewers: thcipriani, mobrovac, hashar, mmodell, #release-engineering-team Reviewed By: thcipriani, mobrovac, #release-engineering-team Tags: #release-engineering-team Maniphest Tasks: T174623 Differential Revision: https://phabricator.wikimedia.org/D769
2017-09-07Support `copies` config entry for multi-stage buildsDan Duvall3-29/+36
Summary: Support a `copies` variant config entry that will result in a multi-stage build, copying both shared library files and application directory from a previously defined variant. This is essentially a shorthand for two `artifacts` entries that are likely to be idiomatic to multi-stage build/prod configurations. Defined a new abstract `build.CopyFrom` instruction and corresponding `docker.DockerCopyFrom` instruction and refactored the writing of these Dockerfile lines to be accomplished using an `InstructionsForPhase` method on `config.ArtifactsConfig`. Implemented new support for `copies` configuration in `config.VariantConfig` and an `InstructionsForPhase` method that returns `build.CopyFrom` instructions for both the shared library and application directories. Fixes T174622 Depends on D759 Test Plan: Run `go test ./...`. Run `blubber blubber.example.yaml production` and ensure the right `COPY --from` lines are included for the final stage. Reviewers: thcipriani, mobrovac, hashar, mmodell, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Maniphest Tasks: T174622 Differential Revision: https://phabricator.wikimedia.org/D768
2017-09-05Compile only unique Docker stages determined from artifactsDan Duvall2-7/+45
Summary: The previous implementation for iterating over defined artifacts did not take into account the possibility for multiple artifacts with the same `from` variant name and would compile stages with duplicate names. This fixes that behavior by iterating over a unique set of names. Test Plan: Run `go test ./...` or `arc unit`. Reviewers: thcipriani, mmodell, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D758
2017-08-30Use real types for build instructionsDan Duvall2-21/+57
Summary: Refactored build instructions to use concrete types and `build.Instruction` as an interface instead of relying on a simple enum and arbitrary string arguments. The formal types result in: 1. Clearer internal data structures 2. Partial compilation and proper argument quoting for all instructions moved into the common `build` package 3. Higher order instructions like `build.RunAll` that easily reduce to compiler specific output Test Plan: Run `arc unit` or `go test ./...` Reviewers: thcipriani, mmodell, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D741
2017-07-12Broaden base test coverageDan Duvall1-20/+0
Summary: Implemented tests for config types and removed deprecated docker compiler tests. Fixes T168001 Test Plan: Run `arc unit --everything` or `go test ./...`. Reviewers: thcipriani, mmodell, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Maniphest Tasks: T168001 Differential Revision: https://phabricator.wikimedia.org/D711
2017-07-11Escape docker outputTyler Cipriani3-12/+115
Summary: This adds a new DockerInstruction interface that can be implmented by different docker instructions. DockerInstruction implements a compile method that returns escaped output for use in a dockerfile. Fixes T167999 Reviewers: dduvall, mmodell, #release-engineering-team Reviewed By: dduvall, #release-engineering-team Tags: #release-engineering-team Maniphest Tasks: T167999 Differential Revision: https://phabricator.wikimedia.org/D705
2017-06-29Run go fmtTyler Cipriani2-1/+3
Summary: One of Golang's "advantages" is not quibbling over style. To this end the `go fmt` command exists. This is the result of me running: find . -name '*.go' -exec go fmt {} \; Test Plan: built package, ran tests Reviewers: dduvall, mmodell, #release-engineering-team Reviewed By: dduvall, mmodell, #release-engineering-team Subscribers: hashar Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D694
2017-06-26Support environment variablesDan Duvall2-15/+13
Summary: Added support for definition of environment variables under `runs.environment`. Corresponding `ENV` instructions will be added to Dockerfile output for the unprivileged build phase. Fixes T168425 Test Plan: Run `go test ./...`. Reviewers: thcipriani, mobrovac, hashar, Jrbranaa, mmodell, #release-engineering-team Reviewed By: mobrovac Tags: #release-engineering-team Maniphest Tasks: T168425 Differential Revision: https://phabricator.wikimedia.org/D691
2017-06-20Set HOME environment variable for runs-as userDan Duvall2-0/+11
Summary: Fixes build issues around home permissions by setting `HOME` to the unprivileged user's home directory once the "privileges dropped" build phase has been reached. Test Plan: Run `go test ./...`. Reviewers: thcipriani, mobrovac, hashar, Jrbranaa, mmodell, #release-engineering-team Reviewed By: mobrovac Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D686
2017-06-20Use correct COPY syntax for quoted pathsDan Duvall2-3/+14
Summary: Fixed compilation of copy build instructions to Dockerfile syntax. Test Plan: Run `go test ./...`. Reviewers: thcipriani, mobrovac, hashar, Jrbranaa, mmodell, #release-engineering-team Reviewed By: mobrovac Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D685
2017-06-15Support single-stage builds for older Docker versionsDan Duvall2-2/+56
Summary: Refactored the Docker compiler to omit the stage name (`AS <stage>`) from the `FROM` instruction when no dependent artifacts are declared. This allows for use with older versions of Docker that don't support multi-stage builds. Test Plan: Run `go test` under the `docker` directory. Reviewers: thcipriani, mobrovac, hashar, Jrbranaa, mmodell, #release-engineering-team Reviewed By: mobrovac Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D684
2017-06-12Fix variant expansion for bool config fieldsDan Duvall1-1/+1
Summary: Refactored bool config fields to use a new `config.Flag` type that keeps track of whether it was set by unmarshalled data or merged from another instance, fixing the behavior of these fields when overwritten by variants. Fixes T166353 Test Plan: Run config unit tests (`cd config; go test`) Reviewers: thcipriani, Joe, hashar, mobrovac, mmodell, #release-engineering-team Reviewed By: mobrovac Tags: #release-engineering-team Maniphest Tasks: T166353 Differential Revision: https://phabricator.wikimedia.org/D680
2017-05-23Use Phabricator import URIsDan Duvall1-2/+2
Summary: Now that the project is hosted on Phabricator the import URIs must be changed. Reviewers: thcipriani, hashar, Jrbranaa, mmodell, #release-engineering-team, demon Reviewed By: #release-engineering-team, demon Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D657
2017-05-10Decouple Docker compiler from apt/npm providersDan Duvall1-8/+21
Establish phases within Docker compiler to allow providers (apt, npm, etc.) to inject their own run/copy instructions into to the Dockerfile compilation process while leaving the compiler agnostic to the providers themselves. The instructions and phases are also generalized to leave room for alternative compilers should they be needed in the future (e.g. aci support via acbuild) but also as a general design constraint to leave compiler implementation concerns out of providers.
2017-05-01Rename `run` config to `runs`Dan Duvall1-7/+7
It sounds more declarative.
2017-04-25Invert `copiestree` feature as `sharedvolume`Dan Duvall1-2/+4
2017-04-19Include entrypointDan Duvall1-0/+5
2017-04-19Allow configuration of whether to copy in source treeDan Duvall1-0/+4
2017-04-19Support compilation to multi-stage DockerfileDan Duvall1-3/+66
Reorganized config type declarations. Added `RunConfig` for declaring working directory and unprivileged runtime user. A system for enforcing restrictions should probably be implemented for this. Implemented Dockerfile compiler, supporting multi-stage output in cases whether artifacts from other variants are configured.
2017-04-18Refactored config structure to use map for variantsDan Duvall1-9/+0
2017-04-18Initial commitDan Duvall1-0/+22
generated by cgit v1.2.1 (git 2.18.0) at 2024-05-04 07:41:14 +0000