| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Summary:
Introduce new `build.EntryPoint` and `build.WorkingDirectory`
instructions to allow configuration to inject them instead of hard
coding their generation in the Docker compiler.
Simplified the Docker compiler to simply iterate over build phases as
returned by a new function `build.Phases()`.
Depends on D990
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, demon, hashar, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D991
|
|
Summary:
After adding a number of new instructions, it seemed there was a lot of
redundant implementation between concrete types. This refactor uses just
one concrete unexported type and a few options for determining the
output format instead of one struct type per possible Docker
instruction.
The exported `docker.Instruction` interface and signature of
`docker.NewInstruction` is unchanged with this refactor. Unit tests for
Docker instruction compilation were simplified to not make type
assertions but were otherwise left to ensure no regression.
Depends on D984
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, demon, hashar, mmodell, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D990
|
|
Summary:
Introduces new `lives` configuration that provides the name/UID/GID of
the user that will own application files and installed dependencies.
This new configuration is distinct from `runs` in that the former
determines application file location ownership and the latter now only
determines runtime process ownership. Default configuration has also
been introduced for both config sections.
In addition to the new configuration, a new `build.CopyAs` instruction
has been introduced that ensures correct UID/GID ownership of files
copied into the container image, and all unqualified `build.Copy`
instructions are wrapped by the new `build.CopyAs` instruction using the
UID/GID appropriate for the current build phase. A new `build.User`
instruction is also introduced and injected into the build at the start
of certain phases to enforce ownership of `build.Run` processes.
This effective process/file ownership model is:
PhasePrivileged - "root"
PhasePrivilegedDropped - lives.as
PhasePreInstall - lives.as
PhaseInstall - lives.as
PhasePostInstall - runs.as
Fixes T187372
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, hashar, demon, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Subscribers: mmodell
Tags: #release-engineering-team
Maniphest Tasks: T187372
Differential Revision: https://phabricator.wikimedia.org/D984
|
|
Summary:
Refactored variant include resolution to correctly order the hierarchy
of includes while still protecting against infinite recursion. The
function was renamed, refactored for clarity, and exported so as to be
better tested and documented with examples.
Test Plan:
Run `go test ./...`.
Check the rendered `config.ResolveIncludes` `godoc` for sanity.
Reviewers: thcipriani, mmodell, hashar, Jrbranaa, zeljkofilipin, demon, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D959
|
|
Summary:
Our current lack of dependency management is leading to issues with
Debian packaging and risks inconsistent builds. Let's use `dep`, the
"official experiment" for go dependency management, and commit the
vendor directory which is small enough following a `dep prune`.
Fixes T180530
Depends on D881
Test Plan:
Run `make` or `go build -v` and examine output to ensure use of `vendor/`
packages.
Reviewers: thcipriani, Joe, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T180530
Differential Revision: https://phabricator.wikimedia.org/D882
|
|
Summary:
In preparation for vendored golang dependencies, we need to move
`arcanist-go` out of the way. A new `.arcvendor` directory seems like a
decent-ish place.
Test Plan: Run `arc unit --everything && arc lint --everything`.
Reviewers: thcipriani, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D881
|
|
|
|
Summary:
Implemented a validation system using the
`github.com/go-playground/validator` package, extending it with custom
validation tags, and implemented translation of validation errors into
somewhat human-friendly messages.
Fixes T175186
Depends on D845
Test Plan: Run the unit tests and try running blubber against some bad config.
Reviewers: thcipriani, hashar, Jrbranaa, Joe, #release-engineering-team, mobrovac
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T175186
Differential Revision: https://phabricator.wikimedia.org/D868
|
|
Summary:
Fixed all linter warnings and advice except for vet's rule about unkeyed
composite literals which was disabled via a `-composites=false` flag in
`.arclint`. Most unkeyed literals (e.g. `build.Run{"command"}`) in this
project just seem too usefully succinct compared to their more verbose
keyed counterparts.
Depends on D841
Test Plan: Run `arc lint --everything` and verify there are no warnings or advice.
Reviewers: thcipriani, hashar, #release-engineering-team
Reviewed By: thcipriani, hashar, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D845
|
|
Summary:
Wrote inline documentation for all the things.
Fixes T168000
Test Plan: Execute `godoc -http=:6060` and proofread all the things.
Reviewers: thcipriani, hashar, #release-engineering-team, demon
Reviewed By: thcipriani, #release-engineering-team, demon
Tags: #release-engineering-team
Maniphest Tasks: T168000
Differential Revision: https://phabricator.wikimedia.org/D841
|
|
|
|
|
|
Summary:
Certain meta data including the Blubber version and variant used at
invocation may be useful in downstream tracking.
Implemented abstract and Docker-specific instructions for labels and
modified the Docker compiler to include these instructions at the end of
the output for the final stage.
Depends on D816
Fixes T178022
Test Plan:
Run unit tests. Run `make && bin/blubber blubber.example.yaml production` and
verify that both `blubber.version` and `blubber.variant` labels are present
and accurate.
Reviewers: thcipriani, hashar, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T178022
Differential Revision: https://phabricator.wikimedia.org/D818
|
|
Summary:
The `go build` tool can accept linker options that dynamically set
variable values at build time. Let's make use of that in our `Makefile`
and `debian/rules` to know and expose meta data such as version and Git
commit at runtime.
Test Plan:
Run `make bin/blubber && bin/blubber --version` and verify that it outputs
"0.0.1-[git head commit]". Build and install the debian package and verify
the same using the installed binary.
Reviewers: thcipriani, hashar, Joe, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D816
|
|
Summary:
Rather than calling `log.Fatal` in `Compile` bubble the error up to
`main` and handle it there.
Test Plan: go test ./...
Reviewers: dduvall, #release-engineering-team
Reviewed By: dduvall, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D784
|
|
Summary:
Initial files generated by `dh-make-golang` have been modified.
Fixes T175609
Depends on D780, D782
Test Plan: Build the package with `gbp buildpackage -us -uc`.
Reviewers: thcipriani, Joe, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T175609
Differential Revision: https://phabricator.wikimedia.org/D778
|
|
Summary:
Debian only provides a package for the GitHub version so it's either
this or we'd have to maintain a patch to change the import paths at
packaging, a moving target for sure.
Since it's only a development dependency, we're not too worried about
versioning it and this is a much more simple solution.
Refs T175609
Test Plan: Run `go get ./... && go test ./...`.
Reviewers: thcipriani, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T175609
Differential Revision: https://phabricator.wikimedia.org/D782
|
|
Summary:
Now that our Phabricator instance handles `go get` requests, we can use
a path without the `.git` suffix as the canonical import path.
Test Plan: Run `go test ./...` and `go build`.
Reviewers: thcipriani, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D780
|
|
Summary:
Problem running blubber cfg.yml test with this config:
```
base: nodejs-slim
variants:
build:
base: nodejs-devel
development:
includes: [build]
entrypoint: [npm, start]
test:
includes: [development]
entrypoint: [npm, test]
```
yields a Dockerfile with `FROM nodejs-slim`; however, I expected that
the base would be `nodejs-devel` since `test` inherits from
`development` which inherits from `build`.
In order for this to work as expected we have to recursively expand
variants as in this patch.
Reviewers: dduvall, #release-engineering-team
Reviewed By: dduvall, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D773
|
|
Summary:
For `npm run` to find executables provided by external dependencies, the
`$NODE_PATH/.bin` directory must be present in `PATH`.
Depends on D769
Test Plan:
Run `go test ./...`.
Build and run Mathoid's test variant at:
https://gerrit.wikimedia.org/r/#/c/376319/2
Reviewers: thcipriani, mobrovac, hashar, mmodell, #release-engineering-team
Reviewed By: thcipriani, mobrovac, mmodell, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D770
|
|
Summary:
Defined new abstract `build.Volume` and corresponding
`docker.DockerVolume` instructions.
Refactored compilation of main `COPY` or `VOLUME` instruction for
application files to use the new instructions and moved injection of
these instructions out of the compiler and into `VariantConfig`. The
latter can be smarter about the following cases:
1. When `copies` is set, simply depend on artifacts for the application
files and do not copy anything from the build host.
2. When `sharedvolume` is `true`, inject a `build.Volume` instruction
for the application working directory.
3. When neither of the above are set, copy application files from the
host.
Fixes T174623
Depends on D768
Test Plan:
Run `go test ./...`. Run `blubber blubber.example.yaml production` and ensure:
1. The `prep` stage has a `COPY . .` instruction.
2. The final stage has no `COPY . .` instruction, only `COPY --from=prep`
instructions.
Reviewers: thcipriani, mobrovac, hashar, mmodell, #release-engineering-team
Reviewed By: thcipriani, mobrovac, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T174623
Differential Revision: https://phabricator.wikimedia.org/D769
|
|
Summary:
Support a `copies` variant config entry that will result in a
multi-stage build, copying both shared library files and application
directory from a previously defined variant. This is essentially a
shorthand for two `artifacts` entries that are likely to be idiomatic to
multi-stage build/prod configurations.
Defined a new abstract `build.CopyFrom` instruction and corresponding
`docker.DockerCopyFrom` instruction and refactored the writing of these
Dockerfile lines to be accomplished using an `InstructionsForPhase`
method on `config.ArtifactsConfig`.
Implemented new support for `copies` configuration in
`config.VariantConfig` and an `InstructionsForPhase` method that returns
`build.CopyFrom` instructions for both the shared library and
application directories.
Fixes T174622
Depends on D759
Test Plan:
Run `go test ./...`. Run `blubber blubber.example.yaml production` and ensure
the right `COPY --from` lines are included for the final stage.
Reviewers: thcipriani, mobrovac, hashar, mmodell, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T174622
Differential Revision: https://phabricator.wikimedia.org/D768
|
|
Summary:
Define `NODE_ENV` based on the configuration and define `NODE_PATH` when
either `npm.install` or `npm.env` are true. The latter behavior is
necessary for final stages of multi-stage builds where npm modules are
not installed but the application requires node-related environment
variables to run.
Depends on D757
Ref T174626
Rename npm module/config to node
Renamed `npm` yaml entry to `node` and `NpmConfig` to `NodeConfig`.
Since we're dealing with general node builds and configuration, this
seems to make more sense.
Test Plan:
Run `go test ./...` or `arc unit`. Run blubber against at least the
`blubber.example.yaml`.
Reviewers: thcipriani, mobrovac, mmodell, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T174626
Differential Revision: https://phabricator.wikimedia.org/D759
|
|
Summary:
The previous implementation for iterating over defined artifacts did not
take into account the possibility for multiple artifacts with the same
`from` variant name and would compile stages with duplicate names. This
fixes that behavior by iterating over a unique set of names.
Test Plan: Run `go test ./...` or `arc unit`.
Reviewers: thcipriani, mmodell, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D758
|
|
Summary:
Install node application dependencies into the common unprivileged
directory defined and managed by `RunsConfig`, and define `NODE_PATH`
such that node applications will search the directory for modules.
This fixes the optimization of image layers by using the original
location for installed modules instead of performing a subsequent move
of the modules back into the project directory. The latter may or may
not result in an additional fs layer depending on the backing storage,
making the previous install/move method unreliable.
Fixes T171632
Depends on D741, D756
Test Plan: Run `go test ./...` or `arc unit`. Build a node application image.
Reviewers: thcipriani, mobrovac, mmodell, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T171632
Differential Revision: https://phabricator.wikimedia.org/D757
|
|
Summary:
Establish `/opt/lib` as the location for installing application
dependencies that are installed via unprivileged execution and from
untrusted sources. The directory is created during the privileged build
phase and owned by the unprivileged runtime user.
Depends on D741
Test Plan: Run `go test ./...` or `arc unit`.
Reviewers: thcipriani, mobrovac, mmodell, #release-engineering-team
Reviewed By: thcipriani, mobrovac, mmodell, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D756
|
|
Summary:
Refactored build instructions to use concrete types and
`build.Instruction` as an interface instead of relying on a simple enum
and arbitrary string arguments. The formal types result in:
1. Clearer internal data structures
2. Partial compilation and proper argument quoting for all instructions
moved into the common `build` package
3. Higher order instructions like `build.RunAll` that easily reduce to
compiler specific output
Test Plan: Run `arc unit` or `go test ./...`
Reviewers: thcipriani, mmodell, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D741
|
|
Summary:
Refactored build instructions in `RunsConfig` to properly quote command
arguments injected from user data.
Established unit tests for `RunsConfig` instruction phases.
Refs T170285. Depends on D711
Omit ENV instruction when `runs.environment` is empty
Fixes T170285
Test Plan:
Run `arc unit`. Create a config without `runs.environment` defined and verify
that no bare `ENV` line ends up in the `Dockerfile` output.
Reviewers: thcipriani, hashar, mmodell, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T170285
Differential Revision: https://phabricator.wikimedia.org/D715
|
|
Summary:
Implemented tests for config types and removed deprecated docker
compiler tests.
Fixes T168001
Test Plan: Run `arc unit --everything` or `go test ./...`.
Reviewers: thcipriani, mmodell, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T168001
Differential Revision: https://phabricator.wikimedia.org/D711
|
|
Summary:
Import `arcanist-go` into `vendor` to provide a unit engine and
`gofmt`/`govet` linters.
Lint with `golint`, `gofmt`, and `govet`.
Test Plan:
Run `arc lint --everything` and revel at the given advice.
Run `arc unit --everything` and hopefully nothing will explode.
Reviewers: thcipriani, mmodell, hashar, Jrbranaa, #release-engineering-team
Reviewed By: mmodell, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D704
|
|
Summary:
This adds a new DockerInstruction interface that can be implmented by
different docker instructions. DockerInstruction implements a compile
method that returns escaped output for use in a dockerfile.
Fixes T167999
Reviewers: dduvall, mmodell, #release-engineering-team
Reviewed By: dduvall, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T167999
Differential Revision: https://phabricator.wikimedia.org/D705
|
|
Summary:
One of Golang's "advantages" is not quibbling over style. To this end
the `go fmt` command exists. This is the result of me running:
find . -name '*.go' -exec go fmt {} \;
Test Plan: built package, ran tests
Reviewers: dduvall, mmodell, #release-engineering-team
Reviewed By: dduvall, mmodell, #release-engineering-team
Subscribers: hashar
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D694
|
|
Summary:
Added support for definition of environment variables under
`runs.environment`. Corresponding `ENV` instructions will be added to
Dockerfile output for the unprivileged build phase.
Fixes T168425
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, mobrovac, hashar, Jrbranaa, mmodell, #release-engineering-team
Reviewed By: mobrovac
Tags: #release-engineering-team
Maniphest Tasks: T168425
Differential Revision: https://phabricator.wikimedia.org/D691
|
|
Summary:
Fixes build issues around home permissions by setting `HOME` to the
unprivileged user's home directory once the "privileges dropped" build
phase has been reached.
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, mobrovac, hashar, Jrbranaa, mmodell, #release-engineering-team
Reviewed By: mobrovac
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D686
|
|
Summary: Fixed compilation of copy build instructions to Dockerfile syntax.
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, mobrovac, hashar, Jrbranaa, mmodell, #release-engineering-team
Reviewed By: mobrovac
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D685
|
|
Summary:
Refactored the Docker compiler to omit the stage name (`AS <stage>`)
from the `FROM` instruction when no dependent artifacts are declared.
This allows for use with older versions of Docker that don't support
multi-stage builds.
Test Plan: Run `go test` under the `docker` directory.
Reviewers: thcipriani, mobrovac, hashar, Jrbranaa, mmodell, #release-engineering-team
Reviewed By: mobrovac
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D684
|
|
Summary:
Refactored bool config fields to use a new `config.Flag` type that keeps
track of whether it was set by unmarshalled data or merged from another
instance, fixing the behavior of these fields when overwritten by
variants.
Fixes T166353
Test Plan: Run config unit tests (`cd config; go test`)
Reviewers: thcipriani, Joe, hashar, mobrovac, mmodell, #release-engineering-team
Reviewed By: mobrovac
Tags: #release-engineering-team
Maniphest Tasks: T166353
Differential Revision: https://phabricator.wikimedia.org/D680
|
|
Summary:
Added a Makefile so you don't need any special mangling of your go path
Test Plan:
None
Reviewers: dduvall, mmodell, #release-engineering-team
Reviewed By: mmodell
Subscribers: Joe
Tags: #release-engineering-team
Revert Plan:
None
Differential Revision: https://phabricator.wikimedia.org/D671
|
|
Summary:
Now that the project is hosted on Phabricator the import URIs must be
changed.
Reviewers: thcipriani, hashar, Jrbranaa, mmodell, #release-engineering-team, demon
Reviewed By: #release-engineering-team, demon
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D657
|
|
Establish phases within Docker compiler to allow providers (apt, npm,
etc.) to inject their own run/copy instructions into to the Dockerfile
compilation process while leaving the compiler agnostic to the providers
themselves.
The instructions and phases are also generalized to leave room for
alternative compilers should they be needed in the future (e.g. aci
support via acbuild) but also as a general design constraint to leave
compiler implementation concerns out of providers.
|
|
It sounds more declarative.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
