summaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorFilesLines
2019-01-09Tweaked logo to play nice with preview renderer on commonsHEADmasterDan Duvall1-51/+53
The paint order for the eye's outer stroke border is not respected by the SVG preview renderer on commons. I re-implemented the eye's border with two independent shapes to avoid the issue. Change-Id: Ie450c4734072b9b215cbbb7e17b4648372610539
2018-12-20Provide a new Blubber logoDan Duvall3-1/+833
The new logo was the winner of a recent round of voting among Release Engineering. (See https://phabricator.wikimedia.org/V19) It incorporates colors from the WM design palette and the ring pattern from the logos of other WM teams, platforms, and tooling. R.I.P. Tyler's dead-whale logo. Change-Id: I0f79dcd9cb521f39059fa4270f65508c3c097a54
2018-12-13Add helm chart to repoTyler Cipriani1-0/+1
Will enable new image build as part of the pipeline CI process. Change-Id: Ie074724169a0aa2b2fe360a1a949f6279c6b605d
2018-12-12Provide OpenAPI spec for BlubberoidDan Duvall7-30/+544
Wrote an OpenAPI 3.0 spec for Blubberoid that provides `x-amples` entries compatible with service-checker. The written spec includes basic schema for Blubber config objects that may be later factored out for use in validation. Note that OpenAPI 3.0 supports only the v4 draft of the JSON Schema standard, so some parts of the configuration could not be fully described. Specifically, v4 does not include the `patternProperties` definition introduced in the JSON Schema v6 draft that would allow us to describe `variants` and `runs.environment` and everything beneath. Blubberoid was refactored slightly to incorporate the new spec as well as assume JSON as the canonical and default configuration format. It was also refactored to include a versioned namespace ("v1") after the server endpoint. Bug: T205920 Change-Id: I28a341aa503b8920d802715660d4c4d62be45475
2018-12-12Support "application/json" in BlubberoidDan Duvall2-1/+79
JSON seems a better option for a web service in general—other toolchains in the Docker/Kubernetes space typically prefer YAML for human-edited configs but convert to JSON on the wire. The "application/json" media type is well established—unlike "application/yaml" which has no official assignment by IANA—and is better supported by the OpenAPI (formerly Swagger) specification. Added content-type media type validation in the Blubberoid HTTP server handler, and added a check for `json.Valid(body)` upon receiving a "application/json" media type. Since any given valid JSON is also valid YAML, Blubberoid simply does a shallow validation of the JSON body before punting to `config.ReadConfig` for YAML unmarshalling and thorough config validation. Bug: T205920 Change-Id: I970acbde497ed446eb8eed568b1328f8c6f5aa55
2018-12-10Use JSON as canonical config formatTyler Cipriani168-31014/+1026
Uses the github.com/ghodss/yaml library to convert YAML to JSON before unmarshaling for the purposes of supporting YAML and JSON input while converting to only support JSON internally. Bug: T207694 Change-Id: I00668014907e9ea54917f5d5067cac08d0668053
2018-12-10Combine blubberoid.yaml and .pipeline/blubber.yamlTyler Cipriani2-24/+14
Change-Id: I726094fd9afa593df31425c86060c4f1d52d8b83
2018-11-16Fix entrypoint merge behaviorTyler Cipriani2-1/+10
Ensure that the CommonConfig being merged can override the Entrypoint. Change-Id: Icff4876e632d0bd590d63e72dccb4d67c8d1a435
2018-11-16Add: a Blubber file for a Blubberoid service Docker imageLars Wirzenius1-0/+22
Change-Id: Ia14c5abe7aa39db490c44e5d947b6b98983cd013
2018-11-16Provide separate Makefile rules for building blubber/blubberoidDan Duvall1-4/+20
Implemented additional rules in the Makefile to be more conventional. Now you can do `make [all]` for both blubber and blubberoid binaries without installing them, or `make blubber` and `make blubberoid` to build them independently. A `make clean` rule was also added. Change-Id: Iaf52d9fbbc1c377c26a0cb21810af2e3d93778e5
2018-10-19Install `golint` from "golang.org/x/lint/golint"Dan Duvall1-1/+1
See https://github.com/golang/lint/issues/415 Change-Id: Ic6d5c90b1409ba1799cfcd5000f7fc1798423e59
2018-10-01Remove support for `sharedvolume` configurationDan Duvall11-74/+12
Support for mounting a shared volume instead of copying application files was meant to provide an option for development use cases. This functionality has never been thoroughly tested or utilized for any use case. It should be removed for now. Relates tangentially to task T204591 that simplifies node support. Change-Id: Ib70cb7bceb504841897a38d732880ba376fe67c8
2018-09-26Merge "Test basic functionality of blubberoid"jenkins-bot1-0/+30
2018-09-20Merge "Remove some types that will be inferred"jenkins-bot2-7/+7
2018-09-17Test basic functionality of blubberoidDan Duvall1-0/+30
Change-Id: I4a946ebbf1545a226457456ffeb8fc6971c46b33
2018-09-17Install `node_modules` to application directoryDan Duvall2-32/+17
The original design decision to install `node_modules` to a central location outside the application directory was for a purported development use case not yet realized. It has led to confusion and unintended failures, and is just generally weird. Let's right this wrong before it fossilizes into debt, like a hysterical raisin in the sun. Bug: T204591 Change-Id: I2e4797833471596b10b6b4a2705055050ca98ac6
2018-09-17Update `.pipeline/blubber.yaml` to v3 formatDan Duvall1-2/+3
Change-Id: I29e0b16f17af572c02c68418e153e26b8e21fa81
2018-09-13Remove some types that will be inferredTyler Cipriani2-7/+7
Adding types to variables that would have their types inferred from function return types causes lint errors in go1.10.3 (my local machine). Change-Id: I71ffbe7843084b4bd22b15ed223ed05d4ccc9aaa
2018-09-13Fix make releaseTyler Cipriani1-1/+2
Missed during the rename from blubberd -> blubberoid. The shell needed to be changed to /bin/bash to accommodate the "{}" shell expansion which doesn't work with sh. Change-Id: I1743d58b17fe794b517e8d1d947ae29135bdc73d
2018-08-29Provide a stateless blubberoid microserviceDan Duvall3-4/+123
The `blubber` command already gets everything it needs from explicit inputs, which makes it an easy candidate for running as a simple microservice. This patch provides exactly that in the form of `blubberoid`, an HTTP server that processes Blubber configuration. To start the daemon: make && blubberoid To use it: curl -i -X POST --data-binary @blubber.example.yaml http://:8748/[variant] Change-Id: Ieea73048d092b974da424ba40ddc90eaf693af0b
2018-08-14Bump Blubber config version to v3Dan Duvall17-42/+42
Change-Id: I91cac046976cf88d513487e27e1b52bf77a6f974
2018-08-14Refactor builder to support file requirements and run pre-installDan Duvall11-142/+243
The builder configuration has proven useful for supporting generic pre-entrypoint commands such as dependency managers not otherwise supported by specific Blubber configuration. Adding additional `builder.requirements` config expands support for such commands by allowing the user to specify files that should be copied into the image before the builder command runs. To support this extra configuration, `builder` had to be changed from a simple string to a mapping. The builder command must now by given as `builder.command`. The pattern of creating parent directories, copying files, and executing one or more commands prior to the entrypoint has become a common one. Some of the implementation of this pattern was moved from `PythonConfig` into shared build macros `build.SortFilesByDir` and `build.SyncFiles`. All config types that must have requirements files copied over independently of the entire source tree (`PythonConfig`, `BuilderConfig`, `NodeConfig`) now delegate to these functions. Change-Id: I67f33034f22cee2851ec866cfb07ab20c23eba8c
2018-08-10Test Blubber in CI using BlubberDan Duvall1-0/+9
Added a `blubber.yaml` that defines a single test variant and entry point for use with the new generic `blubber-test` CI job (see I31b367f2ab8b897fa988e70222f70d84186a631b). Bug: T200452 Change-Id: I60b23e1bca8486e43ebf0f62198e85833f86b921
2018-08-09Makefile lint rule now tests gofmt output and runs go vet correctlyDan Duvall2-3/+3
The `lint` rule was previous surfacing purely whitespace output of `gofmt` as a failure and running `go tool vet` instead of `go vet` (the former accepts source files as arguments, the latter takes package names). A real lint error in `config/node.go` was also fixed. Change-Id: I7f13dcbb45a448f1ecac87e97527a7f4ed4d2c48
2018-08-02Merge "Provide Makefile rules for running linters and tests"Thcipriani3-5/+26
2018-08-01Provide Makefile rules for running linters and testsDan Duvall3-5/+26
With the move to Gerrit and away from Phabricator/Arcanist, we lost an easy way to run linters. New rules have been added to the `Makefile`, `lint`, `unit`, and `test` that run linters, unit tests, and both linters and unit tests, respectively. Bug: T200452 Change-Id: I0742daaa14389841d88f13eba47dee07ac127cf2
2018-08-01Add .gitreview fileAntoine Musso1-0/+6
Change-Id: I4b6a2ece0f0096c49ab4dba06f358e794941f94d
2018-07-31Change import paths from Phabricator to GerritDan Duvall38-111/+62
All import paths and other references to the previously Blubber repo in Phabricator have been changed to point to the new Gerrit project URL. Note that building or testing against this change will require you to move your working directory to `$GOPATH/src/gerrit.wikimedia.org/r/blubber`. Contribution documentation was updated to reflect the changes, and the `.arcvendor` submodule has been removed. Bug: T200452 Change-Id: I3ab23e420b2825e86e2bd7c9b3da9d4de23edaa1
2018-07-26Add slash to directory path if copying > 1 fileTyler Cipriani3-10/+18
Summary: Ran into a Docker error when attempting to build a node project with more than one requirements files: When using COPY with more than one source file, the destination must be a directory and end with a / Reviewers: dduvall, #release-engineering-team Reviewed By: dduvall, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1080
2018-05-31tool -> tooTyler Cipriani1-1/+1
Reviewers: dduvall, #release-engineering-team Reviewed By: dduvall, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1065
2018-05-25Release v0.4.0Dan Duvall1-1/+1
Summary: Bump version to v0.4.0. Test Plan: Run `make && blubber --version` and verify the output. Reviewers: thcipriani, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1060
2018-05-24Attempt to resolve symlinked CURDIR in MakefileDan Duvall1-0/+4
Summary: See upstream bug https://github.com/golang/go/issues/24359 Test Plan: Run `make` from a symlink targetting the blubber directory under `GOPATH` and verify that `blubber --version` outputs the version and commit. Reviewers: thcipriani, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1061
2018-05-24Cross-compile using goxTyler Cipriani3-1/+21
Reviewers: dduvall, #release-engineering-team Reviewed By: dduvall, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1059
2018-05-20Bump config version to v2Dan Duvall17-44/+44
2018-05-20Add now mandatory version config field to READMEGilles Dubuc1-0/+2
Reviewers: thcipriani, dduvall, #release-engineering-team, mmodell Reviewed By: dduvall, #release-engineering-team, mmodell Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1052
2018-05-20Refactor config for node app dependenciesDan Duvall7-57/+63
Summary: Change `node.dependencies` flag to `node.requirements`, the list of files (typically `package.json` and either `npm-shrinkwrap.json` or `package-lock.json`) that declare dependencies to be installed with NPM. Test Plan: Run `go test ./...`. Try it out with something like Mathoid. Reviewers: thcipriani, hashar, mobrovac, Joe, akosiaris, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1058
2018-05-20Add support for Builder arbitrary build commandsGilles Dubuc4-9/+102
Reviewers: thcipriani, dduvall, #release-engineering-team Reviewed By: dduvall, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1054
2018-05-19Add a pip install stepAlexandros Kosiaris2-64/+74
Summary: We want to also install the wheels, not just generate them. To do that we add a pip install -t <target> command. For this to work however, we need to install into /opt/lib/python and amend the environment accordingly to set PYTHONPATH and PATH in order for the libraries and binaries to be accessible. Reviewers: dduvall, thcipriani, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1057
2018-04-30Ensure non-interactive front-end for APTDan Duvall2-0/+6
Summary: Ensure that `DEBIAN_FRONTEND=noninteractive` is set before injecting `apt-get install` instructions. Test Plan: Run unit tests. Reviewers: thcipriani, mmodell, hashar, demon, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1032
2018-04-05Refactor validation tests without YAML parsingDan Duvall13-238/+172
Summary: Since `config.Validate` was changed to take any interface as an argument, many of the validation tests can be refactored in a way that avoids having to parse the full config context in YAML and instead validates each specific config struct directly. The new test pattern is simpler and less prone to future breakage should unrelated parts of the overall config change. Tests that rely on root config context were left unchanged. Test Plan: Run `go test ./...`. Reviewers: thcipriani, demon, hashar, mmodell, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1023
2018-04-05Introduce strict/versioned config parsingDan Duvall18-16/+193
Summary: Introduced a `version` config field that must be specified and match `config.CurrentVersion`. Changed `config.ReadConfig` to use `yaml.UnmarshalStrict` to ensure that errors are surfaced when unknown/bad fields are present in the given YAML config. A smaller `config.VersionConfig` is now unmarshaled first to prevalidate the new `version` field before the entire config is parsed. Fixes T191460 Test Plan: Run `go test ./...`. Run `blubber` against some configuration containing invalid fields and ensure that it surfaces a YAML error. Reviewers: thcipriani, demon, hashar, mmodell, mobrovac, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Maniphest Tasks: T191460 Differential Revision: https://phabricator.wikimedia.org/D1021
2018-03-28Provide a CONTRIBUTING.md guideDan Duvall2-0/+99
Summary: To help new developers get on board, let's provide a `CONTRIBUTING.md` doc. Test Plan: Proofread with your eyeballs. Reviewers: thcipriani, demon, hashar, mmodell, zeljkofilipin, greg, Jrbranaa, #release-engineering-team Reviewed By: thcipriani, hashar, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D993
2018-03-22Release v0.3.0Dan Duvall1-1/+1
Summary: Bump the version to v0.3.0. Test Plan: Run `make && blubber --version` and verify the output. Reviewers: thcipriani, hashar, demon, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1014
2018-03-22Policy example should work on blubber.example.yamlTyler Cipriani1-2/+2
Test Plan: Run: blubber -p policy.example.yaml blubber.example.yaml production Be happy it works. Reviewers: dduvall, #release-engineering-team Reviewed By: dduvall, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1013
2018-03-22No need to export NewValidatorTyler Cipriani2-4/+4
Summary: It occurred to me while looking at code in another patch that there is no need to have this function be public. Test Plan: `go test ./...` Reviewers: dduvall, #release-engineering-team Reviewed By: dduvall, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1012
2018-03-22Provide a `runs.insecurely` to be used with test variantsDan Duvall5-2/+66
Summary: Use cases involving running of test suites and doc generation require more liberal ownership and read/write permission to application files. When `runs.insecurely` is set to `true`, the effective runtime user will be `lives.as`, the same user that owns the application files and installed dependencies. D999 is a complement to this change to allow restrictions on this and other potentially sensitive configuration. Depends on D999, D1002 Test Plan: Run `go test ./...`. Reviewers: thcipriani, mmodell, hashar, #release-engineering-team, demon Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1003
2018-03-22Fix ownership on artifact copiesDan Duvall10-41/+91
Summary: The implementation of D984 did not include enforcing ownership for `build.CopyFrom` instruction and so artifacts copied from one image to another via `copies:` were problematically owned as root. In order to fix this behavior: 1. `config.ArtifactConfig` `build.CopyFrom` instructions are now injected duration `build.PhaseInstall` 2. `config.VariantConfig` calls `build.ApplyUser` for these artifact instructions as well using the `runs.as` user 3. `build.CopyAs` was refactored to wrap any `build.Instruction` which should only really be used with `build.Copy` or `build.CopyFrom`. Test Plan: Run `go test ./...`. Run `blubber` against configuration with a variant that uses `copies` and verify that the `COPY --from` instructions also include a `--chown` flag. Reviewers: thcipriani, mmodell, hashar, #release-engineering-team, demon Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D1002
2018-03-19Allow for configuration policiesDan Duvall90-289/+8876
Summary: Implements a rough interface for validating configuration against arbitrary policy rules. Policies are provided as YAML and passed via the command line as file paths or remote URIs. The format of policies is: enforcements: - path: <path> rule: <rule> Where `<path>` is a YAML-ish path to a config field and `<rule>` is any expression our config validator understands (expressions built in by the validator library and custom tags defined in `config.validation.go`). Example policy: enforcements: - path: variants.production.base rule: oneof=debian:jessie debian:stretch - path: variants.production.runs.as rule: ne=foo - path: variants.production.node.dependencies rule: isfalse Command flag parsing was implemented in `main.go` to support the new `--policy=uri` flag and improve existing handling of `--version` and the usage statement. Test Plan: Run `go test ./...`. Reviewers: thcipriani, demon, hashar, mmodell, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D999
2018-03-06Support Python projectsDan Duvall4-8/+434
Summary: A new root and variant `python` config field is provided with two new fields below, `version` and `requirements`. The former, `version`, should specify the Python executable to use when executing related package installation commands and ostensibly the same executable that will be used to run the application. The latter, `requirements`, should specify all pip requirements files such that a compiler that supports layered filesystems (e.g. Docker) can output separate instructions that will invalidate cache layers for changes to those files independently of changes to the rest of the codebase. Python related instructions will be generated only if either `version` or `requirements` are given. Fixes T186545 Test Plan: Run `go test ./...`. Reviewers: thcipriani, hashar, demon, #release-engineering-team Reviewed By: thcipriani, #release-engineering-team Tags: #release-engineering-team Maniphest Tasks: T186545 Differential Revision: https://phabricator.wikimedia.org/D976
2018-03-06Makefile: install to global GOPATH with correct -ldflagsTyler Cipriani1-17/+3
Summary: Since we now use ldflags to set global variables it's helpful to automate via Make. Using a single global GOPATH seems to jive more with the best-practice established in the go community[0] and make testing/code review easier. [0]. <https://github.com/golang/go/wiki/GOPATH#use-a-single-gopath> Test Plan: run make Reviewers: dduvall, #release-engineering-team Reviewed By: dduvall, #release-engineering-team Tags: #release-engineering-team Differential Revision: https://phabricator.wikimedia.org/D998
generated by cgit v1.2.1 (git 2.18.0) at 2024-04-26 17:44:12 +0000