From dff8a6721ea6cd04d1cba259d4111d03d26833f6 Mon Sep 17 00:00:00 2001 From: Lars Wirzenius Date: Thu, 17 May 2018 16:36:35 +0300 Subject: Add: README --- README | 184 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 184 insertions(+) create mode 100644 README diff --git a/README b/README new file mode 100644 index 0000000..4ecd086 --- /dev/null +++ b/README @@ -0,0 +1,184 @@ +hetznertool - create VMs in the Hetzner Cloud for QvarnLabs Ab +============================================================================= + +For Hetzner we want something to allow us to easy create VMs, run +Ansible against them, and have them have DNS names. Here's a first +stab at what it might look like. First some general observations: + +* Hetzner does not separate between projects and stacks. There is a + project abstraction, but no stack abstraction. However, unlike with + OpenStack, we can add projects easily ourselves. Hetzner calls + projects "contexts", in some places. + + Projects need to be created via the web interface ("cloud console"). + Also via the web interface, ssh keys are uploaded (separately for + each project), and access tokens generated (also for each project). + Each user should have their unique access token. + + After a project is created, the `hcloud` software is used to create + a "context", creation of which requires the access token. + + This part is all manual, at least for now, but it's a one-time + operation. + +* Hetzner also doesn't seem to care about more than VMs. Each VM gets + a public IP, and there's no firewall etc rules to handle. + +* Hetzner has an API, but also has a command line tool (`hcloud`), + which seems like it's going to be nicer to use, at least initially. + (We had small issues with the OpenStack API changing from underneath + us.) + +The workflow with Hetzner could look like this: + +* We create a "project" in the Hetzner cloud. We create per-project + access tokens, and store those using `pass` (in a personal instance, + not shared). + +* We write a tool that reads a description of the set of VMs to be + created for a project. The descrioption might look like this: + + defaults: + image: debian9 + type: cx11 + hosts: + - name: haproxy + type: cx51 + - name: qvarn1 + - name: qvarn2 + +* VM creation tool would be invoked like this: + + hetznertool create liwproject foo.yaml --ssh-key liw-openpgp + + Here `liwproject` is the (short) name for the project (context) in + which to create the VMs, `foo.yaml` is the description file, and + `liw-openpgp` is the ssh key name to install in the new VM by + default. + +* The tool will run `hcloud` with the appropriate parameters, read + from the description file and command line options, and also create + corresponding DNS entries: `liwproject-haproxy.h.qvarnlabs.eu`. + Additionally it will write an Ansible inventory file that can be + used with `ansible-playbook -i` (one inventory file per Hetzner + context, listing VMs in the context in a way that our playbooks can + refer to them with generic names). + +* The tool will wait until the VMs are created. + +* Later on, we can add support for deleting all the VMs in context: + + hetznertool delete liwproject + +In more detail +============================================================================= + +Preparation: Hetzner project creation +----------------------------------------------------------------------------- + +Log into + + https://console.hetzner.cloud/projects + +Create projects as needed, add ssh keys to them, and create access +tokens for each. Use the access tokens to create "contexts" for +`hcloud` (run `hcloud context create FOO`, it will ask for the access +token). + +Use the ssh token at once when creating it: it cannot be recovered +later (but a new one can be created at will). Not that it's not +necessary to store the access token elsewhere, but do guard the +`.config/hcloud/cli.toml` file, which is where `hcloud` stores the +access tokens. `hcloud` does not seem to have a way to store that in +`pass` or similar. Avoid sharing `cli.toml` with anyone. + +This only needs to be done once per project. + +Preparation: Configure `hetznertool` +----------------------------------------------------------------------------- + +Create `~/.config/heznertool/hetznertool.yaml` with the following +content (except adjust paths for your files): + + dnszone-dir: ~/qvarnlabs/code/dnszone + dnszone-file: db.hetzner + ansible-inventory-dir: . + +The tool will write zone file (`db.hetzner`) in the zone directory +(`~/qvarnlabs/code/dnszone`), git comit that, and do a git push. + +The tool will write the Ansible inventory files to the directory +specified with `ansible-inventory-dir` (defaulting to the current +working directory). + +Input file: VM descriptions +----------------------------------------------------------------------------- + +The `default` item has defaults, which will be used of a host doesn't +override them. + +* `image` specifies the VM image to use. +* `type` is the type of VM to create (CPUs, RAM, disk) + +Under `hosts`, a list of dicts, each dict describing one VM: + +* `name` is the name of the VM +* `image`, `type` are optional: if present, they override the value + from `defaults`, and if missing, the value from `defaults` is used + +There may later be more parameters, if they're needed. + +Action: create VMs +----------------------------------------------------------------------------- + +For each VM in the description file, compute the full description +(fill in missing values from `defaults`), and then run: + + hcloud context use CONTEXT + hcloud server create --image IMAGE --name NAME --type TYPE + +The `CONTEXT` comes from the `hetznertool` command line. + + +Action: update DNS +----------------------------------------------------------------------------- + +Get list of servers by running `hcloud server list`, which gives +output like this: + + ID NAME STATUS IPV4 IPV6 + 642236 controller running 94.130.180.87 2a01:4f8:1c0c:4054::/64 + 642237 qvisqve running 195.201.99.89 2a01:4f8:1c0c:75f4::/64 + 642238 apt running 195.201.42.198 2a01:4f8:1c0c:7e66::/64 + 642239 artifacts running 94.130.184.219 2a01:4f8:1c0c:69ba::/64 + 642240 worker1 running 195.201.112.70 2a01:4f8:1c1c:1871::/64 + 642241 worker2 running 195.201.144.105 2a01:4f8:1c1c:1872::/64 + +Parse this to extract the name and IPv4 addresses. Do this for every +context (`hcloud context list`). + +Update `dnszones.git`, file `db.hetzner` to add a line for each +VM in each context: + + {CONTEXT}-{NAME} IN A {IPV4} + +(It is assumed that the `qvarnlabs.eu` DNS servers are configured to +honor the `db.hetzner` file.) + +Commit the new file to git, and push to the server. + +Copy the new file to `ns1.qvarnlabs.net` and `ns2.qvarnlabs.net` and +restart the Bind server on `ns1`. + + +Action: write inventory files +----------------------------------------------------------------------------- + +For each context in the Hetzner cloud, write an Ansible inventory +file `hosts.{CONTEXT}`: + + {SERVER} ansible_ssh_host={IPV4} + +Write these files to the current working directory. The +`ansible_ssh_host` is there so Ansible can be run at once and does not +need to wait for DNS to be updated. -- cgit v1.2.1