From 4f1619a3ba6ce68e08cea4cc4ca3c61b9605c4f1 Mon Sep 17 00:00:00 2001 From: Lars Wirzenius Date: Thu, 17 May 2018 19:10:48 +0300 Subject: Change: update README --- README | 191 ++++++++++++++++++++++------------------------------------------- 1 file changed, 65 insertions(+), 126 deletions(-) diff --git a/README b/README index 4ecd086..ffb63c9 100644 --- a/README +++ b/README @@ -1,77 +1,29 @@ hetznertool - create VMs in the Hetzner Cloud for QvarnLabs Ab ============================================================================= -For Hetzner we want something to allow us to easy create VMs, run -Ansible against them, and have them have DNS names. Here's a first -stab at what it might look like. First some general observations: +`hetznertool` is a little utility to make it easier to manage VMs in +the Hetzner Cloud. It's inspired by the OpenStack Heat tooling we've +used before. The utility can: -* Hetzner does not separate between projects and stacks. There is a - project abstraction, but no stack abstraction. However, unlike with - OpenStack, we can add projects easily ourselves. Hetzner calls - projects "contexts", in some places. +* list all servers in all hcloud contexts +* create servers in a context based on a YAML specification file + * also, update DNS (`CONTEXT-PROJECT.h.qvarnlabs.eu`) + * also, write Ansible inventory files +* delete all servers in a named context - Projects need to be created via the web interface ("cloud console"). - Also via the web interface, ssh keys are uploaded (separately for - each project), and access tokens generated (also for each project). - Each user should have their unique access token. - After a project is created, the `hcloud` software is used to create - a "context", creation of which requires the access token. - - This part is all manual, at least for now, but it's a one-time - operation. - -* Hetzner also doesn't seem to care about more than VMs. Each VM gets - a public IP, and there's no firewall etc rules to handle. - -* Hetzner has an API, but also has a command line tool (`hcloud`), - which seems like it's going to be nicer to use, at least initially. - (We had small issues with the OpenStack API changing from underneath - us.) - -The workflow with Hetzner could look like this: - -* We create a "project" in the Hetzner cloud. We create per-project - access tokens, and store those using `pass` (in a personal instance, - not shared). - -* We write a tool that reads a description of the set of VMs to be - created for a project. The descrioption might look like this: - - defaults: - image: debian9 - type: cx11 - hosts: - - name: haproxy - type: cx51 - - name: qvarn1 - - name: qvarn2 - -* VM creation tool would be invoked like this: - - hetznertool create liwproject foo.yaml --ssh-key liw-openpgp - - Here `liwproject` is the (short) name for the project (context) in - which to create the VMs, `foo.yaml` is the description file, and - `liw-openpgp` is the ssh key name to install in the new VM by - default. - -* The tool will run `hcloud` with the appropriate parameters, read - from the description file and command line options, and also create - corresponding DNS entries: `liwproject-haproxy.h.qvarnlabs.eu`. - Additionally it will write an Ansible inventory file that can be - used with `ansible-playbook -i` (one inventory file per Hetzner - context, listing VMs in the context in a way that our playbooks can - refer to them with generic names). - -* The tool will wait until the VMs are created. - -* Later on, we can add support for deleting all the VMs in context: +Requirements +============================================================================= - hetznertool delete liwproject +* Python 3 +* `hcloud` from + * Download, unpack tarball from above URL. + * Move `bin/hcloud` into a location on the PATH. + * Create one or more contexts by going adding projects, ssh keys, + and access tokens via the Hetzner Cloud console. (See below.) +* Get `hetznertool` with git clone from + `ssh://git@git.qvarnlabs.net/hetznertool` -In more detail -============================================================================= Preparation: Hetzner project creation ----------------------------------------------------------------------------- @@ -94,91 +46,78 @@ access tokens. `hcloud` does not seem to have a way to store that in This only needs to be done once per project. -Preparation: Configure `hetznertool` ------------------------------------------------------------------------------ - -Create `~/.config/heznertool/hetznertool.yaml` with the following -content (except adjust paths for your files): - dnszone-dir: ~/qvarnlabs/code/dnszone - dnszone-file: db.hetzner - ansible-inventory-dir: . +Use +============================================================================= -The tool will write zone file (`db.hetzner`) in the zone directory -(`~/qvarnlabs/code/dnszone`), git comit that, and do a git push. +All `hetznertool` invocations should be done in the directory to where +you cloned it. -The tool will write the Ansible inventory files to the directory -specified with `ansible-inventory-dir` (defaulting to the current -working directory). -Input file: VM descriptions +Config ----------------------------------------------------------------------------- -The `default` item has defaults, which will be used of a host doesn't -override them. - -* `image` specifies the VM image to use. -* `type` is the type of VM to create (CPUs, RAM, disk) - -Under `hosts`, a list of dicts, each dict describing one VM: +Create `~/.config/hetznertool/hetznertool.yaml`: -* `name` is the name of the VM -* `image`, `type` are optional: if present, they override the value - from `defaults`, and if missing, the value from `defaults` is used + ssh-key: liw-openpgp + ansible-inventory-dir: /home/liw/qvarnlabs/code/qvarn-prov/ansible -There may later be more parameters, if they're needed. +(edit values as suits you). -Action: create VMs +List ----------------------------------------------------------------------------- -For each VM in the description file, compute the full description -(fill in missing values from `defaults`), and then run: +List all servers: - hcloud context use CONTEXT - hcloud server create --image IMAGE --name NAME --type TYPE + ./hetznercloud list -The `CONTEXT` comes from the `hetznertool` command line. - - -Action: update DNS +Create ----------------------------------------------------------------------------- -Get list of servers by running `hcloud server list`, which gives -output like this: +Create a server specification file (`qvarn.servers`): + + defaults: + type: cx11 + image: debian-9 + hosts: + - name: haproxy + type: cx51 + - name: qvarn1 + - name: qvarn2 + - name: qvarn3 + - name: qvarn4 + - name: postgres + type: cx51 + - name: qvisvqe - ID NAME STATUS IPV4 IPV6 - 642236 controller running 94.130.180.87 2a01:4f8:1c0c:4054::/64 - 642237 qvisqve running 195.201.99.89 2a01:4f8:1c0c:75f4::/64 - 642238 apt running 195.201.42.198 2a01:4f8:1c0c:7e66::/64 - 642239 artifacts running 94.130.184.219 2a01:4f8:1c0c:69ba::/64 - 642240 worker1 running 195.201.112.70 2a01:4f8:1c1c:1871::/64 - 642241 worker2 running 195.201.144.105 2a01:4f8:1c1c:1872::/64 +Values from `defaults` are used if a dict in the list in `hosts` +doesn't specify a value. -Parse this to extract the name and IPv4 addresses. Do this for every -context (`hcloud context list`). +Create servers (may take a minute per server): -Update `dnszones.git`, file `db.hetzner` to add a line for each -VM in each context: + ./hetznertool create mydev qvarn.server - {CONTEXT}-{NAME} IN A {IPV4} +(where `mydev` is the name of a context). This will write inventory +files and update DNS, creating names like +`mydev-haproxy.h.qvarnlabs.eu`. -(It is assumed that the `qvarnlabs.eu` DNS servers are configured to -honor the `db.hetzner` file.) +Delete +----------------------------------------------------------------------------- -Commit the new file to git, and push to the server. +Delete sll servers in a context: -Copy the new file to `ns1.qvarnlabs.net` and `ns2.qvarnlabs.net` and -restart the Bind server on `ns1`. + ./hetznertool delete mydev +This will also rewrite zone and inventory files to update them to any +remaining servers. -Action: write inventory files +To run Ansible ----------------------------------------------------------------------------- -For each context in the Hetzner cloud, write an Ansible inventory -file `hosts.{CONTEXT}`: + ansible-playbook -i hosts.mydev playbook.yml + - {SERVER} ansible_ssh_host={IPV4} +Problems? +============================================================================= -Write these files to the current working directory. The -`ansible_ssh_host` is there so Ansible can be run at once and does not -need to wait for DNS to be updated. +Ask in the company Matrix room. -- cgit v1.2.1