Merge branch 'liw/fixes' into 'main'v0.3

changes in prep for first public demo

See merge request larswirzenius/v-i!56

6 files changed, 142 insertions, 64 deletions

diff --git a/NEWS.md b/NEWS.md
index f47f7c0..d844266 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -4,6 +4,51 @@ This file summarizes user-visible changes between releases of v-i, the
 vmdb2-based installer of Debian onto bare metal systems.
 
 
+# Version 0.3, released 2023-08-12
+
+## Major changes
+
+* Both the installer can now use wifi. The wifi password used in the
+  installer is copied to the target system. It's now possible to
+  install without wired networking at all, and the booted system
+  connects to wifi automatically.
+
+* SSH host key and host certificate can be set at installation time.
+  This helps connecting to newly installed systems, for show using
+  certificates. Also, users can log in using SSH user certificates
+  instead of keys, if they have certificates made with trusted SSH CA
+  keys.
+
+* The installer provides addresses via its on Ethernet via DHCP, to
+  allow bootstrapping a LAN. It avoids serving addresses via its main
+  Ethernet port, which it uses itself to access the Internet.
+
+* File systems other then ext4 are now supported on the target
+  system, for example btrfs.
+
+* The installer attempts to use "secure discard", when emptying
+  drives. It falls back to plain discard, without an error, if the
+  secure variant fails.
+
+## Minor changes
+
+* Root can now only log in via SSH using a key or certificate:
+  passwords are not allowed for root on SSH. Console login as root
+  doesn't require a password.
+
+* Ansible variables can be given via files, not just in playbooks,
+  when running the installer.
+
+* The installer now allows installing any Debian release. Debian 11
+  and 12 have been tested.
+
+* Improved documentation of the target specification file format and
+  how to use SSH certificates.
+
+* Various sharp corners have been rounded to make development and use
+  of the installer less painful.
+
+
 # Version 0.2, released 2022-08-07
 
 ## New or changed features
diff --git a/README.md b/README.md
index 3e94a8e..58537ad 100644
--- a/README.md
+++ b/README.md
@@ -21,17 +21,17 @@ choice (the author uses Ansible). The basic system allows you to log
 in as root with SSH using the key you provide to **v-i**. The
 installer removes all partitions on all drives on the system, and sets
 up LUKS and LVM2 on all the drives you specify. It sets up the time
-zone and console keyboard layout you specify. It installs the
-`locales-all` package so your locale is available. It sets up the
-system to use the `deb.debian.org` Debian CDN mirror network. Anything
-else you'll have to install and configure yourself.
+zone and console keyboard layout you specify. The installed system is
+fairly basic, but functional. Anything else you'll have to install and
+configure yourself.
 
 **v-i** installs a very basic Debian onto a PC. It's entirely
 non-interactive and unhelpful. The author wrote it so that repeated
 installations would be less of a chore than using the official Debian
 installer. (Actually, the author thought it'd be a quick, easy hack,
 and was too stubborn to give up, when it turned out to be a bit
-tricky.)
+tricky. What was meant to be a weekend hack turned into a multi-year
+project, on and off.)
 
  **v-i** uses **vmdb2** to install onto bare metal hardware.
 [**vmdb2**][] is a program to create a disk image virtual machines
@@ -59,37 +59,8 @@ hostname: exolobe5
 drive: /dev/nvme0n1
 ```
 
-A number of fields are allowed in the specification file:
-
-* `hostname`---the hostname of the installed system. This is so that
-  when the installed system boots, and gets a network address using
-  DHCP, it can provide a name. The author's home network setup
-  automatically adds that hostname to the internal DNS. This avoids a
-  manual DNS configuration step, and the author is lazy.
-  ([dnsmasq](https://dnsmasq.org/) is lovely.)
-* `drive`---the main drive to install to. This is where the EFI and
-  `/boot` partitions are created, and where GRUB gets installed. The
-  rest of the drive will be a physical volume for LVM2.
-* `extra_drives`---a list of any additional physical volumes for LVM2.
-  These will not be partitioned, and will be used entirely as physical
-  volumes.
-* `luks`---the password for full disk encryption for all LVM2 physical
-   volumes. If not set, LUKS is not used. This is a single, fixed
-   password that is in cleartext. _You are expected to change it_
-   after the system is installed and booted. If you'd rather use, say,
-   a hardware token's challenge/response feature or TPM for LUKS,
-   that's better done on a running system.
-* `extra_playbooks`---additional Ansible playbooks to use on the
-  installed system. **v-i** comes with a "standard playbook" (in
-  [`std.yml`][]) that it uses unconditionally, to set up a "standard
-  system" that the author likes. You can provide additional playbooks,
-  for additional configuration at installation time.
-* `ansible_vars`---variables to set for Ansible playbooks.
-  - the `user_pub` variable contains an SSH public key that gets
-    installed into the `root` user `authorized_keys` file on the
-    installed system by the standard playbook
-  - the `user_ca_pubkey` variable contains public key for an SSH CA
-    whose user certificates are to be trusted
+See [spec.md][] for a full description of the target specification
+file.
 
 With all this configuration in a file, which you can keep in git, you
 can install a base system repeatedly to a specific computer, and do it
@@ -108,6 +79,8 @@ tedious.)
 [`build-installer.sh`]: build-installer.sh
 [`v-i`]: v-i
 [`std.yml`]: std.yml
+[spec.md]: spec.md
+[tutorial]: tutorial.md
 [author]: https://liw.fi/
 [Debian installer]: https://www.debian.org/devel/debian-installer/
 [preseed files]: https://wiki.debian.org/DebianInstaller/Preseed
@@ -174,10 +147,11 @@ standard system looks like this:
 * a `root` LV (20 GiB)
 * the rest of the VG not allocated
 * a basic Debian installation
-  - `root` password is locked
-  - network setup on `eth0` using systemd-networkd
+  - network setup on `eth0` using `systemd-networkd`
+  - optionally wifi using `systemd-networkd` and `iwd`
   - SSH host key and host certificate installed if defined
-  - log in to `root` over SSH using a key or user certificate
+  - `root` password is locked, no login on console
+  - log in as `root` allowed over SSH using a key or user certificate
 
 While **vmdb2** can, and does, run Ansible to configure the system
 being installed, in practice some things work better if most
@@ -190,17 +164,19 @@ still being installed in a chroot.
 
 Thus, **v-i** does the following:
 
-* delete any trace of LVM2 from all drives, wipe all SSDs, and
-  generally reset the system to as close to a blank state as possible
-  - there is no question "are you sure?" to give the user a chance to
-    repent: as soon as you run **v-i**, you've lost all your data
+* delete any trace of LVM2 from all drives, erase all SSDs (securely,
+  if possble), and generally reset the system to as close to a blank
+  state as possible
+  - __there is no question "are you sure?" to give the user a chance to
+    repent: as soon as you run **v-i**, you've lost all your data__
 * create a partition table ("label") on the target drive
-* create EFI and boot partitions, needed to boot with UEFI and LUKS
+* create cleartext EFI and boot partitions, needed to boot with UEFI
+  and LUKS
 * create a physical volume for LVM2, and a logical volume for the root
   file system
   - add any additional drives as physical volumes to the volume group
-  - optionally use LUKS for full disk encryption for each physical
-    volume
+  - optionally use LUKS2 for full disk encryption for each physical
+    volume (LUKS2 for `argon2id` support)
 * install the Debian base system
   - run `debootstrap`, install a boot loader, and create fstab and
     `crypttab` files
@@ -211,7 +187,8 @@ Thus, **v-i** does the following:
   - install an SSH server
   - add a chosen SSH public key to the root user's authorized keys
     file
-* run any additional playbooks
+  - other configuration
+* run any additional playbooks provided by the user
 
 **v-i** uses the **vmdb2** caching feature, where the results of
 `debootstrap` and some other steps get stored in a compressed tar
@@ -241,12 +218,14 @@ Also, to build an image to boot off for running the installer:
 * `installer.yml`---the Ansible playbook for creating the installer
   image
 
-See the tutorial about ways to add your SSH public key to the image so
+See the [tutorial][] about ways to add your SSH public key to the image so
 that you can log into the installer via SSH.
 
 You probably mostly only need to modify `v-i` and `std.yml`. The rest
 is to get you and your target machine into a state where you can run
-the installer.
+the installer. If you have a working installer image, you can update
+those two files by copying new versions into place: this is much
+faster than building a whole new installer image.
 
 
 
@@ -257,14 +236,15 @@ yet, but expects the following to be asked.
 
 ### What version of Debian does v-i install?
 
-**v-i** installs Debian 11 (bullseye).
+**v-i** installs the Debian stable release, by default. That's version
+12 (bookworm) at the time of writing this. You can install other
+versions by setting the `debian_release` field in the target
+specification file. Any version from Debian 11 onward should work.
 
 ### What about other releases of Debian?
 
 The Debian 11 (bullseye) release is the earliest release the author
-has gotten to work with **v-i**, and is the only release the author is
-installing on bare metal systems. Later versions of Debian may work,
-we will see.
+has gotten to work with **v-i**.
 
 ### Is only UEFI supported?
 
@@ -365,7 +345,7 @@ wifi_password: notopen
 
 The installed system is plain Debian, and you can configure it to
 support wifi as you would any other Debian system. The `v-i` installer
-does not copy over the wifi credentials to the installed system.
+**does** copy over the wifi credentials to the installed system.
 
 ### I'd like to use v-i, but I need changes
 
diff --git a/installer-ansible.yml b/installer-ansible.yml
index e8fddf7..e924f06 100644
--- a/installer-ansible.yml
+++ b/installer-ansible.yml
@@ -1,8 +1,4 @@
 # Ansible playbook to install stuff for v-i.
-# TODO:
-# - maybe install iwlwifi firmware?
-# - install liw-openpgp.pub and a gpg config to use my Yubikey
-
 
 - hosts: image
   tasks:
@@ -76,7 +72,7 @@
       copy:
         content: |
           XKBMODEL="pc105"
-          XKBLAYOUT="fi"
+          XKBLAYOUT="us"
           XKBVARIANT=""
           XKBOPTIONS=""
           BACKSPACE="guess"
diff --git a/std.yml b/std.yml
index a335b02..08724a9 100644
--- a/std.yml
+++ b/std.yml
@@ -116,7 +116,30 @@
         name: ifupdown
         state: absent
 
-    - name: "configure networkd"
+    - name: "enable the non-free-firmware component on bookworm"
+      when: debian_release != "bullseye"
+      apt_repository:
+        repo: "deb http://deb.debian.org/debian {{ debian_release}} non-free-firmware"
+        state: present
+        update_cache: yes
+
+    - name: "install iwd and firmware for wifi"
+      apt:
+        name:
+          - firmware-brcm80211
+          - firmware-iwlwifi
+          - firmware-libertas
+          - firmware-misc-nonfree
+          - firmware-realtek
+          - firmware-ti-connectivity
+          - iwd
+
+    - name: "enable iwd"
+      systemd:
+        name: iwd
+        enabled: yes
+
+    - name: "configure networkd for Ethernet"
       copy:
         content: |
           [Match]
@@ -126,6 +149,21 @@
           DHCP=yes
         dest: /etc/systemd/network/external.network
 
+    - name: "configure networkd for wireless"
+      copy:
+        content: |
+          [Match]
+          Name=wlan*
+
+          [Network]
+          DHCP=yes
+        dest: /etc/systemd/network/wireless.network
+
+    - name: "copy wireless credentials from host to target"
+      copy:
+        src: /var/lib/iwd/
+        dest: /var/lib/iwd/
+
     - name: "enable networkd"
       systemd:
         name: systemd-networkd
@@ -134,11 +172,13 @@
   vars:
     ansible_python_interpreter: /usr/bin/python3
 
-    # You may want to override these.
     user_locale: |
-      export LC_CTYPE=fi_FI.UTF8
+      export LC_CTYPE=C.UTF8
+
+    # You may want to override these to get a non-US keyboard layout.
     user_keyboard_model: pc105
-    user_keyboard_layout: fi
+    user_keyboard_layout: us
     user_console_codeset: Lat15
 
     passwordless_root: false
+
diff --git a/tutorial.md b/tutorial.md
index 74b966a..c8e48e0 100644
--- a/tutorial.md
+++ b/tutorial.md
@@ -70,6 +70,17 @@ The steps:
 8. Wait for the system to boot. Log in as root, either from the
    virtual console (no password), or via SSH.
 
+   If you need to connect to wifi, you need to log in as root and run
+   the following commands:
+
+    ~~~sh
+    iwctl station wlan0 get-networks
+    iwctl stations wlan0 connect DadItIsThisOne
+    ~~~
+
+   Substitute the name of the network as needed. Enter wifi password
+   when requested.
+
 9. Create a target specification file `foo-target.yaml`. The name can
    be anything, but it must be valid YAML. In the example below, there
    is one NVMe drive that will get the system installed. All drives
@@ -79,6 +90,8 @@ The steps:
     drive: /dev/nvme0n1
     luks: hunter2
     ~~~
+    
+    See [spec.md][] for a full description of the specification file.
 
 10. Install. Add the `--verbose` option to `./v-i` if you want to know
     what's happening. The first time you run this on a given v-i USB
@@ -87,7 +100,7 @@ The steps:
 
     ~~~sh
     rm -f install.log
-    ./v-i --verbose foo-target.yaml
+    v-i --verbose foo-target.yaml
     ~~~
      
     This will take few minutes the first time, assuming a fast
@@ -101,3 +114,6 @@ The steps:
     Remember to change the LUKS password for each drive.
 
 13. Optional: Let me know how it went.
+
+[spec.md]: spec.md
+
diff --git a/v-i b/v-i
index 76a7946..96b76e6 100755
--- a/v-i
+++ b/v-i
@@ -549,6 +549,7 @@ def main():
         log(f"reading Ansible vars from {filename}")
         with open(filename) as f:
             vars_dict = yaml.safe_load(f)
+        vars_dict["debian_release"] = system.debian_release
         ansible_vars.update(vars_dict)
     ansible_vars_json = json.dumps(ansible_vars, indent=4)
     log(f"ansible_vars:\n{ansible_vars_json}")