Skip to content

Gotchas

  • The address is reserved at 10.82.50.100 by a static lease on the MikroTik (dhcp50-external), keyed on the MAC 00:16:3e:5e:a1:01 that main.tf pins on the NIC. Two things must hold for it to match:
    • dhcpV4Config.ClientIdentifier = "mac" in nix/hosts/s3-01/default.nix. networkd's default client ID is a DUID derived from machine-id, which every reinstall regenerates.
    • The lease's Client ID field stays empty. RouterOS matches on client ID whenever one is set; a stale DUID there (ff:…) makes the lease sit at waiting while the host draws a dynamic address from the pool. This is why ClientIdentifier = "mac" once looked like it did not work.

The corefile entry for s3.gewis.nl and the 40_seaweedfs-buckets endpoint default both hardcode .100. After fixing a mismatch, delete the host's dynamic lease and run networkctl reconfigure enX0 (or reboot): a plain renew just extends the lease the host already holds. - A rebuild can drop its own connection. Activation restarts systemd-networkd, which can renew onto a different lease mid-apply. The switch itself completes — you will see activating the configuration... and the unit restart list — and then the deploy dies with Timeout, server <old-ip> not responding and exit 255. Observed going from .100 to .102. The host is fine; only tofu's bookkeeping is unfinished. Confirm with readlink -f /run/current-system on the new address, then re-run tofu apply: it re-reads the address from the guest agent, and re-running the switch on an already-switched host is a fast no-op. - The host answers on the mesh too. NetBird gives it a name from its hostname (s3-01, s3-01.net.nb.gewis.nl) even though NB_EXTRA_DNS_LABELS is unset, and that address does not move with DHCP. It is the better target when chasing a host whose lease just changed — though it takes a moment to come back after netbird-gewis.service restarts. - wss://, not https:// for the XO endpoint. The wrong scheme, or a self-signed cert without XOA_INSECURE=true, gives websocket: bad handshake. - Sizes are in bytes in the provider; this config takes GiB and converts. - A template is required — the VM cannot boot bare. The guest agent goes in via cloud-init, which only fires on first boot, so recreate the VM if you change cloud-init. A reboot will not re-run it. - net.ifnames=0 is what makes IP discovery work. Ubuntu noble's xe-guest-utilities is pinned at 7.20.2, whose vifNamePrefixList is {eth, eno, ens, emp, enx}. It lacks enX, exactly the name Xen assigns on 24.04, so the agent runs fine (XO shows managementAgentDetected: true) but never writes attr/vif/*/ipv4/* to xenstore. XAPI has no host-side IP discovery to fall back on, so expected_ip_cidr would never be satisfied and create would die at the timeout. Cloud-init works around it by dropping net.ifnames=0 into /etc/default/grub.d/, running update-grub and rebooting; the NIC then comes up as eth0, which 7.20.2 does parse. Do not remove that reboot — the kernel parameter only takes effect on the next boot. See xcp-ng/xcp#655 and xe-guest-utilities#164. This affects only the throwaway bootstrap Ubuntu; nixpkgs ships 10.0.1, which does include enX. - 20-minute create timeout in modules/xcpng-vm/main.tf, because a full clone is genuinely slow and can overrun the provider's 5-minute default. - NetBird runs unhardened (nix/modules/netbird.nix): its built-in SSH server has to switch into the login user, which the hardened systemd sandbox blocks. - Rotating the setup key changes nothing on a joined host. The peer is identified by the WireGuard key in /var/lib/netbird-gewis/config.json, and the client only puts a setup key on the wire when management reports the peer as unregistered. netbird-gewis-login.service reinforces this: it runs netbird up only while netbird status says NeedsLogin. So a new key with Allow extra DNS labels does not grant the existing peer that permission — allow_extra_dns_labels is stamped onto the peer record at registration and never re-derived, and ExtraDNSLabels is not updatable by login, sync, or the management API. Adding dnsLabel to a peer registered without the flag makes its next login fail with setup key doesn't allow extra DNS labels. Delete the peer in the dashboard first, then restart the login unit. - State lives in Scaleway, encrypted before it leaves your machine — see State above. Never move it onto the S3 server this project creates.