infra/docs/atlas-dr-lab.md

Isolated Atlas DR lab

This is a scaled rehearsal, not a substitute for a full-data restore. The atlas-dr-lab libvirt VM on Ikaros was left shut off on 2026-09-30. Its persistent volumes are in the default libvirt pool: the current 30 GiB OS volume atlas-dr-lab-os-rebuild2.qcow2, the pre-rebuild OS volume atlas-dr-lab-os.qcow2, and four independent 4 GiB atlas-dr-lab-data{1,2,3,4}.qcow2 volumes. The VM uses libvirt's default NAT network (last DHCP address 192.168.122.168), 2 vCPU, and 4 GiB RAM. The data disks have virtio-atlasdrdata{1,2,3,4} serials. No physical disk or production Atlas storage is attached. The VM has no autostart.

Rebuild inputs and isolation

• Use Rocky's 9.8 GenericCloud Base x86_64 image Rocky-9-GenericCloud-Base-9.8-20260525.0.x86_64.qcow2 from https://download.rockylinux.org/pub/rocky/9.8/images/x86_64/. Verify its .CHECKSUM file; the observed SHA-256 was 92c206cc6f790c61583247eefe87890f8828420662c17cacf247cec78ab4eec8.
• Use a dedicated lab-only inventory merged after the repository inventory, and always --limit atlas_dr_lab. The temporary 2026-09-30 inventory/playbook and logs are in /tmp/atlas-dr-lab-image/; copy a sanitized inventory to durable private storage before /tmp is cleared if the lab will be repeated. Never reuse host_vars/atlas.yml, production Vault secrets, or production disk by-id paths for the lab.
• The lab host belongs to platform_rocky and atlas. It uses dradmin (UID/GID 1000) with the operator's public SSH key and a random, unknown password hash, the libvirt DHCP address, pool zpool, mount root /zpool, the four virtio-atlasdrdata* by-id paths, a 1 GiB backup reservation, and rocky_manage_openzfs_repo: true with only zfs in host_packages. The following gates remain false: sharing, firewall, media stack, ZFS timers, Borg, USB, monitoring, and Prometheus pull. atlas_manage_storage is true. Set atlas_create_pool: true only for the first disposable pool creation, then set it false before any later run.
• A minimal lab playbook selects atlas_dr_lab, become: true, and the existing packages_rocky and profile_atlas roles. Use a separate ANSIBLE_CONFIG without the production Vault password script, and keep host-key checking on with a lab-specific known-hosts file. The 2026-09-30 runs used -i ansible/inventory/hosts.yml -i <lab-inventory.yml> and --limit atlas_dr_lab throughout.

Rehearsal and narrow checks

1. Before any pool operation, compare virsh -c qemu:///system domblklist atlas-dr-lab with the four intended qcow2 paths, and in the guest compare /dev/disk/by-id/virtio-atlasdrdata* with lsblk. Do not proceed if a physical disk or production identity appears.
2. For a first-time disposable build only, run the lab playbook with --tags pool and atlas_create_pool: true, then immediately set the gate false. Run the full lab playbook and check zpool status -P zpool, zfs list -r zpool, SELinux, and failed systemd units.
3. Write a non-sensitive canary under the lab /zpool/archive and snapshot it. Record the pool GUID and canary SHA-256. Export the lab pool cleanly, shut down the VM, and replace only the OS volume with a fresh verified Rocky image. Preserve all four data volumes. Reconfigure cloud-init for a new instance; the seed CD-ROM must use SATA. The SCSI seed attachment tried during this rehearsal was not detected by cloud-init and was replaced with a SATA attachment before proceeding.
4. On the new OS, apply packages_rocky to reinstall OpenZFS. First run zpool import -d /dev/disk/by-id without importing, compare GUID and vdev membership, then use ordinary zpool import -d /dev/disk/by-id zpool. Do not use -f, -F, -X, rollback, or pool creation.
5. Reapply profile_atlas with the lab gates and atlas_create_pool: false. Verify the canary, restored snapshot file in an empty temporary directory, dataset hierarchy, SELinux, and pool health. A second full playbook run should report changed=0. Remove temporary restored files and shut down the VM after testing.

The observed 2026-09-30 pool GUID was 8880368391795119587; the canary SHA-256 was 949701c7a95fadae1fddc21abe846c4312212dbfeb7477948f3188fc3ec34a78. The post-rebuild Ansible run succeeded, a repeat run reported changed=0, 12 datasets and the original snapshot were present, and the pool was healthy. The snapshot-restored file matched content and basic metadata. See atlas-recovery.md for the production runbook and limits.