mirror of
https://github.com/fscotto/infra.git
synced 2026-10-03 13:29:58 +00:00
Compare commits
55 Commits
160d63c02d
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
18eb2d2eb2 | ||
|
|
269fb13665 | ||
|
|
2dfe766b7b | ||
|
|
755f24bc72 | ||
|
|
7bc7f0e645 | ||
|
|
1577eec19d | ||
|
|
e30683c3d1 | ||
|
|
4bd6aafb53 | ||
|
|
9e76309833 | ||
|
|
ed3fee06e8 | ||
|
|
309d64b4ed | ||
|
|
dd33a4f55d | ||
|
|
0028fe8c4d | ||
|
|
12037fcc9a | ||
|
|
3f9a626759 | ||
|
|
31fedb8d44 | ||
|
|
9b5ee77905 | ||
|
|
54fb7d46d7 | ||
|
|
a609e68f42 | ||
|
|
06d3b175cb | ||
|
|
256d758b1a | ||
|
|
9d0013769c | ||
|
|
5b0f415163 | ||
|
|
3401b6137d | ||
|
|
bae6a9f554 | ||
|
|
c37483ba38 | ||
|
|
f491362365 | ||
|
|
5a1047adde | ||
|
|
802cb8c7ba | ||
|
|
0144600a4a | ||
|
|
3d2ef02c98 | ||
|
|
8844d00e24 | ||
|
|
9798fe3a12 | ||
|
|
702283b430 | ||
|
|
797087c66f | ||
|
|
fa1c8c0b82 | ||
|
|
0b6efc9ad8 | ||
|
|
361ee77d72 | ||
|
|
d4e40d423a | ||
|
|
0a5c2ac1a4 | ||
|
|
48a7f57f7e | ||
|
|
defa98c968 | ||
|
|
21e41f4fc1 | ||
|
|
e10c6694f8 | ||
|
|
4c10af3187 | ||
|
|
3ac732751c | ||
|
|
e837b0059b | ||
|
|
7e498514dd | ||
|
|
e7836ea25f | ||
|
|
de2c24d15c | ||
|
|
bd13cb65cc | ||
|
|
77afdda0a3 | ||
|
|
add75d74e9 | ||
|
|
e46a2b22d7 | ||
|
|
a97c64b08f |
435
AGENTS.md
435
AGENTS.md
@@ -25,6 +25,9 @@ Ansible-driven personal infrastructure repo for Fedora and Void desktops, Fedora
|
||||
- Preserve layering `all -> platform -> role -> desktop -> host`.
|
||||
- Keep `ansible/site.yml` small; orchestration belongs there, implementation belongs in roles.
|
||||
- Prefer minimal, targeted edits. Preserve idempotency and existing ordering.
|
||||
- Keep completed one-time cleanup operations out of the playbook. Execute them directly
|
||||
with explicit authorization; retain only the ongoing desired-state configuration and
|
||||
historical documentation, not permanent cleanup flags or tasks.
|
||||
- Use Git Flow branch prefixes: `feature/` for new functionality, `bugfix/` for non-urgent fixes,
|
||||
`hotfix/` for urgent production fixes, `release/` for release preparation, and `support/` for
|
||||
maintained release lines. Do not use abbreviated prefixes such as `feat/`.
|
||||
@@ -48,20 +51,55 @@ Ansible-driven personal infrastructure repo for Fedora and Void desktops, Fedora
|
||||
- Rocky server after activation: `ansible-playbook ansible/site.yml --limit <host> --check --diff`
|
||||
- Atlas NAS: `ansible-playbook ansible/site.yml --limit atlas --check --diff`
|
||||
- Aegis IoT: `ansible-playbook ansible/site.yml --limit aegis --check --diff`
|
||||
- Aegis NFS client layer: `ansible-playbook ansible/site.yml --limit aegis --tags nfs --list-tasks`
|
||||
- Aegis host DNS: `ansible-playbook ansible/site.yml --limit aegis --tags dns --check --diff`
|
||||
- Focused checks:
|
||||
- Emacs is disabled by default; temporary Emacs check: `ansible-playbook ansible/site.yml --limit <host> --tags emacs --check --diff -e emacs_enabled=true`
|
||||
- AI coding agents: `ansible-playbook ansible/site.yml --limit <host> --tags ai_agents --check --diff`
|
||||
- Mail bootstrap: `sh -n scripts/bootstrap_mail.sh` and `shellcheck scripts/bootstrap_mail.sh`
|
||||
- Server compose render: `podman-compose -f /opt/docker/server/docker-compose.yml config` and `systemctl status podman-compose-server`
|
||||
- Server NPM Quadlet: `systemctl status prometheus-npm.service`; the Compose fallback is retired.
|
||||
- Explicit Prometheus legacy cleanup (destructive only without check mode):
|
||||
`ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true`
|
||||
- Atlas media stack:
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff`
|
||||
- Atlas rootless Gitea staging (does not start Gitea):
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags gitea --check --diff`
|
||||
- Atlas canonical Gitea domain (restarts only Gitea on a real configuration change):
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags gitea_public_domain --check --diff`
|
||||
- Atlas iCloudPD storage and boot-started Quadlet:
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags icloudpd --check --diff`
|
||||
- Atlas explicit Gitea host-owner migration (live outage; never a normal run):
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags gitea_owner_migration -e atlas_gitea_owner_migration=true`
|
||||
- Atlas explicit isolated Gitea restore rehearsal (not part of normal runs):
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags gitea_restore -e atlas_gitea_restore_test=true`
|
||||
- Atlas final Gitea replacement gate (dry-run only until a stopped-source export is pulled):
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags gitea_final_restore --check --diff -e atlas_gitea_final_restore=true`
|
||||
- Prometheus final Gitea export helper (dry-run installs only; outage action remains opt-in):
|
||||
`ansible-playbook ansible/site.yml --limit prometheus --tags gitea_final_export --check --diff`
|
||||
- Gitea cutover network configuration before activation:
|
||||
`ansible-playbook ansible/site.yml --limit prometheus --tags gitea_cutover,prometheus_backup --check --diff -e server_gitea_on_atlas=true`
|
||||
and `ansible-playbook ansible/site.yml --limit atlas --tags gitea --check --diff`
|
||||
- Atlas daily Navidrome music copy:
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags music_sync --check --diff`
|
||||
- Atlas network/share hardening:
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags hardening,sharing --check --diff`
|
||||
- Atlas phase-one rootless services:
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 --check --diff`
|
||||
- Prometheus/Atlas WireGuard overlay:
|
||||
`ansible-playbook ansible/site.yml --limit prometheus,atlas --tags wireguard --check --diff`
|
||||
- DuckDNS config only: `ansible-playbook ansible/site.yml --limit prometheus --tags duckdns --check --diff`
|
||||
- Atlas ZFS snapshot retention and scrub timers:
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags snapshots,scrub --check --diff`
|
||||
- Atlas encrypted Borg backup:
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags packages,borg --check --diff`
|
||||
- Atlas Borg progress logging only:
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags borg_logging --check --diff`
|
||||
- Atlas manual offline USB backup and 45Drives Alerts reminder:
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags usb_backup,usb_reminder --check --diff`
|
||||
- Atlas pool, disk, capacity, temperature, and job monitoring:
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags monitoring --check --diff`
|
||||
- Atlas explicit post-restore SELinux relabeling:
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags restorecon --check -e '{"atlas_restorecon_paths":["/zpool/archive"]}'`
|
||||
- Prometheus/Aegis WireGuard gateway:
|
||||
`ansible-playbook ansible/site.yml --limit prometheus,aegis --tags wireguard --check --diff`
|
||||
- Prometheus NPM Quadlet steady state (does not perform a cutover):
|
||||
`ansible-playbook ansible/site.yml --limit prometheus --tags npm_quadlet --check --diff`
|
||||
- DuckDNS config only (skipped on Prometheus): `ansible-playbook ansible/site.yml --limit prometheus --tags duckdns --check --diff`
|
||||
|
||||
## Conventions
|
||||
- Use FQCN Ansible modules.
|
||||
@@ -105,16 +143,24 @@ The dotfile vars follow the same split: `desktop_common_dotfiles` carries mode-i
|
||||
- Windows applications are installed manually and are not managed from the WSL profile.
|
||||
|
||||
## Rocky Server Notes
|
||||
- DuckDNS is rendered by `profile_server` from host-local `server_duckdns_domain` and
|
||||
- Prometheus disables DuckDNS provisioning with `server_duckdns_enabled: false`. Its updater,
|
||||
log and five-minute cron entry were explicitly retired; the external DuckDNS name and Vault
|
||||
token remain untouched. The completed one-time cleanup has no remaining playbook tasks.
|
||||
- When enabled, DuckDNS is rendered by `profile_server` from host-local `server_duckdns_domain` and
|
||||
`vault_duckdns_token`. Keep the rotated token in encrypted Vault or untracked local vars, never in
|
||||
dotfiles. The private `~/duckdns/duck.sh` keeps the existing entrypoint; rendering uses `no_log`
|
||||
and disables diffs. Provisioning does not execute the updater or change its external schedule.
|
||||
- `rocky_server` is a child of both `platform_rocky` and `server`; `prometheus` is its active target.
|
||||
- The target must already provide `server_username` with local sudo access before the profile runs.
|
||||
- The Rocky profile installs Podman and podman-compose, uses firewalld, preserves SELinux enforcement, and renders the
|
||||
existing Nginx Proxy Manager/Gitea Compose stack with a `podman-compose-server` systemd unit. PostgreSQL and
|
||||
Navidrome are no longer part of the desired Prometheus configuration. The role does not stop or remove legacy
|
||||
containers, delete `/opt/postgres/data`, start the Compose stack, update DNS, or cut over traffic.
|
||||
- The Rocky profile installs Podman and podman-compose. Prometheus explicitly retires the legacy
|
||||
Compose unit, files and final-export helper with `server_legacy_stack_retired: true`.
|
||||
Its approved opt-in cleanup removed old application data on 2026-10-03; normal runs do not
|
||||
delete data or recreate the retired files. On Prometheus, Nginx Proxy Manager is now the rootful
|
||||
`prometheus-npm.service` Quadlet with a pinned image digest and the existing `/opt/npm/data` and
|
||||
`/opt/npm/letsencrypt` bind mounts. The rootful `server_web` bridge remains `10.89.0.0/24`.
|
||||
Gitea runs on Atlas; PostgreSQL and Navidrome are absent from the desired Prometheus stack.
|
||||
Normal runs do not delete legacy data, update DNS, or perform an implicit cutover;
|
||||
destructive cleanup requires its explicit tag and opt-in extra-var.
|
||||
- Firewalld enables SSH, Cockpit (`9090/tcp`), HTTP and HTTPS. Nginx Proxy Manager publishes `80/tcp` and
|
||||
`443/tcp`; bind its administration interface only to `127.0.0.1:81` and use `npm-tunnel` from Ikaros or Nymph.
|
||||
Nextcloud remains disabled; do not provision `/srv/nextcloud` directories.
|
||||
@@ -137,10 +183,10 @@ The dotfile vars follow the same split: `desktop_common_dotfiles` carries mode-i
|
||||
- Atlas requires `vault_atlas_admin_password_hash` for Cockpit and, while sharing is enabled,
|
||||
`vault_atlas_samba_password`. The future rootful media stack also requires
|
||||
`vault_atlas_immich_db_password`. Never print these values.
|
||||
- Atlas creates the complete declared hierarchy only under the verified existing or explicitly bootstrapped pool: `work`, `archive`,
|
||||
`archive/app_data`, `archive/app_data/navidrome`, `archive/app_data/syncthing`, `media`, `media/music`,
|
||||
`media/photobook`, `backups`, `backups/services`, and `backup_prometheus`. `backups/services` has a `500G`
|
||||
refreservation. There is no separate legacy `zpool/syncthing` dataset.
|
||||
- Atlas creates the complete declared hierarchy only under the verified existing or explicitly bootstrapped pool: `archive`,
|
||||
`services`, `services/data`, `services/data/navidrome`, `services/data/syncthing`, `media`, `media/music`,
|
||||
`media/photobook`, `backup`, `backup/hosts`, and `backup/hosts/prometheus`. `backup` has a `500G`
|
||||
reservation covering its descendants. `archive` is the SMB-shared raw-data namespace; container state is never beneath it.
|
||||
- The `immich` system account is fixed to UID/GID `1100`, has no login shell or `wheel` membership, and receives only
|
||||
the `video` and `render` supplementary groups. Immich's rootful Quadlets run as `1100:1100`; Server and ML receive
|
||||
`/dev/dri`, while the Photobook external library is read-only at `/external/photobook`.
|
||||
@@ -149,52 +195,308 @@ The dotfile vars follow the same split: `desktop_common_dotfiles` carries mode-i
|
||||
`1100`. Targeted SELinux is enforced persistently; a required reboot is reported but never initiated automatically. The primary LAN interface is assigned explicitly to the managed firewalld zone, and firewall rules are applied before NFS or SMB are started; their service state and TCP listeners are then verified. SMB3 exposes `Archive` to Vault-backed authorized accounts on mandatory encrypted, signed SMB3 over TCP/445 only and admits the configured LAN without host-specific exclusions.
|
||||
- Atlas NPM and Immich share a rootful Podman network. NPM publishes HTTP/HTTPS, but its administration port remains
|
||||
bound to `127.0.0.1:81`; do not expose it directly to the LAN or Internet.
|
||||
- `profile_backend_phase1` is limited to rootless Navidrome and Syncthing user Quadlets on Atlas. Official Navidrome
|
||||
`0.63.2` uses SQLite below `/data` and does not support `ND_DATABASE_URL` or an external PostgreSQL backend; do not
|
||||
recreate the obsolete Prometheus `navidromedb` service. The role requires the storage role's `zpool/media/music`,
|
||||
`zpool/archive/app_data`, `zpool/archive/app_data/navidrome`, and `zpool/archive/app_data/syncthing` datasets at
|
||||
their exact paths. It never creates the pool.
|
||||
- Keep `backend_phase1_start_services` false until the stopped Prometheus Navidrome data directory has been copied to
|
||||
Atlas and its SQLite database verified. The playbook renders the target but never migrates or deletes application
|
||||
data; after cutover, set the flag true to enable and start Navidrome and Syncthing.
|
||||
- Phase 1 must not change Prometheus' existing NPM deployment. NPM continues to be managed exactly by `profile_server`;
|
||||
use `10.0.0.2:4533` for Navidrome and `10.0.0.2:8384` for the Syncthing GUI. Syncthing does not use host networking:
|
||||
its GUI, transfer, QUIC and discovery ports are explicitly published only on `10.0.0.2`; native transfer/discovery does not use the HTTP proxy.
|
||||
- `wireguard_overlay` manages the required `wg0` path between Prometheus and Atlas, persists private keys only on their
|
||||
respective hosts, exchanges only derived public keys, and verifies a real peer handshake. The initial run must
|
||||
include both hosts. Prometheus
|
||||
opens `51820/udp`; after creating the WireGuard firewalld zone, restore Prometheus' rootful Podman networking with
|
||||
`podman network reload --all` so the existing proxy stack retains container DNS. The Atlas backend role admits
|
||||
service ports only in the WireGuard firewalld zone.
|
||||
- `profile_backend_phase1` temporarily runs rootless Navidrome and Syncthing on Atlas until Uranus replaces
|
||||
them. It binds only to Atlas' LAN IP, never `wg0`; Navidrome and the Syncthing GUI admit only Aegis as
|
||||
the source-NAT gateway, while native Syncthing ports admit the configured LAN. It initializes fresh
|
||||
state only and never migrates or deletes source application data. The enabled rootless
|
||||
`atlas-music-sync.timer` copies `/zpool/archive/Music` to `/zpool/media/music` daily at 00:45
|
||||
Europe/Rome without deleting destination files; it requires both datasets to be mounted.
|
||||
- `wireguard_overlay` manages `wg0` between Prometheus (`10.0.0.1`) and Aegis (`10.0.0.2`). It persists private
|
||||
keys only on their respective hosts, exchanges only derived public keys through Ansible, and verifies a real peer
|
||||
handshake. Prometheus opens `51820/udp`; Aegis is the LAN gateway. Its persistent IPv4 forwarding, narrowly scoped
|
||||
WireGuard-to-LAN firewalld policy, and source masquerading permit Prometheus to reach LAN services without a static
|
||||
route on the router. Prometheus includes `192.168.178.0/24` in Aegis' peer `AllowedIPs`; add the Uranus VIP there
|
||||
when it is assigned. After a firewalld reload, restore Prometheus' rootful Podman networking with
|
||||
`podman network reload --all` so the existing proxy stack retains container DNS.
|
||||
|
||||
## Atlas NAS TODO
|
||||
- Provide the required Vault variables and validate the first remote bootstrap on the real Rocky Linux 9 host.
|
||||
Before the first apply, confirm the pool, mountpoints, LAN subnet and firewalld zone. Keep
|
||||
`atlas_manage_media_stack` disabled until `/dev/dri`, the container paths and the Immich database secret are validated.
|
||||
- Validate the complete baseline on the target: OpenZFS kmod loading, existing pool import, dataset
|
||||
mounts, SSH reconnect, Cockpit and all selected 45Drives plugins, NFSv4, SMB and Syncthing.
|
||||
- Finalize dataset properties and the shared UID/GID, group and POSIX ACL model; test the same files
|
||||
through both NFS and SMB before considering multiprotocol access complete.
|
||||
- Add Ansible-managed ZFS snapshot retention and scrub timers. Use Cockpit Scheduler for visibility
|
||||
or manual operations, not as the only source of configuration, and never automate snapshot rollback.
|
||||
- Manage the Syncthing star topology, device IDs, folders, folder modes, ignore rules and protected GUI
|
||||
or API access for the selected clients.
|
||||
- Validate the managed WireGuard path and its LAN/VPN-only firewalld rules before enabling remote services;
|
||||
never expose SSH, Cockpit, NFS, SMB or Syncthing through public port forwarding.
|
||||
- Add the Atlas-initiated least-privilege Prometheus backup pull: Prometheus exposes only prepared
|
||||
Completed validation: the existing RAIDZ2 pool and datasets, SELinux, LAN firewall, SSH, Cockpit with
|
||||
the selected 45Drives plugins, encrypted SMB3 `Archive`, the Aegis-only NFSv4 `photobook` export, and the
|
||||
Prometheus--Aegis WireGuard gateway are operational. The gateway handshake, forwarding, source masquerading,
|
||||
and TCP reachability to Atlas were verified. Temporary Navidrome and Syncthing are available through their
|
||||
manual NPM Proxy Hosts; Syncthing uses `/data/Org` backed by the SMB-shared Archive dataset. Aegis has also
|
||||
validated NFSv4.2 read, write, delete, and `all_squash` mapping to UID/GID `1100` end-to-end. The ZFS
|
||||
snapshot timers are active; a recursive hourly snapshot and scheduled retention prune completed
|
||||
successfully. The first monthly scrub remains a runtime check.
|
||||
|
||||
### Priority 1 - Data protection
|
||||
- [x] Deploy Ansible-managed recursive ZFS snapshots with 24 hourly, 30 daily, 8 weekly, and 12 monthly
|
||||
generations, plus a monthly scrub on the first Sunday at 03:00. The timers and first hourly snapshot were
|
||||
verified on Atlas. Cockpit Scheduler is for visibility or manual operations only, and snapshot
|
||||
rollback is never automated.
|
||||
- [ ] Verify the first monthly ZFS scrub from its actual service result. Scheduled retention pruning
|
||||
was observed on 2026-09-30; timer activation alone does not establish a successful scrub.
|
||||
- [x] Activate and validate the encrypted offsite Borg backup to the Hetzner Storage Box. Atlas uses the
|
||||
dedicated SSH identity, pinned ED25519 host key, Vault-backed `repokey` encryption, and a locked
|
||||
non-login `borg` account with no sudo or supplementary groups. The initial snapshot-consistent backup,
|
||||
Borg repository check, and temporary-directory restore completed successfully; the restored `Archive`
|
||||
tree matched the live data, and temporary snapshots and mounts were removed. The exported recovery key
|
||||
was copied offline. Daily backup retries and logging, 30 daily, 8 weekly and 12 monthly archives,
|
||||
compaction, and monthly repository checks are enabled. Runs report a ZFS-based estimated percentage.
|
||||
On 2026-09-30 a successful incremental run also removed the stale 2026-09-29 snapshot and its own
|
||||
temporary snapshot after exit; the earlier `RuntimeDirectory` cleanup failure is resolved.
|
||||
- [x] Populate `/zpool/archive` with the currently available data so offsite and offline backup tests run
|
||||
against a representative load.
|
||||
- [x] Evaluate Borg against the populated pool. The 2026-09-29 archive took 1 h 32 min for 2.18 TB
|
||||
original / 2.04 TB compressed data, with 13.49 GB deduplicated size; retention and compaction
|
||||
succeeded. On 2026-09-30 a subsequent incremental archive completed in about 22 seconds with
|
||||
successful cleanup. The monitor reported 37% Storage Box quota used. These are observed runs, not
|
||||
a guarantee of future duration or compression ratio.
|
||||
- [x] Add the UUID-bound offline USB backup with versioned rsync, locking, capacity checks, verification,
|
||||
safe unmounting and a tested restore procedure; never trigger it for an arbitrary USB disk. The
|
||||
LUKS/ext4 identities were read-only verified; the manual service and 45Drives Alerts reminder timer were
|
||||
deployed on Atlas. Interactive LUKS unlock is part of the manual service; only the reminder is
|
||||
scheduled for the first Saturday of each month at 10:00 Europe/Rome via the existing 45Drives
|
||||
notifier. A manual test produced an Alerts notification, not an email. The first USB attempt failed
|
||||
on a `security.selinux` xattr and was interrupted; the xattr filter is deployed and the temporary
|
||||
recursive snapshot and open LUKS mapper were cleaned up. A later run reported checksum verification
|
||||
and published the USB version, but failed while removing host-namespace ZFS snapshot mounts. Those
|
||||
exact mounts and snapshots were cleaned up. An `ExecStopPost` helper now removes only the named
|
||||
temporary snapshot after the backup process exits. A new full run checksum-verified and published a
|
||||
USB version; the service ended successfully, the mapper closed, no temporary USB snapshot remained,
|
||||
and the pool was healthy. On 2026-09-25 an independent, read-only USB restore test copied one file from
|
||||
the published `atlas/latest` version into `/var/tmp` and matched contents, owner, mode, size, mtime and
|
||||
POSIX ACL. The temporary copy and mount were removed, the mapper closed, and the pool remained healthy.
|
||||
- [x] Test restores independently from a ZFS snapshot, Borg, and the offline USB backup before relying on
|
||||
any backup path. The earlier Borg temporary-directory restore passed. On 2026-09-25 a separate,
|
||||
read-only ZFS snapshot test restored one file to `/var/tmp`, confirmed matching contents, ownership,
|
||||
mode, mtime and ACL, then removed its temporary copy and on-demand mount. This is a file-level smoke
|
||||
test, not full dataset recovery. An independent USB file restore passed on 2026-09-25 with matching
|
||||
content and metadata; the later scaled OS-rebuild rehearsal is documented under Priority 2.
|
||||
- [x] Add monitoring and alerting for pool health, scrub/resilver, SMART data, temperatures, free space,
|
||||
snapshot/local-backup growth, Hetzner Storage Box quota, and failed maintenance or backup timers.
|
||||
The half-hourly Atlas health monitor and systemd final-failure hooks are deployed. A live probe
|
||||
found no issues; the service and timer succeeded, and a labelled 45Drives Alerts test notification
|
||||
was submitted. Alerts are deduplicated; email delivery is not claimed. The Storage Box quota probe
|
||||
runs `df -m` over the dedicated pinned-key SSH identity and does not open the Borg repository.
|
||||
The failed-job hook was corrected to pass the literal systemd unit name; its expansion was verified
|
||||
on Atlas, but a new real failure notification has not been deliberately triggered.
|
||||
|
||||
### Priority 2 - NAS operability and recovery
|
||||
- [x] Document and test disaster recovery in `docs/atlas-recovery.md`: the operator confirmed Vault
|
||||
and Borg recovery material is available offline; provisional targets are RPO 24h/RTO 72h. On
|
||||
2026-09-30 an isolated small Rocky VM was rebuilt with the Atlas Ansible roles, imported its
|
||||
preserved RAIDZ2 pool without force/rewind, and restored a file from the preserved snapshot;
|
||||
the second Ansible run was idempotent. Earlier independent production ZFS, USB, and Borg file
|
||||
restore tests remain separate evidence. A production-size full restore, unclean import, and
|
||||
measured 24h/72h compliance are not claimed.
|
||||
- [x] Define a controlled Rocky kernel/OpenZFS update and reboot procedure in
|
||||
`docs/atlas-updates.md`. The first real change-window execution is not yet
|
||||
validated; the procedure never reboots automatically or upgrades pool features.
|
||||
- [x] Add the Atlas-initiated least-privilege Prometheus backup pull: Prometheus exposes only prepared
|
||||
read-only dumps through a dedicated account and Atlas retains the private SSH key, pinned host key,
|
||||
atomic pull, verification, retention and systemd service/timer.
|
||||
- Add the encrypted offsite backup with Borg to a Hetzner Storage Box: use a dedicated SSH identity,
|
||||
pin the host key, keep Borg repository credentials and encryption material in Vault, use
|
||||
snapshot-consistent sources, and manage retries, logging, pruning, repository checks and restores.
|
||||
- Add the UUID-bound offline USB backup with versioned rsync, locking, capacity checks, verification,
|
||||
safe unmounting and a tested restore procedure; never trigger it for an arbitrary USB disk.
|
||||
- Add monitoring and alerting for pool health, scrub/resilver, SMART data, temperatures, free space and
|
||||
failed backup timers, plus a controlled Rocky kernel/OpenZFS update and reboot procedure.
|
||||
- Document and test disaster recovery: rebuild Atlas with Ansible, import the existing pool, restore
|
||||
from snapshot/USB/Hetzner, preserve Vault and Borg recovery material offline, and define RPO/RTO.
|
||||
- Optionally design iCloud photo ingestion as a separate workflow after the storage and backup layers
|
||||
are validated; do not make it a dependency of the Atlas baseline.
|
||||
atomic pull, verification, retention and systemd service/timer. The dedicated key/account and unit
|
||||
files are deployed; live read-only SSH, shell denial, and write denial were verified. On 2026-09-30
|
||||
a manual export, Atlas pull, checksum verification, and temporary restore passed; both SQLite
|
||||
databases passed integrity checks and a restored Git repository passed `git fsck`. Both daily
|
||||
timers are enabled for 02:00/03:00 Europe/Rome. On 2026-10-01 their first scheduled export and
|
||||
pull succeeded: Atlas verified the payload checksum and published `20261001T000001Z` as `latest`.
|
||||
- [x] Decide whether a common SMB/NFS namespace is required: no. `Archive` (SMB) and `photobook` (NFS)
|
||||
remain intentionally distinct; `docs/atlas-sharing-decision.md` records the decision. No ACL or export
|
||||
change is authorized by this decision.
|
||||
|
||||
### Priority 3 - Service expansion
|
||||
- [x] Populate `/zpool/media/music` and validate Navidrome. On 2026-09-30, 21,158 files
|
||||
(93,937,810,350 regular-file bytes) were copied from `/zpool/archive/Music` using a temporary
|
||||
ZFS snapshot; a checksum-based rsync dry run found no differences or extra files. Navidrome saw
|
||||
all files through its read-only mount, completed a scan, indexed 18,168 tracks, and responded
|
||||
over HTTP. Some imported playlists still reference obsolete Windows paths. The source was left
|
||||
intact and the temporary snapshot was removed.
|
||||
- [x] Schedule a daily, non-deleting copy from `Archive/Music` to the separate Navidrome music
|
||||
dataset. The rootless `atlas-music-sync.timer` is enabled for 00:45 Europe/Rome; a manual
|
||||
idempotent service run succeeded on 2026-10-01. The first scheduled run triggered at
|
||||
00:45 CEST on 2026-10-02 and exited successfully (`Result=success`, status 0); the next
|
||||
run is scheduled for 2026-10-03 00:45 CEST.
|
||||
- [x] Design the staged Prometheus-to-Atlas Gitea migration in `docs/atlas-gitea-migration.md`.
|
||||
The approved topology keeps NPM on Prometheus and moves HTTPS and public SSH (TCP/2222) together;
|
||||
Gitea runs as an `admin`-owned rootless user Quadlet on Atlas with an internal `gitea` user.
|
||||
The rootful-to-rootless data-layout
|
||||
conversion passed an isolated restore rehearsal. The later partial cutover is tracked below.
|
||||
- [x] Prepare the dedicated Atlas Gitea dataset, non-login UID/GID 1101 with a separate rootless Podman
|
||||
sub-ID range, and disabled user Quadlet. On 2026-10-01 the targeted Ansible run and a second idempotent
|
||||
run passed; the generated unit was inactive, with no staging HTTP/SSH listener. POSIX ACLs on only the
|
||||
service-namespace parents grant this account traversal without access to sibling datasets.
|
||||
- [x] Perform an isolated rootless restore rehearsal from the verified Prometheus backup. On 2026-10-01
|
||||
the SHA-256-checked selective extraction and path/SSH conversion succeeded; SQLite `quick_check`
|
||||
passed, all 33 repositories passed `git fsck`, and source/target public SSH host-key fingerprints
|
||||
matched. The pinned rootless image answered HTTP and listened on internal SSH/2222 with
|
||||
`--network none`; the temporary container was removed and the Quadlet stayed inactive. A second
|
||||
restore run made no changes. This is a rehearsal copy, not the final consistent cutover copy.
|
||||
- [x] Verify ZFS and Borg coverage of the staged Gitea dataset. On 2026-10-01 the managed recursive
|
||||
hourly snapshot `atlas-auto-hourly-20261001T193401Z` included it, and the managed incremental
|
||||
Borg archive `atlas-20261001T193420Z` included its database. A private one-file restore from
|
||||
each independently matched the staged database and passed SQLite `quick_check`; temporary files
|
||||
and snapshot mounts were removed, the Borg service ended successfully, and the pool was healthy.
|
||||
- [x] Include the new Gitea dataset in a UUID-bound offline USB version and test a file restore
|
||||
before accepting production writes. The operator's 2026-10-01 manual run published version
|
||||
`20261001T201220Z-254397` successfully on 2026-10-02. Its Gitea database was restored to a
|
||||
temporary directory from a read-only mount: contents, owner, group, mode, size, mtime and POSIX
|
||||
ACL matched, and SQLite `quick_check` passed. Temporary files and mounts were removed, LUKS
|
||||
was closed, and the pool remained healthy. A redundant run was stopped during verification;
|
||||
its temporary snapshot was cleaned up and the service's resulting failed state was reset.
|
||||
- [x] Install a separate opt-in final Gitea export helper on Prometheus. Its 2026-10-01 targeted
|
||||
deployment and `bash -n` passed while Gitea and NPM stayed running. It refuses an active export
|
||||
timer, stops only Gitea, verifies SQLite, publishes a checksum-verified Gitea-only version for
|
||||
Atlas' existing pull, and leaves the source stopped on success. It was invoked on 2026-10-02
|
||||
after the export timer was stopped; version `20261002T071525Z` was pulled and verified on Atlas.
|
||||
- [x] Prepare the Atlas final-restore gate without replacing the rehearsal: it accepts only a
|
||||
checksum-verified `gitea-cutover` export, refuses a running target, stages and validates the new
|
||||
layout before replacing the marked rehearsal, and rolls back a failed swap. Synthetic success
|
||||
and rollback tests passed on 2026-10-01. On 2026-10-02 the final gate replaced the rehearsal;
|
||||
SQLite `quick_check`, all 33 repository `git fsck` checks, checksum and SSH host-key comparison passed.
|
||||
- [x] Start the rootless Atlas Gitea Quadlet and move the primary HTTPS route. On 2026-10-02 Atlas
|
||||
answered HTTP 200 through the Aegis gateway. NPM stayed on Prometheus; its variable upstream
|
||||
required a managed Nginx `server_proxy.conf` override because runtime DNS ignores Compose
|
||||
`extra_hosts`. The primary public HTTPS page and API returned 200, and `git ls-remote` succeeded
|
||||
for a representative repository after NPM restart; the Navidrome and Syncthing Proxy Hosts also
|
||||
responded. The source
|
||||
Gitea container was removed from the desired Compose stack without deleting its data; the
|
||||
Prometheus backup export timer resumed for NPM only. A post-cutover recursive ZFS snapshot and
|
||||
encrypted Borg archive `atlas-20261002T073044Z` completed successfully.
|
||||
- [x] Move the live Gitea Quadlet and dataset from the legacy host `gitea` account to `admin`
|
||||
after a disposable snapshot-copy test of the pinned derived image. On 2026-10-02 the explicit
|
||||
outage run stopped only legacy Gitea, made safety snapshot
|
||||
`zpool/services/data/gitea@gitea-owner-migration-20261002T100104`, changed dataset ownership,
|
||||
and validated loopback staging (HTTP 200, internal `gitea` UID/GID 1000, SQLite `quick_check`)
|
||||
before promoting the `admin` Quadlet. Production LAN and public HTTPS returned 200; Navidrome
|
||||
and Syncthing remained active, the pool was healthy, and the normal Gitea run changed nothing.
|
||||
The old host account and data on Prometheus remain preserved; the old Atlas Quadlet and its
|
||||
parent-dataset traverse ACL were removed. A subsequent normal run changed nothing.
|
||||
- [x] Validate public Gitea SSH/2222 and an authenticated read from Ikaros. After the VPS
|
||||
firewall was opened on 2026-10-02, TCP/2222 connected, the public ED25519 host-key
|
||||
fingerprint matched Atlas, Gitea authenticated `fscotto` using the `ikaros` key, and
|
||||
`git ls-remote` returned HEAD for `fscotto/infra.git` over public SSH.
|
||||
- [x] Validate authenticated SSH pull and push. On 2026-10-02 the operator reported both
|
||||
operations working through the public SSH endpoint; the earlier agent-run `git ls-remote`
|
||||
remains the independent read-only check. The agent did not perform a test push.
|
||||
- [x] Validate Gitea login and write via HTTPS. On 2026-10-03 the operator confirmed
|
||||
authenticated web login and Git clone/pull/push through the public HTTPS endpoint. Do not
|
||||
restart the stale source Gitea after Atlas has accepted writes.
|
||||
- [ ] Design and deploy Nextcloud as another explicitly temporary Atlas service before Uranus. Give it
|
||||
separate persistent application, database, and cache storage; keep credentials in Vault; publish it only
|
||||
through NPM over the Prometheus--Aegis gateway; and define backup, upgrade, and eventual Uranus-migration
|
||||
procedures before exposing user data. Do not deploy Nextcloud before the data-protection checklist is complete.
|
||||
- [x] Move Gitea canonical HTTPS and SSH hostname to `git.fscotto.co` on
|
||||
2026-10-03 through Ansible. Only Gitea restarted; second run changed nothing.
|
||||
HTTPS and authenticated SSH reads returned the same repository HEAD.
|
||||
The new NPM hostnames passed TLS/HTTP checks; old DuckDNS Proxy Hosts were
|
||||
observed disabled. Details are in `docs/domain-fscotto-co.md`.
|
||||
- [x] Confirm login on the new Gitea hostname and update remaining client remotes/integrations.
|
||||
The operator confirmed completion on 2026-10-03; the agent did not perform a test push.
|
||||
- [x] Remove obsolete DuckDNS NPM Proxy Hosts, unused certificates and the old upstream override.
|
||||
The operator confirmed completion on 2026-10-03; no new agent runtime check was performed.
|
||||
- [x] Review and remove completed one-time procedures from the playbook.
|
||||
The operator confirmed completion on 2026-10-03.
|
||||
- [x] Retire Prometheus' local DuckDNS updater on 2026-10-03 through Ansible:
|
||||
the five-minute cron entry and private updater/log directory were removed.
|
||||
Provisioning is disabled; repeat cleanup changed nothing. HTTPS services, private NPM
|
||||
administration and the export timer stayed healthy. The external name and Vault token
|
||||
remain untouched for possible future use on a local host.
|
||||
- [ ] Keep `atlas_manage_media_stack` disabled until the future Immich deployment has validated `/dev/dri`,
|
||||
container paths, and the required Vault database secret.
|
||||
|
||||
### Priority 4 - Optional workflows
|
||||
- [x] Deploy the declared Atlas iCloudPD state dataset and inactive rootless `admin` Quadlet.
|
||||
Photos belong under `/zpool/archive/Pictures/iCloudPD`; private config/MFA state belongs in
|
||||
`zpool/services/data/icloudpd`. Photobook remains reserved for Immich. Ansible now renders
|
||||
`icloudpd.conf` with the Apple ID from the existing Vault key, but does not store the password
|
||||
or manage MFA. Automatic startup was approved on 2026-10-03; the Quadlet now
|
||||
uses `WantedBy=default.target` and Ansible keeps the service running.
|
||||
The isolated no-network layout test is documented in
|
||||
`docs/atlas-icloudpd-migration.md`. On 2026-10-02 Atlas deployment and a second idempotent run
|
||||
passed; no app config existed at deployment. A manual first start on 2026-10-02 generated
|
||||
`icloudpd.conf`; an Ansible run then replaced it with a private mode-0600 Vault-backed template
|
||||
and an idempotent second run. The image later expanded the config, so Ansible now seeds it
|
||||
only when absent and maintains the declared fields. Its launcher requires `traceroute`; the
|
||||
rootless Quadlet grants only `NET_RAW`, tested in isolation and after restart. The service
|
||||
was subsequently initialized interactively; initial ingestion is tracked below.
|
||||
- [x] Retire Aegis iCloudPD completely. The operator authorized deleting its Quadlet,
|
||||
`/var/lib/icloudpd` data, and MFA state despite an unaudited container overlay. After two
|
||||
interactive-sudo runs on 2026-10-02, the unit is `not-found`/`inactive`, the Quadlet and state
|
||||
directory are absent, and AdGuard remains active. The temporary retirement tasks have since
|
||||
been removed from the Aegis role; it no longer manages iCloudPD.
|
||||
- [x] Validate Atlas iCloudPD authentication and initial ingestion. On 2026-10-03 the active
|
||||
rootless service logged `All photos and videos have been downloaded` at 02:16 and reported
|
||||
completion for the user. The destination held 11,658 files (86,020,430,015 bytes); the preceding 24h
|
||||
logs showed download activity without authentication failures or errors. A later read-only check
|
||||
found the service still active. This confirms the initial download, not the next daily cycle.
|
||||
- [x] Declare HEIC decoding for Fedora graphical desktops without converting the originals on Atlas.
|
||||
The Fedora role installs RPM Fusion Free with a pinned signing-key fingerprint and
|
||||
`libheif-freeworld` on Ikaros and Nymph. The package was confirmed installed on Ikaros on
|
||||
2026-10-03; Nymph deployment and an actual image-opening test were not observed.
|
||||
- [ ] Validate Atlas iCloudPD filesystem/SELinux/SMB access, the next daily sync, ZFS/Borg/USB
|
||||
backup inclusion, and isolated restore of photos and private state. A recursive hourly snapshot
|
||||
of `zpool/archive` exists after ingestion, but no iCloudPD-specific backup version or restore
|
||||
has been verified. The first monthly scrub remains a separate open data-protection check.
|
||||
|
||||
## Prometheus NPM Quadlet cutover
|
||||
- [x] Stage a rootful NPM Quadlet using the exact running image and the existing data/certificate
|
||||
mounts, bridge subnet, public HTTP/HTTPS ports, and loopback-only administration port.
|
||||
The generated service depends on `server-web-network.service` and is wanted by `multi-user.target`.
|
||||
- [x] Take and verify the stopped-source export before switching owners. Version
|
||||
`20261003T091009Z` was pulled to Atlas and its NPM SQLite database checked in isolation.
|
||||
- [x] Cut over NPM to `prometheus-npm.service` on 2026-10-03. The legacy Compose unit is inactive
|
||||
and disabled; the Quadlet is active with zero recorded restarts. Public Gitea and Syncthing
|
||||
HTTPS returned 200 with valid TLS, while public TCP/81 remained unreachable.
|
||||
- [x] Validate the post-cutover backup path. The export and Atlas pull published
|
||||
`20261003T091633Z`; checksum, SQLite `quick_check`, ten proxy hosts, six certificate records,
|
||||
both Quadlet files were present, and the complete Let's Encrypt tree (70 regular files plus
|
||||
12 symlinks) matched the live data. A targeted normal Ansible run changed nothing. Details and rollback
|
||||
boundaries are in `docs/prometheus-npm-quadlet.md`.
|
||||
- [x] Remove only unused Gitea, Navidrome and PostgreSQL images with opt-in
|
||||
Ansible tasks on 2026-10-03. Second run changed nothing; NPM stayed active
|
||||
with zero restarts, HTTP/HTTPS passed, backup timer and SSH proxy stayed active.
|
||||
This image-only step preserved data and fallback; the later approved deletion is tracked below. Validation:
|
||||
`ansible-playbook ansible/site.yml --limit prometheus --tags server_image_cleanup --check --diff -e server_legacy_image_cleanup=true`
|
||||
- [x] Complete explicitly approved old-data and Compose fallback removal on 2026-10-03.
|
||||
Backup paths and mount dependencies were reconciled before deletion; repeat cleanup changed
|
||||
nothing. The normal Compose/template/helper check did not recreate retired files.
|
||||
A separately approved manual export/pull published `20261003T112906Z`; checksum and isolated
|
||||
SQLite restore passed with ten proxy hosts and both Quadlet definitions. NPM, primary HTTPS,
|
||||
WireGuard, SSH proxy and backup timer remained healthy; existing backup archives were preserved.
|
||||
- [x] Retire the unused secondary Gitea hostname `git.ov-ad3410.infomaniak.ch`
|
||||
on 2026-10-03. Its NPM Proxy Host was already soft-deleted and had no
|
||||
associated certificate. Its Ansible domain and runtime override were removed;
|
||||
nginx -t and reload passed without restarting NPM. Primary HTTPS returned 200
|
||||
with valid TLS. At that step only `git.fscotto.duckdns.org` remained declared;
|
||||
the subsequent domain transition and operator-confirmed cleanup are tracked above.
|
||||
- [ ] Observe the first scheduled export and Atlas pull after the cutover; the manual end-to-end
|
||||
cycle passed, but the next unattended cycle has not yet occurred.
|
||||
|
||||
## Cerberus Management Node (Deferred)
|
||||
`cerberus` is postponed until the office in the new house is physically set up. It is not an inventory
|
||||
host and this section is a design and implementation backlog, not authorization to provision it early.
|
||||
|
||||
The planned node is a Lenovo ThinkCentre M700 Tiny with an Intel Core i3-6100T, 8 GB RAM, a 256 GB SSD,
|
||||
and native 1 Gbps Ethernet. It will connect to a multi-input KVM switch using a passive DisplayPort-to-HDMI
|
||||
cable, sharing the monitor and peripherals with Ikaros. Fedora Sericea (immutable Fedora with the Sway
|
||||
Wayland compositor) is the intended OS. Cerberus is an isolated management plane: a dedicated Toolbox
|
||||
environment will run Ansible for future `uranus` cluster provisioning. Rootless Podman will host Grafana,
|
||||
Prometheus, and Loki. The 256 GB local SSD is the hot tier retaining metrics and logs for 30 days; scheduled,
|
||||
validated exports of older historical data will use a dedicated Atlas NFS dataset as cold storage.
|
||||
|
||||
### Implementation plan
|
||||
- [ ] Confirm the office, KVM switch, passive DisplayPort-to-HDMI path, shared monitor/peripherals, and native
|
||||
1 Gbps Ethernet are physically operational before adding Cerberus to inventory.
|
||||
- [ ] Install and update Fedora Sericea with Sway; document the immutable-host lifecycle and keep host changes
|
||||
declarative rather than treating the base OS as a mutable workstation.
|
||||
- [ ] Model Cerberus as its own host with independent platform, role, desktop, network, and storage inputs;
|
||||
do not repurpose Ikaros variables or make it a Uranus cluster member.
|
||||
- [ ] Provision an isolated Toolbox-based Ansible controller with the required collections and a reproducible
|
||||
project checkout; define its least-privilege SSH access, known-host handling, and Vault workflow without
|
||||
storing secrets in the image or repository.
|
||||
- [ ] Define the explicit Uranus provisioning workflow from Cerberus, including inventory boundaries,
|
||||
validation-only runs, and separate approval for any destructive cluster operation.
|
||||
- [ ] Design rootless Podman/Quadlet services for Grafana, Prometheus, and Loki, including persistent local
|
||||
state, service ownership, LAN exposure/authentication, resource limits, updates, and backups.
|
||||
- [ ] Size and enforce a 30-day local hot-retention policy for metrics and logs on the 256 GB SSD; validate
|
||||
actual disk growth and alert before capacity exhaustion.
|
||||
- [ ] Create and validate a dedicated Atlas NFS cold-storage dataset and least-privilege export for Cerberus;
|
||||
do not use a broad existing share or couple it to unrelated Atlas application state.
|
||||
- [ ] Implement scheduled, idempotent exports of data older than 30 days to the Atlas NFS cold tier, with
|
||||
locking, capacity checks, integrity verification, retention rules, failure monitoring, and a tested restore.
|
||||
- [ ] Validate management-plane recovery: rebuild Cerberus, restore observability history from Atlas, and
|
||||
confirm that Uranus provisioning can resume without depending on unreproducible local state.
|
||||
|
||||
## Coding Agent Notes
|
||||
- Shared agent definitions and lifecycle flags live in `ai_agents` in `ansible/inventory/group_vars/all.yml`.
|
||||
@@ -221,14 +523,19 @@ The dotfile vars follow the same split: `desktop_common_dotfiles` carries mode-i
|
||||
- `aegis` is a remote Fedora IoT Raspberry Pi 4 node. Bootstrap it once with
|
||||
`ansible/bootstrap/aegis.bu`; the remaining configuration is applied by `profile_aegis` over SSH.
|
||||
- Fedora IoT is immutable. Do not add it to mutable Fedora package or shared dotfile roles.
|
||||
- `profile_aegis` owns rootful Podman Quadlets, persistent container state under `/var/lib`, the
|
||||
Podman auto-update timer, LAN-restricted firewalld rules, and SSH hardening. Keep
|
||||
`aegis_lan_subnet` and `aegis_adguard_web_port` host-specific; SSH permits only the declared
|
||||
- `profile_aegis` owns the `nfs-utils` and `wireguard-tools` rpm-ostree layers and reports the required reboot
|
||||
without initiating it. `wireguard_overlay` then configures Aegis as the WireGuard LAN gateway with persistent IPv4
|
||||
forwarding, a scoped inter-zone policy, and source masquerading. It also owns rootful Podman Quadlets, persistent container
|
||||
state under `/var/lib`, the Podman auto-update timer, LAN-restricted firewalld rules, and SSH hardening. Keep
|
||||
`aegis_lan_subnet`, `aegis_adguard_web_port`, and `aegis_network_connection_uuid` host-specific;
|
||||
SSH permits only the declared
|
||||
key-authenticated users, never root or password authentication. Keep Apple IDs and other
|
||||
credentials in Vault and use `no_log` for their rendering.
|
||||
- `aegis_adguard_web_port` defaults to `80`. The initial AdGuard Home wizard port `3000` is intentionally unmanaged: open and close it manually only while
|
||||
completing initial setup. Disable the local systemd-resolved stub through `profile_aegis` before
|
||||
AdGuard binds port 53; keep
|
||||
`/etc/resolv.conf` linked to `/run/systemd/resolve/resolv.conf` so Aegis retains router-provided DNS.
|
||||
- iCloudPD requires post-deployment interactive MFA initialization; its cookie/configuration state is
|
||||
persisted in `/var/lib/icloudpd/config`.
|
||||
`/etc/resolv.conf` linked to `/run/systemd/resolve/resolv.conf`. LAN clients may use AdGuard, but
|
||||
Aegis must use the independent upstream DNS declared by `aegis_host_dns_servers` so Greenboot does
|
||||
not depend on the AdGuard container during startup.
|
||||
- Aegis iCloudPD has been retired and is no longer managed by this role. Its service, Quadlet,
|
||||
data, and MFA state were removed with the operator's explicit authorization.
|
||||
|
||||
387
README.it.md
387
README.it.md
@@ -95,6 +95,27 @@ Nota sullo stato attuale del playbook principale:
|
||||
- `ansible/site.yml` applica il profilo server Rocky a `prometheus` con DNF, systemd, dotfiles server e firewalld
|
||||
- `ansible/site.yml` applica il profilo NAS Rocky su `atlas` tramite SSH remoto
|
||||
|
||||
## Nodo pianificato e posticipato: Cerberus
|
||||
|
||||
`cerberus` e un nodo di management **posticipato**, in attesa dell'allestimento
|
||||
fisico dell'ufficio nella nuova casa. Non e ancora presente nell'inventory e non
|
||||
esistono ruoli o playbook che lo prendano come target.
|
||||
|
||||
L'hardware previsto e un Lenovo ThinkCentre M700 Tiny (Intel Core i3-6100T,
|
||||
8 GB di RAM e SSD da 256 GB) con Ethernet nativa a 1 Gbps. Condividera monitor
|
||||
e periferiche di Ikaros tramite uno switch KVM a ingressi multipli, usando un
|
||||
cavo passivo DisplayPort-HDMI per il collegamento video. Il sistema operativo
|
||||
previsto e Fedora Sericea, la variante Fedora immutabile con compositor Wayland
|
||||
Sway.
|
||||
|
||||
Cerberus sara un management plane isolato: Ansible verra eseguito in un ambiente
|
||||
Toolbox dedicato per il provisioning del futuro cluster `uranus`, anziche da
|
||||
Ikaros o da un host non gestito. Lo stack di osservabilita rootless Podman
|
||||
eseguira Grafana, Prometheus e Loki. L'SSD locale sara l'hot storage, con
|
||||
metriche e log conservati per 30 giorni; esportazioni programmate trasferiranno
|
||||
i dati storici piu vecchi su un dataset Atlas montato via NFS come cold storage.
|
||||
Il piano di implementazione, con prerequisiti espliciti, e in `AGENTS.md`.
|
||||
|
||||
## Desktop
|
||||
|
||||
Target operativi:
|
||||
@@ -161,6 +182,10 @@ Le applicazioni Windows sono installate e gestite manualmente; il profilo WSL no
|
||||
|
||||
## Server
|
||||
|
||||
La migrazione dei servizi pubblici a `fscotto.co`, la gestione Ansible
|
||||
degli URL Gitea e i passaggi ancora aperti per ritirare DuckDNS sono in
|
||||
[`docs/domain-fscotto-co.md`](docs/domain-fscotto-co.md).
|
||||
|
||||
Sistema operativo:
|
||||
|
||||
- Rocky Linux 9
|
||||
@@ -180,28 +205,37 @@ Lo stato attuale del profilo server include:
|
||||
- installazione pacchetti Rocky via DNF, EPEL e CRB
|
||||
- installazione di Podman e podman-compose
|
||||
- abilitazione dei servizi systemd dichiarati in inventory/group vars
|
||||
- copia dei dotfiles server e rendering del `docker-compose.yml` per Nginx Proxy Manager e Gitea,
|
||||
piu l'unita `podman-compose-server` (attivazione manuale)
|
||||
- copia dei dotfiles server e rendering del Quadlet rootful `prometheus-npm.service` per Nginx Proxy
|
||||
Manager; il vecchio fallback Compose è stato rimosso con autorizzazione esplicita
|
||||
- attivazione di firewalld con SSH, Cockpit (`9090/tcp`), HTTP e HTTPS abilitati
|
||||
- Syncthing escluso dal profilo server Rocky
|
||||
|
||||
Il Compose desiderato su Prometheus non include piu Navidrome ne il database PostgreSQL obsoleto.
|
||||
Navidrome e Syncthing appartengono ad Atlas; Navidrome ufficiale usa invece SQLite. Il profilo non
|
||||
arresta o rimuove automaticamente eventuali container legacy e non elimina `/opt/postgres/data`.
|
||||
Il 2026-10-03 la pulizia opt-in autorizzata ha rimosso dati e immagini precedenti di Gitea,
|
||||
Navidrome e PostgreSQL, directory obsolete vuote, helper finale Gitea e fallback Compose NPM.
|
||||
I servizi migrati restano su Atlas. `server_legacy_stack_retired: true` evita che i normali task
|
||||
ricreino i residui; la cancellazione richiede `--tags server_legacy_cleanup` e
|
||||
`-e server_legacy_cleanup=true`. NPM attivo e archivi di backup restano intatti.
|
||||
Export, pull Atlas e restore SQLite isolato post-pulizia sono riusciti; il primo ciclo automatico
|
||||
resta da osservare. Evidenze e confini del recovery:
|
||||
[`docs/prometheus-npm-quadlet.md`](docs/prometheus-npm-quadlet.md).
|
||||
|
||||
Nginx Proxy Manager pubblica solo `80/tcp` e `443/tcp`; la sua interfaccia di amministrazione e
|
||||
associata a `127.0.0.1:81` ed e raggiungibile da Ikaros o Nymph con l'alias Bash `npm-tunnel`.
|
||||
Nextcloud resta disabilitato e il profilo non crea directory `/srv/nextcloud`.
|
||||
|
||||
La fase 1 su Atlas non modifica questo deployment NPM ne i suoi dati persistenti. Dopo aver attivato
|
||||
WireGuard e i servizi Atlas, configurare i proxy host NPM correnti con upstream Navidrome
|
||||
`http://10.0.0.2:4533` e upstream per la GUI Syncthing `http://10.0.0.2:8384`. Solo la GUI web di
|
||||
Syncthing usa NPM; il traffico di sincronizzazione resta sulle porte native pubblicate esplicitamente solo
|
||||
sull'indirizzo WireGuard di Atlas. Configurare l'autenticazione Syncthing e una policy di accesso NPM adeguata prima di pubblicare la GUI.
|
||||
La fase 1 su Atlas non modifica i dati persistenti NPM. I proxy host NPM usano gli upstream LAN
|
||||
`http://192.168.178.55:4533` per Navidrome e `http://192.168.178.55:8384` per la GUI Syncthing;
|
||||
Prometheus li raggiunge attraverso Aegis come gateway WireGuard. Solo la GUI web di Syncthing usa
|
||||
NPM; il traffico di sincronizzazione resta sulle porte native esposte sulla LAN dichiarata.
|
||||
Mantenere l'autenticazione Syncthing e una policy di accesso NPM adeguata.
|
||||
|
||||
### DuckDNS
|
||||
|
||||
`profile_server` genera `~/duckdns/duck.sh` con permessi `0700`, mantenendo il percorso dello
|
||||
`server_duckdns_enabled: false` disabilita il provisioning su Prometheus, che usa IP statico
|
||||
e `fscotto.co`. Updater, log e cron ogni cinque minuti sono stati rimossi una sola volta;
|
||||
non restano task o flag di pulizia. Il nome DuckDNS esterno e il token Vault restano invariati.
|
||||
|
||||
Sui server con `server_duckdns_enabled: true`, `profile_server` genera `~/duckdns/duck.sh` con permessi `0700`, mantenendo il percorso dello
|
||||
script e `duck.log`. Definire `server_duckdns_domain` negli host vars del server e salvare il
|
||||
**nuovo token rigenerato** in `vault_duckdns_token`, nel Vault cifrato `secrets/vault.yml`
|
||||
(`ansible-vault edit secrets/vault.yml`) oppure negli override non versionati `secrets/vault.local.yml`.
|
||||
@@ -246,90 +280,282 @@ ansible-playbook ansible/site.yml --limit prometheus -e server_username=myuser -
|
||||
|
||||
## NAS
|
||||
|
||||
`atlas` e un NAS Rocky Linux 9 raggiunto tramite SSH. Normalmente il pool ZFS esiste gia e il profilo
|
||||
gestisce solo i dataset figli. Un bootstrap RAIDZ2 una tantum e disponibile solo con conferma esplicita
|
||||
(`atlas_create_pool=true`) e quattro percorsi reali e verificati `/dev/disk/by-id/...` in
|
||||
`atlas_zpool_disks`. Non partiziona, forza, distrugge, esegue rollback o modifica il layout vdev di un
|
||||
pool esistente. I client Linux usano NFSv4, quelli Windows/WSL SMB; entrambi restano limitati alla LAN
|
||||
configurata.
|
||||
`atlas` è un NAS Rocky Linux 9 raggiunto via SSH. Normalmente il pool esiste già e il profilo gestisce
|
||||
solo i dataset figli. La creazione iniziale del RAIDZ2 richiede esplicitamente `atlas_create_pool=true`
|
||||
e quattro percorsi `/dev/disk/by-id/...` verificati in `atlas_zpool_disks`. Il ruolo non partiziona,
|
||||
forza, distrugge, ripristina né modifica il layout vdev di un pool esistente. I client Linux usano NFSv4,
|
||||
quelli Windows/WSL SMB; l'accesso è limitato alla LAN configurata.
|
||||
|
||||
Per il primo avvio fornire `vault_atlas_authorized_ssh_keys`, `vault_atlas_admin_password_hash`,
|
||||
`vault_atlas_samba_password` e `vault_atlas_immich_db_password`. Eseguire il bootstrap tramite
|
||||
l'amministratore esistente:
|
||||
Per il primo avvio servono `vault_atlas_admin_password_hash`, `vault_atlas_samba_password` e
|
||||
`vault_atlas_immich_db_password`; il primo è un hash compatibile con `/etc/shadow`, non una password
|
||||
Cockpit in chiaro. Il bootstrap usa l'amministratore preesistente:
|
||||
|
||||
```bash
|
||||
ansible-playbook ansible/site.yml --limit atlas \
|
||||
-e atlas_connection_username=<existing-admin>
|
||||
```
|
||||
|
||||
`vault_atlas_admin_password_hash` deve essere un hash compatibile con `/etc/shadow`, non una
|
||||
password Cockpit in chiaro. Le esecuzioni successive usano `atlas_admin_username`. Atlas dichiara
|
||||
abilitati storage, condivisioni e regole firewall LAN. Prima della prima applicazione verificare pool e
|
||||
mountpoint esistenti, subnet LAN e zona firewalld attiva. `atlas_manage_media_stack` resta disabilitato
|
||||
finche non saranno validati `/dev/dri`, i percorsi dei container e il segreto del database Immich.
|
||||
Le esecuzioni successive usano `atlas_admin_username`. Storage, condivisioni e firewall LAN sono
|
||||
abilitati; prima dell'applicazione verificare pool, mountpoint, subnet e zona firewalld. La creazione
|
||||
del pool è protetta da un gate esplicito e avviene solo se è assente. Atlas non fa più parte della VPN
|
||||
WireGuard: la vecchia interfaccia è stata ritirata manualmente dopo la verifica del collegamento tra
|
||||
Prometheus e Aegis. Le chiavi SSH autorizzate sono in file separati sotto
|
||||
`~/.ssh/authorized_keys.d/`. `atlas_manage_media_stack` resta disabilitato finché `/dev/dri`, percorsi
|
||||
dei container e segreto del database Immich non sono validati.
|
||||
|
||||
Con la gestione storage attiva, Atlas crea l'intera gerarchia sotto il pool `zpool` esistente o creato esplicitamente:
|
||||
`work`, `archive`, `archive/app_data`, i dataset applicativi separati
|
||||
`archive/app_data/navidrome` e `archive/app_data/syncthing`, `media`, `media/music`,
|
||||
`media/photobook`, `backups`, `backups/services` e `backup_prometheus`. I dataset applicativi e
|
||||
di archivio usano `zstd`; media, Syncthing e backup dei servizi usano `lz4`;
|
||||
`backups/services` mantiene inoltre una `refreservation` di `500G`.
|
||||
Atlas impone SELinux targeted in modo persistente e segnala, senza avviarlo, l’eventuale reboot necessario per attivarlo. Assegna esplicitamente l’interfaccia LAN primaria alla zona firewalld gestita e applica hardening persistente del kernel di rete: rifiuta redirect e source-route, registra i martian, usa reverse-path filtering loose per WireGuard e disabilita il forwarding IPv4. SSH consente solo l’amministratore dichiarato tramite chiave pubblica; root, password, agent e forwarding
|
||||
remoto sono disabilitati, mentre il forwarding locale resta disponibile per tunnel amministrativi privati. SMB3 pubblica `Archive` solo agli account Samba configurati con password in Vault e
|
||||
ammette la LAN configurata su SMB3 cifrato e firmato, esclusivamente su TCP/445. NFSv4 esporta soltanto
|
||||
`media/photobook` all'IP configurato di Aegis su TCP/2049, con `all_squash` verso UID/GID anonimi `1100`.
|
||||
Sotto `zpool` Atlas crea `archive` (SMB), `services/data` con i dataset applicativi
|
||||
`services/data/navidrome` e `services/data/syncthing`, `media`, `media/music`, `media/photobook` e
|
||||
`backup/hosts/prometheus`. Archivio e applicazioni usano `zstd`; media, Syncthing e backup host usano
|
||||
`lz4`. `backup` ha una riserva di `500G` che copre i discendenti. SELinux targeted è persistente;
|
||||
l'eventuale riavvio necessario viene segnalato, non eseguito. Atlas assegna l'interfaccia primaria
|
||||
alla zona firewalld gestita, rifiuta redirect e source route, registra i martian, mantiene il reverse-path
|
||||
filter loose e disabilita il forwarding IPv4. SSH consente soltanto l'amministratore dichiarato con
|
||||
chiave pubblica: root, password, agent forwarding e remote forwarding sono disabilitati, mentre il
|
||||
forwarding locale resta disponibile per i tunnel amministrativi. SMB3 espone `Archive` agli account
|
||||
autorizzati da Vault sulla LAN, solo su TCP/445 con cifratura e firma obbligatorie. NFSv4 espone
|
||||
soltanto `media/photobook` all'IP di Aegis su TCP/2049, con `all_squash` verso UID/GID `1100`.
|
||||
|
||||
L'account di sistema `immich` usa UID/GID `1100`, shell senza login, nessuna appartenenza a `wheel` e
|
||||
i gruppi supplementari `video` e `render`. I Quadlet rootful di Immich Server, ML, cache compatibile
|
||||
Redis, PostgreSQL e NPM condividono una rete Podman. Immich viene eseguito come `1100:1100`; Server e
|
||||
ML ricevono `/dev/dri` e Photobook e montato in sola lettura su `/external/photobook`. NPM pubblica `80` e
|
||||
`443`, mentre l'amministrazione resta vincolata a `127.0.0.1:81` per l'accesso tramite tunnel SSH.
|
||||
L'account di sistema `immich` usa UID/GID `1100`, non ha shell di login né gruppo `wheel` e riceve i
|
||||
gruppi `video` e `render`. Lo stack Immich futuro prevede Quadlet rootful per Server, ML, cache,
|
||||
PostgreSQL e NPM su una rete Podman comune. Immich gira come `1100:1100`, Server e ML ricevono
|
||||
`/dev/dri` e Photobook è montato in sola lettura su `/external/photobook`. NPM pubblica `80` e `443`;
|
||||
l'interfaccia amministrativa resta su `127.0.0.1:81`, raggiungibile via tunnel SSH.
|
||||
|
||||
La fase 1 e limitata ai Quadlet utente rootless di Navidrome e Syncthing su Atlas. E abilitata nella
|
||||
configurazione host di Atlas e puo essere impostata a `false` solo per una sospensione intenzionale. Navidrome ufficiale `0.63.2` usa il database SQLite sotto `/data` e
|
||||
non supporta `ND_DATABASE_URL` ne un backend PostgreSQL esterno. Il servizio obsoleto `navidromedb`
|
||||
e quindi rimosso da Prometheus invece di essere replicato su Atlas. Il ruolo deriva i percorsi dal
|
||||
pool `zpool`, montato in `/zpool`: musica in sola lettura da `/zpool/media/music`, stato
|
||||
applicativo Navidrome e `navidrome.db` in `/zpool/archive/app_data/navidrome` e dati Syncthing in
|
||||
`/zpool/archive/app_data/syncthing`. `profile_atlas` crea questi dataset quando
|
||||
`atlas_manage_storage` e attivo; il ruolo backend verifica i mountpoint esatti prima di avviare i
|
||||
container. Il ruolo backend non crea mai il pool. Il ruolo separato `wireguard_overlay`
|
||||
gestisce `wg0` tra Prometheus (`10.0.0.1`) e Atlas (`10.0.0.2`), genera una sola volta le chiavi
|
||||
private sui rispettivi host e scambia tramite Ansible soltanto quelle pubbliche. Solo Prometheus apre
|
||||
pubblicamente `51820/udp`. Le porte backend sono ammesse esclusivamente nella zona firewalld WireGuard.
|
||||
Atlas ospita temporaneamente Navidrome e Syncthing rootless fino alla sostituzione con Uranus. I
|
||||
servizi sono inizializzati **ex novo**, senza migrare lo stato precedente, rispettivamente sotto
|
||||
`/zpool/services/data/navidrome` e `/zpool/services/data/syncthing`; la musica in
|
||||
`/zpool/media/music` è stata popolata separatamente da `/zpool/archive/Music` il 2026-09-30;
|
||||
Navidrome ha completato la scansione. Il timer rootless `atlas-music-sync.timer` copia i file nuovi
|
||||
o modificati ogni giorno alle 00:45 Europe/Rome, senza eliminare quelli presenti solo nella
|
||||
destinazione; entrambi i dataset ZFS devono essere montati. La prima esecuzione schedulata è
|
||||
riuscita il 2026-10-02. Alcune playlist originali contengono
|
||||
ancora vecchi percorsi Windows. I servizi sono vincolati all'indirizzo LAN di Atlas
|
||||
(`192.168.178.55`), mai a WireGuard. `wireguard_overlay` collega invece Prometheus (`10.0.0.1`)
|
||||
e Aegis (`10.0.0.2`): le chiavi private restano sui rispettivi host e Ansible scambia solo le pubbliche.
|
||||
Prometheus apre `51820/udp`; Aegis inoltra soltanto il traffico overlay→LAN dichiarato e applica
|
||||
source NAT, evitando interfacce VPN su Atlas/Uranus e route statiche sul router. Navidrome (`4533/tcp`)
|
||||
e la GUI Syncthing (`8384/tcp`) ammettono solo Aegis, mentre le porte native Syncthing sono limitate
|
||||
alla LAN. Dopo la verifica dei servizi, configurare manualmente i Proxy Host NPM verso
|
||||
`http://192.168.178.55:4533` e `http://192.168.178.55:8384`. Il peer Prometheus include la LAN
|
||||
negli `AllowedIPs`; aggiungere la VIP Uranus quando esisterà. Dopo il reload di firewalld, Ansible
|
||||
ricarica le reti Podman rootful di Prometheus per conservare DNS e connettività del proxy.
|
||||
|
||||
`backend_phase1_start_services` resta falso durante il trasferimento dello stato applicativo, quindi
|
||||
la prima esecuzione reale del backend genera i Quadlet senza creare un database Atlas vuoto. Dopo aver
|
||||
arrestato Navidrome su Prometheus, copiare l'intera directory `/opt/navidrome/data/` in
|
||||
`/zpool/archive/app_data/navidrome/`, preservando `navidrome.db` e gli eventuali file SQLite laterali.
|
||||
Impostare quindi questa variabile a vero e rieseguire il ruolo per abilitare e avviare Navidrome e
|
||||
Syncthing. Il playbook non copia e non elimina mai i dati applicativi.
|
||||
La migrazione Gitea da Prometheus ad Atlas è descritta in
|
||||
[`docs/atlas-gitea-migration.md`](docs/atlas-gitea-migration.md). Gitea usa un Quadlet rootless
|
||||
di `admin` su un dataset dedicato; l'immagine derivata mantiene UID/GID 1000 ma chiama l'utente
|
||||
interno `gitea`. NPM resta su Prometheus e l'HTTPS pubblico primario serve Atlas. L'SSH pubblico
|
||||
su TCP/2222 autentica la chiave `ikaros` e un `git ls-remote` è riuscito; l'operatore ha
|
||||
confermato pull e push SSH. Login e scrittura Git via HTTPS sono stati confermati il 2026-10-03. I dati sorgente restano
|
||||
conservati su Prometheus senza avviarne il vecchio container.
|
||||
|
||||
Validare e generare i servizi Atlas con:
|
||||
Validare il gateway con:
|
||||
|
||||
```bash
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags storage
|
||||
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit prometheus,atlas --tags wireguard
|
||||
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 --check --diff
|
||||
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1
|
||||
ansible-playbook ansible/site.yml --limit prometheus,aegis --tags wireguard --check --diff
|
||||
```
|
||||
|
||||
Per il cutover, arrestare il vecchio Navidrome prima di copiare la sua directory dati, verificare
|
||||
l'ownership dell'account `admin` su Atlas e confermare la presenza del database SQLite copiato prima
|
||||
di impostare `backend_phase1_start_services: true` in `host_vars/atlas.yml`. Conservare i dati sorgente
|
||||
e il container legacy `navidromedb` fermo finche Navidrome su Atlas e una prova di restore non sono
|
||||
stati validati.
|
||||
La prima esecuzione reale WireGuard deve includere entrambi i peer. Se Aegis ha appena installato il
|
||||
layer `wireguard-tools`, riavviarlo manualmente e rieseguire senza `--check`: il ruolo attende un
|
||||
handshake effettivo.
|
||||
|
||||
Restano da completare retention delle snapshot, topologia Syncthing, validazione WireGuard/firewall,
|
||||
pull di backup da Prometheus, backup cifrati con Borg su una Hetzner Storage Box, backup USB,
|
||||
monitoraggio e test di disaster recovery. Il backlog operativo dettagliato e in `AGENTS.md`.
|
||||
Gli snapshot ZFS ricorsivi coprono l'intero pool: 24 orari al minuto 05, 30 giornalieri alle 00:15,
|
||||
8 settimanali la domenica alle 01:00 e 12 mensili il primo giorno alle 02:00. La retention elimina
|
||||
solo gli snapshot con prefisso gestito `atlas-auto` e non esegue rollback. Lo scrub OpenZFS mensile è
|
||||
previsto la prima domenica alle 03:00; il timer settimanale incompatibile è disabilitato. Il primo
|
||||
snapshot orario ricorsivo è riuscito e la pulizia pianificata della retention è stata osservata il
|
||||
2026-09-30. Il primo scrub mensile richiede ancora una verifica a runtime.
|
||||
|
||||
```bash
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags snapshots,scrub --check --diff
|
||||
```
|
||||
|
||||
Il backup Borg cifrato usa il sub-account Hetzner `u660064-sub1`, il repository relativo `./borg-data`
|
||||
e Borg remoto 1.4 su SSH porta 23. La chiave ED25519 del server è fissata; una chiave client dedicata
|
||||
appartiene all'account `borg`, bloccato e senza login, sudo o gruppi supplementari. La chiave privata
|
||||
resta in `/etc/atlas-borg`; la passphrase proviene da `vault_atlas_borg_passphrase` ed è resa in un
|
||||
file `0600`. Solo il wrapper root crea snapshot e mount; avvia il client come `borg` con il minimo
|
||||
accesso temporaneo in lettura, senza concedergli gestione ZFS o sudo.
|
||||
|
||||
Il backup giornaliero parte alle 04:30 con un ritardo casuale fino a 30 minuti. Crea uno snapshot ZFS
|
||||
ricorsivo temporaneo e ricostruisce tutti i dataset sotto `/zpool` in un albero di bind mount in sola
|
||||
lettura, per inserirli in un unico archivio coerente. Il wrapper smonta ricorsivamente l'albero privato;
|
||||
un helper `ExecStopPost` mirato rimuove eventuali mount dello snapshot nel namespace host e lo snapshot
|
||||
temporaneo dopo l'uscita del processo. Borg conserva 30 archivi giornalieri, 8 settimanali e 12
|
||||
mensili, poi compatta il repository. Il controllo completo di metadati e repository si svolge il 15
|
||||
di ogni mese alle 06:00. Le operazioni usano un lock comune, journal e retry systemd limitati. Le
|
||||
nuove esecuzioni riportano al massimo una riga di avanzamento al minuto: percentuale **stimata**,
|
||||
dataset, file elaborati e byte originali/compressi/deduplicati. Il denominatore è la somma dei
|
||||
`logicalreferenced` ZFS dello snapshot, non un totale Borg: può superare il 100% e non comprende
|
||||
retention, compattazione o controlli. Le righe di progresso non riportano i nomi dei file; eventuali
|
||||
warning possono farlo. Seguire il job con `sudo journalctl -fu atlas-borg-backup.service`; modifiche
|
||||
all'helper non cambiano un'esecuzione già avviata.
|
||||
|
||||
Attivazione iniziale esplicita:
|
||||
|
||||
1. Inserire una passphrase unica in `secrets/vault.yml` con `ansible-vault edit`.
|
||||
2. Generare e mostrare solo la chiave pubblica con
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags borg_key`.
|
||||
3. Installarla nel sub-account Hetzner, poi applicare con
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags packages,borg`.
|
||||
4. Copiare `secrets/recovery/atlas-borg-repokey.export` su un supporto davvero offline: la copia
|
||||
locale ignorata da Git non è di per sé un backup offline.
|
||||
|
||||
Il ruolo inizializza solo un repository `repokey` assente, non accetta password SSH né host key non
|
||||
fissate e non avvia manualmente il primo backup. Validazione:
|
||||
|
||||
```bash
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags packages,borg --check --diff
|
||||
```
|
||||
|
||||
L'attivazione iniziale è riuscita: backup e controllo del repository, restore completo in una
|
||||
directory temporanea confrontato con l'albero `Archive`, esportazione offline della chiave di recupero
|
||||
e pulizia di snapshot/mount temporanei. Il 2026-09-25 un test separato da snapshot ZFS giornaliero ha
|
||||
copiato un file di `/zpool/archive` in `/var/tmp`, verificando contenuto, proprietario, modalità,
|
||||
mtime e ACL POSIX; copia e mount temporanei sono stati rimossi senza interrompere Borg. Non è un test
|
||||
di ripristino dell'intero dataset.
|
||||
|
||||
L'archivio del pool popolato del 2026-09-29 ha richiesto 1 h 32 min per 2,18 TB originali / 2,04 TB
|
||||
compressi, con 13,49 GB di dimensione deduplicata. Retention e compattazione sono riuscite, ma un
|
||||
errore di permessi su `RuntimeDirectory` ha impedito la pulizia dello snapshot dopo il job. Dopo la
|
||||
correzione, l'archivio incrementale del 2026-09-30 è terminato in circa 22 secondi, ha rimosso lo
|
||||
snapshot residuo e quello corrente ed è terminato con stato 0. Il monitor ha rilevato il 37% della
|
||||
quota Storage Box utilizzata. Questi risultati non predicono durata o compressione dei prossimi run.
|
||||
|
||||
Il backup USB offline è distribuito come **servizio solo manuale** (`atlas_manage_usb_backup: true`):
|
||||
Ansible non formatta, sblocca, monta né avvia automaticamente il disco. Il disco esistente è stato
|
||||
verificato in sola lettura il 2026-09-23: UUID LUKS `577b3c43-ea37-4611-81a9-39d555cdfbd4`,
|
||||
UUID ext4 interno `758e2d2e-a427-4797-aad9-39c3a9f17c7e`, mapper `zpool-backup`. All'ispezione
|
||||
era montato in `/mnt/zpool-backup`; il servizio richiede invece che il mapper **non sia montato** prima
|
||||
dell'avvio. Se serve, `systemd-ask-password` chiede interattivamente la passphrase LUKS tramite
|
||||
l'agente di `systemctl start` e la passa direttamente a `cryptsetup`, senza salvarla, esporla negli
|
||||
argomenti o memorizzarla nella cache. Lo script monta il disco privatamente, crea uno snapshot ZFS
|
||||
ricorsivo, copia tutti i dataset in `atlas/snapshots/<timestamp>/` con `rsync --link-dest`, verifica
|
||||
con un dry-run basato sui checksum, aggiorna atomicamente `atlas/latest`, smonta e chiude LUKS. Un
|
||||
errore non sostituisce `latest` né cancella versioni complete precedenti. Borg e USB possono operare
|
||||
contemporaneamente su snapshot distinti, ma la lettura concorrente può ridurre il throughput.
|
||||
|
||||
La copia USB conserva le ACL ma non gli attributi estesi generici, compreso `security.selinux`: la
|
||||
policy della destinazione deve ricreare le etichette dopo un restore. Per un percorso esplicito:
|
||||
|
||||
```bash
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags restorecon \
|
||||
-e '{"atlas_restorecon_paths":["/zpool/archive"]}'
|
||||
```
|
||||
|
||||
Il task accetta solo percorsi sotto la radice del pool Atlas, esegue `restorecon -RFv` solo su quelli
|
||||
indicati ed è altrimenti inattivo; non va lanciato sull'intero pool durante i run ordinari. Le vecchie
|
||||
versioni USB non vengono eliminate automaticamente senza una retention deliberata. Il controllo di
|
||||
capacità include il trasferimento stimato e una riserva libera di 10 GiB. Dopo un backup riuscito,
|
||||
scollegare fisicamente il disco per renderlo davvero offline.
|
||||
|
||||
Validare la configurazione senza avviare il backup e, separatamente, un eventuale relabel pianificato:
|
||||
|
||||
```bash
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags usb_backup,usb_reminder --check --diff
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags restorecon --check \
|
||||
-e '{"atlas_restorecon_paths":["/zpool/archive"]}'
|
||||
```
|
||||
|
||||
Prima dell'avvio manuale smontare in sicurezza `/mnt/zpool-backup`, se ancora montato. Con il mapper
|
||||
chiuso, `sudo systemctl start atlas-usb-backup.service` chiede la passphrase e avvia il backup; né la
|
||||
password LUKS né un keyfile vanno in Ansible. Seguire con
|
||||
`sudo journalctl -fu atlas-usb-backup.service`. **Non esiste un timer di backup USB.** Soltanto
|
||||
`atlas-usb-reminder.timer` è schedulato il primo sabato del mese alle 10:00 `Europe/Rome`: invia un
|
||||
promemoria al notifier 45Drives Houston, senza avviare il backup. Un test manuale ha prodotto una
|
||||
notifica in 45Drives Alerts, **non un'email**; il log conferma l'invio della notifica, non la consegna
|
||||
di posta. Il primo evento pianificato era il 2026-10-03 alle 10:00 CEST. Controllare timer e risultato
|
||||
con `systemctl list-timers atlas-usb-reminder.timer` e in 45Drives Alerts.
|
||||
|
||||
Il primo tentativo USB del 2026-09-23 fallì su `security.selinux` e, dopo l'interruzione, lasciò
|
||||
snapshot e mapper aperti. Applicato il filtro rsync, furono rimossi lo snapshot fallito, il mapper
|
||||
smontato e lo stato failed; non rimase una copia valida di quel tentativo. Un run del 2026-09-24
|
||||
pubblicò una versione verificata ma fallì nella distruzione dello snapshot a causa di mount
|
||||
`.zfs/snapshot` aperti nel namespace host. Dopo la pulizia non forzata, è stato aggiunto un helper
|
||||
`ExecStopPost` mirato e testato con uno snapshot usa-e-getta. Un run successivo del 2026-09-24 ha
|
||||
verificato i checksum, pubblicato la versione ed è terminato con successo: mapper chiuso, nessuno
|
||||
snapshot USB temporaneo e pool sano. Il 2026-09-25 un test di restore indipendente ha aperto il disco
|
||||
in sola lettura, montato ext4 con `ro,noload`, copiato un file di 5.707.945 byte da `atlas/latest` in
|
||||
una directory vuota sotto `/var/tmp` e confrontato contenuto, proprietario, modalità, dimensione,
|
||||
mtime e ACL POSIX. Il test ha rimosso copia e mount temporanei, chiuso LUKS e lasciato il pool sano
|
||||
mentre Borg continuava. È un test su file, non un esercizio completo di disaster recovery.
|
||||
|
||||
Il monitoraggio Atlas è eseguito ogni 30 minuti da `atlas-health-monitor.timer`. Sonde in sola
|
||||
lettura controllano stato/errori del pool e dei vdev, scrub/resilver, SMART dei quattro dischi del
|
||||
pool e dell'NVMe di sistema, temperature dei dischi e CPU, spazio di sistema/pool/snapshot, crescita
|
||||
di `zpool/backup` e quota Hetzner tramite `df -m` via SSH con l'account `borg` e la chiave fissata.
|
||||
La query remota non apre il repository Borg né il suo lock. Gli alert di crescita richiedono una
|
||||
baseline di circa 24 ore. Sono controllati anche attivazione e freschezza dei timer; hook systemd
|
||||
`OnFailure` segnalano errori di snapshot, scrub, Borg, USB, promemoria e monitoraggio. Il monitor non
|
||||
riavvia Borg; avvisa solo se un run supera 14 giorni. Soglie e percorsi stabili dei dischi sono nelle
|
||||
variabili host. Gli avvisi usano 45Drives Houston con deduplicazione; **la consegna email non è stata
|
||||
verificata**. Il controllo live del 2026-09-25 non ha trovato problemi e ha inviato una notifica di
|
||||
prova. Il 2026-09-30 il monitor ha rilevato zero problemi e una quota Storage Box occupata al 37%.
|
||||
L'hook per i job falliti ora passa il nome letterale della unità systemd; l'espansione è stata
|
||||
verificata senza inviare un falso allarme.
|
||||
|
||||
```bash
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags monitoring --check --diff
|
||||
sudo /usr/local/libexec/atlas-health-monitor --dry-run
|
||||
sudo journalctl -u atlas-health-monitor.service -n 100 --no-pager
|
||||
systemctl list-timers atlas-health-monitor.timer
|
||||
```
|
||||
|
||||
`--dry-run` non invia alert e non modifica lo stato del monitor. Un controllo reale si avvia con
|
||||
`sudo systemctl start atlas-health-monitor.service`, senza avviare servizi di backup. Per una prova
|
||||
etichettata di 45Drives Alerts usare
|
||||
`sudo /usr/local/libexec/atlas-health-monitor --test-notification`.
|
||||
|
||||
### Timer systemd di Atlas
|
||||
|
||||
Tutti i dieci timer gestiti sono abilitati. Gli orari sono locali ad Atlas (`Europe/Rome`); Borg e
|
||||
monitoraggio aggiungono il ritardo casuale indicato. Tutti hanno `Persistent=true`: un evento perso
|
||||
viene recuperato quando il timer torna attivo.
|
||||
|
||||
| Timer | Pianificazione (`OnCalendar`) | Azione |
|
||||
| --- | --- | --- |
|
||||
| `atlas-zfs-snapshot-hourly.timer` | `*-*-* *:05:00` — ogni ora al minuto 05 | Snapshot ricorsivo orario e retention |
|
||||
| `atlas-zfs-snapshot-daily.timer` | `*-*-* 00:15:00` — ogni giorno alle 00:15 | Snapshot ricorsivo giornaliero e retention |
|
||||
| `atlas-zfs-snapshot-weekly.timer` | `Sun *-*-* 01:00:00` — domenica alle 01:00 | Snapshot ricorsivo settimanale e retention |
|
||||
| `atlas-zfs-snapshot-monthly.timer` | `*-*-01 02:00:00` — primo giorno del mese alle 02:00 | Snapshot ricorsivo mensile e retention |
|
||||
| `zfs-scrub-monthly@zpool.timer` | `Sun *-*-01..07 03:00:00` — prima domenica alle 03:00 | Scrub ZFS |
|
||||
| `atlas-borg-backup.timer` | `*-*-* 04:30:00` — ogni giorno alle 04:30, più 0–30 min casuali | Backup cifrato offsite |
|
||||
| `atlas-borg-check.timer` | `*-*-15 06:00:00` — giorno 15 alle 06:00, più 0–30 min casuali | Controllo repository Borg |
|
||||
| `atlas-usb-reminder.timer` | `Sat *-*-01..07 10:00:00 Europe/Rome` — primo sabato alle 10:00 | Solo promemoria 45Drives Alerts |
|
||||
| `atlas-health-monitor.timer` | `*:0/30` — ogni mezz'ora, più 0–5 min casuali | Controlli di salute in sola lettura |
|
||||
| `atlas-prometheus-pull.timer` | `*-*-* 03:00:00 Europe/Rome` — ogni giorno alle 03:00 | Pull e verifica del backup preparato su Prometheus |
|
||||
|
||||
`atlas-usb-backup.service` **non ha timer** e va avviato manualmente. Il timer del fornitore
|
||||
`zfs-scrub-weekly@zpool.timer` è disabilitato a favore dello scrub mensile. Il timer di preparazione
|
||||
su Prometheus è attivo alle 02:00 Europe/Rome; il primo ciclo pianificato è riuscito il 2026-10-01.
|
||||
Un export, pull e ripristino temporaneo post-cutover NPM Quadlet sono riusciti il 2026-10-03;
|
||||
il primo ciclo pianificato dopo quel cutover resta da osservare. Durante un backup Borg attivo,
|
||||
`systemctl list-timers` può mostrare `-` per il prossimo evento senza che il timer sia disabilitato.
|
||||
Per vedere la pianificazione corrente: `systemctl list-timers --all` su Atlas.
|
||||
|
||||
Nextcloud è previsto come servizio temporaneo su Atlas prima di Uranus, ma solo dopo la validazione
|
||||
della protezione dei dati: richiede storage applicativo, database e cache separati, segreti Vault,
|
||||
pubblicazione solo tramite NPM e Aegis, procedure di backup, aggiornamento e migrazione. Non
|
||||
distribuirlo prima di completare la checklist di protezione dei dati.
|
||||
|
||||
Atlas è la destinazione dichiarata per iCloudPD. Ansible gestisce dataset, Quadlet rootless e
|
||||
`icloudpd.conf` privato con Apple ID dal Vault: foto in `/zpool/archive/Pictures/iCloudPD`,
|
||||
stato in `zpool/services/data/icloudpd`. Il primo avvio è stato manuale; password e MFA restano
|
||||
gestiti interattivamente, senza avvio automatico al boot. L'inizializzazione è stata completata e
|
||||
il download iniziale di foto e video è terminato il 2026-10-03. Su Aegis
|
||||
il servizio, il Quadlet e `/var/lib/icloudpd` sono stati rimossi e verificati; il playbook Aegis
|
||||
non contiene più task iCloudPD. L'accesso SMB e il ripristino dai backup dei nuovi dati restano
|
||||
da verificare. L'export NFS Photobook resta
|
||||
invariato. Dettagli in [`docs/atlas-icloudpd-migration.md`](docs/atlas-icloudpd-migration.md).
|
||||
|
||||
Il primo ciclo pianificato del backup di Prometheus e una prova di disaster recovery a dimensione reale
|
||||
restano da verificare. Il 2026-09-30 una VM Rocky isolata ha superato ricostruzione OS con Ansible,
|
||||
import del pool RAIDZ2 fittizio e ripristino da snapshot; RPO 24 ore/RTO 72 ore restano obiettivi
|
||||
provvisori, non tempi misurati. Dettagli e limiti sono in `docs/atlas-recovery.md`. Il backlog
|
||||
prioritizzato è in `AGENTS.md`.
|
||||
|
||||
---
|
||||
|
||||
@@ -426,8 +652,8 @@ Questo significa che, allo stato attuale:
|
||||
- `deadalus` riceve il profilo Fedora WSL tramite play dev dedicati
|
||||
- il server Rocky (`prometheus`) e gestito con pacchetti, servizi, dotfiles server e firewalld
|
||||
- il NAS Rocky (`atlas`) usa un pool ZFS gia esistente, condivisioni NFSv4/SMB limitate alla LAN e Cockpit/45Drives
|
||||
- lo stack Compose server include soltanto `gitea` e `nginx-proxy-manager`; Navidrome e Syncthing
|
||||
della fase 1 sono Quadlet rootless su Atlas
|
||||
- NPM è un Quadlet rootful su Prometheus, mentre Gitea, Navidrome e Syncthing sono Quadlet
|
||||
rootless su Atlas; il fallback Compose server è stato rimosso
|
||||
|
||||
# Dotfiles
|
||||
|
||||
@@ -533,9 +759,10 @@ ansible-playbook ansible/site.yml --limit <host> --tags <tag1>,<tag2> --check --
|
||||
ansible-playbook ansible/site.yml --limit <host> --start-at-task "<task name>" --check --diff
|
||||
ansible-lint ansible/roles/<role>
|
||||
yamllint ansible/path/to/file.yml
|
||||
podman-compose -f /opt/docker/server/docker-compose.yml config
|
||||
ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 --check --diff
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags music_sync --check --diff
|
||||
```
|
||||
|
||||
## Tag supportati dal playbook
|
||||
|
||||
439
README.md
439
README.md
@@ -67,6 +67,27 @@ The official ChatGPT desktop RPM is enabled only on `ikaros` and `nymph`. The
|
||||
playbook configures OpenAI's signed RPM repository and imports its pinned RPM
|
||||
signing key before installation; subsequent updates are handled by DNF.
|
||||
|
||||
## Deferred planned node: Cerberus
|
||||
|
||||
`cerberus` is a **postponed** management-plane node, pending the physical setup
|
||||
of the office in the new house. It is not yet an inventory host and no role or
|
||||
playbook targets it.
|
||||
|
||||
The planned hardware is a Lenovo ThinkCentre M700 Tiny (Intel Core i3-6100T,
|
||||
8 GB RAM, and a 256 GB SSD) with native 1 Gbps Ethernet. It will share Ikaros'
|
||||
monitor and peripherals through a multi-input KVM switch, using a passive
|
||||
DisplayPort-to-HDMI cable for its video connection. Fedora Sericea, the
|
||||
immutable Fedora variant with the Sway Wayland compositor, is the intended
|
||||
operating system.
|
||||
|
||||
Cerberus will be an isolated management plane: Ansible will run from a
|
||||
dedicated Toolbox environment to provision the future `uranus` cluster, rather
|
||||
than from Ikaros or an unmanaged host. Its rootless Podman observability stack
|
||||
will run Grafana, Prometheus, and Loki. The local SSD is the hot tier and
|
||||
retains metrics and logs for 30 days; scheduled exports will place older
|
||||
historical data on an NFS-mounted Atlas dataset as the cold tier. The detailed,
|
||||
implementation-gated plan is maintained in `AGENTS.md`.
|
||||
|
||||
## Desktop profiles
|
||||
|
||||
- `ikaros`: stable Fedora Workstation + GNOME desktop.
|
||||
@@ -104,28 +125,36 @@ That gives it Fedora packages through DNF, Docker from the official repository,
|
||||
|
||||
## Server
|
||||
|
||||
The public service domain transition to `fscotto.co`, Gitea canonical URL
|
||||
management, and remaining DuckDNS retirement steps are documented in
|
||||
[`docs/domain-fscotto-co.md`](docs/domain-fscotto-co.md).
|
||||
|
||||
`prometheus` is the Rocky Linux 9 server. It has no graphical environment and gets server-specific
|
||||
dotfiles and templates. The profile provisions configuration only: it does not transfer data, start
|
||||
the Compose stack, update DNS, or perform a cutover.
|
||||
dotfiles and templates. The profile does not transfer application data, update DNS, or perform an
|
||||
implicit service cutover.
|
||||
|
||||
The server profile installs platform-specific packages, Podman and podman-compose, declared systemd
|
||||
services, and firewalld. The manually activated `podman-compose-server` unit contains the existing
|
||||
Nginx Proxy Manager and Gitea services. The desired Compose file no longer includes Navidrome,
|
||||
Syncthing, or the obsolete Navidrome PostgreSQL database. Navidrome and Syncthing belong to Atlas;
|
||||
official Navidrome uses SQLite instead. Applying the profile does not stop or remove legacy
|
||||
containers and does not delete `/opt/postgres/data`.
|
||||
services, and firewalld. Nginx Proxy Manager runs as the rootful `prometheus-npm.service` Quadlet.
|
||||
On 2026-10-03 the operator-approved opt-in cleanup removed old Gitea, Navidrome and PostgreSQL
|
||||
data/images, empty legacy directories, the Gitea final-export helper and the Compose rollback files.
|
||||
The migrated services stay on Atlas. `server_legacy_stack_retired: true` prevents normal runs from
|
||||
recreating retired files. Data deletion requires `--tags server_legacy_cleanup` and
|
||||
`-e server_legacy_cleanup=true`; image-only cleanup has its own `server_image_cleanup` tag and flag.
|
||||
Active NPM resources and existing backup archives remain preserved.
|
||||
The post-cleanup export/pull and isolated SQLite restore passed; the first unattended cycle remains
|
||||
pending. Evidence and recovery boundaries:
|
||||
[`docs/prometheus-npm-quadlet.md`](docs/prometheus-npm-quadlet.md).
|
||||
|
||||
|
||||
Firewalld enables SSH, Cockpit (`9090/tcp`), HTTP and HTTPS. Nginx Proxy Manager publishes only
|
||||
`80/tcp` and `443/tcp`; its administration interface is bound to `127.0.0.1:81` and can be reached
|
||||
from Ikaros or Nymph with the `npm-tunnel` Bash alias. Nextcloud remains disabled and the profile
|
||||
does not provision any `/srv/nextcloud` directories.
|
||||
|
||||
The Atlas phase-one work does not change this NPM deployment or its persistent data. Once WireGuard
|
||||
and the Atlas services are active, configure the current NPM proxy hosts with Navidrome upstream
|
||||
`http://10.0.0.2:4533` and Syncthing GUI upstream `http://10.0.0.2:8384`. Only the Syncthing web GUI
|
||||
uses NPM; synchronization traffic remains on explicitly published native ports bound only to the Atlas
|
||||
WireGuard address. Configure both Syncthing authentication and an appropriate NPM access policy before
|
||||
publishing its GUI.
|
||||
NPM remains managed only by `profile_server`. Its WireGuard peer is Aegis (`10.0.0.2`), which forwards
|
||||
selected requests to LAN addresses and source-NATs them so no static route is required on the router.
|
||||
Use an Atlas LAN address for any current NAS-backed upstream; when Uranus receives its VIP, add that VIP
|
||||
to Prometheus' Aegis peer `AllowedIPs` and declare the corresponding proxy target separately.
|
||||
|
||||
Server identity comes from `server_username`, `server_user_group`, and `server_user_home` in `ansible/inventory/group_vars/server.yml`. `server_username` defaults to `username`, but it can be overridden, for example:
|
||||
|
||||
@@ -142,7 +171,11 @@ Prometheus authorizes its declared SSH public keys through separate files below
|
||||
|
||||
### DuckDNS
|
||||
|
||||
`profile_server` renders `~/duckdns/duck.sh` with mode `0700`, keeping the existing updater path
|
||||
`server_duckdns_enabled: false` disables provisioning on Prometheus, which uses its static IP
|
||||
and `fscotto.co`. The local updater, log and five-minute cron job were removed once;
|
||||
no cleanup tasks or flags remain. The external DuckDNS name and Vault token remain untouched.
|
||||
|
||||
For servers with `server_duckdns_enabled: true`, `profile_server` renders `~/duckdns/duck.sh` with mode `0700`, keeping the existing updater path
|
||||
and `duck.log`. Set `server_duckdns_domain` in the server's host vars and store the **rotated**
|
||||
`vault_duckdns_token` in encrypted `secrets/vault.yml` (using `ansible-vault edit secrets/vault.yml`)
|
||||
or untracked `secrets/vault.local.yml`. Never commit the rendered script or put the token on a
|
||||
@@ -191,16 +224,22 @@ ansible/bootstrap/generate-aegis-ign.sh --write IMAGE DEVICE
|
||||
```
|
||||
|
||||
The controller manages it remotely as `pi@aegis`; unlike local desktop profiles, Aegis is
|
||||
intentionally an SSH inventory target. `profile_aegis` manages rootful Podman Quadlets for AdGuard
|
||||
Home and iCloudPD, persistent data under `/var/lib`, the Podman auto-update timer, LAN-restricted
|
||||
firewalld rules, SSH key-only access for `pi`, and `wake-ikaros`. Set the host-local
|
||||
`aegis_lan_subnet` and `aegis_adguard_web_port` values before applying it. The playbook permits
|
||||
intentionally an SSH inventory target. `profile_aegis` manages a rootful Podman Quadlet for AdGuard
|
||||
Home, its persistent data under `/var/lib`, the Podman auto-update timer, LAN-restricted
|
||||
firewalld rules, SSH key-only access for `pi`, the `nfs-utils` and `wireguard-tools` rpm-ostree layers,
|
||||
and `wake-ikaros`. `wireguard_overlay` makes Aegis the internal endpoint and LAN gateway for Prometheus:
|
||||
it enables persistent IPv4 forwarding, installs a scoped WireGuard-to-LAN firewalld policy, and source-NATs
|
||||
forwarded tunnel traffic so the router needs no static route. A new layered package deployment requires a manual reboot; the
|
||||
role reports this condition but never reboots Aegis automatically. Set the host-local
|
||||
`aegis_lan_subnet`, `aegis_adguard_web_port`, and `aegis_network_connection_uuid` values before
|
||||
applying it. The playbook permits
|
||||
AdGuard Home HTTP on port `80`; the initial wizard port `3000` is intentionally unmanaged and must be
|
||||
opened and closed manually during initial setup. The profile disables the local systemd-resolved DNS
|
||||
stub and points `/etc/resolv.conf` to its full resolver data, freeing port 53
|
||||
for AdGuard while retaining DNS learned from the router. Define
|
||||
`vault_aegis_icloudpd_apple_id` in Vault before applying it. iCloudPD still requires interactive MFA
|
||||
initialization after its first deployment.
|
||||
stub and points `/etc/resolv.conf` to its full resolver data, freeing port 53 for AdGuard. LAN clients
|
||||
may use AdGuard on Aegis, while Aegis itself uses the independent upstream DNS declared by
|
||||
`aegis_host_dns_servers`; this prevents Greenboot from depending on the AdGuard container during
|
||||
startup. Reboot Aegis after changing its NetworkManager DNS profile. iCloudPD was retired from Aegis;
|
||||
the Aegis role no longer contains iCloudPD tasks. Atlas iCloudPD config is Vault-backed; MFA is manual.
|
||||
|
||||
New Aegis images create the `admin` account in Butane. Before configuring a newly imaged node, run its
|
||||
first playbook execution with `-e ansible_user=admin`; the SSH hardening role then permits that same
|
||||
@@ -213,6 +252,21 @@ ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit aegis --check --diff --ask-become-pass
|
||||
```
|
||||
|
||||
Apply only the independent host DNS configuration, then reboot Aegis manually:
|
||||
|
||||
```bash
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit aegis --tags dns --ask-become-pass
|
||||
```
|
||||
|
||||
Layer the Aegis NFS and WireGuard client tools independently, then reboot Aegis manually when the role reports
|
||||
that the new deployment is ready:
|
||||
|
||||
```bash
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit aegis --tags nfs --ask-become-pass
|
||||
```
|
||||
|
||||
## NAS
|
||||
|
||||
`atlas` is a Rocky Linux 9 NAS reached through SSH. Normally its pool already exists and the profile
|
||||
@@ -223,17 +277,9 @@ clients use NFSv4 and Windows/WSL clients use SMB; both are restricted to the co
|
||||
|
||||
For the first run, provide `vault_atlas_admin_password_hash`, `vault_atlas_samba_password`, and
|
||||
`vault_atlas_immich_db_password`. Bootstrap the host through its
|
||||
existing administrator. Open `51820/udp` towards Prometheus in the provider firewall first, then
|
||||
include both WireGuard peers in the same idempotent playbook run:
|
||||
|
||||
```bash
|
||||
ansible-playbook ansible/site.yml --limit prometheus,atlas \
|
||||
-e atlas_connection_username=<existing-admin> \
|
||||
-e atlas_create_pool=true
|
||||
```
|
||||
|
||||
The explicit pool gate is safe to repeat: the role creates the RAIDZ2 pool only when it is absent.
|
||||
WireGuard waits for a real peer handshake before the play continues.
|
||||
existing administrator. The explicit pool gate is safe to repeat: the role creates the RAIDZ2 pool only when
|
||||
it is absent. Atlas no longer participates in the WireGuard overlay; its old interface is retired manually only after
|
||||
Prometheus and Aegis have completed the replacement handshake.
|
||||
|
||||
`vault_atlas_admin_password_hash` must be an `/etc/shadow`-compatible hash, not a clear-text
|
||||
Cockpit password. Subsequent runs use `atlas_admin_username`. Atlas declares storage, sharing, and its
|
||||
@@ -243,11 +289,11 @@ paths, and the Immich database secret are validated. Atlas reads its declared SS
|
||||
separate files below `~/.ssh/authorized_keys.d/`.
|
||||
|
||||
With storage management enabled, Atlas creates the complete dataset hierarchy below the existing or
|
||||
explicitly bootstrapped `zpool`: `work`, `archive`, `archive/app_data`, the separate `archive/app_data/navidrome` and
|
||||
`archive/app_data/syncthing` application datasets, `media`, `media/music`, `media/photobook`,
|
||||
`backups`, `backups/services`, and `backup_prometheus`. Application/archive datasets use `zstd`,
|
||||
while media, Syncthing and service-backup datasets use `lz4`; `backups/services` also has a `500G`
|
||||
refreservation. Atlas enforces targeted SELinux persistently and reports, without initiating, any reboot required to activate it. It assigns its primary LAN interface explicitly to the managed firewalld zone and applies persistent kernel network hardening: redirects and source routes are rejected, martians logged, reverse-path filtering remains loose for WireGuard, and IPv4 forwarding is disabled. SSH permits only the declared administrator using public-key authentication; root login, passwords,
|
||||
explicitly bootstrapped `zpool`: SMB-shared `archive`, private `services/data` with separate
|
||||
`services/data/navidrome` and `services/data/syncthing` application datasets, `media`, `media/music`,
|
||||
`media/photobook`, and `backup/hosts/prometheus`. Application/archive datasets use `zstd`, while media,
|
||||
Syncthing, and host-backup datasets use `lz4`; `backup` has a `500G` reservation covering its descendants.
|
||||
Atlas enforces targeted SELinux persistently and reports, without initiating, any reboot required to activate it. It assigns its primary LAN interface explicitly to the managed firewalld zone and applies persistent kernel network hardening: redirects and source routes are rejected, martians logged, reverse-path filtering remains loose for WireGuard, and IPv4 forwarding is disabled. SSH permits only the declared administrator using public-key authentication; root login, passwords,
|
||||
agent and remote forwarding are disabled, while local forwarding remains available for private administrative tunnels. SMB3 exposes `Archive` only to the configured Vault-backed
|
||||
Samba accounts on encrypted, signed SMB3 over TCP/445 only and admits the configured LAN without host-specific
|
||||
exclusions. NFSv4 exports only `media/photobook` to the configured Aegis IP over TCP/2049, using
|
||||
@@ -259,53 +305,281 @@ and NPM Quadlets share one Podman network. Immich runs as `1100:1100`; Server an
|
||||
and Photobook is mounted read-only at `/external/photobook`. NPM publishes ports `80` and `443`; its
|
||||
administration interface remains restricted to `127.0.0.1:81` for SSH-tunnel access.
|
||||
|
||||
Phase 1 is limited to rootless Navidrome and Syncthing user Quadlets on Atlas. It is enabled in the
|
||||
Atlas host configuration and can be set to `false` only for a deliberate suspension. Official Navidrome `0.63.2` uses its SQLite database below `/data`; it does
|
||||
not support `ND_DATABASE_URL` or an external PostgreSQL backend. The obsolete `navidromedb` service
|
||||
was therefore removed from Prometheus instead of being reproduced on Atlas. The role derives all
|
||||
storage paths from the `zpool` mounted at `/zpool`: music is read-only at
|
||||
`/zpool/media/music`, Navidrome application state and `navidrome.db` are stored at
|
||||
`/zpool/archive/app_data/navidrome`, and Syncthing persists at
|
||||
`/zpool/archive/app_data/syncthing`. `profile_atlas` creates these datasets when
|
||||
`atlas_manage_storage` is enabled; the backend role verifies their exact mountpoints before starting
|
||||
containers. The backend role never creates the pool. The separate `wireguard_overlay` role manages `wg0`
|
||||
between Prometheus (`10.0.0.1`) and Atlas (`10.0.0.2`), generating private keys once
|
||||
on their respective hosts and exchanging only public keys through Ansible. Prometheus alone opens
|
||||
`51820/udp` publicly. When the WireGuard zone is created, Ansible reloads firewalld and immediately
|
||||
reloads Prometheus' rootful Podman networks so the existing proxy stack retains container DNS and
|
||||
connectivity. Backend ports are admitted only in the WireGuard firewalld zone.
|
||||
Atlas temporarily hosts rootless Navidrome and Syncthing until Uranus replaces them. They bind only to
|
||||
Atlas' LAN address (`192.168.178.55`); WireGuard remains exclusively between Prometheus (`10.0.0.1`)
|
||||
and Aegis (`10.0.0.2`). Their state is initialized ex novo in `/zpool/services/data/navidrome` and
|
||||
`/zpool/services/data/syncthing`; no source application state is migrated. The music library at
|
||||
`/zpool/media/music` was populated separately from `/zpool/archive/Music` on 2026-09-30;
|
||||
Navidrome completed its library scan. The rootless `atlas-music-sync.timer` copies new and changed
|
||||
files daily at 00:45 Europe/Rome, without deleting destination-only files. Both ZFS datasets must
|
||||
be mounted. Its first scheduled run succeeded on 2026-10-02. Some source playlists still contain
|
||||
obsolete Windows paths.
|
||||
|
||||
`backend_phase1_start_services` stays false during the application-state transfer, so the first real
|
||||
backend run renders the Quadlets without creating an empty Atlas database. After stopping Navidrome
|
||||
on Prometheus, copy the complete `/opt/navidrome/data/` directory into
|
||||
`/zpool/archive/app_data/navidrome/`, preserving `navidrome.db` and any SQLite sidecar files. Then set
|
||||
this variable to true and rerun the role to enable and start Navidrome and Syncthing. The playbook
|
||||
never copies or deletes application data.
|
||||
The Gitea move from Prometheus to Atlas is tracked in
|
||||
[`docs/atlas-gitea-migration.md`](docs/atlas-gitea-migration.md). The final consistent copy runs in
|
||||
Atlas' dedicated dataset under `admin`'s rootless user Quadlet. Its pinned derived image uses an
|
||||
internal Unix user named `gitea` (UID/GID 1000), while clone URLs keep `git@`. NPM remains on Prometheus and the primary
|
||||
public HTTPS route serves Atlas. Public SSH/2222 authenticates the `ikaros` key and serves
|
||||
`git ls-remote`; the operator also confirmed SSH pull and push. HTTPS login and Git writes were
|
||||
confirmed on 2026-10-03. The old Gitea data remains on Prometheus, but its container
|
||||
is absent from the desired stack.
|
||||
|
||||
Validate and render the Atlas services with:
|
||||
The separate `wireguard_overlay` role manages `wg0` between Prometheus (`10.0.0.1`) and Aegis
|
||||
(`10.0.0.2`), generating private keys once on their respective hosts and exchanging only public keys
|
||||
through Ansible. Prometheus alone opens `51820/udp`. Aegis forwards only the declared overlay-to-LAN
|
||||
traffic and source-NATs it, so Atlas and future Uranus nodes require neither a VPN interface nor a router
|
||||
static route. Atlas permits Navidrome (`4533/tcp`) and the Syncthing GUI (`8384/tcp`) only from Aegis;
|
||||
Syncthing native ports are limited to the LAN. Configure NPM manually with
|
||||
`http://192.168.178.55:4533` and `http://192.168.178.55:8384` after the services are healthy.
|
||||
Prometheus' peer includes the LAN subnet in `AllowedIPs`; add the Uranus VIP there when it exists.
|
||||
When the WireGuard zone is created, Ansible reloads firewalld and immediately reloads Prometheus'
|
||||
rootful Podman networks so the existing proxy stack retains container DNS and connectivity.
|
||||
|
||||
Validate the gateway with:
|
||||
|
||||
```bash
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags storage
|
||||
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit prometheus,atlas --tags wireguard
|
||||
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 --check --diff
|
||||
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1
|
||||
ansible-playbook ansible/site.yml --limit prometheus,aegis --tags wireguard --check --diff
|
||||
```
|
||||
|
||||
For the cutover, stop the old Navidrome writer before copying its data directory, verify ownership by
|
||||
the Atlas `admin` account and confirm that the copied SQLite database is present before changing
|
||||
`backend_phase1_start_services: true` in `host_vars/atlas.yml`. Keep the source data and the stopped
|
||||
legacy `navidromedb` container until Navidrome on Atlas and a restore test have been validated.
|
||||
The first real WireGuard run must include both peers. If Fedora IoT has just layered `wireguard-tools`,
|
||||
reboot Aegis manually and rerun the command without `--check`; the role then waits for a real peer
|
||||
handshake.
|
||||
|
||||
Snapshot retention, Syncthing topology, WireGuard/firewall validation, Prometheus backup pulls,
|
||||
encrypted Borg backups to a Hetzner Storage Box, USB backup, monitoring, and disaster-recovery tests
|
||||
remain follow-up work. The detailed operational backlog is kept in `AGENTS.md`.
|
||||
Atlas declares recursive, systemd-timed ZFS snapshots for the complete pool hierarchy: 24 hourly
|
||||
snapshots at minute 05, 30 daily snapshots at 00:15, 8 weekly snapshots on Sunday at 01:00, and 12
|
||||
monthly snapshots on the first day at 02:00. The retention helper prunes only snapshots carrying its
|
||||
managed `atlas-auto` prefix and never rolls back a dataset. The OpenZFS monthly scrub timer is scheduled
|
||||
for the first Sunday at 03:00; the conflicting weekly scrub timer is disabled explicitly. The first recursive
|
||||
hourly snapshot completed successfully on Atlas, and scheduled retention pruning was observed on
|
||||
2026-09-30. The first monthly scrub still awaits runtime evidence. Validate this layer independently with:
|
||||
|
||||
```bash
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags snapshots,scrub --check --diff
|
||||
```
|
||||
|
||||
Atlas also declares an encrypted Borg backup to the dedicated Hetzner Storage Box sub-account
|
||||
`u660064-sub1`. The repository is the sub-account-relative `./borg-data` path and uses the explicitly
|
||||
selected remote Borg 1.4 binary over SSH port 23. The ED25519 server key is pinned; a dedicated client
|
||||
key is generated for the locked, non-login `borg` system account, and its private half never leaves
|
||||
`/etc/atlas-borg`. The account has no sudo or supplementary groups and owns only its SSH identity,
|
||||
passphrase, cache, and Borg state. Borg receives its passphrase through a mode `0600` file rendered from
|
||||
`vault_atlas_borg_passphrase`.
|
||||
|
||||
The daily backup starts at 04:30 with up to 30 minutes of randomized delay. It creates a temporary,
|
||||
recursive ZFS snapshot and reconstructs every dataset below `/zpool` as a read-only bind-mounted tree,
|
||||
so parent and child datasets enter one consistent Borg archive. The wrapper recursively unmounts its
|
||||
private source tree; a narrowly scoped `ExecStopPost` helper removes any remaining host-namespace ZFS
|
||||
snapshot mounts and the named temporary snapshot after the backup process exits. Only the root wrapper
|
||||
performs snapshot and mount operations; it launches the Borg client as `borg` with temporary read-search
|
||||
capability and no ZFS, sudo, or pool-management privileges. Borg retains 30 daily, 8 weekly, and 12
|
||||
monthly archives, then compacts the standard
|
||||
read-write repository. A full metadata and repository check runs as `borg` on the fifteenth day of each
|
||||
month at 06:00. Both operations use a common lock, journal logging, and bounded systemd retries.
|
||||
New backup runs also log the create phase and a compact progress line at most once per minute: an
|
||||
**estimated** percentage, dataset, files processed, and original/compressed/deduplicated bytes. The
|
||||
denominator is the summed ZFS `logicalreferenced` size of the backup's own recursive snapshot, not a
|
||||
Borg-reported total: the estimate can exceed 100% and does not cover retention, compaction, or checks.
|
||||
Progress lines omit individual filenames; warnings may still name affected files.
|
||||
Follow the current run with
|
||||
`sudo journalctl -fu atlas-borg-backup.service` on Atlas; changes to the helper do not alter a run
|
||||
already in progress.
|
||||
|
||||
Initial activation remains explicit:
|
||||
|
||||
1. Add a strong unique `vault_atlas_borg_passphrase` with `ansible-vault edit secrets/vault.yml`.
|
||||
2. Generate and display only the dedicated public key with
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags borg_key`.
|
||||
3. Install that public key in the Hetzner sub-account, then apply with
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags packages,borg`.
|
||||
4. Copy the ignored `secrets/recovery/atlas-borg-repokey.export` file to genuinely offline storage.
|
||||
The controller-side copy is not an offline backup by itself.
|
||||
|
||||
The role initializes only the missing `repokey` repository and never accepts an unpinned host key or
|
||||
password authentication. It does not start the first backup manually. Validate the rendered state with:
|
||||
|
||||
```bash
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags packages,borg --check --diff
|
||||
```
|
||||
|
||||
Atlas runtime activation is complete: the initial backup and repository check succeeded, a full restore
|
||||
to a temporary directory was validated against the live `Archive` tree, the recovery-key export was copied
|
||||
to offline storage, and the temporary snapshot and bind mounts were cleaned up.
|
||||
The populated-pool archive on 2026-09-29 took 1 h 32 min for 2.18 TB original / 2.04 TB compressed
|
||||
data, with a 13.49 GB deduplicated archive size. Retention and compaction succeeded, but a
|
||||
`RuntimeDirectory` permission error prevented post-exit snapshot cleanup. After correction, the
|
||||
2026-09-30 incremental archive completed in about 22 seconds, removed the stale and current temporary
|
||||
snapshots, and ended with service status 0. The monitor reported 37% Storage Box quota used. These
|
||||
observations do not predict the duration or compression ratio of future runs.
|
||||
On 2026-09-25 a separate ZFS restore smoke test copied a small file from an automatic daily
|
||||
`zpool/archive` snapshot to `/var/tmp`, then confirmed matching contents, ownership, mode, mtime and
|
||||
POSIX ACL. The temporary copy and on-demand snapshot mount were removed; Borg kept running. This
|
||||
does not validate a full dataset recovery.
|
||||
|
||||
The offline USB backup is deployed as a manual-only service (`atlas_manage_usb_backup: true`):
|
||||
Ansible never formats, unlocks, mounts, backs up to, or schedules the disk. Atlas' existing USB disk was verified
|
||||
read-only on 2026-09-23 as LUKS UUID `577b3c43-ea37-4611-81a9-39d555cdfbd4`, containing ext4 UUID
|
||||
`758e2d2e-a427-4797-aad9-39c3a9f17c7e` through mapper `zpool-backup`. It was mounted at
|
||||
`/mnt/zpool-backup` at inspection time. The service deliberately requires the verified mapper to be
|
||||
**not mounted** before starting. When necessary, `systemd-ask-password` requests the LUKS passphrase
|
||||
through the `systemctl start` password agent; it is piped directly to `cryptsetup` without saving it,
|
||||
passing it as a command argument, or caching it. The service then mounts the disk privately, takes a recursive ZFS snapshot,
|
||||
copies every dataset to a versioned `atlas/snapshots/<timestamp>/` directory using `rsync --link-dest`,
|
||||
verifies the result with a checksum-based dry run, atomically updates `atlas/latest`, unmounts and closes
|
||||
LUKS. A failed run never replaces `latest` or removes an earlier complete version. Borg and the USB
|
||||
backup may run concurrently from separate snapshots; both reading the same pool can reduce throughput.
|
||||
The USB copy preserves ACLs but not generic extended attributes; `security.selinux` is also intentionally
|
||||
excluded because the target SELinux policy must recreate labels during a restore. Do not restore data into
|
||||
service paths without relabeling. After restoring an explicit dataset path, apply its destination policy with:
|
||||
|
||||
```bash
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags restorecon \
|
||||
-e '{"atlas_restorecon_paths":["/zpool/archive"]}'
|
||||
```
|
||||
|
||||
The task accepts only paths below the Atlas pool mount root, runs `restorecon -RFv` only for the paths
|
||||
provided at invocation, and is otherwise a no-op. It must not be used on the whole pool during routine runs.
|
||||
Old USB versions are not pruned automatically, to avoid deleting the only offline
|
||||
copy without an explicitly chosen retention policy; capacity checks include an estimated transfer size
|
||||
and a 10 GiB free-space reserve. The disk must be physically disconnected after a successful backup
|
||||
to make the copy offline.
|
||||
|
||||
To check the USB backup and reminder configuration without starting a backup, run:
|
||||
|
||||
```bash
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags usb_backup,usb_reminder --check --diff
|
||||
```
|
||||
|
||||
To validate a planned, explicit post-restore relabel operation without changing labels, run:
|
||||
|
||||
```bash
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags restorecon --check \
|
||||
-e '{"atlas_restorecon_paths":["/zpool/archive"]}'
|
||||
```
|
||||
|
||||
Before the first **manual** service start, safely unmount the currently mounted
|
||||
`/mnt/zpool-backup`; never run it on an arbitrary mounted disk. Future starts
|
||||
can begin with the mapper closed: `sudo systemctl start atlas-usb-backup.service` prompts for the
|
||||
passphrase interactively and then performs the backup. Neither the LUKS password nor a key file belongs
|
||||
in Ansible. Inspect the run with
|
||||
`sudo journalctl -fu atlas-usb-backup.service`. There is intentionally no timer. Independently test a
|
||||
read-only mount and restore from `atlas/latest` into an empty temporary directory before marking the
|
||||
USB recovery path complete. Only `atlas-usb-reminder.timer` is enabled, for the first Saturday of each
|
||||
month at 10:00 Europe/Rome. Its warning notification uses the existing 45Drives Houston notifier.
|
||||
A manual test confirmed a notification in 45Drives Alerts, **not** an email. The reminder service log
|
||||
reports notification submission, not email delivery; the role does not depend on SMTP/OAuth settings.
|
||||
The reminder never starts the backup. Check its schedule with
|
||||
`systemctl list-timers atlas-usb-reminder.timer` and the result in 45Drives Alerts.
|
||||
The timer was verified active with its first scheduled run at 2026-10-03 10:00 CEST. No email
|
||||
delivery is claimed.
|
||||
The first manual USB attempt on 2026-09-23 did not complete: rsync was denied while removing
|
||||
`security.selinux` on the USB filesystem, then the interrupted service left its recursive
|
||||
`atlas-usb-20260923T185748Z-2469168` snapshot and the `zpool-backup` LUKS mapper open. The
|
||||
rsync xattr filter was deployed afterward. The incomplete USB directory was absent on inspection;
|
||||
the exact failed snapshot was removed, the verified and unmounted mapper closed, and the service
|
||||
failed state cleared. A final check found no remnant snapshot, mount, mapper, or staging directory.
|
||||
The failed attempt was not a valid backup, and no USB restore had been tested at that point.
|
||||
On 2026-09-24 a later run reported a checksum-verified, published USB version and closed the LUKS
|
||||
mapper, but the service failed while destroying its temporary ZFS snapshot: OpenZFS still had
|
||||
on-demand `.zfs/snapshot` mounts open in the host namespace. Those exact temporary snapshots were
|
||||
unmounted normally and removed; no force or rollback was used. The backup service now records its
|
||||
snapshot name and runs a narrowly scoped `ExecStopPost` cleanup after the private backup process
|
||||
exits. The cleanup helper was tested with a disposable recursive snapshot and an active snapshot
|
||||
mount. A complete run on 2026-09-24 later checksum-verified and published a new USB version; the
|
||||
service ended successfully, the LUKS mapper closed, no temporary USB snapshot remained, and the pool
|
||||
was healthy. On 2026-09-25 an independent restore test opened the configured USB disk read-only, mounted
|
||||
ext4 with `ro,noload`, restored a 5,707,945-byte file from the published `atlas/latest` version to an
|
||||
empty `/var/tmp` directory, and matched its content, owner, mode, size, mtime, and POSIX ACL against
|
||||
the USB source. The test removed its temporary copy and mount, closed the LUKS mapper, and left the
|
||||
pool healthy while Borg continued running. This is a file-level recovery smoke test, not a full dataset
|
||||
or disaster-recovery exercise.
|
||||
|
||||
Atlas health monitoring runs every 30 minutes through `atlas-health-monitor.timer`. Its read-only probes
|
||||
check pool/vdev state and errors, scrub/resilver status, four pool disks and the system NVMe via SMART,
|
||||
disk and CPU temperatures, system/pool/snapshot space, local `zpool/backup` growth, and the Hetzner
|
||||
Storage Box quota via `df -m` over the dedicated `borg` account's pinned-key SSH connection. The remote
|
||||
query never opens the Borg repository or its lock. Growth alerts compare against a roughly 24-hour
|
||||
baseline and therefore begin only after enough samples exist. The monitor also checks
|
||||
maintenance/backup timer activation and freshness; systemd `OnFailure` hooks report snapshot,
|
||||
scrub, Borg, USB, reminder, and monitoring services when they enter the failed state. An ongoing
|
||||
Borg run is never restarted by the monitor; only a run exceeding 14 days raises a warning.
|
||||
Thresholds and stable disk paths are declared in Atlas host variables. Alerts use the existing 45Drives
|
||||
Houston notifier and repeated issues are deduplicated; **email delivery is not verified**. The
|
||||
2026-09-25 live probe found no issues and a labelled test notification was submitted. On 2026-09-30
|
||||
the monitor reported zero issues and 37% Storage Box quota used. The failed-job hook now passes the
|
||||
literal systemd unit name; its expansion was verified without sending a false failure notification.
|
||||
|
||||
```bash
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags monitoring --check --diff
|
||||
sudo /usr/local/libexec/atlas-health-monitor --dry-run
|
||||
sudo journalctl -u atlas-health-monitor.service -n 100 --no-pager
|
||||
systemctl list-timers atlas-health-monitor.timer
|
||||
```
|
||||
|
||||
`--dry-run` sends no alerts and does not change monitor state. A real check is
|
||||
`sudo systemctl start atlas-health-monitor.service`; do not start the backup services merely to test
|
||||
monitoring. For a labelled 45Drives Alerts delivery test, use
|
||||
`sudo /usr/local/libexec/atlas-health-monitor --test-notification`.
|
||||
|
||||
### Atlas systemd timers
|
||||
|
||||
All ten managed timers below are enabled. Times are local to Atlas (`Europe/Rome`); Borg and monitoring
|
||||
add the indicated randomized delay. Every timer has `Persistent=true`, so a missed calendar run is
|
||||
scheduled after the timer becomes active again.
|
||||
|
||||
| Timer | Schedule (`OnCalendar`) | Action |
|
||||
| --- | --- | --- |
|
||||
| `atlas-zfs-snapshot-hourly.timer` | `*-*-* *:05:00` — every hour at :05 | Recursive hourly snapshot and retention |
|
||||
| `atlas-zfs-snapshot-daily.timer` | `*-*-* 00:15:00` — daily at 00:15 | Recursive daily snapshot and retention |
|
||||
| `atlas-zfs-snapshot-weekly.timer` | `Sun *-*-* 01:00:00` — Sunday at 01:00 | Recursive weekly snapshot and retention |
|
||||
| `atlas-zfs-snapshot-monthly.timer` | `*-*-01 02:00:00` — first day of the month at 02:00 | Recursive monthly snapshot and retention |
|
||||
| `zfs-scrub-monthly@zpool.timer` | `Sun *-*-01..07 03:00:00` — first Sunday at 03:00 | ZFS scrub |
|
||||
| `atlas-borg-backup.timer` | `*-*-* 04:30:00` — daily at 04:30, plus 0–30 min random delay | Encrypted offsite backup |
|
||||
| `atlas-borg-check.timer` | `*-*-15 06:00:00` — 15th of the month at 06:00, plus 0–30 min random delay | Borg repository check |
|
||||
| `atlas-usb-reminder.timer` | `Sat *-*-01..07 10:00:00 Europe/Rome` — first Saturday at 10:00 | 45Drives Alerts reminder only |
|
||||
| `atlas-health-monitor.timer` | `*:0/30` — every half-hour, plus 0–5 min random delay | Read-only health checks |
|
||||
| `atlas-prometheus-pull.timer` | `*-*-* 03:00:00 Europe/Rome` — daily at 03:00 | Pull and verify the prepared Prometheus backup |
|
||||
|
||||
`atlas-usb-backup.service` has **no timer**: the encrypted USB backup must be started manually.
|
||||
The vendor's `zfs-scrub-weekly@zpool.timer` is intentionally disabled in favor of the monthly scrub.
|
||||
The Prometheus export timer runs at 02:00 Europe/Rome. Its first scheduled export and Atlas pull
|
||||
passed on 2026-10-01; a manual post-NPM-Quadlet export, pull, and temporary restore passed on
|
||||
2026-10-03. The first scheduled cycle after that cutover remains to be observed. While a
|
||||
Borg backup is still running, `systemctl list-timers` may show `-` for its next trigger; this does not
|
||||
mean the timer has been disabled. Inspect the current schedule on Atlas with
|
||||
`systemctl list-timers --all`.
|
||||
|
||||
A temporary Nextcloud deployment on Atlas is also planned before Uranus: it requires separately
|
||||
declared persistent application, database, and cache storage, Vault-backed credentials, NPM-only
|
||||
publishing through Aegis, and defined backup, upgrade, and eventual migration procedures. Do not deploy
|
||||
it before the data-protection checklist is complete.
|
||||
|
||||
Atlas is the declared iCloud photo-ingestion host. Ansible manages the rootless Quadlet, a private
|
||||
Vault-backed `icloudpd.conf`, photos under `/zpool/archive/Pictures/iCloudPD`, and separate state in
|
||||
`zpool/services/data/icloudpd`. The service was started manually; Ansible does not enable automatic
|
||||
startup or manage the password and MFA keyring. The operator initialized MFA interactively; on
|
||||
2026-10-03 the initial photo/video download completed. Aegis iCloudPD, including its service data,
|
||||
has been removed and verified; the Aegis role no longer manages it. Backup/restore and SMB access
|
||||
for the new data remain unverified. The Photobook NFS export remains untouched. See
|
||||
[`docs/atlas-icloudpd-migration.md`](docs/atlas-icloudpd-migration.md).
|
||||
|
||||
The first scheduled Prometheus backup runs and production-size disaster-recovery tests remain follow-up work. The prioritized
|
||||
operational backlog is kept in `AGENTS.md`.
|
||||
|
||||
Priority 2 procedures and decisions are recorded in
|
||||
[`docs/atlas-recovery.md`](docs/atlas-recovery.md),
|
||||
[`docs/atlas-updates.md`](docs/atlas-updates.md), and
|
||||
[`docs/atlas-sharing-decision.md`](docs/atlas-sharing-decision.md).
|
||||
The provisional Atlas recovery objectives are RPO 24 hours and RTO 72 hours;
|
||||
an isolated small-VM OS rebuild, pool import, Ansible reapplication, and
|
||||
snapshot restore passed, but full-size recovery time is unmeasured. `Archive` (SMB) and
|
||||
`photobook` (NFS) remain deliberately separate.
|
||||
The Prometheus pull architecture and manual export/pull/restore evidence are in
|
||||
[`docs/prometheus-backup.md`](docs/prometheus-backup.md). Both daily timers are
|
||||
enabled; their first scheduled runs remain to be verified.
|
||||
|
||||
## How layering works
|
||||
|
||||
@@ -392,8 +666,8 @@ ansible-playbook ansible/site.yml --limit deadalus --tags ai_agents --check --di
|
||||
| `profile_workstation_dev_wsl` | WSL development setup. |
|
||||
| `profile_server` | Server setup. |
|
||||
| `profile_atlas` | Rocky Linux 9 NAS setup. |
|
||||
| `profile_backend_phase1` | Rootless Navidrome and Syncthing on Atlas. |
|
||||
| `wireguard_overlay` | Prometheus/Atlas WireGuard overlay. |
|
||||
| `profile_backend_phase1` | Temporary rootless Atlas Navidrome and Syncthing services. |
|
||||
| `wireguard_overlay` | Prometheus/Aegis WireGuard LAN gateway. |
|
||||
| `profile_aegis` | Fedora IoT always-on LAN node. |
|
||||
| `dotfiles_common` | Shared user dotfiles. |
|
||||
|
||||
@@ -405,8 +679,8 @@ platform_void -> packages_void + services_runit
|
||||
platform_void & graphical_desktop -> profile_desktop_common + profile_desktop_sway + profile_desktop_niri + profile_desktop_host
|
||||
platform_fedora -> packages_fedora + services_systemd
|
||||
platform_rocky -> packages_rocky + services_systemd
|
||||
wireguard_overlay -> wireguard_overlay (after platform_rocky)
|
||||
role_aegis -> profile_aegis
|
||||
wireguard_overlay -> wireguard_overlay (after Aegis profile and platform_rocky)
|
||||
atlas -> profile_atlas
|
||||
role_backend_phase1 -> profile_backend_phase1 (after atlas)
|
||||
rocky_server -> dotfiles_common + profile_server (after platform_rocky)
|
||||
@@ -476,8 +750,9 @@ ansible-playbook ansible/site.yml --limit <host> --tags <tag1>,<tag2> --check --
|
||||
ansible-playbook ansible/site.yml --limit <host> --start-at-task "<task name>" --check --diff
|
||||
ansible-lint ansible/roles/<role>
|
||||
yamllint ansible/path/to/file.yml
|
||||
podman-compose -f /opt/docker/server/docker-compose.yml config
|
||||
ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags music_sync --check --diff
|
||||
```
|
||||
|
||||
## Tags
|
||||
@@ -511,7 +786,7 @@ ansible-playbook ansible/site.yml --list-tags
|
||||
| `sharing` | Atlas NFSv4 and SMB3 configuration. |
|
||||
| `storage` | Atlas child ZFS datasets. |
|
||||
| `tmux` | tmux configuration and plugins. |
|
||||
| `wireguard` | Prometheus/Atlas WireGuard overlay. |
|
||||
| `wireguard` | Prometheus/Aegis WireGuard LAN gateway. |
|
||||
| `wsl` | WSL bootstrap and configuration. |
|
||||
|
||||
## Bootstrapping a new machine
|
||||
|
||||
@@ -9,6 +9,7 @@ fedora_packages_base:
|
||||
- jq
|
||||
- make
|
||||
- nodejs
|
||||
- openssl
|
||||
- ripgrep
|
||||
|
||||
fedora_manage_docker_repo: true
|
||||
|
||||
@@ -40,4 +40,3 @@ server_firewalld_services:
|
||||
server_firewalld_ports: []
|
||||
server_sshd_service_name: sshd
|
||||
server_compose_selinux_mount_option: Z
|
||||
server_syncthing_enabled: false
|
||||
|
||||
@@ -6,6 +6,11 @@ effective_username: "{{ server_username }}"
|
||||
effective_user_group: "{{ server_user_group }}"
|
||||
effective_user_home: "{{ server_user_home }}"
|
||||
server_container_stack_dir: /opt/docker/server
|
||||
server_npm_quadlet_stage: false
|
||||
server_npm_quadlet_cutover: false
|
||||
server_legacy_stack_retired: false
|
||||
server_legacy_cleanup: false
|
||||
server_duckdns_enabled: true
|
||||
ai_agents: {}
|
||||
vim_plugins_enabled: false
|
||||
|
||||
@@ -75,25 +80,42 @@ server_directories:
|
||||
group: root
|
||||
mode: "0755"
|
||||
|
||||
server_syncthing_enabled: true
|
||||
server_syncthing_directories:
|
||||
- path: /opt/syncthing/config
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
- path: /srv/syncthing
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
- path: /srv/syncthing/data
|
||||
owner: "1000"
|
||||
group: "1000"
|
||||
mode: "0755"
|
||||
|
||||
server_sshd_settings:
|
||||
PermitRootLogin: "no"
|
||||
|
||||
server_sshd_allow_users:
|
||||
- "{{ server_username }}"
|
||||
server_backup_export_enabled: false
|
||||
server_backup_username: prometheus-backup
|
||||
server_backup_public_key_name: atlas-pull
|
||||
server_backup_export_root: /var/lib/prometheus-backup-export
|
||||
server_backup_rrsync_path: /usr/share/doc/rsync/support/rrsync
|
||||
server_backup_export_calendar: "*-*-* 02:00:00 Europe/Rome"
|
||||
server_backup_export_start_timer: false
|
||||
# Explicit Gitea cutover helper: installed separately from any outage action.
|
||||
server_gitea_cutover_tools_enabled: false
|
||||
server_gitea_final_export: false
|
||||
server_gitea_on_atlas: false
|
||||
server_gitea_atlas_address: "{{ hostvars['atlas'].ansible_host }}"
|
||||
server_gitea_npm_domains: []
|
||||
server_gitea_ssh_public_port: 2222
|
||||
server_gitea_ssh_target_port: 2222
|
||||
server_backup_export_source_keep: 3
|
||||
server_backup_export_paths: >-
|
||||
{{ ['opt/npm/data', 'opt/npm/letsencrypt']
|
||||
+ ([] if server_gitea_on_atlas | bool else ['opt/gitea/data', 'home/git/.ssh'])
|
||||
+ ([] if server_legacy_stack_retired | bool else
|
||||
['opt/docker/server/docker-compose.yml',
|
||||
'etc/systemd/system/podman-compose-server.service'])
|
||||
+ (['etc/containers/systemd/prometheus-npm.container',
|
||||
'etc/containers/systemd/server-web.network']
|
||||
if server_npm_quadlet_stage | bool else [])
|
||||
+ ['etc/ssh/sshd_config', 'etc/ssh/sshd_config.d',
|
||||
'etc/firewalld', 'etc/wireguard/wg0.conf'] }}
|
||||
server_backup_export_excludes: >-
|
||||
{{ ['opt/npm/data/logs']
|
||||
+ ([] if server_gitea_on_atlas | bool else
|
||||
['opt/gitea/data/gitea/log', 'opt/gitea/data/gitea/tmp',
|
||||
'opt/gitea/data/gitea/sessions', 'opt/gitea/data/gitea/indexers']) }}
|
||||
server_ssh_authorized_keys: []
|
||||
server_ssh_authorized_key_directory: "{{ server_user_home }}/.ssh/authorized_keys.d"
|
||||
|
||||
@@ -12,9 +12,10 @@ workstation_dev_wsl_packages:
|
||||
- python3-pip
|
||||
- tmux
|
||||
|
||||
# Java 11 and Maven are managed by Mise on this Fedora WSL profile. Keep their
|
||||
# versions pinned; update them deliberately.
|
||||
# Java 11, Java 25 and Maven are managed by Mise on this Fedora WSL profile.
|
||||
# Keep their versions pinned; update them deliberately.
|
||||
workstation_mise_java_version: temurin-11.0.31+11
|
||||
workstation_mise_java_25_version: 25.0.2
|
||||
workstation_mise_maven_version: 3.9.16
|
||||
|
||||
workstation_is_wsl: true
|
||||
|
||||
@@ -9,6 +9,32 @@ ansible_ssh_use_tty: false
|
||||
|
||||
aegis_lan_subnet: 192.168.178.0/24
|
||||
aegis_adguard_web_port: 80
|
||||
aegis_network_connection_name: FRITZ!Box 7530 NR
|
||||
aegis_network_connection_uuid: a52fda3d-3eb6-351f-bf04-753edcb76944
|
||||
aegis_host_dns_servers:
|
||||
- 192.168.178.1
|
||||
aegis_host_dns_search_domains:
|
||||
- fritz.box
|
||||
|
||||
# Aegis is the internal WireGuard endpoint and NAT gateway for the LAN.
|
||||
wireguard_overlay_enabled: true
|
||||
wireguard_address: 10.0.0.2/24
|
||||
wireguard_enable_ipv4_forwarding: true
|
||||
wireguard_reload_rootful_podman_networks: true
|
||||
wireguard_forwarding_policies:
|
||||
- name: aegis-wireguard-to-lan
|
||||
ingress_zone: wireguard
|
||||
egress_zone: public
|
||||
source: 10.0.0.0/24
|
||||
destination: "{{ aegis_lan_subnet }}"
|
||||
masquerade: true
|
||||
wireguard_peers:
|
||||
- name: prometheus
|
||||
host: prometheus
|
||||
endpoint: "{{ hostvars['prometheus']['ansible_host'] }}:{{ hostvars['prometheus']['wireguard_listen_port'] }}"
|
||||
allowed_ips:
|
||||
- 10.0.0.0/24
|
||||
persistent_keepalive: 25
|
||||
aegis_ssh_authorized_keys:
|
||||
- name: ikaros
|
||||
key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINrIxXjA3ffPwziKGR5gzc4gAoBehQPlnEMcXF4Wl0ZS ikaros"
|
||||
@@ -16,5 +42,3 @@ aegis_ssh_authorized_keys:
|
||||
key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEH/7GJfGt0ZVmKeEzceoFkFkeCXFryKK9vAbaip+HCx nymph"
|
||||
- name: siren
|
||||
key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA95wYlzpfN3rjUhpMeP4KHn8I6ZrjQXoDTgwgRIa++b siren"
|
||||
|
||||
aegis_icloudpd_apple_id: "{{ vault_aegis_icloudpd_apple_id | default('') }}"
|
||||
|
||||
@@ -35,35 +35,124 @@ atlas_zpool_disks:
|
||||
- /dev/disk/by-id/ata-ST4000VN006-3CW104_WW6A6VJK
|
||||
- /dev/disk/by-id/ata-ST4000VN006-3CW104_WW6AC1LM
|
||||
atlas_create_pool: false
|
||||
atlas_zfs_dataset_work: work
|
||||
atlas_zfs_dataset_archive: archive
|
||||
atlas_zfs_dataset_app_data: archive/app_data
|
||||
atlas_zfs_dataset_navidrome: archive/app_data/navidrome
|
||||
atlas_zfs_dataset_syncthing: archive/app_data/syncthing
|
||||
atlas_zfs_dataset_services: services
|
||||
atlas_zfs_dataset_app_data: services/data
|
||||
atlas_zfs_dataset_navidrome: services/data/navidrome
|
||||
atlas_zfs_dataset_syncthing: services/data/syncthing
|
||||
atlas_zfs_dataset_media: media
|
||||
atlas_zfs_dataset_music: media/music
|
||||
atlas_zfs_dataset_backup_prometheus: backup_prometheus
|
||||
atlas_zfs_dataset_backup: backup
|
||||
atlas_zfs_dataset_host_backups: backup/hosts
|
||||
atlas_zfs_dataset_backup_prometheus: backup/hosts/prometheus
|
||||
atlas_zfs_backup_reservation: 500G
|
||||
atlas_zfs_dataset_photobook: media/photobook
|
||||
atlas_zfs_dataset_backups: backups
|
||||
atlas_zfs_dataset_service_backups: backups/services
|
||||
atlas_mount_root: /zpool
|
||||
atlas_manage_storage: true
|
||||
# Rootless Gitea was restored from the stopped-source export before production activation.
|
||||
atlas_manage_gitea: true
|
||||
atlas_gitea_production_enabled: true
|
||||
atlas_gitea_public_domain: git.fscotto.co
|
||||
atlas_prometheus_pull_start_timer: true
|
||||
atlas_manage_zfs_snapshots: true
|
||||
atlas_zfs_snapshot_prefix: atlas-auto
|
||||
atlas_zfs_snapshot_policies:
|
||||
- name: hourly
|
||||
calendar: "*-*-* *:05:00"
|
||||
keep: 24
|
||||
- name: daily
|
||||
calendar: "*-*-* 00:15:00"
|
||||
keep: 30
|
||||
- name: weekly
|
||||
calendar: "Sun *-*-* 01:00:00"
|
||||
keep: 8
|
||||
- name: monthly
|
||||
calendar: "*-*-01 02:00:00"
|
||||
keep: 12
|
||||
atlas_manage_zfs_scrub: true
|
||||
atlas_zfs_scrub_calendar: "Sun *-*-01..07 03:00:00"
|
||||
atlas_manage_borg_backup: true
|
||||
atlas_borg_repository_host: u660064-sub1.your-storagebox.de
|
||||
atlas_borg_repository_user: u660064-sub1
|
||||
atlas_borg_repository_port: 23
|
||||
atlas_borg_repository_path: ./borg-data
|
||||
atlas_borg_remote_path: borg-1.4
|
||||
# Verified against Hetzner's published ED25519 fingerprint on 2026-09-17:
|
||||
# SHA256:XqONwb1S0zuj5A1CDxpOSuD2hnAArV1A3wKY7Z3sdgM
|
||||
atlas_borg_host_key: >-
|
||||
[u660064-sub1.your-storagebox.de]:23 ssh-ed25519
|
||||
AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs
|
||||
atlas_borg_backup_calendar: "*-*-* 04:30:00"
|
||||
atlas_borg_check_calendar: "*-*-15 06:00:00"
|
||||
atlas_borg_randomized_delay: 30m
|
||||
atlas_borg_keep_daily: 30
|
||||
atlas_borg_keep_weekly: 8
|
||||
atlas_borg_keep_monthly: 12
|
||||
atlas_manage_usb_backup: true
|
||||
# Read-only lsblk verification on Atlas, 2026-09-23. Never store the LUKS password here.
|
||||
atlas_usb_backup_luks_uuid: 577b3c43-ea37-4611-81a9-39d555cdfbd4
|
||||
atlas_usb_backup_fs_uuid: 758e2d2e-a427-4797-aad9-39c3a9f17c7e
|
||||
atlas_usb_backup_mapper_name: zpool-backup
|
||||
atlas_manage_usb_reminder: true
|
||||
atlas_usb_reminder_calendar: "Sat *-*-01..07 10:00:00 Europe/Rome"
|
||||
atlas_manage_monitoring: true
|
||||
atlas_manage_prometheus_backup_pull: true
|
||||
# Prometheus ED25519 host key read through the controller's strict SSH trust on 2026-09-30.
|
||||
# Fingerprint: SHA256:rfedk7DHI9mLB3UHk/4F3HHlSIiswtCAFsAXvfh6iXk
|
||||
atlas_prometheus_ssh_host_key: >-
|
||||
179.237.102.172 ssh-ed25519
|
||||
AAAAC3NzaC1lZDI1NTE5AAAAIC4b+QXlPupoEx71W9NKs9tTeYjBqTkVMqbGB97nMNWv
|
||||
# Physical pool disks and the system NVMe; the disconnected USB disk is intentionally excluded.
|
||||
atlas_monitor_smart_devices:
|
||||
- { name: pool-1, path: "{{ atlas_zpool_disks[0] }}", warning_c: 50, critical_c: 55 }
|
||||
- { name: pool-2, path: "{{ atlas_zpool_disks[1] }}", warning_c: 50, critical_c: 55 }
|
||||
- { name: pool-3, path: "{{ atlas_zpool_disks[2] }}", warning_c: 50, critical_c: 55 }
|
||||
- { name: pool-4, path: "{{ atlas_zpool_disks[3] }}", warning_c: 50, critical_c: 55 }
|
||||
- name: system-nvme
|
||||
path: /dev/disk/by-id/nvme-Patriot_M.2_P320_256GB_P320ADB26011606111
|
||||
warning_c: 70
|
||||
critical_c: 85
|
||||
atlas_monitor_timers:
|
||||
- { name: atlas-zfs-snapshot-hourly.timer, max_age_hours: 3 }
|
||||
- { name: atlas-zfs-snapshot-daily.timer, max_age_hours: 36 }
|
||||
- { name: atlas-zfs-snapshot-weekly.timer, max_age_hours: 216 }
|
||||
- { name: atlas-zfs-snapshot-monthly.timer, max_age_hours: 960 }
|
||||
- { name: zfs-scrub-monthly@zpool.timer, max_age_hours: 960 }
|
||||
- { name: atlas-borg-backup.timer, max_age_hours: 48 }
|
||||
- { name: atlas-borg-check.timer, max_age_hours: 960 }
|
||||
# The first manual USB reminder is not due until October; activation is checked, not age.
|
||||
- { name: atlas-usb-reminder.timer, max_age_hours: 0 }
|
||||
atlas_monitor_failure_units:
|
||||
- atlas-zfs-snapshot@.service
|
||||
- zfs-scrub@zpool.service
|
||||
- atlas-borg-backup.service
|
||||
- atlas-borg-check.service
|
||||
- atlas-usb-backup.service
|
||||
- atlas-usb-reminder.service
|
||||
- atlas-health-monitor.service
|
||||
atlas_monitor_remote_capacity:
|
||||
user: "{{ atlas_borg_repository_user }}"
|
||||
host: "{{ atlas_borg_repository_host }}"
|
||||
run_as: "{{ atlas_borg_username }}"
|
||||
ssh_wrapper: "{{ atlas_borg_ssh_wrapper_path }}"
|
||||
warning_percent: 80
|
||||
critical_percent: 90
|
||||
growth_warning_gib_day: 500
|
||||
atlas_manage_sharing: true
|
||||
atlas_manage_media_stack: false
|
||||
# Planned after data-protection validation: move iCloudPD photo ingestion from
|
||||
# Aegis to Atlas, with photos under /zpool/archive/Pictures/iCloudPD and
|
||||
# application/MFA state in a separate dataset. Do not deploy or cut over yet.
|
||||
|
||||
wireguard_overlay_enabled: true
|
||||
wireguard_address: 10.0.0.2/24
|
||||
wireguard_peers:
|
||||
- name: prometheus
|
||||
host: prometheus
|
||||
endpoint: "{{ hostvars['prometheus']['ansible_host'] }}:{{ hostvars['prometheus']['wireguard_listen_port'] }}"
|
||||
allowed_ips:
|
||||
- 10.0.0.0/24
|
||||
persistent_keepalive: 25
|
||||
|
||||
# WireGuard is retired on Atlas. These rootless services are a temporary home
|
||||
# until Uranus replaces them.
|
||||
backend_phase1_enabled: true
|
||||
backend_phase1_start_services: false
|
||||
backend_phase1_wireguard_address: 10.0.0.2
|
||||
backend_phase1_start_services: true
|
||||
backend_phase1_bind_address: "{{ ansible_host }}"
|
||||
backend_phase1_firewalld_zone: "{{ atlas_firewalld_zone }}"
|
||||
backend_phase1_npm_source_ip: "{{ atlas_aegis_ip }}"
|
||||
backend_phase1_syncthing_native_subnet: "{{ atlas_lan_subnet }}"
|
||||
backend_phase1_music_sync_enabled: true
|
||||
|
||||
rocky_manage_openzfs_repo: true
|
||||
rocky_manage_syncthing_binary: false
|
||||
@@ -73,13 +162,21 @@ rocky_podman_packages:
|
||||
|
||||
host_packages:
|
||||
- cockpit
|
||||
- cockpit-podman
|
||||
- cockpit-storaged
|
||||
- realmd
|
||||
- pcp
|
||||
- python3-pcp
|
||||
- cryptsetup
|
||||
- nfs-utils
|
||||
- policycoreutils
|
||||
- policycoreutils-python-utils
|
||||
- python3-libselinux
|
||||
- setroubleshoot-server
|
||||
- samba
|
||||
- samba-client
|
||||
- samba-common-tools
|
||||
- borgbackup
|
||||
- zfs
|
||||
|
||||
atlas_nfs_exports:
|
||||
@@ -113,4 +210,5 @@ atlas_firewalld_rich_rules:
|
||||
host_enabled_services:
|
||||
- sshd
|
||||
- cockpit.socket
|
||||
- pmlogger.service
|
||||
- zfs.target
|
||||
|
||||
@@ -6,7 +6,28 @@ ansible_port: 22
|
||||
ansible_ssh_private_key_file: /home/fscotto/.ssh/id_ed25519
|
||||
|
||||
server_username: rocky
|
||||
server_legacy_stack_retired: true
|
||||
# Destructive deletion runs only with an explicit extra-var and cleanup tag.
|
||||
server_legacy_cleanup: false
|
||||
# Explicit opt-in cleanup; no data, volumes, networks or NPM images are removed.
|
||||
server_legacy_image_cleanup: false
|
||||
server_legacy_images:
|
||||
- docker.gitea.com/gitea:1.25.2
|
||||
- docker.io/deluan/navidrome:latest
|
||||
- docker.io/library/postgres:13
|
||||
server_npm_quadlet_stage: true
|
||||
server_npm_quadlet_image: docker.io/jc21/nginx-proxy-manager@sha256:52b2c59994f3d36acfcf70a1626f29734df0ed8c71bacc0269f78b6f939858bb
|
||||
# The stopped-source export and live Quadlet cutover passed on 2026-10-03.
|
||||
server_npm_quadlet_cutover: true
|
||||
server_backup_export_enabled: true
|
||||
server_backup_export_start_timer: true
|
||||
# Install the final-copy helper only; it is never run by a normal playbook invocation.
|
||||
server_gitea_cutover_tools_enabled: true
|
||||
server_gitea_on_atlas: true
|
||||
server_gitea_npm_domains:
|
||||
- git.fscotto.duckdns.org
|
||||
server_duckdns_domain: fscotto
|
||||
server_duckdns_enabled: false
|
||||
server_ssh_authorized_keys:
|
||||
- name: ikaros
|
||||
key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINrIxXjA3ffPwziKGR5gzc4gAoBehQPlnEMcXF4Wl0ZS ikaros"
|
||||
@@ -21,15 +42,23 @@ wireguard_listen_port: 51820
|
||||
wireguard_enable_ipv4_forwarding: true
|
||||
wireguard_reload_rootful_podman_networks: true
|
||||
wireguard_peers:
|
||||
- name: atlas
|
||||
host: atlas
|
||||
- name: aegis
|
||||
host: aegis
|
||||
# TODO: Aggiungere VIP di Uranus in futuro
|
||||
allowed_ips:
|
||||
- 10.0.0.2/32
|
||||
- 192.168.178.0/24
|
||||
|
||||
host_packages:
|
||||
- cockpit
|
||||
- cockpit-navigator
|
||||
- cockpit-podman
|
||||
- cockpit-storaged
|
||||
- realmd
|
||||
- pcp
|
||||
- python3-pcp
|
||||
- setroubleshoot-server
|
||||
|
||||
host_enabled_services:
|
||||
- cockpit.socket
|
||||
- pmlogger.service
|
||||
|
||||
@@ -99,7 +99,7 @@ all:
|
||||
wireguard_overlay:
|
||||
hosts:
|
||||
prometheus:
|
||||
atlas:
|
||||
aegis:
|
||||
|
||||
role_backend_phase1:
|
||||
hosts:
|
||||
|
||||
@@ -39,11 +39,41 @@
|
||||
state: enabled
|
||||
when: "'workstation_dev_wsl' in group_names"
|
||||
|
||||
- name: Install distribution signing keys for Fedora desktop codecs
|
||||
tags: [packages, heic]
|
||||
ansible.builtin.dnf:
|
||||
name: distribution-gpg-keys
|
||||
state: present
|
||||
when: "'graphical_desktop' in group_names"
|
||||
|
||||
- name: Import RPM Fusion Free signing key for Fedora desktop codecs
|
||||
tags: [packages, heic]
|
||||
ansible.builtin.rpm_key:
|
||||
key: /usr/share/distribution-gpg-keys/rpmfusion/RPM-GPG-KEY-rpmfusion-free-fedora-2020
|
||||
fingerprint: E9A491A3DE247814E7E067EAE06F8ECDD651FF2E
|
||||
state: present
|
||||
when: "'graphical_desktop' in group_names"
|
||||
|
||||
- name: Enable RPM Fusion Free for Fedora desktop codecs
|
||||
tags: [packages, heic]
|
||||
ansible.builtin.dnf:
|
||||
name: "https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-{{ ansible_facts['distribution_major_version'] }}.noarch.rpm"
|
||||
state: present
|
||||
when: "'graphical_desktop' in group_names"
|
||||
|
||||
- name: Refresh dnf package metadata
|
||||
tags: [packages]
|
||||
ansible.builtin.dnf:
|
||||
update_cache: true
|
||||
|
||||
- name: Install HEIC decoder on Fedora desktops
|
||||
tags: [packages, heic]
|
||||
ansible.builtin.dnf:
|
||||
name: libheif-freeworld
|
||||
state: present
|
||||
update_cache: true
|
||||
when: "'graphical_desktop' in group_names"
|
||||
|
||||
- name: Install packages on Fedora
|
||||
tags: [packages]
|
||||
ansible.builtin.dnf:
|
||||
|
||||
@@ -1,10 +1,13 @@
|
||||
---
|
||||
aegis_hostname: aegis
|
||||
aegis_layered_packages:
|
||||
- nfs-utils
|
||||
- wireguard-tools
|
||||
aegis_network_connection_name: ""
|
||||
aegis_network_connection_uuid: ""
|
||||
aegis_host_dns_servers: []
|
||||
aegis_host_dns_search_domains: []
|
||||
aegis_adguard_image: docker.io/adguard/adguardhome:latest
|
||||
aegis_icloudpd_image: docker.io/boredazfcuk/icloudpd:latest
|
||||
aegis_icloudpd_folder_structure: '{:%Y/%m/%d}'
|
||||
aegis_icloudpd_synchronisation_interval: 86400
|
||||
aegis_icloudpd_apple_id: ""
|
||||
aegis_ikaros_mac_address: aa:bb:cc:dd:ee:ff
|
||||
aegis_wol_port: 9
|
||||
|
||||
|
||||
@@ -9,13 +9,12 @@
|
||||
name: sshd.service
|
||||
state: reloaded
|
||||
|
||||
- name: Restart Aegis Quadlet services
|
||||
- name: Restart Aegis AdGuard Quadlet
|
||||
ansible.builtin.systemd:
|
||||
name: "{{ item }}"
|
||||
state: restarted
|
||||
daemon_reload: true
|
||||
loop:
|
||||
- adguardhome.service
|
||||
- icloudpd.service
|
||||
loop_control:
|
||||
label: "{{ item }}"
|
||||
|
||||
@@ -1,22 +1,99 @@
|
||||
---
|
||||
- name: Require Aegis iCloudPD Apple ID
|
||||
tags: [aegis, icloudpd]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- aegis_icloudpd_apple_id | length > 0
|
||||
fail_msg: Define vault_aegis_icloudpd_apple_id before applying the Aegis profile.
|
||||
no_log: true
|
||||
- name: Manage Aegis layered packages
|
||||
tags: [aegis, packages, nfs, wireguard]
|
||||
community.general.rpm_ostree_pkg:
|
||||
name: "{{ aegis_layered_packages }}"
|
||||
state: present
|
||||
register: aegis_layered_packages_result
|
||||
when: aegis_layered_packages | length > 0
|
||||
|
||||
- name: Report reboot required for Aegis layered packages
|
||||
tags: [aegis, packages, nfs, wireguard]
|
||||
ansible.builtin.debug:
|
||||
msg: Reboot Aegis to activate the newly layered packages, then rerun the playbook.
|
||||
when: aegis_layered_packages_result.needs_reboot | default(false)
|
||||
|
||||
- name: Require completed Aegis network placeholders
|
||||
tags: [aegis, firewall, services]
|
||||
tags: [aegis, dns, firewall, network, services]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- aegis_lan_subnet != 'CHANGEME_LAN_SUBNET'
|
||||
- aegis_firewalld_zone | length > 0
|
||||
- aegis_adguard_web_port | int > 0
|
||||
- aegis_adguard_web_port | int < 65536
|
||||
- aegis_network_connection_name | length > 0
|
||||
- aegis_network_connection_uuid | length > 0
|
||||
- aegis_host_dns_servers | length > 0
|
||||
- ansible_facts["default_ipv4"]["address"] not in aegis_host_dns_servers
|
||||
- aegis_ssh_allowed_users | length > 0
|
||||
fail_msg: Define the Aegis LAN subnet, firewalld zone, AdGuard web port, and SSH users.
|
||||
fail_msg: >-
|
||||
Define the Aegis LAN subnet, firewalld zone, AdGuard web port, independent host DNS,
|
||||
NetworkManager connection, and SSH users. Aegis must not use its own address as upstream DNS.
|
||||
|
||||
- name: Verify the declared Aegis NetworkManager connection exists
|
||||
tags: [aegis, dns, network, services]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- nmcli
|
||||
- --get-values
|
||||
- connection.id
|
||||
- connection
|
||||
- show
|
||||
- uuid
|
||||
- "{{ aegis_network_connection_uuid }}"
|
||||
register: aegis_network_connection
|
||||
changed_when: false
|
||||
failed_when: >-
|
||||
aegis_network_connection.rc != 0
|
||||
or aegis_network_connection.stdout != aegis_network_connection_name
|
||||
|
||||
- name: Read the current Aegis host DNS configuration
|
||||
tags: [aegis, dns, network, services]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- nmcli
|
||||
- --get-values
|
||||
- ipv4.ignore-auto-dns,ipv4.dns,ipv4.dns-search,ipv6.ignore-auto-dns
|
||||
- connection
|
||||
- show
|
||||
- uuid
|
||||
- "{{ aegis_network_connection_uuid }}"
|
||||
register: aegis_host_dns_current
|
||||
changed_when: false
|
||||
|
||||
- name: Configure independent upstream DNS for the Aegis host
|
||||
tags: [aegis, dns, network, services]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- nmcli
|
||||
- connection
|
||||
- modify
|
||||
- uuid
|
||||
- "{{ aegis_network_connection_uuid }}"
|
||||
- ipv4.ignore-auto-dns
|
||||
- "yes"
|
||||
- ipv4.dns
|
||||
- "{{ aegis_host_dns_servers | join(',') }}"
|
||||
- ipv4.dns-search
|
||||
- "{{ aegis_host_dns_search_domains | join(',') }}"
|
||||
- ipv6.ignore-auto-dns
|
||||
- "yes"
|
||||
register: aegis_host_dns_profile
|
||||
when: >-
|
||||
aegis_host_dns_current.stdout_lines !=
|
||||
[
|
||||
'yes',
|
||||
aegis_host_dns_servers | join(','),
|
||||
aegis_host_dns_search_domains | join(','),
|
||||
'yes'
|
||||
]
|
||||
changed_when: true
|
||||
|
||||
- name: Report Aegis reboot required for host DNS changes
|
||||
tags: [aegis, dns, network, services]
|
||||
ansible.builtin.debug:
|
||||
msg: Reboot Aegis to activate its independent upstream DNS before testing another OS update.
|
||||
when: aegis_host_dns_profile.changed | default(false)
|
||||
|
||||
- name: Set Aegis hostname
|
||||
tags: [aegis, services]
|
||||
@@ -34,8 +111,6 @@
|
||||
loop:
|
||||
- /var/lib/adguard/work
|
||||
- /var/lib/adguard/conf
|
||||
- /var/lib/icloudpd/data
|
||||
- /var/lib/icloudpd/config
|
||||
|
||||
- name: Create Quadlet configuration directory
|
||||
tags: [aegis, containers]
|
||||
@@ -57,12 +132,9 @@
|
||||
loop:
|
||||
- src: adguardhome.container.j2
|
||||
dest: adguardhome.container
|
||||
- src: icloudpd.container.j2
|
||||
dest: icloudpd.container
|
||||
loop_control:
|
||||
label: "{{ item.dest }}"
|
||||
no_log: "{{ item.dest == 'icloudpd.container' }}"
|
||||
notify: Restart Aegis Quadlet services
|
||||
notify: Restart Aegis AdGuard Quadlet
|
||||
|
||||
- name: Create Aegis systemd-resolved configuration directory
|
||||
tags: [aegis, adguard, dns, services]
|
||||
@@ -83,7 +155,7 @@
|
||||
mode: "0644"
|
||||
notify:
|
||||
- Restart Aegis systemd-resolved
|
||||
- Restart Aegis Quadlet services
|
||||
- Restart Aegis AdGuard Quadlet
|
||||
|
||||
- name: Point Aegis resolver at the full systemd-resolved configuration
|
||||
tags: [aegis, adguard, dns, services]
|
||||
@@ -276,7 +348,7 @@
|
||||
group: root
|
||||
mode: "0755"
|
||||
|
||||
- name: Enable Aegis Quadlet services and automatic updates
|
||||
- name: Enable Aegis AdGuard Quadlet and automatic updates
|
||||
tags: [aegis, containers, services]
|
||||
ansible.builtin.systemd:
|
||||
name: "{{ item }}"
|
||||
@@ -285,7 +357,6 @@
|
||||
daemon_reload: true
|
||||
loop:
|
||||
- adguardhome.service
|
||||
- icloudpd.service
|
||||
- podman-auto-update.timer
|
||||
loop_control:
|
||||
label: "{{ item }}"
|
||||
|
||||
@@ -1,20 +0,0 @@
|
||||
# Managed by Ansible. Do not edit manually.
|
||||
[Unit]
|
||||
Description=iCloud Photos Downloader
|
||||
Wants=network-online.target
|
||||
After=network-online.target
|
||||
|
||||
[Container]
|
||||
Image={{ aegis_icloudpd_image }}
|
||||
Environment=apple_id={{ aegis_icloudpd_apple_id }}
|
||||
Environment=folder_structure={{ aegis_icloudpd_folder_structure }}
|
||||
Environment=synchronisation_interval={{ aegis_icloudpd_synchronisation_interval }}
|
||||
Volume=/var/lib/icloudpd/data:/home/root/iCloud:Z
|
||||
Volume=/var/lib/icloudpd/config:/config:Z
|
||||
AutoUpdate=registry
|
||||
|
||||
[Service]
|
||||
Restart=always
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -47,31 +47,164 @@ atlas_hardening_sysctls:
|
||||
net.ipv6.conf.all.accept_source_route: "0"
|
||||
net.ipv6.conf.default.accept_source_route: "0"
|
||||
atlas_zfs_pool: CHANGEME_ZFS_POOL
|
||||
atlas_zfs_dataset_work: work
|
||||
atlas_zfs_dataset_archive: archive
|
||||
atlas_zfs_dataset_app_data: archive/app_data
|
||||
atlas_zfs_dataset_navidrome: archive/app_data/navidrome
|
||||
atlas_zfs_dataset_syncthing: archive/app_data/syncthing
|
||||
atlas_zfs_dataset_services: services
|
||||
atlas_zfs_dataset_app_data: services/data
|
||||
atlas_zfs_dataset_navidrome: services/data/navidrome
|
||||
atlas_zfs_dataset_syncthing: services/data/syncthing
|
||||
atlas_zfs_dataset_media: media
|
||||
atlas_zfs_dataset_music: media/music
|
||||
atlas_zfs_dataset_backup_prometheus: backup_prometheus
|
||||
atlas_zfs_dataset_backup: backup
|
||||
atlas_zfs_dataset_host_backups: backup/hosts
|
||||
atlas_zfs_dataset_backup_prometheus: backup/hosts/prometheus
|
||||
atlas_zfs_backup_reservation: 500G
|
||||
atlas_zfs_dataset_photobook: media/photobook
|
||||
atlas_zfs_dataset_backups: backups
|
||||
atlas_zfs_dataset_service_backups: backups/services
|
||||
atlas_zfs_service_backups_refreservation: 500G
|
||||
atlas_mount_root: /CHANGEME_ATLAS_MOUNT_ROOT
|
||||
|
||||
atlas_manage_zfs_snapshots: false
|
||||
atlas_zfs_snapshot_prefix: atlas-auto
|
||||
atlas_zfs_snapshot_policies: []
|
||||
atlas_manage_zfs_scrub: false
|
||||
atlas_zfs_scrub_calendar: ""
|
||||
|
||||
atlas_manage_borg_backup: false
|
||||
atlas_borg_username: borg
|
||||
atlas_borg_group: borg
|
||||
atlas_borg_home: /var/lib/atlas-borg
|
||||
atlas_borg_repository_host: CHANGEME_BORG_HOST
|
||||
atlas_borg_repository_user: CHANGEME_BORG_USER
|
||||
atlas_borg_repository_port: 23
|
||||
atlas_borg_repository_path: ./borg-data
|
||||
atlas_borg_remote_path: borg-1.4
|
||||
atlas_borg_host_key: ""
|
||||
atlas_borg_ssh_private_key_path: /etc/atlas-borg/id_ed25519
|
||||
atlas_borg_known_hosts_path: /etc/atlas-borg/known_hosts
|
||||
atlas_borg_passphrase_path: /etc/atlas-borg/passphrase
|
||||
atlas_borg_ssh_wrapper_path: /usr/local/libexec/atlas-borg-ssh
|
||||
atlas_borg_passphrase: "{{ vault_atlas_borg_passphrase | default('') }}"
|
||||
atlas_borg_encryption_mode: repokey
|
||||
atlas_borg_archive_prefix: atlas
|
||||
atlas_borg_snapshot_prefix: atlas-borg
|
||||
atlas_borg_compression: auto,zstd,3
|
||||
atlas_borg_backup_calendar: ""
|
||||
atlas_borg_check_calendar: ""
|
||||
atlas_borg_randomized_delay: 30m
|
||||
atlas_borg_keep_daily: 30
|
||||
atlas_borg_keep_weekly: 8
|
||||
atlas_borg_keep_monthly: 12
|
||||
atlas_borg_config_dir: /var/lib/atlas-borg
|
||||
atlas_borg_cache_dir: /var/cache/atlas-borg
|
||||
atlas_borg_lock_path: /var/lib/atlas-borg/backup.lock
|
||||
atlas_borg_recovery_export_path: "{{ playbook_dir }}/../secrets/recovery/atlas-borg-repokey.export"
|
||||
|
||||
# Manual-only offline backup. No USB device is formatted or mounted by Ansible.
|
||||
atlas_manage_usb_backup: false
|
||||
atlas_usb_backup_luks_uuid: ""
|
||||
atlas_usb_backup_fs_uuid: ""
|
||||
atlas_usb_backup_mapper_name: atlas-usb-backup
|
||||
atlas_usb_backup_min_free_bytes: 10737418240
|
||||
atlas_usb_backup_snapshot_prefix: atlas-usb
|
||||
atlas_manage_usb_reminder: false
|
||||
atlas_usb_reminder_calendar: ""
|
||||
atlas_usb_reminder_notifier: /opt/45drives/houston/houston-notify
|
||||
|
||||
# Read-only health probes and 45Drives Alerts; disabled outside Atlas host vars.
|
||||
atlas_manage_monitoring: false
|
||||
atlas_monitor_calendar: "*:0/30"
|
||||
atlas_monitor_notifier: "{{ atlas_usb_reminder_notifier }}"
|
||||
atlas_monitor_smart_devices: []
|
||||
atlas_monitor_timers: []
|
||||
atlas_monitor_failure_units: []
|
||||
atlas_monitor_effective_timers: >-
|
||||
{{ atlas_monitor_timers
|
||||
+ ([{'name': 'atlas-prometheus-pull.timer', 'max_age_hours': 26}]
|
||||
if atlas_manage_prometheus_backup_pull | bool and atlas_prometheus_pull_start_timer | bool
|
||||
else []) }}
|
||||
atlas_monitor_effective_failure_units: >-
|
||||
{{ atlas_monitor_failure_units
|
||||
+ (['atlas-prometheus-pull.service']
|
||||
if atlas_manage_prometheus_backup_pull | bool else []) }}
|
||||
atlas_monitor_remote_capacity: {}
|
||||
atlas_monitor_pool_warning_percent: 80
|
||||
atlas_monitor_pool_critical_percent: 90
|
||||
atlas_monitor_root_warning_percent: 80
|
||||
atlas_monitor_root_critical_percent: 90
|
||||
atlas_monitor_snapshot_warning_percent: 10
|
||||
atlas_monitor_snapshot_critical_percent: 20
|
||||
atlas_monitor_snapshot_growth_warning_gib_day: 100
|
||||
atlas_monitor_backup_growth_warning_gib_day: 100
|
||||
atlas_monitor_cpu_warning_c: 85
|
||||
atlas_monitor_cpu_critical_c: 95
|
||||
atlas_monitor_borg_max_runtime_days: 14
|
||||
|
||||
# Explicit post-restore relabeling only; never relabel datasets during ordinary runs.
|
||||
atlas_restorecon_paths: []
|
||||
|
||||
atlas_archive_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_archive }}"
|
||||
atlas_services_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_services }}"
|
||||
atlas_app_data_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_app_data }}"
|
||||
atlas_navidrome_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_navidrome }}"
|
||||
atlas_syncthing_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_syncthing }}"
|
||||
atlas_media_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_media }}"
|
||||
atlas_music_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_music }}"
|
||||
atlas_work_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_work }}"
|
||||
atlas_backup_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_backup }}"
|
||||
atlas_host_backups_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_host_backups }}"
|
||||
atlas_backup_prometheus_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_backup_prometheus }}"
|
||||
atlas_manage_prometheus_backup_pull: false
|
||||
atlas_prometheus_pull_ssh_dir: /etc/atlas-prometheus-pull
|
||||
atlas_prometheus_pull_private_key_path: "{{ atlas_prometheus_pull_ssh_dir }}/id_ed25519"
|
||||
atlas_prometheus_pull_known_hosts_path: "{{ atlas_prometheus_pull_ssh_dir }}/known_hosts"
|
||||
atlas_prometheus_ssh_host_key: ""
|
||||
atlas_prometheus_pull_source_user: prometheus-backup
|
||||
atlas_prometheus_pull_source_port: 22
|
||||
atlas_prometheus_pull_calendar: "*-*-* 03:00:00 Europe/Rome"
|
||||
atlas_prometheus_pull_start_timer: false
|
||||
atlas_prometheus_pull_keep_daily: 30
|
||||
atlas_prometheus_pull_keep_weekly: 8
|
||||
atlas_prometheus_pull_keep_monthly: 12
|
||||
atlas_prometheus_pull_max_age_hours: 24
|
||||
atlas_photobook_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_photobook }}"
|
||||
atlas_backups_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_backups }}"
|
||||
atlas_service_backups_mountpoint: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_service_backups }}"
|
||||
|
||||
# Rootless Gitea runs in admin's user manager; the image maps internal gitea to UID/GID 1000.
|
||||
atlas_manage_gitea: false
|
||||
atlas_gitea_username: "{{ atlas_admin_username }}"
|
||||
atlas_gitea_group: "{{ atlas_admin_group }}"
|
||||
atlas_gitea_uid: "{{ atlas_admin_uid }}"
|
||||
atlas_gitea_gid: "{{ atlas_admin_gid }}"
|
||||
atlas_gitea_home: "{{ atlas_admin_home }}"
|
||||
atlas_gitea_container_uid: 1000
|
||||
atlas_gitea_container_gid: 1000
|
||||
atlas_gitea_legacy_username: gitea
|
||||
atlas_gitea_legacy_uid: 1101
|
||||
atlas_gitea_legacy_home: /var/lib/atlas-gitea
|
||||
atlas_gitea_owner_migration: false
|
||||
atlas_gitea_dataset: "{{ atlas_zfs_pool }}/services/data/gitea"
|
||||
atlas_gitea_mountpoint: "{{ atlas_app_data_mountpoint }}/gitea"
|
||||
atlas_gitea_quadlet_dir: "{{ atlas_gitea_home }}/.config/containers/systemd"
|
||||
atlas_gitea_image: localhost/atlas-gitea:1.25.2-user-gitea-v1
|
||||
atlas_gitea_image_build_dir: "{{ atlas_gitea_home }}/.local/share/atlas-gitea-image"
|
||||
atlas_gitea_production_enabled: false
|
||||
atlas_gitea_public_domain: ""
|
||||
atlas_gitea_bind_address: "{{ ansible_host }}"
|
||||
atlas_gitea_http_port: 3000
|
||||
atlas_gitea_ssh_port: 2222
|
||||
atlas_gitea_staging_bind_address: 127.0.0.1
|
||||
atlas_gitea_staging_http_port: 3001
|
||||
atlas_gitea_staging_ssh_port: 2223
|
||||
atlas_gitea_restore_test: false
|
||||
atlas_gitea_final_restore: false
|
||||
atlas_gitea_restore_helper: /usr/local/libexec/atlas-gitea-restore-test
|
||||
|
||||
# Declare storage and an inactive Quadlet only. The operator supplies the
|
||||
# private configuration, handles MFA, and starts the user service manually.
|
||||
atlas_icloudpd_dataset: "{{ atlas_zfs_pool }}/services/data/icloudpd"
|
||||
atlas_icloudpd_state_dir: "{{ atlas_app_data_mountpoint }}/icloudpd"
|
||||
atlas_icloudpd_config_dir: "{{ atlas_icloudpd_state_dir }}/config"
|
||||
atlas_icloudpd_photos_dir: "{{ atlas_archive_mountpoint }}/Pictures/iCloudPD"
|
||||
atlas_icloudpd_image: >-
|
||||
docker.io/boredazfcuk/icloudpd@sha256:9966c31ddf0b5b306ac2410b4edd5d626806d96e80c92b83cbb689972dc9389f
|
||||
atlas_icloudpd_quadlet_dir: "{{ atlas_admin_home }}/.config/containers/systemd"
|
||||
atlas_icloudpd_timezone: Europe/Rome
|
||||
|
||||
atlas_45drives_repo_url: https://repo.45drives.com/repofiles/rocky/45drives-enterprise.repo
|
||||
atlas_45drives_repo_file: /etc/yum.repos.d/45drives-enterprise.repo
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
FROM docker.gitea.com/gitea@sha256:f1943db2d2f1e447e857b3f0aee4ebb7b184500f86e5b80eae110fd435435906
|
||||
|
||||
# Preserve the official image's UID/GID, paths and entrypoint; change only the
|
||||
# internal Unix identity. The host-side rootless owner is Atlas admin.
|
||||
USER 0
|
||||
RUN sed -i 's/^git:x:1000:1000:/gitea:x:1000:1000:/' /etc/passwd \
|
||||
&& sed -i 's/^git:x:1000:/gitea:x:1000:/' /etc/group \
|
||||
&& grep -q '^gitea:x:1000:1000:' /etc/passwd \
|
||||
&& grep -q '^gitea:x:1000:' /etc/group
|
||||
USER 1000:1000
|
||||
59
ansible/roles/profile_atlas/files/atlas-borg-progress.py
Normal file
59
ansible/roles/profile_atlas/files/atlas-borg-progress.py
Normal file
@@ -0,0 +1,59 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Turn Borg's JSON progress stream into bounded, readable journal entries."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
|
||||
|
||||
def size(value):
|
||||
if not isinstance(value, (int, float)):
|
||||
return "unknown"
|
||||
return f"{value / (1024 ** 3):.2f} GiB"
|
||||
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--estimated-total-bytes", type=int, required=True)
|
||||
args = parser.parse_args()
|
||||
if args.estimated_total_bytes <= 0:
|
||||
parser.error("estimated total must be positive")
|
||||
|
||||
last_progress = 0.0
|
||||
for line in sys.stdin:
|
||||
try:
|
||||
event = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
print(line.rstrip(), flush=True)
|
||||
continue
|
||||
|
||||
kind = event.get("type")
|
||||
if kind == "archive_progress":
|
||||
now = time.monotonic()
|
||||
if now - last_progress < 60 and not event.get("finished"):
|
||||
continue
|
||||
path = event.get("path") or ""
|
||||
parts = path.split("/")
|
||||
dataset = parts[1] if len(parts) > 1 and parts[0] == "source" else "unknown"
|
||||
original_size = event.get("original_size")
|
||||
if isinstance(original_size, (int, float)) and original_size >= 0:
|
||||
percent = original_size / args.estimated_total_bytes * 100
|
||||
estimated_progress = (
|
||||
f"{percent:.1f}%" if percent < 100 else ">=100% (ZFS estimate exceeded)"
|
||||
)
|
||||
else:
|
||||
estimated_progress = "unknown"
|
||||
print(
|
||||
"Borg create progress: "
|
||||
f"estimated={estimated_progress} dataset={dataset} "
|
||||
f"files={event.get('nfiles', 'unknown')} "
|
||||
f"original={size(original_size)} "
|
||||
f"compressed={size(event.get('compressed_size'))} "
|
||||
f"deduplicated={size(event.get('deduplicated_size'))}",
|
||||
flush=True,
|
||||
)
|
||||
last_progress = now
|
||||
elif kind == "log_message":
|
||||
print(f"Borg {event.get('levelname', 'INFO')}: {event.get('message', '')}", flush=True)
|
||||
elif kind == "progress_message" and event.get("message"):
|
||||
print(f"Borg: {event['message']}", flush=True)
|
||||
249
ansible/roles/profile_atlas/files/atlas-gitea-restore-test.py
Normal file
249
ansible/roles/profile_atlas/files/atlas-gitea-restore-test.py
Normal file
@@ -0,0 +1,249 @@
|
||||
#!/usr/bin/python3
|
||||
"""Rehearse a selective rootful-to-rootless Gitea restore, never a cutover."""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path, PurePosixPath
|
||||
import re
|
||||
import shutil
|
||||
import sqlite3
|
||||
import tarfile
|
||||
import tempfile
|
||||
|
||||
|
||||
SOURCE_PREFIX = PurePosixPath("opt/gitea/data")
|
||||
HOST_KEYS = (
|
||||
"ssh_host_ed25519_key",
|
||||
"ssh_host_rsa_key",
|
||||
"ssh_host_ecdsa_key",
|
||||
)
|
||||
SERVER_SETTINGS = {
|
||||
"START_SSH_SERVER": "true",
|
||||
"BUILTIN_SSH_SERVER_USER": "git",
|
||||
"SSH_USER": "git",
|
||||
"SSH_PORT": "2222",
|
||||
"SSH_LISTEN_PORT": "2222",
|
||||
"SSH_SERVER_HOST_KEYS": ", ".join(
|
||||
f"/var/lib/gitea/ssh/{key}" for key in HOST_KEYS
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def sha256(path):
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def expected_digest(backup):
|
||||
checksum = (backup / "payload.sha256").read_text().strip().split()
|
||||
if len(checksum) != 2 or checksum[1] != "payload.tar":
|
||||
raise ValueError("Unexpected Prometheus backup checksum manifest")
|
||||
if not re.fullmatch(r"[0-9a-f]{64}", checksum[0]):
|
||||
raise ValueError("Invalid Prometheus backup SHA-256")
|
||||
return checksum[0]
|
||||
|
||||
|
||||
def convert_config(config):
|
||||
original = config.read_text()
|
||||
output = []
|
||||
section = ""
|
||||
server_seen = set()
|
||||
server_found = False
|
||||
run_user_seen = False
|
||||
|
||||
def append_missing_server_settings():
|
||||
for key, value in SERVER_SETTINGS.items():
|
||||
if key not in server_seen:
|
||||
output.append(f"{key} = {value}\n")
|
||||
|
||||
for line in original.splitlines(keepends=True):
|
||||
match = re.match(r"^\s*\[([^]]+)\]\s*$", line)
|
||||
if match:
|
||||
if not run_user_seen:
|
||||
output.append("RUN_USER = gitea\n")
|
||||
run_user_seen = True
|
||||
if section == "server":
|
||||
append_missing_server_settings()
|
||||
section = match.group(1).lower()
|
||||
server_found |= section == "server"
|
||||
output.append(line)
|
||||
continue
|
||||
setting = re.match(r"^(\s*)([A-Z_]+)(\s*=\s*)(.*?)(\r?\n?)$", line)
|
||||
if setting and section == "" and setting.group(2) == "RUN_USER":
|
||||
run_user_seen = True
|
||||
line = f"{setting.group(1)}RUN_USER{setting.group(3)}gitea{setting.group(5)}"
|
||||
elif setting and section == "server" and setting.group(2) in SERVER_SETTINGS:
|
||||
key = setting.group(2)
|
||||
server_seen.add(key)
|
||||
line = f"{setting.group(1)}{key}{setting.group(3)}{SERVER_SETTINGS[key]}{setting.group(5)}"
|
||||
else:
|
||||
line = line.replace("/data/", "/var/lib/gitea/")
|
||||
output.append(line)
|
||||
if section == "server":
|
||||
append_missing_server_settings()
|
||||
if not server_found:
|
||||
raise ValueError("Gitea server configuration missing")
|
||||
config.write_text("".join(output))
|
||||
config.chmod(0o600)
|
||||
|
||||
|
||||
def extract_gitea(tar_path, staged_data):
|
||||
count = 0
|
||||
with tarfile.open(tar_path, mode="r") as archive:
|
||||
for member in archive:
|
||||
name = PurePosixPath(member.name)
|
||||
if name == SOURCE_PREFIX:
|
||||
continue
|
||||
if SOURCE_PREFIX not in name.parents:
|
||||
continue
|
||||
relative = name.relative_to(SOURCE_PREFIX)
|
||||
if not relative.parts or any(part in (".", "..") for part in relative.parts):
|
||||
raise ValueError("Unsafe Gitea backup path")
|
||||
if not (member.isdir() or member.isfile()):
|
||||
raise ValueError("Unexpected Gitea backup member type")
|
||||
destination = staged_data.joinpath(*relative.parts)
|
||||
if member.isdir():
|
||||
destination.mkdir(parents=True, exist_ok=True)
|
||||
destination.chmod(0o700)
|
||||
continue
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
with archive.extractfile(member) as source, destination.open("xb") as target:
|
||||
shutil.copyfileobj(source, target)
|
||||
destination.chmod(member.mode & 0o777)
|
||||
count += 1
|
||||
if count == 0:
|
||||
raise ValueError("No Gitea files in backup")
|
||||
|
||||
|
||||
def validate(staged_data, staged_config):
|
||||
database = staged_data / "gitea/gitea.db"
|
||||
repositories = staged_data / "git/repositories"
|
||||
if not database.is_file() or not repositories.is_dir():
|
||||
raise ValueError("Missing SQLite database or Git repositories")
|
||||
with sqlite3.connect(f"file:{database}?mode=ro", uri=True) as connection:
|
||||
if connection.execute("PRAGMA quick_check").fetchone()[0] != "ok":
|
||||
raise ValueError("Gitea SQLite quick_check failed")
|
||||
if connection.execute("SELECT count(*) FROM repository").fetchone()[0] < 1:
|
||||
raise ValueError("Gitea backup contains no repository records")
|
||||
if not any(repositories.rglob("*.git")):
|
||||
raise ValueError("Gitea backup contains no Git repository directories")
|
||||
if not (staged_config / "app.ini").is_file():
|
||||
raise ValueError("Gitea app.ini missing")
|
||||
for name in HOST_KEYS:
|
||||
if not (staged_data / "ssh" / name).is_file():
|
||||
raise ValueError("Gitea SSH host key missing")
|
||||
|
||||
|
||||
def chown_tree(root, uid, gid):
|
||||
for directory, dirs, files in os.walk(root):
|
||||
os.chown(directory, uid, gid)
|
||||
for name in dirs + files:
|
||||
os.chown(os.path.join(directory, name), uid, gid)
|
||||
|
||||
|
||||
def replace_rehearsal(target, stage, digest, uid, gid):
|
||||
previous_data = target / ".previous-rehearsal-data"
|
||||
previous_config = target / ".previous-rehearsal-config"
|
||||
if previous_data.exists() or previous_config.exists():
|
||||
raise ValueError("An interrupted Gitea replacement needs manual recovery")
|
||||
os.rename(target / "data", previous_data)
|
||||
try:
|
||||
os.rename(target / "config", previous_config)
|
||||
os.rename(stage / "data", target / "data")
|
||||
os.rename(stage / "config", target / "config")
|
||||
final_marker = target / ".final-sha256"
|
||||
final_marker.write_text(digest + "\n")
|
||||
final_marker.chmod(0o600)
|
||||
os.chown(final_marker, uid, gid)
|
||||
(target / ".rehearsal-sha256").unlink()
|
||||
except Exception:
|
||||
for name, previous in (("data", previous_data), ("config", previous_config)):
|
||||
current = target / name
|
||||
if previous.exists():
|
||||
if current.exists():
|
||||
shutil.rmtree(current)
|
||||
os.rename(previous, current)
|
||||
(target / ".final-sha256").unlink(missing_ok=True)
|
||||
raise
|
||||
shutil.rmtree(previous_data)
|
||||
shutil.rmtree(previous_config)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--backup", type=Path, required=True)
|
||||
parser.add_argument("--target", type=Path, required=True)
|
||||
parser.add_argument("--uid", type=int, required=True)
|
||||
parser.add_argument("--gid", type=int, required=True)
|
||||
parser.add_argument("--replace-rehearsal", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
backup = args.backup.resolve(strict=True)
|
||||
target = args.target.resolve(strict=True)
|
||||
if not str(backup).startswith("/zpool/backup/hosts/prometheus/snapshots/"):
|
||||
raise ValueError("Refusing backup outside the Atlas Prometheus snapshots")
|
||||
if str(target) != "/zpool/services/data/gitea":
|
||||
raise ValueError("Refusing target outside the dedicated Gitea dataset")
|
||||
if args.uid != 1000 or args.gid != 1000:
|
||||
raise ValueError("Unexpected admin-owned Gitea account IDs")
|
||||
expected = expected_digest(backup)
|
||||
if sha256(backup / "payload.tar") != expected:
|
||||
raise ValueError("Prometheus backup SHA-256 mismatch")
|
||||
|
||||
marker = target / (".final-sha256" if args.replace_rehearsal else ".rehearsal-sha256")
|
||||
if marker.exists():
|
||||
if marker.read_text().strip() != expected:
|
||||
raise ValueError("A different Gitea restore already occupies this dataset")
|
||||
validate(target / "data", target / "config")
|
||||
print("unchanged")
|
||||
return
|
||||
if args.replace_rehearsal:
|
||||
metadata = json.loads((backup / "metadata.json").read_text())
|
||||
if metadata.get("purpose") != "gitea-cutover":
|
||||
raise ValueError("Final restore requires an explicit Gitea cutover export")
|
||||
if not (target / ".rehearsal-sha256").is_file():
|
||||
raise ValueError("Only a marked rehearsal may be replaced")
|
||||
if not all((target / name).is_dir() for name in ("data", "config")):
|
||||
raise ValueError("Prepared Gitea volume paths are missing")
|
||||
else:
|
||||
if (target / ".final-sha256").exists():
|
||||
raise ValueError("Refusing a rehearsal restore over final Gitea data")
|
||||
for name in ("data", "config"):
|
||||
directory = target / name
|
||||
if not directory.is_dir() or any(directory.iterdir()):
|
||||
raise ValueError("Gitea target is not empty; refusing overwrite")
|
||||
|
||||
with tempfile.TemporaryDirectory(prefix=".rehearsal-", dir=target) as temporary:
|
||||
stage = Path(temporary)
|
||||
staged_data = stage / "data"
|
||||
staged_config = stage / "config"
|
||||
staged_data.mkdir()
|
||||
staged_config.mkdir()
|
||||
extract_gitea(backup / "payload.tar", staged_data)
|
||||
source_config = staged_data / "gitea/conf/app.ini"
|
||||
if not source_config.is_file():
|
||||
raise ValueError("Source Gitea app.ini missing")
|
||||
shutil.copy2(source_config, staged_config / "app.ini")
|
||||
source_config.unlink()
|
||||
convert_config(staged_config / "app.ini")
|
||||
validate(staged_data, staged_config)
|
||||
chown_tree(stage, args.uid, args.gid)
|
||||
if args.replace_rehearsal:
|
||||
replace_rehearsal(target, stage, expected, args.uid, args.gid)
|
||||
else:
|
||||
for name in ("data", "config"):
|
||||
(target / name).rmdir()
|
||||
os.rename(stage / name, target / name)
|
||||
marker.write_text(expected + "\n")
|
||||
marker.chmod(0o600)
|
||||
os.chown(marker, args.uid, args.gid)
|
||||
print("restored")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
396
ansible/roles/profile_atlas/files/atlas-health-monitor.py
Normal file
396
ansible/roles/profile_atlas/files/atlas-health-monitor.py
Normal file
@@ -0,0 +1,396 @@
|
||||
#!/usr/bin/python3
|
||||
"""Read-only Atlas health probes with deduplicated 45Drives Alerts."""
|
||||
|
||||
import argparse
|
||||
import fcntl
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
CONFIG_PATH = Path("/etc/atlas-health-monitor.json")
|
||||
STATE_DIR = Path("/var/lib/atlas-health-monitor")
|
||||
STATE_PATH = STATE_DIR / "state.json"
|
||||
GIB = 1024**3
|
||||
|
||||
|
||||
def run(*argv, timeout=40):
|
||||
return subprocess.run(argv, capture_output=True, text=True, timeout=timeout, check=False)
|
||||
|
||||
|
||||
def issue(issues, key, severity, message):
|
||||
issues[key] = {"severity": severity, "message": message}
|
||||
|
||||
|
||||
def notify(config, event, severity, subject, message):
|
||||
now = datetime.now(timezone.utc)
|
||||
payload = {
|
||||
"timestamp": now.isoformat(timespec="seconds"),
|
||||
"unixtime": int(now.timestamp()),
|
||||
"event": event,
|
||||
"severity": severity,
|
||||
"subject": subject,
|
||||
"email_message": message,
|
||||
}
|
||||
result = run(config["notifier"], json.dumps(payload, ensure_ascii=False), timeout=30)
|
||||
if result.returncode:
|
||||
raise RuntimeError(f"45Drives notifier exited {result.returncode}: {result.stderr.strip()}")
|
||||
|
||||
|
||||
def parse_fields(text):
|
||||
return dict(line.split("=", 1) for line in text.splitlines() if "=" in line)
|
||||
|
||||
|
||||
def systemd_fields(unit, *properties):
|
||||
result = run("systemctl", "show", unit, *(f"-p{item}" for item in properties))
|
||||
if result.returncode:
|
||||
raise RuntimeError(f"systemctl show {unit} exited {result.returncode}")
|
||||
return parse_fields(result.stdout)
|
||||
|
||||
|
||||
def unix_time(text):
|
||||
if not text or text == "n/a":
|
||||
return None
|
||||
result = run("date", "-d", text, "+%s")
|
||||
if result.returncode:
|
||||
raise ValueError(f"Cannot parse systemd timestamp: {text}")
|
||||
return int(result.stdout.strip())
|
||||
|
||||
|
||||
def check_pool(config, issues, measurements):
|
||||
pool = config["pool"]
|
||||
listing = run("zpool", "list", "-H", "-p", "-o", "size,alloc,capacity,health", pool)
|
||||
if listing.returncode:
|
||||
issue(issues, "pool.probe", "critical", f"Cannot query ZFS pool {pool}")
|
||||
return
|
||||
try:
|
||||
size, alloc, capacity, health = listing.stdout.strip().split("\t")
|
||||
size, alloc, capacity = int(size), int(alloc), int(capacity)
|
||||
except (ValueError, TypeError):
|
||||
issue(issues, "pool.probe", "critical", "Invalid ZFS pool capacity response")
|
||||
return
|
||||
measurements.update(pool_size_bytes=size, pool_alloc_bytes=alloc, pool_capacity_percent=capacity)
|
||||
if health != "ONLINE":
|
||||
issue(issues, "pool.health", "critical", f"ZFS pool {pool} state is {health}")
|
||||
if capacity >= config["pool_critical_percent"]:
|
||||
issue(issues, "pool.capacity", "critical", f"ZFS pool {pool} is {capacity}% full")
|
||||
elif capacity >= config["pool_warning_percent"]:
|
||||
issue(issues, "pool.capacity", "warning", f"ZFS pool {pool} is {capacity}% full")
|
||||
|
||||
status = run("zpool", "status", "-P", pool)
|
||||
if status.returncode:
|
||||
issue(issues, "pool.status", "critical", f"Cannot query detailed ZFS status for {pool}")
|
||||
return
|
||||
bad_vdevs = []
|
||||
for line in status.stdout.splitlines():
|
||||
match = re.match(r"^\s*(\S+)\s+(ONLINE|DEGRADED|FAULTED|OFFLINE|UNAVAIL|REMOVED)\s+(\d+)\s+(\d+)\s+(\d+)", line)
|
||||
if match:
|
||||
name, state, reads, writes, checksums = match.groups()
|
||||
if state != "ONLINE" or any(int(value) for value in (reads, writes, checksums)):
|
||||
bad_vdevs.append(f"{name}: {state}, READ={reads}, WRITE={writes}, CKSUM={checksums}")
|
||||
if bad_vdevs:
|
||||
issue(issues, "pool.vdevs", "critical", "ZFS vdev errors: " + "; ".join(bad_vdevs))
|
||||
errors = re.search(r"^errors:\s*(.*)$", status.stdout, re.MULTILINE)
|
||||
if not errors or errors.group(1).strip() != "No known data errors":
|
||||
issue(issues, "pool.data_errors", "critical", "ZFS status reports data errors; inspect zpool status -v")
|
||||
if re.search(r"^\s*scan:\s*resilver in progress", status.stdout, re.MULTILINE | re.IGNORECASE):
|
||||
issue(issues, "pool.resilver", "warning", "ZFS resilver is in progress; inspect zpool status")
|
||||
scan = re.search(r"^\s*scan:\s*(.*)$", status.stdout, re.MULTILINE)
|
||||
if scan and re.search(r"\bwith [1-9][0-9]* errors\b", scan.group(1)):
|
||||
issue(issues, "pool.scan_errors", "critical", f"ZFS scan reported errors: {scan.group(1)}")
|
||||
|
||||
|
||||
def check_capacity(config, issues, measurements):
|
||||
pool = config["pool"]
|
||||
listing = run("zfs", "list", "-H", "-p", "-o", "name,usedbysnapshots", "-r", pool)
|
||||
if listing.returncode:
|
||||
issue(issues, "snapshot.probe", "warning", "Cannot query ZFS snapshot space")
|
||||
else:
|
||||
try:
|
||||
snapshots = sum(int(line.split("\t")[1]) for line in listing.stdout.splitlines())
|
||||
measurements["snapshots_bytes"] = snapshots
|
||||
size = measurements.get("pool_size_bytes")
|
||||
if size:
|
||||
percent = snapshots * 100 // size
|
||||
measurements["snapshots_percent"] = percent
|
||||
if percent >= config["snapshot_critical_percent"]:
|
||||
issue(issues, "snapshot.capacity", "critical", f"Snapshots use {percent}% of pool size")
|
||||
elif percent >= config["snapshot_warning_percent"]:
|
||||
issue(issues, "snapshot.capacity", "warning", f"Snapshots use {percent}% of pool size")
|
||||
except (ValueError, IndexError):
|
||||
issue(issues, "snapshot.probe", "warning", "Invalid ZFS snapshot-space response")
|
||||
backup = run("zfs", "list", "-H", "-p", "-o", "used", config["backup_dataset"])
|
||||
if backup.returncode:
|
||||
issue(issues, "backup.capacity_probe", "warning", "Cannot query local backup dataset space")
|
||||
else:
|
||||
try:
|
||||
measurements["backup_bytes"] = int(backup.stdout.strip())
|
||||
except ValueError:
|
||||
issue(issues, "backup.capacity_probe", "warning", "Invalid local backup space response")
|
||||
|
||||
try:
|
||||
filesystem = os.statvfs("/")
|
||||
total = filesystem.f_blocks * filesystem.f_frsize
|
||||
available = filesystem.f_bavail * filesystem.f_frsize
|
||||
used_percent = (total - available) * 100 // total
|
||||
measurements["root_capacity_percent"] = used_percent
|
||||
if used_percent >= config["root_critical_percent"]:
|
||||
issue(issues, "root.capacity", "critical", f"Atlas system filesystem is {used_percent}% full")
|
||||
elif used_percent >= config["root_warning_percent"]:
|
||||
issue(issues, "root.capacity", "warning", f"Atlas system filesystem is {used_percent}% full")
|
||||
except (OSError, ZeroDivisionError):
|
||||
issue(issues, "root.capacity_probe", "warning", "Cannot query Atlas system filesystem space")
|
||||
|
||||
|
||||
def check_remote_capacity(config, issues, measurements):
|
||||
"""Query only the Storage Box quota; do not open or inspect the Borg repository."""
|
||||
remote = config["remote_capacity"]
|
||||
try:
|
||||
result = run("runuser", "-u", remote["run_as"], "--", remote["ssh_wrapper"],
|
||||
f"{remote['user']}@{remote['host']}", "df", "-m", timeout=65)
|
||||
if result.returncode:
|
||||
raise ValueError(f"SSH df exited {result.returncode}")
|
||||
lines = result.stdout.strip().splitlines()
|
||||
if len(lines) != 2:
|
||||
raise ValueError("Unexpected Storage Box df output")
|
||||
fields = lines[1].split()
|
||||
if len(fields) < 5:
|
||||
raise ValueError("Incomplete Storage Box df output")
|
||||
total_mib, used_mib, available_mib = (int(value) for value in fields[1:4])
|
||||
percent = int(fields[4].rstrip("%"))
|
||||
if total_mib <= 0 or not 0 <= percent <= 100 or available_mib < 0:
|
||||
raise ValueError("Invalid Storage Box quota values")
|
||||
except (OSError, ValueError, subprocess.TimeoutExpired):
|
||||
issue(issues, "remote.capacity_probe", "warning", "Cannot query Hetzner Storage Box quota via pinned-key SSH")
|
||||
return
|
||||
measurements.update(remote_capacity_percent=percent, remote_bytes=used_mib * 1024**2,
|
||||
remote_available_bytes=available_mib * 1024**2)
|
||||
if percent >= remote["critical_percent"]:
|
||||
issue(issues, "remote.capacity", "critical", f"Hetzner Storage Box quota is {percent}% full")
|
||||
elif percent >= remote["warning_percent"]:
|
||||
issue(issues, "remote.capacity", "warning", f"Hetzner Storage Box quota is {percent}% full")
|
||||
|
||||
|
||||
def check_smart(config, issues, measurements):
|
||||
for device in config["smart_devices"]:
|
||||
name, path = device["name"], device["path"]
|
||||
try:
|
||||
result = run("smartctl", "-j", "-a", path, timeout=60)
|
||||
data = json.loads(result.stdout)
|
||||
status = int(data.get("smartctl", {}).get("exit_status", result.returncode))
|
||||
except (subprocess.TimeoutExpired, json.JSONDecodeError, ValueError) as exc:
|
||||
issue(issues, f"smart.{name}.probe", "critical", f"SMART probe failed for {name}: {type(exc).__name__}")
|
||||
continue
|
||||
if status:
|
||||
severity = "critical" if status & 0b00001111 else "warning"
|
||||
issue(issues, f"smart.{name}.status", severity, f"SMART reported exit status {status} for {name}")
|
||||
passed = data.get("smart_status", {}).get("passed")
|
||||
if passed is False:
|
||||
issue(issues, f"smart.{name}.health", "critical", f"SMART self-assessment failed for {name}")
|
||||
elif passed is None:
|
||||
issue(issues, f"smart.{name}.health", "warning", f"SMART self-assessment unavailable for {name}")
|
||||
temperature = data.get("temperature", {}).get("current")
|
||||
if isinstance(temperature, (int, float)):
|
||||
measurements[f"smart_{name}_c"] = temperature
|
||||
if temperature >= device["critical_c"]:
|
||||
issue(issues, f"smart.{name}.temperature", "critical", f"{name} temperature is {temperature} C")
|
||||
elif temperature >= device["warning_c"]:
|
||||
issue(issues, f"smart.{name}.temperature", "warning", f"{name} temperature is {temperature} C")
|
||||
else:
|
||||
issue(issues, f"smart.{name}.temperature", "warning", f"Temperature unavailable for {name}")
|
||||
for attribute in data.get("ata_smart_attributes", {}).get("table", []):
|
||||
attribute_id = attribute.get("id")
|
||||
if attribute_id in (5, 187, 197, 198):
|
||||
raw = attribute.get("raw", {}).get("value", 0)
|
||||
if isinstance(raw, int) and raw > 0:
|
||||
severity = "critical" if attribute_id in (197, 198) else "warning"
|
||||
issue(issues, f"smart.{name}.ata_{attribute_id}", severity,
|
||||
f"{name} SMART attribute {attribute_id} raw count is {raw}")
|
||||
nvme = data.get("nvme_smart_health_information_log", {})
|
||||
if isinstance(nvme, dict):
|
||||
if int(nvme.get("critical_warning", 0)):
|
||||
issue(issues, f"smart.{name}.nvme_warning", "critical", f"{name} NVMe critical warning is nonzero")
|
||||
if int(nvme.get("media_errors", 0)):
|
||||
issue(issues, f"smart.{name}.nvme_media", "critical", f"{name} NVMe media errors are nonzero")
|
||||
|
||||
|
||||
def check_cpu(config, issues, measurements):
|
||||
sensors = []
|
||||
for hwmon in Path("/sys/class/hwmon").glob("hwmon*"):
|
||||
try:
|
||||
if (hwmon / "name").read_text().strip() != "coretemp":
|
||||
continue
|
||||
sensors.extend(int(path.read_text().strip()) / 1000 for path in hwmon.glob("temp*_input"))
|
||||
except (OSError, ValueError):
|
||||
continue
|
||||
if not sensors:
|
||||
issue(issues, "cpu.temperature_probe", "warning", "CPU temperature sensors are unavailable")
|
||||
return
|
||||
hottest = max(sensors)
|
||||
measurements["cpu_max_c"] = hottest
|
||||
if hottest >= config["cpu_critical_c"]:
|
||||
issue(issues, "cpu.temperature", "critical", f"CPU temperature is {hottest:g} C")
|
||||
elif hottest >= config["cpu_warning_c"]:
|
||||
issue(issues, "cpu.temperature", "warning", f"CPU temperature is {hottest:g} C")
|
||||
|
||||
|
||||
def check_jobs(config, issues, measurements, now):
|
||||
for timer in config["timers"]:
|
||||
name = timer["name"]
|
||||
try:
|
||||
fields = systemd_fields(name, "ActiveState", "UnitFileState", "LastTriggerUSec", "ActiveEnterTimestamp")
|
||||
if fields.get("ActiveState") != "active" or fields.get("UnitFileState") != "enabled":
|
||||
issue(issues, f"timer.{name}", "critical", f"Timer {name} is not active and enabled")
|
||||
max_age = int(timer["max_age_hours"]) * 3600
|
||||
if max_age:
|
||||
last = unix_time(fields.get("LastTriggerUSec"))
|
||||
if last is None:
|
||||
last = unix_time(fields.get("ActiveEnterTimestamp"))
|
||||
if last is not None and now - last > max_age:
|
||||
issue(issues, f"timer.{name}.stale", "warning",
|
||||
f"Timer {name} has not fired in {int((now-last)/3600)} hours")
|
||||
except (RuntimeError, ValueError, subprocess.TimeoutExpired):
|
||||
issue(issues, f"timer.{name}.probe", "warning", f"Cannot query timer {name}")
|
||||
for unit in config["failure_units"]:
|
||||
if unit.endswith("@.service"):
|
||||
continue
|
||||
try:
|
||||
fields = systemd_fields(unit, "ActiveState", "Result", "ExecMainStartTimestamp")
|
||||
state = fields.get("ActiveState")
|
||||
if state == "failed" or (state == "inactive" and fields.get("Result") not in (None, "", "success")):
|
||||
issue(issues, f"service.{unit}", "critical", f"Service {unit} failed: {fields.get('Result')}")
|
||||
if unit == "atlas-borg-backup.service" and fields.get("ActiveState") == "activating":
|
||||
started = unix_time(fields.get("ExecMainStartTimestamp"))
|
||||
if started is not None and now - started > config["borg_max_runtime_days"] * 86400:
|
||||
issue(issues, "backup.borg_long_running", "warning",
|
||||
"Borg has run longer than its configured limit")
|
||||
except (RuntimeError, ValueError, subprocess.TimeoutExpired):
|
||||
issue(issues, f"service.{unit}.probe", "warning", f"Cannot query service {unit}")
|
||||
|
||||
|
||||
def check_growth(config, issues, measurements, samples, now):
|
||||
previous = [sample for sample in samples if 20 * 3600 <= now - sample.get("time", now) <= 48 * 3600]
|
||||
if previous:
|
||||
baseline = min(previous, key=lambda sample: abs(now - sample["time"] - 86400))
|
||||
days = (now - baseline["time"]) / 86400
|
||||
for name, threshold in (("snapshots", config["snapshot_growth_warning_gib_day"]),
|
||||
("backup", config["backup_growth_warning_gib_day"]),
|
||||
("remote", config["remote_capacity"]["growth_warning_gib_day"])):
|
||||
current, old = measurements.get(f"{name}_bytes"), baseline.get(f"{name}_bytes")
|
||||
if isinstance(current, int) and isinstance(old, int) and days > 0:
|
||||
growth_gib_day = (current - old) / GIB / days
|
||||
measurements[f"{name}_growth_gib_day"] = round(growth_gib_day, 1)
|
||||
if growth_gib_day >= threshold:
|
||||
issue(issues, f"{name}.growth", "warning",
|
||||
f"Local {name} usage grew {growth_gib_day:.1f} GiB/day over {days:.1f} days")
|
||||
|
||||
|
||||
def allowed_failure_unit(config, unit):
|
||||
for allowed in config["failure_units"]:
|
||||
if allowed == unit:
|
||||
return True
|
||||
if allowed.endswith("@.service") and unit.startswith(allowed[:-9] + "@") and unit.endswith(".service"):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def load_state():
|
||||
if not STATE_PATH.exists():
|
||||
return {"active": {}, "samples": []}
|
||||
with STATE_PATH.open(encoding="utf-8") as stream:
|
||||
state = json.load(stream)
|
||||
if not isinstance(state.get("active"), dict) or not isinstance(state.get("samples"), list):
|
||||
raise ValueError("Invalid Atlas monitor state; refusing to overwrite it")
|
||||
return state
|
||||
|
||||
|
||||
def save_state(state):
|
||||
with tempfile.NamedTemporaryFile("w", dir=STATE_DIR, prefix=".state-", delete=False,
|
||||
encoding="utf-8") as stream:
|
||||
path = Path(stream.name)
|
||||
os.chmod(path, 0o600)
|
||||
json.dump(state, stream, sort_keys=True)
|
||||
stream.write("\n")
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.replace(path, STATE_PATH)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--dry-run", action="store_true", help="probe without notifications or state changes")
|
||||
parser.add_argument("--test-notification", action="store_true", help="submit a labelled test alert")
|
||||
parser.add_argument("--job-failed", metavar="UNIT", help="notify about a failed configured service")
|
||||
args = parser.parse_args()
|
||||
with CONFIG_PATH.open(encoding="utf-8") as stream:
|
||||
config = json.load(stream)
|
||||
if args.test_notification:
|
||||
notify(config, "atlas_monitor_test", "warning", "Test monitoraggio Atlas",
|
||||
"Notifica di prova: il monitoraggio Atlas raggiunge 45Drives Alerts. Non conferma l'invio email.")
|
||||
print("Atlas monitor test submitted to 45Drives Alerts; email delivery is not verified.")
|
||||
return 0
|
||||
if args.job_failed:
|
||||
if not allowed_failure_unit(config, args.job_failed):
|
||||
raise ValueError("Unconfigured Atlas failure unit")
|
||||
notify(config, "atlas_job_failed", "critical", f"Job Atlas fallito: {args.job_failed}",
|
||||
f"Il servizio {args.job_failed} e' fallito. Controlla: "
|
||||
f"sudo journalctl -u {args.job_failed} -n 100 --no-pager")
|
||||
print(f"Atlas job failure submitted to 45Drives Alerts: {args.job_failed}")
|
||||
return 0
|
||||
|
||||
now = int(time.time())
|
||||
issues, measurements = {}, {}
|
||||
check_pool(config, issues, measurements)
|
||||
check_capacity(config, issues, measurements)
|
||||
check_remote_capacity(config, issues, measurements)
|
||||
check_smart(config, issues, measurements)
|
||||
check_cpu(config, issues, measurements)
|
||||
check_jobs(config, issues, measurements, now)
|
||||
if args.dry_run:
|
||||
print(json.dumps({"issues": issues, "measurements": measurements}, sort_keys=True))
|
||||
return 0
|
||||
|
||||
STATE_DIR.mkdir(mode=0o700, exist_ok=True)
|
||||
with (STATE_DIR / "monitor.lock").open("w") as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
state = load_state()
|
||||
check_growth(config, issues, measurements, state["samples"], now)
|
||||
active, failed_notifications = state["active"], []
|
||||
for key, details in issues.items():
|
||||
old = active.get(key)
|
||||
if old is None or old.get("severity") != details["severity"]:
|
||||
try:
|
||||
notify(config, "atlas_health_issue", details["severity"],
|
||||
f"Atlas: {key}", details["message"])
|
||||
active[key] = details
|
||||
print(f"ALERT {details['severity']} {key}: {details['message']}", flush=True)
|
||||
except (RuntimeError, subprocess.TimeoutExpired) as exc:
|
||||
failed_notifications.append(key)
|
||||
print(f"NOTIFICATION FAILED {key}: {exc}", file=sys.stderr, flush=True)
|
||||
for key in set(active) - set(issues):
|
||||
print(f"RECOVERED {key}", flush=True)
|
||||
del active[key]
|
||||
state["samples"] = [sample for sample in state["samples"] if now - sample.get("time", 0) < 48 * 3600]
|
||||
state["samples"].append({"time": now, **{key: value for key, value in measurements.items()
|
||||
if key in ("snapshots_bytes", "backup_bytes", "remote_bytes")}})
|
||||
save_state(state)
|
||||
print(f"Atlas health: issues={len(issues)} notifications_failed={len(failed_notifications)} "
|
||||
f"pool={measurements.get('pool_capacity_percent', 'unknown')}% "
|
||||
f"remote={measurements.get('remote_capacity_percent', 'unknown')}% "
|
||||
f"snapshots={measurements.get('snapshots_bytes', 'unknown')} bytes "
|
||||
f"backup={measurements.get('backup_bytes', 'unknown')} bytes", flush=True)
|
||||
return 1 if failed_notifications else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except (OSError, RuntimeError, ValueError, subprocess.TimeoutExpired) as error:
|
||||
print(f"Atlas health monitor failed: {error}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
56
ansible/roles/profile_atlas/files/atlas-prometheus-prune.py
Normal file
56
ansible/roles/profile_atlas/files/atlas-prometheus-prune.py
Normal file
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Prune only verified, named Prometheus backup versions after publication."""
|
||||
|
||||
import datetime as dt
|
||||
import pathlib
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
|
||||
def main() -> None:
|
||||
if len(sys.argv) != 5:
|
||||
raise SystemExit("Usage: atlas-prometheus-prune SNAPSHOTS DAILY WEEKLY MONTHLY")
|
||||
root = pathlib.Path(sys.argv[1])
|
||||
counts = [int(value) for value in sys.argv[2:]]
|
||||
if not root.is_dir() or root.is_symlink() or min(counts) < 1:
|
||||
raise SystemExit("Invalid backup directory or retention counts")
|
||||
versions = []
|
||||
for entry in root.iterdir():
|
||||
if not entry.is_dir() or entry.is_symlink():
|
||||
continue
|
||||
if not re.fullmatch(r"[0-9]{8}T[0-9]{6}Z", entry.name):
|
||||
continue
|
||||
try:
|
||||
when = dt.datetime.strptime(entry.name, "%Y%m%dT%H%M%SZ")
|
||||
except ValueError:
|
||||
continue
|
||||
if not all((entry / name).is_file() for name in ("payload.tar", "payload.sha256", "metadata.json")):
|
||||
continue
|
||||
versions.append((when, entry))
|
||||
versions.sort(reverse=True)
|
||||
if not versions:
|
||||
raise SystemExit("No published backup versions found; refusing to prune")
|
||||
|
||||
keep = {entry for _, entry in versions[: counts[0]]}
|
||||
for count, key in (
|
||||
(counts[1], lambda when: when.isocalendar()[:2]),
|
||||
(counts[2], lambda when: (when.year, when.month)),
|
||||
):
|
||||
periods = set()
|
||||
for when, entry in versions:
|
||||
period = key(when)
|
||||
if period in periods:
|
||||
continue
|
||||
periods.add(period)
|
||||
keep.add(entry)
|
||||
if len(periods) >= count:
|
||||
break
|
||||
|
||||
for _, entry in versions:
|
||||
if entry not in keep:
|
||||
shutil.rmtree(entry)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,4 +1,14 @@
|
||||
---
|
||||
- name: Reload Atlas admin user manager
|
||||
become_user: "{{ atlas_admin_username }}"
|
||||
ansible.builtin.systemd:
|
||||
scope: user
|
||||
daemon_reload: true
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_admin_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ atlas_admin_uid }}/bus"
|
||||
when: not ansible_check_mode
|
||||
|
||||
- name: Reload SSH service
|
||||
ansible.builtin.systemd:
|
||||
name: sshd
|
||||
@@ -28,6 +38,18 @@
|
||||
name: smb
|
||||
state: restarted
|
||||
|
||||
- name: Restart Atlas Borg timers
|
||||
ansible.builtin.systemd:
|
||||
name: "{{ item }}"
|
||||
state: restarted
|
||||
daemon_reload: true
|
||||
loop:
|
||||
- atlas-borg-backup.timer
|
||||
- atlas-borg-check.timer
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Restart Atlas media Quadlets
|
||||
ansible.builtin.systemd:
|
||||
name: "{{ item }}"
|
||||
|
||||
538
ansible/roles/profile_atlas/tasks/borg_backup.yml
Normal file
538
ansible/roles/profile_atlas/tasks/borg_backup.yml
Normal file
@@ -0,0 +1,538 @@
|
||||
---
|
||||
- name: Validate Atlas Borg backup configuration
|
||||
tags: [atlas, storage, backup, borg, borg_key]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_manage_storage | bool
|
||||
- atlas_zfs_pool != 'CHANGEME_ZFS_POOL'
|
||||
- atlas_mount_root.startswith('/')
|
||||
- atlas_borg_username is match('^[a-z_][a-z0-9_-]*$')
|
||||
- atlas_borg_group is match('^[a-z_][a-z0-9_-]*$')
|
||||
- atlas_borg_username not in ['root', atlas_admin_username]
|
||||
- atlas_borg_group != 'wheel'
|
||||
- atlas_borg_home.startswith('/var/lib/')
|
||||
- atlas_borg_repository_host is match('^[A-Za-z0-9.-]+$')
|
||||
- atlas_borg_repository_user is match('^[A-Za-z0-9_-]+$')
|
||||
- atlas_borg_repository_port | int > 0
|
||||
- atlas_borg_repository_port | int < 65536
|
||||
- atlas_borg_repository_path is match('^\./[A-Za-z0-9][A-Za-z0-9._/-]*$')
|
||||
- "'/../' not in ('/' ~ atlas_borg_repository_path ~ '/')"
|
||||
- atlas_borg_remote_path is match('^borg-[0-9]+\.[0-9]+$')
|
||||
- atlas_borg_host_key.startswith(
|
||||
'[' ~ atlas_borg_repository_host ~ ']:' ~ (atlas_borg_repository_port | string) ~ ' ssh-ed25519 '
|
||||
)
|
||||
- atlas_borg_ssh_private_key_path.startswith('/etc/atlas-borg/')
|
||||
- atlas_borg_known_hosts_path.startswith('/etc/atlas-borg/')
|
||||
- atlas_borg_passphrase_path.startswith('/etc/atlas-borg/')
|
||||
- atlas_borg_ssh_wrapper_path.startswith('/usr/local/libexec/')
|
||||
- atlas_borg_encryption_mode == 'repokey'
|
||||
- atlas_borg_archive_prefix is match('^[a-z0-9][a-z0-9_-]*$')
|
||||
- atlas_borg_snapshot_prefix is match('^[a-z0-9][a-z0-9_-]*$')
|
||||
- atlas_borg_keep_daily | int > 0
|
||||
- atlas_borg_keep_weekly | int > 0
|
||||
- atlas_borg_keep_monthly | int > 0
|
||||
fail_msg: >-
|
||||
Atlas Borg needs a safe relative repository path, a pinned ED25519 host
|
||||
key, positive retention counts, and valid dedicated SSH settings.
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Create the Atlas Borg system group
|
||||
tags: [atlas, storage, backup, borg, borg_key]
|
||||
ansible.builtin.group:
|
||||
name: "{{ atlas_borg_group }}"
|
||||
system: true
|
||||
state: present
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Create the least-privilege Atlas Borg account
|
||||
tags: [atlas, storage, backup, borg, borg_key]
|
||||
ansible.builtin.user:
|
||||
name: "{{ atlas_borg_username }}"
|
||||
group: "{{ atlas_borg_group }}"
|
||||
groups: []
|
||||
append: false
|
||||
comment: Atlas Borg backup service
|
||||
home: "{{ atlas_borg_home }}"
|
||||
create_home: false
|
||||
shell: /sbin/nologin
|
||||
password_lock: true
|
||||
system: true
|
||||
state: present
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Read Atlas Borg account group membership
|
||||
tags: [atlas, storage, backup, borg, borg_key]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- id
|
||||
- -nG
|
||||
- "{{ atlas_borg_username }}"
|
||||
register: atlas_borg_account_groups
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Require the Atlas Borg account to have no supplementary groups
|
||||
tags: [atlas, storage, backup, borg, borg_key]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_borg_account_groups.stdout.split() == [atlas_borg_group]
|
||||
fail_msg: >-
|
||||
The Atlas Borg service account must belong only to its private primary
|
||||
group and must never receive wheel or other supplementary membership.
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Validate Atlas Borg systemd calendars
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- systemd-analyze
|
||||
- calendar
|
||||
- "{{ item }}"
|
||||
loop:
|
||||
- "{{ atlas_borg_backup_calendar }}"
|
||||
- "{{ atlas_borg_check_calendar }}"
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Create Atlas Borg configuration directory
|
||||
tags: [atlas, storage, backup, borg, borg_key]
|
||||
ansible.builtin.file:
|
||||
path: /etc/atlas-borg
|
||||
state: directory
|
||||
owner: root
|
||||
group: "{{ atlas_borg_group }}"
|
||||
mode: "0750"
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Generate the dedicated Atlas Borg SSH identity
|
||||
tags: [atlas, storage, backup, borg, borg_key]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- ssh-keygen
|
||||
- -q
|
||||
- -t
|
||||
- ed25519
|
||||
- -N
|
||||
- ""
|
||||
- -C
|
||||
- atlas-borg@atlas
|
||||
- -f
|
||||
- "{{ atlas_borg_ssh_private_key_path }}"
|
||||
creates: "{{ atlas_borg_ssh_private_key_path }}"
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Protect the Atlas Borg private SSH identity
|
||||
tags: [atlas, storage, backup, borg, borg_key]
|
||||
ansible.builtin.file:
|
||||
path: "{{ atlas_borg_ssh_private_key_path }}"
|
||||
owner: "{{ atlas_borg_username }}"
|
||||
group: "{{ atlas_borg_group }}"
|
||||
mode: "0600"
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Set permissions on the Atlas Borg public SSH identity
|
||||
tags: [atlas, storage, backup, borg, borg_key]
|
||||
ansible.builtin.file:
|
||||
path: "{{ atlas_borg_ssh_private_key_path }}.pub"
|
||||
owner: "{{ atlas_borg_username }}"
|
||||
group: "{{ atlas_borg_group }}"
|
||||
mode: "0644"
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Read the dedicated Atlas Borg public SSH identity
|
||||
tags: [atlas, storage, backup, borg, borg_key]
|
||||
ansible.builtin.slurp:
|
||||
src: "{{ atlas_borg_ssh_private_key_path }}.pub"
|
||||
register: atlas_borg_public_key
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Report the public SSH identity to install in the Hetzner sub-account
|
||||
tags: [atlas, storage, backup, borg, borg_key]
|
||||
ansible.builtin.debug:
|
||||
msg: "{{ atlas_borg_public_key.content | b64decode | trim }}"
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Pin the Hetzner Storage Box SSH host key
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.copy:
|
||||
content: "{{ atlas_borg_host_key }}\n"
|
||||
dest: "{{ atlas_borg_known_hosts_path }}"
|
||||
owner: "{{ atlas_borg_username }}"
|
||||
group: "{{ atlas_borg_group }}"
|
||||
mode: "0600"
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Create Atlas Borg state directories
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.file:
|
||||
path: "{{ item }}"
|
||||
state: directory
|
||||
owner: "{{ atlas_borg_username }}"
|
||||
group: "{{ atlas_borg_group }}"
|
||||
mode: "0700"
|
||||
loop:
|
||||
- "{{ atlas_borg_config_dir }}"
|
||||
- "{{ atlas_borg_cache_dir }}"
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Create the shared Atlas Borg operation lock
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.copy:
|
||||
content: ""
|
||||
dest: "{{ atlas_borg_lock_path }}"
|
||||
owner: "{{ atlas_borg_username }}"
|
||||
group: "{{ atlas_borg_group }}"
|
||||
mode: "0600"
|
||||
force: false
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Require the Atlas Borg encryption passphrase from Vault
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_borg_passphrase | length >= 20
|
||||
fail_msg: >-
|
||||
Define vault_atlas_borg_passphrase with a strong unique value in the
|
||||
encrypted Vault before activating the Borg repository.
|
||||
no_log: true
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Install the Atlas Borg passphrase
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.copy:
|
||||
content: "{{ atlas_borg_passphrase }}\n"
|
||||
dest: "{{ atlas_borg_passphrase_path }}"
|
||||
owner: "{{ atlas_borg_username }}"
|
||||
group: "{{ atlas_borg_group }}"
|
||||
mode: "0600"
|
||||
diff: false
|
||||
no_log: true
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Install the Atlas Borg backup helper
|
||||
tags: [atlas, storage, backup, borg, borg_logging]
|
||||
ansible.builtin.template:
|
||||
src: atlas-borg-backup.sh.j2
|
||||
dest: /usr/local/sbin/atlas-borg-backup
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0750"
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Install the Atlas Borg snapshot cleanup helper
|
||||
tags: [atlas, storage, backup, borg, borg_logging]
|
||||
ansible.builtin.template:
|
||||
src: atlas-borg-snapshot-cleanup.sh.j2
|
||||
dest: /usr/local/sbin/atlas-borg-snapshot-cleanup
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0750"
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Install the Atlas Borg check helper
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.template:
|
||||
src: atlas-borg-check.sh.j2
|
||||
dest: /usr/local/sbin/atlas-borg-check
|
||||
owner: root
|
||||
group: "{{ atlas_borg_group }}"
|
||||
mode: "0750"
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Create the local libexec directory for the Atlas Borg SSH wrapper
|
||||
tags: [atlas, storage, backup, borg, borg_logging]
|
||||
ansible.builtin.file:
|
||||
path: "{{ atlas_borg_ssh_wrapper_path | dirname }}"
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Install the Atlas Borg progress formatter
|
||||
tags: [atlas, storage, backup, borg, borg_logging]
|
||||
ansible.builtin.copy:
|
||||
src: atlas-borg-progress.py
|
||||
dest: /usr/local/libexec/atlas-borg-progress
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Install the capability-dropping Atlas Borg SSH wrapper
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.template:
|
||||
src: atlas-borg-ssh.sh.j2
|
||||
dest: "{{ atlas_borg_ssh_wrapper_path }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Install Atlas Borg systemd units
|
||||
tags: [atlas, storage, backup, borg, borg_logging]
|
||||
ansible.builtin.template:
|
||||
src: "{{ item }}.j2"
|
||||
dest: "/etc/systemd/system/{{ item }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
loop:
|
||||
- atlas-borg-backup.service
|
||||
- atlas-borg-backup.timer
|
||||
- atlas-borg-check.service
|
||||
- atlas-borg-check.timer
|
||||
notify: Restart Atlas Borg timers
|
||||
when: atlas_manage_borg_backup | bool
|
||||
|
||||
- name: Verify dedicated SSH access to the Hetzner Storage Box
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- ssh
|
||||
- -T
|
||||
- -i
|
||||
- "{{ atlas_borg_ssh_private_key_path }}"
|
||||
- -p
|
||||
- "{{ atlas_borg_repository_port | string }}"
|
||||
- -o
|
||||
- BatchMode=yes
|
||||
- -o
|
||||
- IdentitiesOnly=yes
|
||||
- -o
|
||||
- StrictHostKeyChecking=yes
|
||||
- -o
|
||||
- "UserKnownHostsFile={{ atlas_borg_known_hosts_path }}"
|
||||
- "{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}"
|
||||
- pwd
|
||||
register: atlas_borg_ssh_probe
|
||||
become: true
|
||||
become_user: "{{ atlas_borg_username }}"
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Require the dedicated public key on the Hetzner sub-account
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_borg_ssh_probe.rc == 0
|
||||
fail_msg: >-
|
||||
Install the reported Atlas Borg public key in the Hetzner sub-account
|
||||
before rerunning the Borg tasks. Password authentication is never used.
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Probe the remote Atlas Borg repository path
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- ssh
|
||||
- -T
|
||||
- -i
|
||||
- "{{ atlas_borg_ssh_private_key_path }}"
|
||||
- -p
|
||||
- "{{ atlas_borg_repository_port | string }}"
|
||||
- -o
|
||||
- BatchMode=yes
|
||||
- -o
|
||||
- IdentitiesOnly=yes
|
||||
- -o
|
||||
- StrictHostKeyChecking=yes
|
||||
- -o
|
||||
- "UserKnownHostsFile={{ atlas_borg_known_hosts_path }}"
|
||||
- "{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}"
|
||||
- stat
|
||||
- "{{ atlas_borg_repository_path }}"
|
||||
register: atlas_borg_repository_path_probe
|
||||
become: true
|
||||
become_user: "{{ atlas_borg_username }}"
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Probe the Atlas Borg repository
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- /usr/bin/borg
|
||||
- --remote-path
|
||||
- "{{ atlas_borg_remote_path }}"
|
||||
- info
|
||||
- >-
|
||||
ssh://{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}:
|
||||
{{- atlas_borg_repository_port }}/{{ atlas_borg_repository_path }}
|
||||
environment:
|
||||
BORG_CACHE_DIR: "{{ atlas_borg_cache_dir }}"
|
||||
BORG_CONFIG_DIR: "{{ atlas_borg_config_dir }}"
|
||||
BORG_PASSCOMMAND: "cat {{ atlas_borg_passphrase_path }}"
|
||||
BORG_RSH: "{{ atlas_borg_ssh_wrapper_path }}"
|
||||
register: atlas_borg_repository_probe
|
||||
become: true
|
||||
become_user: "{{ atlas_borg_username }}"
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
- atlas_borg_repository_path_probe.rc == 0
|
||||
|
||||
- name: Reject an existing path that is not the configured Borg repository
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_borg_repository_probe.rc == 0
|
||||
fail_msg: >-
|
||||
The remote repository path already exists but Borg could not open it.
|
||||
Refusing to initialize over existing data; verify the path, passphrase,
|
||||
and repository state manually.
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
- atlas_borg_repository_path_probe.rc == 0
|
||||
|
||||
- name: Initialize the encrypted Atlas Borg repository
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- /usr/bin/borg
|
||||
- --remote-path
|
||||
- "{{ atlas_borg_remote_path }}"
|
||||
- init
|
||||
- --encryption
|
||||
- "{{ atlas_borg_encryption_mode }}"
|
||||
- >-
|
||||
ssh://{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}:
|
||||
{{- atlas_borg_repository_port }}/{{ atlas_borg_repository_path }}
|
||||
environment:
|
||||
BORG_CACHE_DIR: "{{ atlas_borg_cache_dir }}"
|
||||
BORG_CONFIG_DIR: "{{ atlas_borg_config_dir }}"
|
||||
BORG_PASSCOMMAND: "cat {{ atlas_borg_passphrase_path }}"
|
||||
BORG_RSH: "{{ atlas_borg_ssh_wrapper_path }}"
|
||||
become: true
|
||||
become_user: "{{ atlas_borg_username }}"
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
- atlas_borg_repository_path_probe.rc != 0
|
||||
|
||||
- name: Verify the encrypted Atlas Borg repository
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- /usr/bin/borg
|
||||
- --remote-path
|
||||
- "{{ atlas_borg_remote_path }}"
|
||||
- info
|
||||
- >-
|
||||
ssh://{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}:
|
||||
{{- atlas_borg_repository_port }}/{{ atlas_borg_repository_path }}
|
||||
environment:
|
||||
BORG_CACHE_DIR: "{{ atlas_borg_cache_dir }}"
|
||||
BORG_CONFIG_DIR: "{{ atlas_borg_config_dir }}"
|
||||
BORG_PASSCOMMAND: "cat {{ atlas_borg_passphrase_path }}"
|
||||
BORG_RSH: "{{ atlas_borg_ssh_wrapper_path }}"
|
||||
become: true
|
||||
become_user: "{{ atlas_borg_username }}"
|
||||
changed_when: false
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Check for the local Atlas Borg recovery-key export
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.stat:
|
||||
path: "{{ atlas_borg_recovery_export_path }}"
|
||||
register: atlas_borg_recovery_export
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Export the Atlas Borg recovery key for offline preservation
|
||||
tags: [atlas, storage, backup, borg]
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
- not atlas_borg_recovery_export.stat.exists
|
||||
no_log: true
|
||||
block:
|
||||
- name: Create the local recovery-material directory
|
||||
ansible.builtin.file:
|
||||
path: "{{ atlas_borg_recovery_export_path | dirname }}"
|
||||
state: directory
|
||||
mode: "0700"
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
|
||||
- name: Export the encrypted Borg repository key on Atlas
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- /usr/bin/borg
|
||||
- --remote-path
|
||||
- "{{ atlas_borg_remote_path }}"
|
||||
- key
|
||||
- export
|
||||
- >-
|
||||
ssh://{{ atlas_borg_repository_user }}@{{ atlas_borg_repository_host }}:
|
||||
{{- atlas_borg_repository_port }}/{{ atlas_borg_repository_path }}
|
||||
- "{{ atlas_borg_config_dir }}/atlas-borg-repokey.export"
|
||||
environment:
|
||||
BORG_CACHE_DIR: "{{ atlas_borg_cache_dir }}"
|
||||
BORG_CONFIG_DIR: "{{ atlas_borg_config_dir }}"
|
||||
BORG_PASSCOMMAND: "cat {{ atlas_borg_passphrase_path }}"
|
||||
BORG_RSH: "{{ atlas_borg_ssh_wrapper_path }}"
|
||||
become: true
|
||||
become_user: "{{ atlas_borg_username }}"
|
||||
|
||||
- name: Fetch the encrypted Borg recovery key from Atlas
|
||||
ansible.builtin.fetch:
|
||||
src: "{{ atlas_borg_config_dir }}/atlas-borg-repokey.export"
|
||||
dest: "{{ atlas_borg_recovery_export_path }}"
|
||||
flat: true
|
||||
|
||||
- name: Protect the local Borg recovery-key export
|
||||
ansible.builtin.file:
|
||||
path: "{{ atlas_borg_recovery_export_path }}"
|
||||
mode: "0600"
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
always:
|
||||
- name: Remove the temporary recovery-key export from Atlas
|
||||
ansible.builtin.file:
|
||||
path: "{{ atlas_borg_config_dir }}/atlas-borg-repokey.export"
|
||||
state: absent
|
||||
|
||||
- name: Enable Atlas Borg backup and check timers
|
||||
tags: [atlas, storage, backup, borg]
|
||||
ansible.builtin.systemd:
|
||||
name: "{{ item }}"
|
||||
enabled: true
|
||||
state: started
|
||||
daemon_reload: true
|
||||
loop:
|
||||
- atlas-borg-backup.timer
|
||||
- atlas-borg-check.timer
|
||||
when:
|
||||
- atlas_manage_borg_backup | bool
|
||||
- not ansible_check_mode
|
||||
159
ansible/roles/profile_atlas/tasks/gitea.yml
Normal file
159
ansible/roles/profile_atlas/tasks/gitea.yml
Normal file
@@ -0,0 +1,159 @@
|
||||
---
|
||||
- name: Prepare the isolated rootless Atlas Gitea target
|
||||
tags: [atlas, gitea]
|
||||
when: atlas_manage_gitea | bool
|
||||
block:
|
||||
- name: Require the existing Atlas application-data dataset
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_manage_storage | bool
|
||||
- atlas_gitea_dataset == atlas_zfs_pool ~ '/services/data/gitea'
|
||||
- atlas_gitea_mountpoint == atlas_app_data_mountpoint ~ '/gitea'
|
||||
- atlas_gitea_username == atlas_admin_username
|
||||
- atlas_gitea_group == atlas_admin_group
|
||||
- atlas_gitea_uid | int == atlas_admin_uid | int
|
||||
- atlas_gitea_gid | int == atlas_admin_gid | int
|
||||
- atlas_gitea_container_uid | int == 1000
|
||||
- atlas_gitea_container_gid | int == 1000
|
||||
- atlas_gitea_staging_bind_address == '127.0.0.1'
|
||||
- not (atlas_gitea_production_enabled | bool) or atlas_manage_firewall | bool
|
||||
- not (atlas_gitea_production_enabled | bool) or atlas_gitea_bind_address == ansible_host
|
||||
fail_msg: >-
|
||||
Rootless Gitea requires Atlas storage, the admin user manager, the
|
||||
dedicated dataset, and loopback-only staging ports.
|
||||
|
||||
- name: Inspect the final-restore marker before production activation
|
||||
ansible.builtin.stat:
|
||||
path: "{{ atlas_gitea_mountpoint }}/.final-sha256"
|
||||
register: atlas_gitea_final_marker
|
||||
when: atlas_gitea_production_enabled | bool
|
||||
|
||||
- name: Refuse production activation without the final consistent restore
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_gitea_final_marker.stat.isreg | default(false)
|
||||
fail_msg: Restore the final stopped-source Gitea export before enabling production.
|
||||
when: atlas_gitea_production_enabled | bool
|
||||
|
||||
- name: Verify the production Gitea dataset belongs to admin
|
||||
ansible.builtin.stat:
|
||||
path: "{{ atlas_gitea_mountpoint }}"
|
||||
register: atlas_gitea_dataset_owner
|
||||
when: atlas_gitea_production_enabled | bool
|
||||
|
||||
- name: Refuse to overlap the legacy host-account service
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_gitea_dataset_owner.stat.uid | int == atlas_admin_uid | int
|
||||
- atlas_gitea_dataset_owner.stat.gid | int == atlas_admin_gid | int
|
||||
fail_msg: >-
|
||||
Run the explicit Gitea owner migration before enabling the admin
|
||||
Quadlet; never chown an active legacy service in a normal run.
|
||||
when: atlas_gitea_production_enabled | bool
|
||||
|
||||
- name: Remove the retired account's parent-dataset traverse ACL
|
||||
ansible.posix.acl:
|
||||
path: "{{ item }}"
|
||||
etype: user
|
||||
entity: "{{ atlas_gitea_legacy_username }}"
|
||||
state: absent
|
||||
loop:
|
||||
- "{{ atlas_services_mountpoint }}"
|
||||
- "{{ atlas_app_data_mountpoint }}"
|
||||
when: atlas_gitea_production_enabled | bool
|
||||
|
||||
- name: Enable POSIX ACLs only on the service-namespace parents
|
||||
community.general.zfs:
|
||||
name: "{{ item }}"
|
||||
state: present
|
||||
extra_zfs_properties:
|
||||
acltype: posix
|
||||
loop:
|
||||
- "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_services }}"
|
||||
- "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_app_data }}"
|
||||
|
||||
- name: Create the dedicated Gitea ZFS dataset
|
||||
community.general.zfs:
|
||||
name: "{{ atlas_gitea_dataset }}"
|
||||
state: present
|
||||
extra_zfs_properties:
|
||||
compression: zstd
|
||||
mountpoint: "{{ atlas_gitea_mountpoint }}"
|
||||
|
||||
- name: Restrict the Gitea dataset and create rootless volume paths
|
||||
ansible.builtin.file:
|
||||
path: "{{ item }}"
|
||||
state: directory
|
||||
owner: "{{ atlas_gitea_username }}"
|
||||
group: "{{ atlas_gitea_group }}"
|
||||
mode: "0700"
|
||||
loop:
|
||||
- "{{ atlas_gitea_mountpoint }}"
|
||||
- "{{ atlas_gitea_mountpoint }}/data"
|
||||
- "{{ atlas_gitea_mountpoint }}/config"
|
||||
- "{{ atlas_gitea_home }}/.config"
|
||||
- "{{ atlas_gitea_home }}/.config/containers"
|
||||
- "{{ atlas_gitea_quadlet_dir }}"
|
||||
|
||||
- name: Ensure lingering for the admin rootless account
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- loginctl
|
||||
- enable-linger
|
||||
- "{{ atlas_gitea_username }}"
|
||||
creates: "/var/lib/systemd/linger/{{ atlas_gitea_username }}"
|
||||
|
||||
- name: Start the admin rootless user manager
|
||||
ansible.builtin.systemd:
|
||||
name: "user@{{ atlas_gitea_uid }}.service"
|
||||
state: started
|
||||
when: not ansible_check_mode
|
||||
|
||||
- name: Prepare the admin-owned Gitea image
|
||||
ansible.builtin.import_tasks: gitea_image.yml
|
||||
|
||||
- name: Render the rootless Gitea Quadlet
|
||||
ansible.builtin.template:
|
||||
src: atlas-gitea.container.j2
|
||||
dest: "{{ atlas_gitea_quadlet_dir }}/atlas-gitea.container"
|
||||
owner: "{{ atlas_gitea_username }}"
|
||||
group: "{{ atlas_gitea_group }}"
|
||||
mode: "0644"
|
||||
|
||||
- name: Permit only Aegis to reach production Gitea HTTP and SSH
|
||||
ansible.posix.firewalld:
|
||||
rich_rule: >-
|
||||
rule family="ipv4" source address="{{ atlas_aegis_ip }}"
|
||||
port port="{{ item }}" protocol="tcp" accept
|
||||
zone: "{{ atlas_firewalld_zone }}"
|
||||
state: "{{ 'enabled' if atlas_gitea_production_enabled | bool else 'disabled' }}"
|
||||
permanent: true
|
||||
immediate: true
|
||||
loop:
|
||||
- "{{ atlas_gitea_http_port }}"
|
||||
- "{{ atlas_gitea_ssh_port }}"
|
||||
when: atlas_manage_firewall | bool
|
||||
|
||||
- name: Reload the rootless Gitea user manager without starting Gitea
|
||||
become_user: "{{ atlas_gitea_username }}"
|
||||
ansible.builtin.systemd:
|
||||
scope: user
|
||||
daemon_reload: true
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_gitea_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ atlas_gitea_uid }}/bus"
|
||||
when: not ansible_check_mode
|
||||
|
||||
- name: Start and enable the rootless Gitea user Quadlet after final restore
|
||||
become_user: "{{ atlas_gitea_username }}"
|
||||
ansible.builtin.systemd:
|
||||
name: atlas-gitea.service
|
||||
scope: user
|
||||
state: started
|
||||
enabled: true
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_gitea_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ atlas_gitea_uid }}/bus"
|
||||
when:
|
||||
- atlas_gitea_production_enabled | bool
|
||||
- not ansible_check_mode
|
||||
51
ansible/roles/profile_atlas/tasks/gitea_image.yml
Normal file
51
ansible/roles/profile_atlas/tasks/gitea_image.yml
Normal file
@@ -0,0 +1,51 @@
|
||||
---
|
||||
- name: Create the admin-owned Gitea image build directory
|
||||
ansible.builtin.file:
|
||||
path: "{{ atlas_gitea_image_build_dir }}"
|
||||
state: directory
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
mode: "0700"
|
||||
|
||||
- name: Install the pinned rootless Gitea Containerfile
|
||||
ansible.builtin.copy:
|
||||
src: Containerfile.gitea-rootless
|
||||
dest: "{{ atlas_gitea_image_build_dir }}/Containerfile"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
mode: "0644"
|
||||
|
||||
- name: Check the admin-owned Gitea image
|
||||
become_user: "{{ atlas_admin_username }}"
|
||||
ansible.builtin.command:
|
||||
argv: [podman, image, exists, "{{ atlas_gitea_image }}"]
|
||||
args:
|
||||
chdir: "{{ atlas_gitea_image_build_dir }}"
|
||||
environment:
|
||||
HOME: "{{ atlas_admin_home }}"
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_admin_uid }}"
|
||||
register: atlas_gitea_image_present
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
|
||||
- name: Build the pinned Gitea image with the internal gitea identity
|
||||
become_user: "{{ atlas_admin_username }}"
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- podman
|
||||
- build
|
||||
- --pull=always
|
||||
- --tag
|
||||
- "{{ atlas_gitea_image }}"
|
||||
- --file
|
||||
- Containerfile
|
||||
- .
|
||||
args:
|
||||
chdir: "{{ atlas_gitea_image_build_dir }}"
|
||||
environment:
|
||||
HOME: "{{ atlas_admin_home }}"
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_admin_uid }}"
|
||||
when:
|
||||
- atlas_gitea_image_present.rc != 0
|
||||
- not ansible_check_mode
|
||||
275
ansible/roles/profile_atlas/tasks/gitea_owner_migration.yml
Normal file
275
ansible/roles/profile_atlas/tasks/gitea_owner_migration.yml
Normal file
@@ -0,0 +1,275 @@
|
||||
---
|
||||
# Run only in an approved outage with -e atlas_gitea_owner_migration=true.
|
||||
- name: Move live Gitea from the legacy host account to admin
|
||||
tags: [atlas, gitea_owner_migration]
|
||||
when: atlas_gitea_owner_migration | bool
|
||||
block:
|
||||
- name: Refuse a check-mode owner migration
|
||||
ansible.builtin.assert:
|
||||
that: not ansible_check_mode
|
||||
fail_msg: The owner migration requires an explicit live outage.
|
||||
|
||||
- name: Inspect the Gitea dataset owner
|
||||
ansible.builtin.stat:
|
||||
path: "{{ atlas_gitea_mountpoint }}"
|
||||
register: atlas_gitea_migration_owner
|
||||
|
||||
- name: Require either the legacy owner or an already migrated dataset
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_gitea_migration_owner.stat.isdir | default(false)
|
||||
- atlas_gitea_migration_owner.stat.uid | int in [atlas_gitea_legacy_uid | int, atlas_admin_uid | int]
|
||||
fail_msg: Refusing to modify a Gitea dataset with an unexpected owner.
|
||||
|
||||
- name: Migrate only a legacy-owned Gitea dataset
|
||||
when: atlas_gitea_migration_owner.stat.uid | int == atlas_gitea_legacy_uid | int
|
||||
block:
|
||||
- name: Require the final cutover marker and configuration
|
||||
ansible.builtin.stat:
|
||||
path: "{{ item }}"
|
||||
loop:
|
||||
- "{{ atlas_gitea_mountpoint }}/.final-sha256"
|
||||
- "{{ atlas_gitea_mountpoint }}/config/app.ini"
|
||||
register: atlas_gitea_migration_files
|
||||
|
||||
- name: Refuse migration without both final data and configuration
|
||||
ansible.builtin.assert:
|
||||
that: atlas_gitea_migration_files.results | map(attribute='stat.isreg') | min
|
||||
|
||||
- name: Check that admin has no existing Gitea Quadlet
|
||||
ansible.builtin.stat:
|
||||
path: "{{ atlas_gitea_quadlet_dir }}/atlas-gitea.container"
|
||||
register: atlas_gitea_admin_quadlet
|
||||
|
||||
- name: Refuse to overwrite an existing admin Quadlet
|
||||
ansible.builtin.assert:
|
||||
that: not atlas_gitea_admin_quadlet.stat.exists
|
||||
|
||||
- name: Check pool health before the outage
|
||||
ansible.builtin.command:
|
||||
argv: [zpool, status, -x, "{{ atlas_zfs_pool }}"]
|
||||
register: atlas_gitea_pool_before
|
||||
changed_when: false
|
||||
failed_when: "'is healthy' not in atlas_gitea_pool_before.stdout"
|
||||
|
||||
- name: Ensure the admin Gitea image is available before stopping the source
|
||||
ansible.builtin.import_tasks: gitea_image.yml
|
||||
|
||||
- name: Stop, snapshot and test the admin-owned staging service
|
||||
block:
|
||||
- name: Stop and disable the legacy Gitea user service
|
||||
become_user: "{{ atlas_gitea_legacy_username }}"
|
||||
ansible.builtin.systemd:
|
||||
name: atlas-gitea.service
|
||||
scope: user
|
||||
state: stopped
|
||||
enabled: false
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_gitea_legacy_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ atlas_gitea_legacy_uid }}/bus"
|
||||
|
||||
- name: Record the migration snapshot name
|
||||
ansible.builtin.set_fact:
|
||||
atlas_gitea_migration_snapshot: >-
|
||||
{{ atlas_gitea_dataset }}@gitea-owner-migration-{{ ansible_facts.date_time.iso8601_basic_short }}
|
||||
|
||||
- name: Snapshot the stopped Gitea dataset for manual recovery
|
||||
ansible.builtin.command:
|
||||
argv: [zfs, snapshot, "{{ atlas_gitea_migration_snapshot }}"]
|
||||
|
||||
- name: Transfer only the Gitea dataset to admin
|
||||
ansible.builtin.file:
|
||||
path: "{{ atlas_gitea_mountpoint }}"
|
||||
state: directory
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
recurse: true
|
||||
|
||||
- name: Set the actual internal Unix process user
|
||||
ansible.builtin.lineinfile:
|
||||
path: "{{ atlas_gitea_mountpoint }}/config/app.ini"
|
||||
regexp: '^RUN_USER\s*='
|
||||
line: RUN_USER = gitea
|
||||
mode: "0600"
|
||||
no_log: true
|
||||
diff: false
|
||||
|
||||
- name: Preserve public git clone URLs independently of the Unix user
|
||||
community.general.ini_file:
|
||||
path: "{{ atlas_gitea_mountpoint }}/config/app.ini"
|
||||
section: server
|
||||
option: "{{ item }}"
|
||||
value: git
|
||||
mode: "0600"
|
||||
no_extra_spaces: false
|
||||
loop: [BUILTIN_SSH_SERVER_USER, SSH_USER]
|
||||
no_log: true
|
||||
diff: false
|
||||
|
||||
- name: Render admin's loopback-only staging Quadlet
|
||||
ansible.builtin.template:
|
||||
src: atlas-gitea.container.j2
|
||||
dest: "{{ atlas_gitea_quadlet_dir }}/atlas-gitea.container"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
mode: "0644"
|
||||
vars:
|
||||
atlas_gitea_production_enabled: false
|
||||
|
||||
- name: Reload the admin user manager for staging
|
||||
become_user: "{{ atlas_admin_username }}"
|
||||
ansible.builtin.systemd:
|
||||
scope: user
|
||||
daemon_reload: true
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_admin_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ atlas_admin_uid }}/bus"
|
||||
|
||||
- name: Start admin's loopback-only staging service
|
||||
become_user: "{{ atlas_admin_username }}"
|
||||
ansible.builtin.systemd:
|
||||
name: atlas-gitea.service
|
||||
scope: user
|
||||
state: started
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_admin_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ atlas_admin_uid }}/bus"
|
||||
|
||||
- name: Verify staging HTTP before promotion
|
||||
ansible.builtin.uri:
|
||||
url: "http://127.0.0.1:{{ atlas_gitea_staging_http_port }}/"
|
||||
status_code: 200
|
||||
register: atlas_gitea_staging_http
|
||||
retries: 30
|
||||
delay: 2
|
||||
until: atlas_gitea_staging_http is succeeded
|
||||
|
||||
- name: Verify the container really runs as internal gitea
|
||||
become_user: "{{ atlas_admin_username }}"
|
||||
ansible.builtin.command:
|
||||
argv: [podman, exec, atlas-gitea, id, -un]
|
||||
environment:
|
||||
HOME: "{{ atlas_admin_home }}"
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_admin_uid }}"
|
||||
register: atlas_gitea_internal_user
|
||||
changed_when: false
|
||||
failed_when: atlas_gitea_internal_user.stdout != 'gitea'
|
||||
|
||||
- name: Verify the migrated SQLite database
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- sqlite3
|
||||
- "{{ atlas_gitea_mountpoint }}/data/gitea/gitea.db"
|
||||
- PRAGMA quick_check;
|
||||
register: atlas_gitea_migration_sqlite
|
||||
changed_when: false
|
||||
failed_when: atlas_gitea_migration_sqlite.stdout != 'ok'
|
||||
|
||||
rescue:
|
||||
- name: Stop admin's failed staging service
|
||||
become_user: "{{ atlas_admin_username }}"
|
||||
ansible.builtin.systemd:
|
||||
name: atlas-gitea.service
|
||||
scope: user
|
||||
state: stopped
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_admin_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ atlas_admin_uid }}/bus"
|
||||
failed_when: false
|
||||
|
||||
- name: Restore the original Gitea configuration from the safety snapshot
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- cp
|
||||
- -a
|
||||
- "{{ atlas_gitea_mountpoint }}/.zfs/snapshot/{{ atlas_gitea_migration_snapshot.split('@')[1] }}/config/app.ini"
|
||||
- "{{ atlas_gitea_mountpoint }}/config/app.ini"
|
||||
when: atlas_gitea_migration_snapshot is defined
|
||||
|
||||
- name: Return the Gitea dataset to the legacy account
|
||||
ansible.builtin.file:
|
||||
path: "{{ atlas_gitea_mountpoint }}"
|
||||
state: directory
|
||||
owner: "{{ atlas_gitea_legacy_username }}"
|
||||
group: "{{ atlas_gitea_legacy_username }}"
|
||||
recurse: true
|
||||
|
||||
- name: Restart the legacy Gitea service
|
||||
become_user: "{{ atlas_gitea_legacy_username }}"
|
||||
ansible.builtin.systemd:
|
||||
name: atlas-gitea.service
|
||||
scope: user
|
||||
state: started
|
||||
enabled: true
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_gitea_legacy_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ atlas_gitea_legacy_uid }}/bus"
|
||||
|
||||
- name: Report the failed migration and preserved snapshot
|
||||
ansible.builtin.fail:
|
||||
msg: >-
|
||||
Admin staging failed; legacy Gitea was restarted. Inspect
|
||||
{{ atlas_gitea_migration_snapshot | default('the host journal') }}.
|
||||
|
||||
- name: Stop admin's validated staging service
|
||||
become_user: "{{ atlas_admin_username }}"
|
||||
ansible.builtin.systemd:
|
||||
name: atlas-gitea.service
|
||||
scope: user
|
||||
state: stopped
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_admin_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ atlas_admin_uid }}/bus"
|
||||
|
||||
- name: Render admin's production Gitea Quadlet
|
||||
ansible.builtin.template:
|
||||
src: atlas-gitea.container.j2
|
||||
dest: "{{ atlas_gitea_quadlet_dir }}/atlas-gitea.container"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
mode: "0644"
|
||||
vars:
|
||||
atlas_gitea_production_enabled: true
|
||||
|
||||
- name: Reload admin's production user manager
|
||||
become_user: "{{ atlas_admin_username }}"
|
||||
ansible.builtin.systemd:
|
||||
scope: user
|
||||
daemon_reload: true
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_admin_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ atlas_admin_uid }}/bus"
|
||||
|
||||
- name: Enable and start admin's production Gitea
|
||||
become_user: "{{ atlas_admin_username }}"
|
||||
ansible.builtin.systemd:
|
||||
name: atlas-gitea.service
|
||||
scope: user
|
||||
state: started
|
||||
enabled: true
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_admin_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ atlas_admin_uid }}/bus"
|
||||
|
||||
- name: Verify production HTTP before retiring the old Quadlet
|
||||
ansible.builtin.uri:
|
||||
url: "http://{{ atlas_gitea_bind_address }}:{{ atlas_gitea_http_port }}/"
|
||||
status_code: 200
|
||||
register: atlas_gitea_production_http
|
||||
retries: 30
|
||||
delay: 2
|
||||
until: atlas_gitea_production_http is succeeded
|
||||
|
||||
- name: Remove only the disabled legacy Quadlet
|
||||
ansible.builtin.file:
|
||||
path: "{{ atlas_gitea_legacy_home }}/.config/containers/systemd/atlas-gitea.container"
|
||||
state: absent
|
||||
|
||||
- name: Reload the legacy user manager after Quadlet removal
|
||||
become_user: "{{ atlas_gitea_legacy_username }}"
|
||||
ansible.builtin.systemd:
|
||||
scope: user
|
||||
daemon_reload: true
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_gitea_legacy_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ atlas_gitea_legacy_uid }}/bus"
|
||||
58
ansible/roles/profile_atlas/tasks/gitea_public_domain.yml
Normal file
58
ansible/roles/profile_atlas/tasks/gitea_public_domain.yml
Normal file
@@ -0,0 +1,58 @@
|
||||
---
|
||||
- name: Manage the public domain of the restored production Gitea
|
||||
tags: [atlas, gitea, gitea_public_domain]
|
||||
when:
|
||||
- atlas_manage_gitea | bool
|
||||
- atlas_gitea_production_enabled | bool
|
||||
- atlas_gitea_public_domain | length > 0
|
||||
block:
|
||||
- name: Require an explicit public Gitea hostname
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_gitea_public_domain is match('^[a-zA-Z0-9][a-zA-Z0-9.-]*\.[a-zA-Z]{2,}$')
|
||||
|
||||
- name: Inspect the restored private Gitea configuration
|
||||
ansible.builtin.stat:
|
||||
path: "{{ atlas_gitea_mountpoint }}/config/app.ini"
|
||||
follow: false
|
||||
register: atlas_gitea_public_config
|
||||
|
||||
- name: Refuse to create or replace an unprepared Gitea configuration
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_gitea_public_config.stat.isreg | default(false)
|
||||
- atlas_gitea_public_config.stat.uid | int == atlas_gitea_uid | int
|
||||
- atlas_gitea_public_config.stat.mode == '0600'
|
||||
|
||||
# app.ini contains secrets: preserve all unrelated settings and suppress diffs.
|
||||
- name: Set only the declared public Gitea server fields
|
||||
community.general.ini_file:
|
||||
path: "{{ atlas_gitea_mountpoint }}/config/app.ini"
|
||||
section: server
|
||||
option: "{{ item.option }}"
|
||||
value: "{{ item.value }}"
|
||||
create: false
|
||||
backup: true
|
||||
owner: "{{ atlas_gitea_username }}"
|
||||
group: "{{ atlas_gitea_group }}"
|
||||
mode: "0600"
|
||||
loop:
|
||||
- { option: DOMAIN, value: "{{ atlas_gitea_public_domain }}" }
|
||||
- { option: ROOT_URL, value: "https://{{ atlas_gitea_public_domain }}/" }
|
||||
- { option: SSH_DOMAIN, value: "{{ atlas_gitea_public_domain }}" }
|
||||
register: atlas_gitea_public_domain_update
|
||||
no_log: true
|
||||
diff: false
|
||||
|
||||
- name: Restart only Gitea when its public configuration changes
|
||||
become_user: "{{ atlas_gitea_username }}"
|
||||
ansible.builtin.systemd:
|
||||
name: atlas-gitea.service
|
||||
scope: user
|
||||
state: restarted
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_gitea_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ atlas_gitea_uid }}/bus"
|
||||
when:
|
||||
- atlas_gitea_public_domain_update is changed
|
||||
- not ansible_check_mode
|
||||
107
ansible/roles/profile_atlas/tasks/gitea_restore.yml
Normal file
107
ansible/roles/profile_atlas/tasks/gitea_restore.yml
Normal file
@@ -0,0 +1,107 @@
|
||||
---
|
||||
- name: Restore Gitea from a verified Prometheus backup only on explicit request
|
||||
tags: [atlas, gitea_restore, gitea_final_restore]
|
||||
when: atlas_gitea_restore_test | bool or atlas_gitea_final_restore | bool
|
||||
block:
|
||||
- name: Require the prepared rootless Gitea target
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_manage_gitea | bool
|
||||
- not (atlas_gitea_restore_test | bool and atlas_gitea_final_restore | bool)
|
||||
- atlas_gitea_staging_bind_address == '127.0.0.1'
|
||||
- atlas_gitea_mountpoint == atlas_app_data_mountpoint ~ '/gitea'
|
||||
fail_msg: Prepare the isolated, loopback-only rootless Gitea target first.
|
||||
|
||||
- name: Confirm the rootless Gitea service is inactive
|
||||
become_user: "{{ atlas_gitea_username }}"
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- systemctl
|
||||
- --user
|
||||
- is-active
|
||||
- atlas-gitea.service
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_gitea_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ atlas_gitea_uid }}/bus"
|
||||
register: atlas_gitea_restore_service_state
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
when: not ansible_check_mode
|
||||
|
||||
- name: Refuse to overwrite an active rootless Gitea service
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_gitea_restore_service_state.stdout == 'inactive'
|
||||
fail_msg: The rootless Gitea user service must be known and inactive before restoring data.
|
||||
when: not ansible_check_mode
|
||||
|
||||
- name: Check for a manually running rootless Gitea container
|
||||
become_user: "{{ atlas_gitea_username }}"
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- podman
|
||||
- ps
|
||||
- --quiet
|
||||
- --filter
|
||||
- name=atlas-gitea
|
||||
args:
|
||||
chdir: "{{ atlas_gitea_home }}"
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_gitea_uid }}"
|
||||
register: atlas_gitea_restore_container_state
|
||||
changed_when: false
|
||||
when: not ansible_check_mode
|
||||
|
||||
- name: Refuse to overwrite a running rootless Gitea container
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_gitea_restore_container_state.stdout | length == 0
|
||||
fail_msg: Stop every rootless Atlas Gitea container before restoring data.
|
||||
when: not ansible_check_mode
|
||||
|
||||
- name: Install the selective rootless Gitea restore helper
|
||||
ansible.builtin.copy:
|
||||
src: atlas-gitea-restore-test.py
|
||||
dest: "{{ atlas_gitea_restore_helper }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0700"
|
||||
|
||||
- name: Restore only Gitea data into the isolated target
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- "{{ atlas_gitea_restore_helper }}"
|
||||
- --backup
|
||||
- "{{ atlas_backup_prometheus_mountpoint }}/latest"
|
||||
- --target
|
||||
- "{{ atlas_gitea_mountpoint }}"
|
||||
- --uid
|
||||
- "{{ atlas_gitea_uid | string }}"
|
||||
- --gid
|
||||
- "{{ atlas_gitea_gid | string }}"
|
||||
register: atlas_gitea_restore_result
|
||||
changed_when: atlas_gitea_restore_result.stdout == 'restored'
|
||||
no_log: true
|
||||
when:
|
||||
- atlas_gitea_restore_test | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Replace the marked rehearsal with the final consistent Gitea export
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- "{{ atlas_gitea_restore_helper }}"
|
||||
- --backup
|
||||
- "{{ atlas_backup_prometheus_mountpoint }}/latest"
|
||||
- --target
|
||||
- "{{ atlas_gitea_mountpoint }}"
|
||||
- --uid
|
||||
- "{{ atlas_gitea_uid | string }}"
|
||||
- --gid
|
||||
- "{{ atlas_gitea_gid | string }}"
|
||||
- --replace-rehearsal
|
||||
register: atlas_gitea_final_restore_result
|
||||
changed_when: atlas_gitea_final_restore_result.stdout == 'restored'
|
||||
no_log: true
|
||||
when:
|
||||
- atlas_gitea_final_restore | bool
|
||||
- not ansible_check_mode
|
||||
188
ansible/roles/profile_atlas/tasks/icloudpd.yml
Normal file
188
ansible/roles/profile_atlas/tasks/icloudpd.yml
Normal file
@@ -0,0 +1,188 @@
|
||||
---
|
||||
- name: Require exact Atlas iCloudPD paths and rootless identity
|
||||
tags: [atlas, icloudpd]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_manage_storage | bool
|
||||
- atlas_icloudpd_dataset == atlas_zfs_pool ~ '/services/data/icloudpd'
|
||||
- atlas_icloudpd_state_dir == atlas_app_data_mountpoint ~ '/icloudpd'
|
||||
- atlas_icloudpd_config_dir == atlas_icloudpd_state_dir ~ '/config'
|
||||
- atlas_icloudpd_photos_dir == atlas_archive_mountpoint ~ '/Pictures/iCloudPD'
|
||||
- atlas_admin_uid | int == 1000
|
||||
- atlas_admin_gid | int == 1000
|
||||
- atlas_icloudpd_image is search('@sha256:[0-9a-f]{64}$')
|
||||
fail_msg: Verify the fixed, separate Atlas iCloudPD photo and state paths.
|
||||
|
||||
- name: Declare rootless Atlas iCloudPD storage and boot-started Quadlet
|
||||
tags: [atlas, icloudpd]
|
||||
block:
|
||||
- name: Inspect the existing Archive and application-data datasets
|
||||
community.general.zfs_facts:
|
||||
name: "{{ item.dataset }}"
|
||||
properties: name,mounted,mountpoint
|
||||
loop:
|
||||
- dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_archive }}"
|
||||
mountpoint: "{{ atlas_archive_mountpoint }}"
|
||||
- dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_app_data }}"
|
||||
mountpoint: "{{ atlas_app_data_mountpoint }}"
|
||||
loop_control:
|
||||
label: "{{ item.dataset }}"
|
||||
register: atlas_icloudpd_parent_datasets
|
||||
|
||||
- name: Refuse missing or unmounted iCloudPD parent datasets
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- item.ansible_facts.ansible_zfs_datasets | length == 1
|
||||
- item.ansible_facts.ansible_zfs_datasets[0].mounted == 'yes'
|
||||
- item.ansible_facts.ansible_zfs_datasets[0].mountpoint == item.item.mountpoint
|
||||
loop: "{{ atlas_icloudpd_parent_datasets.results }}"
|
||||
loop_control:
|
||||
label: "{{ item.item.dataset }}"
|
||||
|
||||
- name: Inspect the existing Pictures namespace and proposed target
|
||||
ansible.builtin.stat:
|
||||
path: "{{ item }}"
|
||||
follow: false
|
||||
loop:
|
||||
- "{{ atlas_archive_mountpoint }}/Pictures"
|
||||
- "{{ atlas_icloudpd_photos_dir }}"
|
||||
- "{{ atlas_icloudpd_photos_dir }}/.atlas-icloudpd-managed"
|
||||
register: atlas_icloudpd_photo_paths
|
||||
|
||||
- name: Refuse to adopt unrelated Pictures data or a symlink
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_icloudpd_photo_paths.results[0].stat.isdir | default(false)
|
||||
- atlas_icloudpd_photo_paths.results[0].stat.uid | int == atlas_admin_uid | int
|
||||
- >-
|
||||
not atlas_icloudpd_photo_paths.results[1].stat.exists or
|
||||
(atlas_icloudpd_photo_paths.results[1].stat.isdir | default(false) and
|
||||
atlas_icloudpd_photo_paths.results[2].stat.isreg | default(false))
|
||||
fail_msg: >-
|
||||
Pictures must exist and be admin-owned; an existing iCloudPD target
|
||||
must carry its managed marker. Never adopt or replace unrelated data.
|
||||
|
||||
- name: Create a dedicated ZFS dataset for iCloudPD configuration and MFA
|
||||
community.general.zfs:
|
||||
name: "{{ atlas_icloudpd_dataset }}"
|
||||
state: present
|
||||
extra_zfs_properties:
|
||||
compression: zstd
|
||||
mountpoint: "{{ atlas_icloudpd_state_dir }}"
|
||||
|
||||
- name: Restrict iCloudPD state and the new photo subtree
|
||||
ansible.builtin.file:
|
||||
path: "{{ item.path }}"
|
||||
state: directory
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
mode: "{{ item.mode }}"
|
||||
loop:
|
||||
- path: "{{ atlas_icloudpd_state_dir }}"
|
||||
mode: "0700"
|
||||
- path: "{{ atlas_icloudpd_config_dir }}"
|
||||
mode: "0700"
|
||||
- path: "{{ atlas_icloudpd_photos_dir }}"
|
||||
mode: "0750"
|
||||
- path: "{{ atlas_icloudpd_quadlet_dir }}"
|
||||
mode: "0700"
|
||||
loop_control:
|
||||
label: "{{ item.path }}"
|
||||
|
||||
- name: Mark only the newly managed iCloudPD photo subtree
|
||||
ansible.builtin.copy:
|
||||
content: "Atlas iCloudPD photo subtree; do not remove source photos.\n"
|
||||
dest: "{{ atlas_icloudpd_photos_dir }}/.atlas-icloudpd-managed"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
mode: "0600"
|
||||
force: false
|
||||
|
||||
- name: Install the image's required mounted-filesystem failsafe
|
||||
ansible.builtin.copy:
|
||||
content: ""
|
||||
dest: "{{ atlas_icloudpd_photos_dir }}/.mounted"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
mode: "0644"
|
||||
force: false
|
||||
|
||||
- name: Require the Vault-backed iCloudPD Apple ID
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- vault_atlas_icloudpd_apple_id is defined
|
||||
- vault_atlas_icloudpd_apple_id | length > 0
|
||||
- vault_atlas_icloudpd_apple_id != 'REPLACE_ME'
|
||||
- vault_atlas_icloudpd_apple_id.splitlines() | length == 1
|
||||
fail_msg: Configure the existing iCloudPD Apple ID in Vault.
|
||||
no_log: true
|
||||
|
||||
- name: Seed private Atlas iCloudPD configuration when absent
|
||||
ansible.builtin.template:
|
||||
src: atlas-icloudpd.conf.j2
|
||||
dest: "{{ atlas_icloudpd_config_dir }}/icloudpd.conf"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
mode: "0600"
|
||||
force: false
|
||||
no_log: true
|
||||
diff: false
|
||||
|
||||
- name: Keep declared iCloudPD options in the image-managed configuration
|
||||
ansible.builtin.lineinfile:
|
||||
path: "{{ atlas_icloudpd_config_dir }}/icloudpd.conf"
|
||||
regexp: "^{{ item.key }}="
|
||||
line: "{{ item.key }}={{ item.value }}"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
mode: "0600"
|
||||
loop:
|
||||
- {key: apple_id, value: "{{ vault_atlas_icloudpd_apple_id }}"}
|
||||
- {key: authentication_type, value: MFA}
|
||||
- {key: user, value: user}
|
||||
- {key: user_id, value: "1000"}
|
||||
- {key: group, value: group}
|
||||
- {key: group_id, value: "1000"}
|
||||
- {key: download_path, value: /home/user/iCloud}
|
||||
- {key: folder_structure, value: "{:%Y/%m/%d}"}
|
||||
- {key: directory_permissions, value: "750"}
|
||||
- {key: file_permissions, value: "640"}
|
||||
- {key: download_interval, value: "86400"}
|
||||
- {key: auto_delete, value: "false"}
|
||||
- {key: delete_after_download, value: "false"}
|
||||
loop_control:
|
||||
label: "{{ item.key }}"
|
||||
no_log: true
|
||||
diff: false
|
||||
|
||||
- name: Render the rootless Atlas iCloudPD Quadlet
|
||||
ansible.builtin.template:
|
||||
src: atlas-icloudpd.container.j2
|
||||
dest: "{{ atlas_icloudpd_quadlet_dir }}/atlas-icloudpd.container"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
mode: "0644"
|
||||
register: atlas_icloudpd_quadlet
|
||||
|
||||
- name: Reload the Atlas admin user manager after iCloudPD Quadlet changes
|
||||
become_user: "{{ atlas_admin_username }}"
|
||||
ansible.builtin.systemd:
|
||||
scope: user
|
||||
daemon_reload: true
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_admin_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ atlas_admin_uid }}/bus"
|
||||
when:
|
||||
- atlas_icloudpd_quadlet.changed
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Keep the rootless Atlas iCloudPD service running
|
||||
become_user: "{{ atlas_admin_username }}"
|
||||
ansible.builtin.systemd:
|
||||
name: atlas-icloudpd.service
|
||||
scope: user
|
||||
state: started
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ atlas_admin_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ atlas_admin_uid }}/bus"
|
||||
when: not ansible_check_mode
|
||||
@@ -14,6 +14,42 @@
|
||||
- name: Import Atlas storage tasks
|
||||
ansible.builtin.import_tasks: storage.yml
|
||||
|
||||
- name: Import explicit Atlas Gitea owner migration
|
||||
ansible.builtin.import_tasks: gitea_owner_migration.yml
|
||||
|
||||
- name: Import staged Atlas rootless Gitea tasks
|
||||
ansible.builtin.import_tasks: gitea.yml
|
||||
|
||||
- name: Import the declared Atlas Gitea public domain
|
||||
ansible.builtin.import_tasks: gitea_public_domain.yml
|
||||
|
||||
- name: Import Atlas iCloudPD storage and boot-started Quadlet tasks
|
||||
ansible.builtin.import_tasks: icloudpd.yml
|
||||
|
||||
- name: Import explicit Atlas Gitea restore rehearsal tasks
|
||||
ansible.builtin.import_tasks: gitea_restore.yml
|
||||
|
||||
- name: Import Atlas ZFS maintenance tasks
|
||||
ansible.builtin.import_tasks: zfs_maintenance.yml
|
||||
|
||||
- name: Import Atlas Borg backup tasks
|
||||
ansible.builtin.import_tasks: borg_backup.yml
|
||||
|
||||
- name: Import Atlas offline USB backup tasks
|
||||
ansible.builtin.import_tasks: usb_backup.yml
|
||||
|
||||
- name: Import Atlas Prometheus backup pull identity tasks
|
||||
ansible.builtin.import_tasks: prometheus_pull_identity.yml
|
||||
|
||||
- name: Import Atlas Prometheus backup pull job tasks
|
||||
ansible.builtin.import_tasks: prometheus_pull_job.yml
|
||||
|
||||
- name: Import Atlas health monitoring tasks
|
||||
ansible.builtin.import_tasks: monitoring.yml
|
||||
|
||||
- name: Import Atlas post-restore SELinux relabeling tasks
|
||||
ansible.builtin.import_tasks: restorecon.yml
|
||||
|
||||
- name: Import Atlas file sharing tasks
|
||||
ansible.builtin.import_tasks: sharing.yml
|
||||
|
||||
|
||||
201
ansible/roles/profile_atlas/tasks/monitoring.yml
Normal file
201
ansible/roles/profile_atlas/tasks/monitoring.yml
Normal file
@@ -0,0 +1,201 @@
|
||||
---
|
||||
- name: Validate Atlas health monitoring policy
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_manage_storage | bool
|
||||
- atlas_zfs_pool != 'CHANGEME_ZFS_POOL'
|
||||
- atlas_monitor_calendar | length > 0
|
||||
- atlas_monitor_smart_devices | length > 0
|
||||
- atlas_monitor_effective_timers | length > 0
|
||||
- atlas_monitor_effective_failure_units | length > 0
|
||||
- atlas_monitor_remote_capacity.user == atlas_borg_repository_user
|
||||
- atlas_monitor_remote_capacity.host == atlas_borg_repository_host
|
||||
- atlas_monitor_remote_capacity.run_as == atlas_borg_username
|
||||
- atlas_monitor_remote_capacity.ssh_wrapper == atlas_borg_ssh_wrapper_path
|
||||
- >-
|
||||
0 < atlas_monitor_remote_capacity.warning_percent | int
|
||||
< atlas_monitor_remote_capacity.critical_percent | int < 100
|
||||
- atlas_monitor_remote_capacity.growth_warning_gib_day | int > 0
|
||||
- atlas_monitor_notifier.startswith('/opt/45drives/houston/')
|
||||
- 0 < atlas_monitor_pool_warning_percent | int < atlas_monitor_pool_critical_percent | int < 100
|
||||
- 0 < atlas_monitor_root_warning_percent | int < atlas_monitor_root_critical_percent | int < 100
|
||||
- 0 < atlas_monitor_snapshot_warning_percent | int < atlas_monitor_snapshot_critical_percent | int < 100
|
||||
- atlas_monitor_snapshot_growth_warning_gib_day | int > 0
|
||||
- atlas_monitor_backup_growth_warning_gib_day | int > 0
|
||||
- 0 < atlas_monitor_cpu_warning_c | int < atlas_monitor_cpu_critical_c | int
|
||||
- atlas_monitor_borg_max_runtime_days | int > 0
|
||||
fail_msg: >-
|
||||
Atlas health monitoring needs real devices, job units, a valid calendar,
|
||||
positive ordered thresholds, and the existing Houston notifier.
|
||||
when: atlas_manage_monitoring | bool
|
||||
|
||||
- name: Validate monitored Atlas SMART devices
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- item.name is match('^[a-z0-9][a-z0-9_-]*$')
|
||||
- item.path.startswith('/dev/disk/by-id/')
|
||||
- 0 < item.warning_c | int < item.critical_c | int
|
||||
fail_msg: "Every monitored disk needs a stable by-id path and ordered temperature thresholds."
|
||||
loop: "{{ atlas_monitor_smart_devices }}"
|
||||
loop_control:
|
||||
label: "{{ item.name }}"
|
||||
when: atlas_manage_monitoring | bool
|
||||
|
||||
- name: Validate monitored Atlas timer names and age thresholds
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- item.name is match('^[a-zA-Z0-9@_.-]+\\.timer$')
|
||||
- item.max_age_hours | int >= 0
|
||||
loop: "{{ atlas_monitor_effective_timers }}"
|
||||
loop_control:
|
||||
label: "{{ item.name }}"
|
||||
when: atlas_manage_monitoring | bool
|
||||
|
||||
- name: Validate monitored Atlas failure unit names
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- item is match('^[a-zA-Z0-9@_.-]+\\.service$')
|
||||
loop: "{{ atlas_monitor_effective_failure_units }}"
|
||||
when: atlas_manage_monitoring | bool
|
||||
|
||||
- name: Validate Atlas health monitor calendar
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.command:
|
||||
argv: [systemd-analyze, calendar, "{{ atlas_monitor_calendar }}"]
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
when: atlas_manage_monitoring | bool
|
||||
|
||||
- name: Install SMART tooling for Atlas health checks
|
||||
tags: [atlas, monitoring, packages]
|
||||
ansible.builtin.dnf:
|
||||
name: smartmontools
|
||||
state: present
|
||||
when: atlas_manage_monitoring | bool
|
||||
|
||||
- name: Inspect the existing 45Drives notifier for monitoring
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.stat:
|
||||
path: "{{ atlas_monitor_notifier }}"
|
||||
register: atlas_monitor_notifier_file
|
||||
when: atlas_manage_monitoring | bool
|
||||
|
||||
- name: Require the existing 45Drives notifier for monitoring
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_monitor_notifier_file.stat.executable | default(false)
|
||||
fail_msg: "The existing 45Drives Houston notifier must be executable."
|
||||
when: atlas_manage_monitoring | bool
|
||||
|
||||
- name: Create private Atlas health monitor state directory
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.file:
|
||||
path: /var/lib/atlas-health-monitor
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0700"
|
||||
when: atlas_manage_monitoring | bool
|
||||
|
||||
- name: Install Atlas health monitor configuration
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.template:
|
||||
src: atlas-health-monitor.json.j2
|
||||
dest: /etc/atlas-health-monitor.json
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0600"
|
||||
when: atlas_manage_monitoring | bool
|
||||
|
||||
- name: Install Atlas health monitor helper
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.copy:
|
||||
src: atlas-health-monitor.py
|
||||
dest: /usr/local/libexec/atlas-health-monitor
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0750"
|
||||
when: atlas_manage_monitoring | bool
|
||||
|
||||
- name: Install Atlas health monitoring units
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.template:
|
||||
src: "{{ item }}.j2"
|
||||
dest: "/etc/systemd/system/{{ item }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
loop:
|
||||
- atlas-health-monitor.service
|
||||
- atlas-health-monitor.timer
|
||||
- atlas-monitor-failure@.service
|
||||
when: atlas_manage_monitoring | bool
|
||||
|
||||
- name: Create failure hook directories for monitored Atlas jobs
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.file:
|
||||
path: "/etc/systemd/system/{{ item }}.d"
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
loop: "{{ atlas_monitor_effective_failure_units }}"
|
||||
when: atlas_manage_monitoring | bool
|
||||
|
||||
- name: Notify 45Drives Alerts when an Atlas job fails
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.template:
|
||||
src: atlas-monitor-failure.conf.j2
|
||||
dest: "/etc/systemd/system/{{ item }}.d/atlas-monitor.conf"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
loop: "{{ atlas_monitor_effective_failure_units }}"
|
||||
when: atlas_manage_monitoring | bool
|
||||
|
||||
- name: Reload systemd after installing Atlas monitoring
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.systemd:
|
||||
daemon_reload: true
|
||||
when:
|
||||
- atlas_manage_monitoring | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Enable the Atlas health monitoring timer
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.systemd:
|
||||
name: atlas-health-monitor.timer
|
||||
enabled: true
|
||||
state: started
|
||||
when:
|
||||
- atlas_manage_monitoring | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Validate the deployed Atlas health monitoring units
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- systemd-analyze
|
||||
- verify
|
||||
- atlas-health-monitor.service
|
||||
- atlas-health-monitor.timer
|
||||
- atlas-monitor-failure@.service
|
||||
changed_when: false
|
||||
when:
|
||||
- atlas_manage_monitoring | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Probe Atlas health without sending notifications
|
||||
tags: [atlas, monitoring]
|
||||
ansible.builtin.command:
|
||||
argv: [/usr/local/libexec/atlas-health-monitor, --dry-run]
|
||||
register: atlas_monitor_dry_run
|
||||
changed_when: false
|
||||
when:
|
||||
- atlas_manage_monitoring | bool
|
||||
- not ansible_check_mode
|
||||
@@ -0,0 +1,66 @@
|
||||
---
|
||||
- name: Validate Atlas Prometheus pull identity inputs
|
||||
tags: [atlas, backup, prometheus_backup, prometheus_backup_key]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_prometheus_pull_ssh_dir.startswith('/etc/')
|
||||
- atlas_prometheus_pull_private_key_path.startswith(atlas_prometheus_pull_ssh_dir ~ '/')
|
||||
- atlas_prometheus_pull_known_hosts_path.startswith(atlas_prometheus_pull_ssh_dir ~ '/')
|
||||
- atlas_prometheus_ssh_host_key.startswith(
|
||||
(hostvars['prometheus'].ansible_host | string) ~ ' ssh-ed25519 '
|
||||
)
|
||||
fail_msg: Pin the verified Prometheus ED25519 SSH host key before enabling the pull.
|
||||
when: atlas_manage_prometheus_backup_pull | bool
|
||||
|
||||
- name: Create private Atlas Prometheus pull SSH directory
|
||||
tags: [atlas, backup, prometheus_backup, prometheus_backup_key]
|
||||
ansible.builtin.file:
|
||||
path: "{{ atlas_prometheus_pull_ssh_dir }}"
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0700"
|
||||
when: atlas_manage_prometheus_backup_pull | bool
|
||||
|
||||
- name: Generate Atlas-only Prometheus pull SSH identity
|
||||
tags: [atlas, backup, prometheus_backup, prometheus_backup_key]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- ssh-keygen
|
||||
- -q
|
||||
- -t
|
||||
- ed25519
|
||||
- -N
|
||||
- ""
|
||||
- -C
|
||||
- atlas-prometheus-pull@atlas
|
||||
- -f
|
||||
- "{{ atlas_prometheus_pull_private_key_path }}"
|
||||
creates: "{{ atlas_prometheus_pull_private_key_path }}"
|
||||
when: atlas_manage_prometheus_backup_pull | bool
|
||||
|
||||
- name: Protect Atlas-only Prometheus pull SSH identity
|
||||
tags: [atlas, backup, prometheus_backup, prometheus_backup_key]
|
||||
ansible.builtin.file:
|
||||
path: "{{ item.path }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "{{ item.mode }}"
|
||||
loop:
|
||||
- { path: "{{ atlas_prometheus_pull_private_key_path }}", mode: "0600" }
|
||||
- { path: "{{ atlas_prometheus_pull_private_key_path }}.pub", mode: "0644" }
|
||||
loop_control:
|
||||
label: "{{ item.path }}"
|
||||
when:
|
||||
- atlas_manage_prometheus_backup_pull | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Pin Prometheus SSH host key on Atlas
|
||||
tags: [atlas, backup, prometheus_backup, prometheus_backup_key]
|
||||
ansible.builtin.copy:
|
||||
content: "{{ atlas_prometheus_ssh_host_key }}\n"
|
||||
dest: "{{ atlas_prometheus_pull_known_hosts_path }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0600"
|
||||
when: atlas_manage_prometheus_backup_pull | bool
|
||||
90
ansible/roles/profile_atlas/tasks/prometheus_pull_job.yml
Normal file
90
ansible/roles/profile_atlas/tasks/prometheus_pull_job.yml
Normal file
@@ -0,0 +1,90 @@
|
||||
---
|
||||
- name: Validate Atlas Prometheus backup pull inputs
|
||||
tags: [atlas, backup, prometheus_backup]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_manage_storage | bool
|
||||
- atlas_prometheus_pull_source_user is match('^[a-z_][a-z0-9_-]*$')
|
||||
- atlas_prometheus_pull_source_port | int > 0
|
||||
- atlas_prometheus_pull_source_port | int < 65536
|
||||
- atlas_prometheus_pull_keep_daily | int > 0
|
||||
- atlas_prometheus_pull_keep_weekly | int > 0
|
||||
- atlas_prometheus_pull_keep_monthly | int > 0
|
||||
- atlas_prometheus_pull_max_age_hours | int > 0
|
||||
- atlas_backup_prometheus_mountpoint.startswith(atlas_mount_root ~ '/')
|
||||
fail_msg: Define the Atlas backup destination, source account, and retention before enabling the pull.
|
||||
when: atlas_manage_prometheus_backup_pull | bool
|
||||
|
||||
- name: Validate Atlas Prometheus backup pull calendar
|
||||
tags: [atlas, backup, prometheus_backup]
|
||||
ansible.builtin.command:
|
||||
argv: [systemd-analyze, calendar, "{{ atlas_prometheus_pull_calendar }}"]
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
when: atlas_manage_prometheus_backup_pull | bool
|
||||
|
||||
- name: Create private Atlas Prometheus backup version directory
|
||||
tags: [atlas, backup, prometheus_backup]
|
||||
ansible.builtin.file:
|
||||
path: "{{ atlas_backup_prometheus_mountpoint }}/snapshots"
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0700"
|
||||
when: atlas_manage_prometheus_backup_pull | bool
|
||||
|
||||
- name: Install Atlas Prometheus backup pull helper
|
||||
tags: [atlas, backup, prometheus_backup]
|
||||
ansible.builtin.template:
|
||||
src: atlas-prometheus-pull.sh.j2
|
||||
dest: /usr/local/sbin/atlas-prometheus-pull
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0750"
|
||||
when: atlas_manage_prometheus_backup_pull | bool
|
||||
|
||||
- name: Install Atlas Prometheus backup retention helper
|
||||
tags: [atlas, backup, prometheus_backup]
|
||||
ansible.builtin.copy:
|
||||
src: atlas-prometheus-prune.py
|
||||
dest: /usr/local/libexec/atlas-prometheus-prune
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0750"
|
||||
when: atlas_manage_prometheus_backup_pull | bool
|
||||
|
||||
- name: Install Atlas Prometheus backup pull systemd units
|
||||
tags: [atlas, backup, prometheus_backup]
|
||||
ansible.builtin.template:
|
||||
src: "{{ item }}.j2"
|
||||
dest: "/etc/systemd/system/{{ item }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
loop:
|
||||
- atlas-prometheus-pull.service
|
||||
- atlas-prometheus-pull.timer
|
||||
loop_control:
|
||||
label: "{{ item }}"
|
||||
register: atlas_prometheus_pull_units
|
||||
when: atlas_manage_prometheus_backup_pull | bool
|
||||
|
||||
- name: Reload systemd after Atlas Prometheus pull unit changes
|
||||
tags: [atlas, backup, prometheus_backup]
|
||||
ansible.builtin.systemd:
|
||||
daemon_reload: true
|
||||
when:
|
||||
- atlas_manage_prometheus_backup_pull | bool
|
||||
- atlas_prometheus_pull_units is changed
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Enable Atlas Prometheus pull timer only after explicit activation
|
||||
tags: [atlas, backup, prometheus_backup]
|
||||
ansible.builtin.systemd:
|
||||
name: atlas-prometheus-pull.timer
|
||||
enabled: true
|
||||
state: started
|
||||
when:
|
||||
- atlas_manage_prometheus_backup_pull | bool
|
||||
- atlas_prometheus_pull_start_timer | bool
|
||||
- not ansible_check_mode
|
||||
27
ansible/roles/profile_atlas/tasks/restorecon.yml
Normal file
27
ansible/roles/profile_atlas/tasks/restorecon.yml
Normal file
@@ -0,0 +1,27 @@
|
||||
---
|
||||
- name: Validate requested Atlas post-restore relabel paths
|
||||
tags: [atlas, restorecon, recovery]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- item is string
|
||||
- item.startswith(atlas_mount_root ~ '/')
|
||||
- item != atlas_mount_root
|
||||
fail_msg: >-
|
||||
Post-restore relabeling accepts only explicit paths below the Atlas pool
|
||||
mount root. Do not relabel the whole pool during routine provisioning.
|
||||
loop: "{{ atlas_restorecon_paths }}"
|
||||
when: atlas_restorecon_paths | length > 0
|
||||
|
||||
- name: Restore SELinux labels on explicitly restored Atlas paths
|
||||
tags: [atlas, restorecon, recovery]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- restorecon
|
||||
- -RFv
|
||||
- "{{ item }}"
|
||||
register: atlas_restorecon_result
|
||||
changed_when: atlas_restorecon_result.stdout | length > 0
|
||||
loop: "{{ atlas_restorecon_paths }}"
|
||||
when:
|
||||
- atlas_restorecon_paths | length > 0
|
||||
- not ansible_check_mode
|
||||
@@ -3,12 +3,6 @@
|
||||
tags: [atlas, storage]
|
||||
ansible.builtin.set_fact:
|
||||
atlas_zfs_datasets:
|
||||
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_work }}"
|
||||
mountpoint: "{{ atlas_work_mountpoint }}"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
properties:
|
||||
mountpoint: "{{ atlas_work_mountpoint }}"
|
||||
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_archive }}"
|
||||
mountpoint: "{{ atlas_archive_mountpoint }}"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
@@ -16,6 +10,13 @@
|
||||
properties:
|
||||
compression: zstd
|
||||
mountpoint: "{{ atlas_archive_mountpoint }}"
|
||||
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_services }}"
|
||||
mountpoint: "{{ atlas_services_mountpoint }}"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
properties:
|
||||
compression: zstd
|
||||
mountpoint: "{{ atlas_services_mountpoint }}"
|
||||
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_app_data }}"
|
||||
mountpoint: "{{ atlas_app_data_mountpoint }}"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
@@ -51,11 +52,27 @@
|
||||
properties:
|
||||
compression: lz4
|
||||
mountpoint: "{{ atlas_music_mountpoint }}"
|
||||
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_backup }}"
|
||||
mountpoint: "{{ atlas_backup_mountpoint }}"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
properties:
|
||||
compression: lz4
|
||||
mountpoint: "{{ atlas_backup_mountpoint }}"
|
||||
reservation: "{{ atlas_zfs_backup_reservation }}"
|
||||
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_host_backups }}"
|
||||
mountpoint: "{{ atlas_host_backups_mountpoint }}"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
properties:
|
||||
compression: lz4
|
||||
mountpoint: "{{ atlas_host_backups_mountpoint }}"
|
||||
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_backup_prometheus }}"
|
||||
mountpoint: "{{ atlas_backup_prometheus_mountpoint }}"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
properties:
|
||||
compression: lz4
|
||||
mountpoint: "{{ atlas_backup_prometheus_mountpoint }}"
|
||||
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_photobook }}"
|
||||
mountpoint: "{{ atlas_photobook_mountpoint }}"
|
||||
@@ -64,21 +81,6 @@
|
||||
properties:
|
||||
compression: lz4
|
||||
mountpoint: "{{ atlas_photobook_mountpoint }}"
|
||||
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_backups }}"
|
||||
mountpoint: "{{ atlas_backups_mountpoint }}"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
properties:
|
||||
compression: lz4
|
||||
mountpoint: "{{ atlas_backups_mountpoint }}"
|
||||
- name: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_service_backups }}"
|
||||
mountpoint: "{{ atlas_service_backups_mountpoint }}"
|
||||
owner: "{{ atlas_admin_username }}"
|
||||
group: "{{ atlas_admin_group }}"
|
||||
properties:
|
||||
compression: lz4
|
||||
mountpoint: "{{ atlas_service_backups_mountpoint }}"
|
||||
refreservation: "{{ atlas_zfs_service_backups_refreservation }}"
|
||||
|
||||
- name: Require completed Atlas storage placeholders
|
||||
tags: [atlas, storage]
|
||||
|
||||
145
ansible/roles/profile_atlas/tasks/usb_backup.yml
Normal file
145
ansible/roles/profile_atlas/tasks/usb_backup.yml
Normal file
@@ -0,0 +1,145 @@
|
||||
---
|
||||
- name: Validate Atlas offline USB backup configuration
|
||||
tags: [atlas, storage, backup, usb_backup]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_manage_storage | bool
|
||||
- atlas_zfs_pool != 'CHANGEME_ZFS_POOL'
|
||||
- atlas_mount_root.startswith('/')
|
||||
- atlas_usb_backup_luks_uuid is match('^[0-9a-fA-F]{8}(-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12}$')
|
||||
- atlas_usb_backup_fs_uuid is match('^[0-9a-fA-F]{8}(-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12}$')
|
||||
- atlas_usb_backup_luks_uuid != atlas_usb_backup_fs_uuid
|
||||
- atlas_usb_backup_mapper_name is match('^[a-z][a-z0-9_-]*$')
|
||||
- atlas_usb_backup_min_free_bytes | int > 0
|
||||
- atlas_usb_backup_snapshot_prefix is match('^[a-z0-9][a-z0-9_-]*$')
|
||||
- atlas_usb_backup_snapshot_prefix != atlas_borg_snapshot_prefix
|
||||
- atlas_usb_backup_snapshot_prefix != atlas_zfs_snapshot_prefix
|
||||
fail_msg: >-
|
||||
The manual Atlas USB backup needs verified LUKS and ext4 UUIDs, a safe
|
||||
mapper name, positive free-space reserve, and a unique snapshot prefix.
|
||||
when: atlas_manage_usb_backup | bool
|
||||
|
||||
- name: Install rsync for the Atlas offline USB backup
|
||||
tags: [atlas, storage, backup, usb_backup]
|
||||
ansible.builtin.dnf:
|
||||
name: rsync
|
||||
state: present
|
||||
when: atlas_manage_usb_backup | bool
|
||||
|
||||
- name: Install the manual Atlas offline USB backup helper
|
||||
tags: [atlas, storage, backup, usb_backup]
|
||||
ansible.builtin.template:
|
||||
src: atlas-usb-backup.sh.j2
|
||||
dest: /usr/local/sbin/atlas-usb-backup
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0750"
|
||||
when: atlas_manage_usb_backup | bool
|
||||
|
||||
- name: Install the Atlas USB snapshot cleanup helper
|
||||
tags: [atlas, storage, backup, usb_backup]
|
||||
ansible.builtin.template:
|
||||
src: atlas-usb-snapshot-cleanup.sh.j2
|
||||
dest: /usr/local/sbin/atlas-usb-snapshot-cleanup
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0750"
|
||||
when: atlas_manage_usb_backup | bool
|
||||
|
||||
- name: Install the manual Atlas offline USB backup service
|
||||
tags: [atlas, storage, backup, usb_backup]
|
||||
ansible.builtin.template:
|
||||
src: atlas-usb-backup.service.j2
|
||||
dest: /etc/systemd/system/atlas-usb-backup.service
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
when: atlas_manage_usb_backup | bool
|
||||
|
||||
- name: Reload systemd for the Atlas offline USB backup service
|
||||
tags: [atlas, storage, backup, usb_backup]
|
||||
ansible.builtin.systemd:
|
||||
daemon_reload: true
|
||||
when:
|
||||
- atlas_manage_usb_backup | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Validate the 45Drives Atlas USB reminder configuration
|
||||
tags: [atlas, backup, usb_reminder]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_manage_usb_backup | bool
|
||||
- atlas_usb_reminder_calendar | length > 0
|
||||
- atlas_usb_reminder_notifier.startswith('/opt/45drives/houston/')
|
||||
fail_msg: >-
|
||||
Enable the manual USB backup and declare a systemd calendar before
|
||||
enabling its 45Drives Alerts reminder.
|
||||
when: atlas_manage_usb_reminder | bool
|
||||
|
||||
- name: Validate the Atlas USB reminder calendar
|
||||
tags: [atlas, backup, usb_reminder]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- systemd-analyze
|
||||
- calendar
|
||||
- "{{ atlas_usb_reminder_calendar }}"
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
when: atlas_manage_usb_reminder | bool
|
||||
|
||||
- name: Inspect the existing 45Drives notifier
|
||||
tags: [atlas, backup, usb_reminder]
|
||||
ansible.builtin.stat:
|
||||
path: "{{ atlas_usb_reminder_notifier }}"
|
||||
register: atlas_usb_reminder_notifier_file
|
||||
when: atlas_manage_usb_reminder | bool
|
||||
|
||||
- name: Require the configured 45Drives notifier for USB reminders
|
||||
tags: [atlas, backup, usb_reminder]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_usb_reminder_notifier_file.stat.executable | default(false)
|
||||
fail_msg: >-
|
||||
The existing 45Drives Houston notifier must be executable.
|
||||
when: atlas_manage_usb_reminder | bool
|
||||
|
||||
- name: Install the 45Drives Atlas USB reminder helper
|
||||
tags: [atlas, backup, usb_reminder]
|
||||
ansible.builtin.template:
|
||||
src: atlas-usb-reminder.py.j2
|
||||
dest: /usr/local/libexec/atlas-usb-reminder
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0750"
|
||||
when: atlas_manage_usb_reminder | bool
|
||||
|
||||
- name: Install the 45Drives Atlas USB reminder service
|
||||
tags: [atlas, backup, usb_reminder]
|
||||
ansible.builtin.template:
|
||||
src: atlas-usb-reminder.service.j2
|
||||
dest: /etc/systemd/system/atlas-usb-reminder.service
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
when: atlas_manage_usb_reminder | bool
|
||||
|
||||
- name: Install the 45Drives Atlas USB reminder timer
|
||||
tags: [atlas, backup, usb_reminder]
|
||||
ansible.builtin.template:
|
||||
src: atlas-usb-reminder.timer.j2
|
||||
dest: /etc/systemd/system/atlas-usb-reminder.timer
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
when: atlas_manage_usb_reminder | bool
|
||||
|
||||
- name: Enable only the Atlas USB notification reminder timer
|
||||
tags: [atlas, backup, usb_reminder]
|
||||
ansible.builtin.systemd:
|
||||
name: atlas-usb-reminder.timer
|
||||
enabled: true
|
||||
state: started
|
||||
daemon_reload: true
|
||||
when:
|
||||
- atlas_manage_usb_reminder | bool
|
||||
- not ansible_check_mode
|
||||
175
ansible/roles/profile_atlas/tasks/zfs_maintenance.yml
Normal file
175
ansible/roles/profile_atlas/tasks/zfs_maintenance.yml
Normal file
@@ -0,0 +1,175 @@
|
||||
---
|
||||
- name: Validate Atlas ZFS snapshot policy
|
||||
tags: [atlas, storage, snapshots]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_manage_storage | bool
|
||||
- atlas_zfs_pool != 'CHANGEME_ZFS_POOL'
|
||||
- atlas_zfs_snapshot_prefix is match('^[a-z0-9][a-z0-9_-]*$')
|
||||
- atlas_zfs_snapshot_policies | length > 0
|
||||
- >-
|
||||
(atlas_zfs_snapshot_policies | map(attribute='name') | unique | list | length)
|
||||
== (atlas_zfs_snapshot_policies | length)
|
||||
fail_msg: >-
|
||||
Enable Atlas storage and declare a non-empty snapshot policy with a safe
|
||||
prefix and unique policy names before managing automatic snapshots.
|
||||
when: atlas_manage_zfs_snapshots | bool
|
||||
|
||||
- name: Validate Atlas ZFS snapshot policy entries
|
||||
tags: [atlas, storage, snapshots]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- item.name is match('^[a-z][a-z0-9_-]*$')
|
||||
- item.keep | int > 0
|
||||
- item.calendar | length > 0
|
||||
fail_msg: >-
|
||||
Every Atlas snapshot policy needs a safe name, a positive retention
|
||||
count, and a systemd calendar expression.
|
||||
loop: "{{ atlas_zfs_snapshot_policies }}"
|
||||
loop_control:
|
||||
label: "{{ item.name | default('unnamed') }}"
|
||||
when: atlas_manage_zfs_snapshots | bool
|
||||
|
||||
- name: Validate Atlas ZFS snapshot calendars
|
||||
tags: [atlas, storage, snapshots]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- systemd-analyze
|
||||
- calendar
|
||||
- "{{ item.calendar }}"
|
||||
loop: "{{ atlas_zfs_snapshot_policies }}"
|
||||
loop_control:
|
||||
label: "{{ item.name }}: {{ item.calendar }}"
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
when: atlas_manage_zfs_snapshots | bool
|
||||
|
||||
- name: Install Atlas ZFS snapshot and retention helper
|
||||
tags: [atlas, storage, snapshots]
|
||||
ansible.builtin.template:
|
||||
src: atlas-zfs-snapshot.sh.j2
|
||||
dest: /usr/local/sbin/atlas-zfs-snapshot
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0750"
|
||||
when: atlas_manage_zfs_snapshots | bool
|
||||
|
||||
- name: Install Atlas ZFS snapshot systemd service
|
||||
tags: [atlas, storage, snapshots]
|
||||
ansible.builtin.template:
|
||||
src: atlas-zfs-snapshot@.service.j2
|
||||
dest: /etc/systemd/system/atlas-zfs-snapshot@.service
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
when: atlas_manage_zfs_snapshots | bool
|
||||
|
||||
- name: Install Atlas ZFS snapshot systemd timers
|
||||
tags: [atlas, storage, snapshots]
|
||||
ansible.builtin.template:
|
||||
src: atlas-zfs-snapshot.timer.j2
|
||||
dest: "/etc/systemd/system/atlas-zfs-snapshot-{{ item.name }}.timer"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
loop: "{{ atlas_zfs_snapshot_policies }}"
|
||||
loop_control:
|
||||
label: "{{ item.name }}"
|
||||
when: atlas_manage_zfs_snapshots | bool
|
||||
|
||||
- name: Enable Atlas ZFS snapshot systemd timers
|
||||
tags: [atlas, storage, snapshots]
|
||||
ansible.builtin.systemd:
|
||||
name: "atlas-zfs-snapshot-{{ item.name }}.timer"
|
||||
enabled: true
|
||||
state: started
|
||||
daemon_reload: true
|
||||
loop: "{{ atlas_zfs_snapshot_policies }}"
|
||||
loop_control:
|
||||
label: "{{ item.name }}"
|
||||
when:
|
||||
- atlas_manage_zfs_snapshots | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Validate Atlas ZFS scrub policy
|
||||
tags: [atlas, storage, scrub]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- atlas_manage_storage | bool
|
||||
- atlas_zfs_pool != 'CHANGEME_ZFS_POOL'
|
||||
- atlas_zfs_scrub_calendar | length > 0
|
||||
fail_msg: >-
|
||||
Enable Atlas storage and declare a systemd calendar expression before
|
||||
managing periodic ZFS scrubs.
|
||||
when: atlas_manage_zfs_scrub | bool
|
||||
|
||||
- name: Validate Atlas ZFS scrub calendar
|
||||
tags: [atlas, storage, scrub]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- systemd-analyze
|
||||
- calendar
|
||||
- "{{ atlas_zfs_scrub_calendar }}"
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
when: atlas_manage_zfs_scrub | bool
|
||||
|
||||
- name: Require OpenZFS scrub systemd units
|
||||
tags: [atlas, storage, scrub]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- systemctl
|
||||
- cat
|
||||
- "{{ item }}"
|
||||
loop:
|
||||
- "zfs-scrub@{{ atlas_zfs_pool }}.service"
|
||||
- "zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer"
|
||||
- "zfs-scrub-weekly@{{ atlas_zfs_pool }}.timer"
|
||||
loop_control:
|
||||
label: "{{ item }}"
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
when: atlas_manage_zfs_scrub | bool
|
||||
|
||||
- name: Create Atlas ZFS scrub timer override directory
|
||||
tags: [atlas, storage, scrub]
|
||||
ansible.builtin.file:
|
||||
path: "/etc/systemd/system/zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer.d"
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
when: atlas_manage_zfs_scrub | bool
|
||||
|
||||
- name: Configure Atlas ZFS monthly scrub schedule
|
||||
tags: [atlas, storage, scrub]
|
||||
ansible.builtin.template:
|
||||
src: atlas-zfs-scrub-timer.conf.j2
|
||||
dest: >-
|
||||
/etc/systemd/system/zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer.d/override.conf
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
when: atlas_manage_zfs_scrub | bool
|
||||
|
||||
- name: Disable the conflicting weekly OpenZFS scrub timer
|
||||
tags: [atlas, storage, scrub]
|
||||
ansible.builtin.systemd:
|
||||
name: "zfs-scrub-weekly@{{ atlas_zfs_pool }}.timer"
|
||||
enabled: false
|
||||
state: stopped
|
||||
daemon_reload: true
|
||||
when:
|
||||
- atlas_manage_zfs_scrub | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Enable the Atlas monthly OpenZFS scrub timer
|
||||
tags: [atlas, storage, scrub]
|
||||
ansible.builtin.systemd:
|
||||
name: "zfs-scrub-monthly@{{ atlas_zfs_pool }}.timer"
|
||||
enabled: true
|
||||
state: started
|
||||
daemon_reload: true
|
||||
when:
|
||||
- atlas_manage_zfs_scrub | bool
|
||||
- not ansible_check_mode
|
||||
@@ -0,0 +1,41 @@
|
||||
[Unit]
|
||||
Description=Back up Atlas ZFS datasets to the encrypted Borg repository
|
||||
Documentation=man:borg-create(1) man:borg-prune(1) man:borg-compact(1)
|
||||
Requires=zfs.target
|
||||
Wants=network-online.target
|
||||
After=zfs.target network-online.target
|
||||
StartLimitIntervalSec=6h
|
||||
StartLimitBurst=3
|
||||
ConditionFileIsExecutable=/usr/local/sbin/atlas-borg-backup
|
||||
ConditionPathExists={{ atlas_borg_passphrase_path }}
|
||||
ConditionPathExists={{ atlas_borg_ssh_private_key_path }}
|
||||
ConditionPathExists={{ atlas_borg_known_hosts_path }}
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/atlas-borg-backup
|
||||
ExecStopPost=+/usr/local/sbin/atlas-borg-snapshot-cleanup
|
||||
User=root
|
||||
Group=root
|
||||
UMask=0077
|
||||
SuccessExitStatus=1
|
||||
Restart=on-failure
|
||||
RestartSec=30m
|
||||
TimeoutStartSec=infinity
|
||||
RuntimeDirectory=atlas-borg
|
||||
RuntimeDirectoryMode=0750
|
||||
Nice=15
|
||||
IOSchedulingClass=best-effort
|
||||
IOSchedulingPriority=7
|
||||
NoNewPrivileges=true
|
||||
PrivateMounts=true
|
||||
PrivateTmp=true
|
||||
ProtectHome=true
|
||||
ProtectSystem=strict
|
||||
ReadWritePaths={{ atlas_borg_cache_dir }} {{ atlas_borg_config_dir }} /run/atlas-borg /run/lock
|
||||
ProtectKernelTunables=true
|
||||
ProtectKernelModules=true
|
||||
ProtectControlGroups=true
|
||||
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
|
||||
RestrictRealtime=true
|
||||
LockPersonality=true
|
||||
215
ansible/roles/profile_atlas/templates/atlas-borg-backup.sh.j2
Normal file
215
ansible/roles/profile_atlas/templates/atlas-borg-backup.sh.j2
Normal file
@@ -0,0 +1,215 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
export LC_ALL=C.utf8
|
||||
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
|
||||
export BORG_CACHE_DIR={{ atlas_borg_cache_dir | quote }}
|
||||
export BORG_CONFIG_DIR={{ atlas_borg_config_dir | quote }}
|
||||
export BORG_PASSCOMMAND={{ ('cat ' ~ atlas_borg_passphrase_path) | quote }}
|
||||
export BORG_RSH={{ atlas_borg_ssh_wrapper_path | quote }}
|
||||
|
||||
readonly pool={{ atlas_zfs_pool | quote }}
|
||||
readonly mount_root={{ atlas_mount_root | quote }}
|
||||
readonly repository={{ ('ssh://' ~ atlas_borg_repository_user ~ '@' ~ atlas_borg_repository_host
|
||||
~ ':' ~ (atlas_borg_repository_port | string) ~ '/' ~ atlas_borg_repository_path) | quote }}
|
||||
readonly remote_path={{ atlas_borg_remote_path | quote }}
|
||||
readonly archive_prefix={{ atlas_borg_archive_prefix | quote }}
|
||||
readonly snapshot_prefix={{ atlas_borg_snapshot_prefix | quote }}
|
||||
readonly compression={{ atlas_borg_compression | quote }}
|
||||
readonly stage=/run/atlas-borg/source
|
||||
readonly snapshot_marker=/run/atlas-borg/snapshot-name
|
||||
readonly borg_user={{ atlas_borg_username | quote }}
|
||||
readonly borg_group={{ atlas_borg_group | quote }}
|
||||
readonly borg_home={{ atlas_borg_home | quote }}
|
||||
readonly borg_lock={{ atlas_borg_lock_path | quote }}
|
||||
readonly progress_filter=/usr/local/libexec/atlas-borg-progress
|
||||
|
||||
snapshot_name=""
|
||||
mounted_targets=()
|
||||
|
||||
# Invoked through the EXIT trap below.
|
||||
# shellcheck disable=SC2329
|
||||
cleanup() {
|
||||
local status=$?
|
||||
local cleanup_status=0
|
||||
local index
|
||||
local source_mount_failed=false
|
||||
trap - EXIT HUP INT TERM
|
||||
set +e
|
||||
|
||||
{% raw %}
|
||||
for ((index = ${#mounted_targets[@]} - 1; index >= 0; index--)); do
|
||||
{% endraw %}
|
||||
if mountpoint -q "${mounted_targets[$index]}" && ! umount -R "${mounted_targets[$index]}"; then
|
||||
printf 'Source snapshot mount cleanup failed: %s\n' "${mounted_targets[$index]}" >&2
|
||||
source_mount_failed=true
|
||||
fi
|
||||
if mountpoint -q "${mounted_targets[$index]}"; then
|
||||
printf 'Source snapshot mount is still active: %s\n' "${mounted_targets[$index]}" >&2
|
||||
source_mount_failed=true
|
||||
else
|
||||
rmdir -- "${mounted_targets[$index]}" 2>/dev/null || true
|
||||
fi
|
||||
done
|
||||
if [[ "$source_mount_failed" == false ]]; then
|
||||
if [[ -d "$stage" ]]; then
|
||||
rmdir -- "$stage" 2>/dev/null || cleanup_status=2
|
||||
fi
|
||||
else
|
||||
cleanup_status=2
|
||||
printf 'Source bind mount cleanup failed; keeping the snapshot for recovery\n' >&2
|
||||
fi
|
||||
|
||||
if ((status == 0 && cleanup_status != 0)); then
|
||||
status=$cleanup_status
|
||||
fi
|
||||
exit "$status"
|
||||
}
|
||||
|
||||
trap cleanup EXIT
|
||||
trap 'exit 143' HUP INT TERM
|
||||
|
||||
run_as_borg() {
|
||||
setpriv \
|
||||
--reuid "$borg_user" \
|
||||
--regid "$borg_group" \
|
||||
--clear-groups \
|
||||
--inh-caps=-all,+dac_read_search \
|
||||
--ambient-caps=-all,+dac_read_search \
|
||||
--bounding-set=-all,+dac_read_search \
|
||||
-- env HOME="$borg_home" USER="$borg_user" LOGNAME="$borg_user" "$@"
|
||||
}
|
||||
|
||||
exec 8>"$borg_lock"
|
||||
flock 8
|
||||
exec 9>/run/lock/atlas-zfs-snapshot.lock
|
||||
|
||||
zpool list -H -o name "$pool" >/dev/null
|
||||
rm -rf "$stage"
|
||||
mkdir -p "$stage"
|
||||
# Keep systemd's root:root ownership of RuntimeDirectory: changing it makes
|
||||
# ExecStopPost re-chown its contents, which SELinux denies for the marker.
|
||||
setfacl -m "u:${borg_user}:rx" /run/atlas-borg
|
||||
chown root:"$borg_group" "$stage"
|
||||
chmod 0750 /run/atlas-borg "$stage"
|
||||
|
||||
flock 9
|
||||
while IFS= read -r stale_snapshot; do
|
||||
stale_suffix="${stale_snapshot#"${pool}@${snapshot_prefix}-"}"
|
||||
if [[ "$stale_suffix" =~ ^[0-9]{8}T[0-9]{6}Z$ ]]; then
|
||||
zfs destroy -r "$stale_snapshot"
|
||||
printf 'Removed stale Borg source snapshot %s\n' "$stale_snapshot"
|
||||
fi
|
||||
done < <(
|
||||
zfs list -H -t snapshot -o name -r "$pool" |
|
||||
grep -E "^${pool}@${snapshot_prefix}-[0-9]{8}T[0-9]{6}Z$" || true
|
||||
)
|
||||
|
||||
timestamp="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
readonly timestamp
|
||||
snapshot_name="${snapshot_prefix}-${timestamp}"
|
||||
readonly snapshot_name
|
||||
printf '%s\n' "$snapshot_name" >"$snapshot_marker"
|
||||
zfs snapshot -r "${pool}@${snapshot_name}"
|
||||
flock -u 9
|
||||
printf 'Created recursive Borg source snapshot %s@%s\n' "$pool" "$snapshot_name"
|
||||
|
||||
while IFS=$'\t' read -r dataset dataset_mountpoint mounted; do
|
||||
if [[ "$mounted" != yes ]]; then
|
||||
printf 'Dataset %s is not mounted; refusing an incomplete backup\n' "$dataset" >&2
|
||||
exit 65
|
||||
fi
|
||||
if [[ "$dataset_mountpoint" != "$mount_root" && "$dataset_mountpoint" != "$mount_root/"* ]]; then
|
||||
printf 'Dataset %s has unexpected mountpoint %s\n' "$dataset" "$dataset_mountpoint" >&2
|
||||
exit 65
|
||||
fi
|
||||
|
||||
dataset_suffix="${dataset#"$pool"}"
|
||||
source_path="${dataset_mountpoint}/.zfs/snapshot/${snapshot_name}"
|
||||
target_path="${stage}${dataset_suffix}"
|
||||
mkdir -p "$target_path"
|
||||
mount --bind "$source_path" "$target_path"
|
||||
mounted_targets+=("$target_path")
|
||||
mount -o remount,bind,ro "$target_path"
|
||||
done < <(zfs list -H -o name,mountpoint,mounted -s name -r "$pool")
|
||||
|
||||
estimated_source_bytes=0
|
||||
while IFS=$'\t' read -r source_snapshot logical_bytes; do
|
||||
if [[ "$source_snapshot" == *"@${snapshot_name}" ]]; then
|
||||
[[ "$logical_bytes" =~ ^[0-9]+$ ]] || {
|
||||
printf 'Invalid logical size for Borg source snapshot %s\n' "$source_snapshot" >&2
|
||||
exit 74
|
||||
}
|
||||
estimated_source_bytes=$((estimated_source_bytes + logical_bytes))
|
||||
fi
|
||||
done < <(zfs list -H -p -t snapshot -o name,logicalreferenced -r "$pool")
|
||||
((estimated_source_bytes > 0)) || {
|
||||
printf 'Could not estimate the Borg source snapshot size\n' >&2
|
||||
exit 74
|
||||
}
|
||||
printf 'Estimated Borg source logical size: %s bytes (ZFS; progress percentage is approximate)\n' \
|
||||
"$estimated_source_bytes"
|
||||
|
||||
archive="${archive_prefix}-${timestamp}"
|
||||
readonly archive
|
||||
borg_status=0
|
||||
|
||||
printf 'Starting Borg archive %s from snapshot %s@%s\n' "$archive" "$pool" "$snapshot_name"
|
||||
set +e
|
||||
(
|
||||
cd /run/atlas-borg
|
||||
run_as_borg borg --remote-path "$remote_path" --lock-wait 600 --log-json --progress create \
|
||||
--show-rc \
|
||||
--stats \
|
||||
--checkpoint-interval 900 \
|
||||
--compression "$compression" \
|
||||
"${repository}::${archive}" \
|
||||
source 2>&1
|
||||
) | /usr/bin/python3 -u "$progress_filter" --estimated-total-bytes "$estimated_source_bytes"
|
||||
create_pipeline_status=("${PIPESTATUS[@]}")
|
||||
set -e
|
||||
create_status=${create_pipeline_status[0]}
|
||||
if ((create_pipeline_status[1] != 0)); then
|
||||
printf 'Borg progress logging failed with status %s\n' "${create_pipeline_status[1]}" >&2
|
||||
exit 2
|
||||
fi
|
||||
if ((create_status >= 2)); then
|
||||
exit "$create_status"
|
||||
fi
|
||||
borg_status=$create_status
|
||||
|
||||
printf 'Borg archive %s created; applying retention\n' "$archive"
|
||||
set +e
|
||||
run_as_borg borg --remote-path "$remote_path" --lock-wait 600 prune \
|
||||
--show-rc \
|
||||
--list \
|
||||
--glob-archives "${archive_prefix}-*" \
|
||||
--keep-daily {{ atlas_borg_keep_daily | int }} \
|
||||
--keep-weekly {{ atlas_borg_keep_weekly | int }} \
|
||||
--keep-monthly {{ atlas_borg_keep_monthly | int }} \
|
||||
"$repository"
|
||||
prune_status=$?
|
||||
set -e
|
||||
if ((prune_status >= 2)); then
|
||||
exit "$prune_status"
|
||||
fi
|
||||
if ((prune_status > borg_status)); then
|
||||
borg_status=$prune_status
|
||||
fi
|
||||
|
||||
printf 'Borg retention complete; compacting repository\n'
|
||||
set +e
|
||||
run_as_borg borg --remote-path "$remote_path" --lock-wait 600 compact \
|
||||
--show-rc \
|
||||
"$repository"
|
||||
compact_status=$?
|
||||
set -e
|
||||
if ((compact_status >= 2)); then
|
||||
exit "$compact_status"
|
||||
fi
|
||||
if ((compact_status > borg_status)); then
|
||||
borg_status=$compact_status
|
||||
fi
|
||||
|
||||
printf 'Borg backup %s completed with status %s\n' "$archive" "$borg_status"
|
||||
exit "$borg_status"
|
||||
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Schedule the encrypted Atlas Borg backup
|
||||
|
||||
[Timer]
|
||||
OnCalendar={{ atlas_borg_backup_calendar }}
|
||||
Persistent=true
|
||||
RandomizedDelaySec={{ atlas_borg_randomized_delay }}
|
||||
AccuracySec=1min
|
||||
Unit=atlas-borg-backup.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,32 @@
|
||||
[Unit]
|
||||
Description=Check the encrypted Atlas Borg repository
|
||||
Documentation=man:borg-check(1)
|
||||
Wants=network-online.target
|
||||
After=network-online.target atlas-borg-backup.service
|
||||
ConditionFileIsExecutable=/usr/local/sbin/atlas-borg-check
|
||||
ConditionPathExists={{ atlas_borg_passphrase_path }}
|
||||
ConditionPathExists={{ atlas_borg_ssh_private_key_path }}
|
||||
ConditionPathExists={{ atlas_borg_known_hosts_path }}
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/atlas-borg-check
|
||||
User={{ atlas_borg_username }}
|
||||
Group={{ atlas_borg_group }}
|
||||
UMask=0077
|
||||
SuccessExitStatus=1
|
||||
TimeoutStartSec=infinity
|
||||
Nice=15
|
||||
IOSchedulingClass=best-effort
|
||||
IOSchedulingPriority=7
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectHome=true
|
||||
ProtectSystem=strict
|
||||
ReadWritePaths={{ atlas_borg_cache_dir }} {{ atlas_borg_config_dir }}
|
||||
ProtectKernelTunables=true
|
||||
ProtectKernelModules=true
|
||||
ProtectControlGroups=true
|
||||
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
|
||||
RestrictRealtime=true
|
||||
LockPersonality=true
|
||||
23
ansible/roles/profile_atlas/templates/atlas-borg-check.sh.j2
Normal file
23
ansible/roles/profile_atlas/templates/atlas-borg-check.sh.j2
Normal file
@@ -0,0 +1,23 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
export LC_ALL=C
|
||||
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
|
||||
export BORG_CACHE_DIR={{ atlas_borg_cache_dir | quote }}
|
||||
export BORG_CONFIG_DIR={{ atlas_borg_config_dir | quote }}
|
||||
export BORG_PASSCOMMAND={{ ('cat ' ~ atlas_borg_passphrase_path) | quote }}
|
||||
export BORG_RSH={{ atlas_borg_ssh_wrapper_path | quote }}
|
||||
|
||||
readonly repository={{ ('ssh://' ~ atlas_borg_repository_user ~ '@' ~ atlas_borg_repository_host
|
||||
~ ':' ~ (atlas_borg_repository_port | string) ~ '/' ~ atlas_borg_repository_path) | quote }}
|
||||
readonly remote_path={{ atlas_borg_remote_path | quote }}
|
||||
readonly archive_prefix={{ atlas_borg_archive_prefix | quote }}
|
||||
readonly borg_lock={{ atlas_borg_lock_path | quote }}
|
||||
|
||||
exec 8>"$borg_lock"
|
||||
flock 8
|
||||
|
||||
exec borg --remote-path "$remote_path" --lock-wait 600 check \
|
||||
--show-rc \
|
||||
--glob-archives "${archive_prefix}-*" \
|
||||
"$repository"
|
||||
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Schedule checks of the encrypted Atlas Borg repository
|
||||
|
||||
[Timer]
|
||||
OnCalendar={{ atlas_borg_check_calendar }}
|
||||
Persistent=true
|
||||
RandomizedDelaySec={{ atlas_borg_randomized_delay }}
|
||||
AccuracySec=1min
|
||||
Unit=atlas-borg-check.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,51 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
|
||||
|
||||
readonly pool={{ atlas_zfs_pool | quote }}
|
||||
readonly mount_root={{ atlas_mount_root | quote }}
|
||||
readonly snapshot_prefix={{ atlas_borg_snapshot_prefix | quote }}
|
||||
readonly marker=/run/atlas-borg/snapshot-name
|
||||
|
||||
[[ -e "$marker" ]] || exit 0
|
||||
[[ -f "$marker" && ! -L "$marker" ]] || {
|
||||
printf 'Unsafe Atlas Borg snapshot marker; leaving snapshots unchanged\n' >&2
|
||||
exit 2
|
||||
}
|
||||
IFS= read -r snapshot_name <"$marker"
|
||||
[[ "$snapshot_name" =~ ^${snapshot_prefix}-[0-9]{8}T[0-9]{6}Z$ ]] || {
|
||||
printf 'Invalid Atlas Borg snapshot marker; leaving snapshots unchanged\n' >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
exec 9>/run/lock/atlas-zfs-snapshot.lock
|
||||
flock 9
|
||||
if zfs list -H -t snapshot -o name "${pool}@${snapshot_name}" >/dev/null 2>&1; then
|
||||
# The private bind mounts are gone, but ZFS may leave its on-demand
|
||||
# .zfs/snapshot mounts in the host namespace until explicitly unmounted.
|
||||
snapshot_mounts=()
|
||||
snapshot_sources=()
|
||||
while IFS=$'\t' read -r dataset dataset_mountpoint; do
|
||||
[[ "$dataset_mountpoint" == "$mount_root" || "$dataset_mountpoint" == "$mount_root/"* ]] || continue
|
||||
snapshot_mounts+=("${dataset_mountpoint}/.zfs/snapshot/${snapshot_name}")
|
||||
snapshot_sources+=("${dataset}@${snapshot_name}")
|
||||
done < <(zfs list -H -o name,mountpoint -s name -r "$pool")
|
||||
|
||||
{% raw %}
|
||||
for ((index = ${#snapshot_mounts[@]} - 1; index >= 0; index--)); do
|
||||
{% endraw %}
|
||||
mounted_source="$(findmnt -rn -M "${snapshot_mounts[$index]}" -o SOURCE || true)"
|
||||
[[ -n "$mounted_source" ]] || continue
|
||||
[[ "$mounted_source" == "${snapshot_sources[$index]}" ]] || {
|
||||
printf 'Unexpected source on Atlas Borg snapshot mount: %s\n' \
|
||||
"${snapshot_mounts[$index]}" >&2
|
||||
exit 2
|
||||
}
|
||||
umount "${snapshot_mounts[$index]}"
|
||||
done
|
||||
|
||||
zfs destroy -r "${pool}@${snapshot_name}"
|
||||
printf 'Removed recursive Atlas Borg source snapshot %s@%s after backup exit\n' \
|
||||
"$pool" "$snapshot_name"
|
||||
fi
|
||||
19
ansible/roles/profile_atlas/templates/atlas-borg-ssh.sh.j2
Normal file
19
ansible/roles/profile_atlas/templates/atlas-borg-ssh.sh.j2
Normal file
@@ -0,0 +1,19 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Borg receives CAP_DAC_READ_SEARCH only for local snapshot traversal. Drop it
|
||||
# before starting the network transport so SSH runs as the plain service user.
|
||||
exec setpriv \
|
||||
--inh-caps=-all \
|
||||
--ambient-caps=-all \
|
||||
-- /usr/bin/ssh \
|
||||
-i {{ atlas_borg_ssh_private_key_path | quote }} \
|
||||
-p {{ atlas_borg_repository_port | int }} \
|
||||
-o BatchMode=yes \
|
||||
-o IdentitiesOnly=yes \
|
||||
-o StrictHostKeyChecking=yes \
|
||||
-o UserKnownHostsFile={{ atlas_borg_known_hosts_path | quote }} \
|
||||
-o ConnectTimeout=30 \
|
||||
-o ServerAliveInterval=60 \
|
||||
-o ServerAliveCountMax=3 \
|
||||
"$@"
|
||||
@@ -0,0 +1,30 @@
|
||||
# Managed by Ansible. Staging does not start automatically.
|
||||
[Unit]
|
||||
Description=Atlas rootless Gitea
|
||||
RequiresMountsFor={{ atlas_gitea_mountpoint }}
|
||||
|
||||
[Container]
|
||||
ContainerName=atlas-gitea
|
||||
Image={{ atlas_gitea_image }}
|
||||
UserNS=keep-id:uid={{ atlas_gitea_container_uid }},gid={{ atlas_gitea_container_gid }}
|
||||
{% if atlas_gitea_production_enabled | bool %}
|
||||
PublishPort={{ atlas_gitea_bind_address }}:{{ atlas_gitea_http_port }}:3000
|
||||
PublishPort={{ atlas_gitea_bind_address }}:{{ atlas_gitea_ssh_port }}:2222
|
||||
{% else %}
|
||||
PublishPort={{ atlas_gitea_staging_bind_address }}:{{ atlas_gitea_staging_http_port }}:3000
|
||||
PublishPort={{ atlas_gitea_staging_bind_address }}:{{ atlas_gitea_staging_ssh_port }}:2222
|
||||
{% endif %}
|
||||
Volume={{ atlas_gitea_mountpoint }}/data:/var/lib/gitea:Z
|
||||
Volume={{ atlas_gitea_mountpoint }}/config:/etc/gitea:Z
|
||||
NoNewPrivileges=true
|
||||
DropCapability=all
|
||||
|
||||
[Service]
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
TimeoutStartSec=900
|
||||
{% if atlas_gitea_production_enabled | bool %}
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
{% endif %}
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"pool": {{ atlas_zfs_pool | to_json }},
|
||||
"backup_dataset": {{ (atlas_zfs_pool ~ '/' ~ atlas_zfs_dataset_backup) | to_json }},
|
||||
"notifier": {{ atlas_monitor_notifier | to_json }},
|
||||
"smart_devices": {{ atlas_monitor_smart_devices | to_json }},
|
||||
"timers": {{ atlas_monitor_effective_timers | to_json }},
|
||||
"failure_units": {{ atlas_monitor_effective_failure_units | to_json }},
|
||||
"remote_capacity": {{ atlas_monitor_remote_capacity | to_json }},
|
||||
"pool_warning_percent": {{ atlas_monitor_pool_warning_percent | int }},
|
||||
"pool_critical_percent": {{ atlas_monitor_pool_critical_percent | int }},
|
||||
"root_warning_percent": {{ atlas_monitor_root_warning_percent | int }},
|
||||
"root_critical_percent": {{ atlas_monitor_root_critical_percent | int }},
|
||||
"snapshot_warning_percent": {{ atlas_monitor_snapshot_warning_percent | int }},
|
||||
"snapshot_critical_percent": {{ atlas_monitor_snapshot_critical_percent | int }},
|
||||
"snapshot_growth_warning_gib_day": {{ atlas_monitor_snapshot_growth_warning_gib_day | int }},
|
||||
"backup_growth_warning_gib_day": {{ atlas_monitor_backup_growth_warning_gib_day | int }},
|
||||
"cpu_warning_c": {{ atlas_monitor_cpu_warning_c | int }},
|
||||
"cpu_critical_c": {{ atlas_monitor_cpu_critical_c | int }},
|
||||
"borg_max_runtime_days": {{ atlas_monitor_borg_max_runtime_days | int }}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
[Unit]
|
||||
Description=Check Atlas pool, disks, capacity, temperatures and maintenance jobs
|
||||
Wants=houston-dbus.service network-online.target
|
||||
After=zfs.target houston-dbus.service network-online.target
|
||||
ConditionFileIsExecutable=/usr/local/libexec/atlas-health-monitor
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/libexec/atlas-health-monitor
|
||||
User=root
|
||||
Group=root
|
||||
UMask=0077
|
||||
StateDirectory=atlas-health-monitor
|
||||
StateDirectoryMode=0700
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectHome=true
|
||||
ProtectSystem=strict
|
||||
ReadWritePaths=/var/lib/atlas-health-monitor
|
||||
ProtectKernelTunables=true
|
||||
ProtectKernelModules=true
|
||||
ProtectControlGroups=true
|
||||
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
|
||||
RestrictRealtime=true
|
||||
LockPersonality=true
|
||||
@@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=Schedule Atlas health checks
|
||||
|
||||
[Timer]
|
||||
OnCalendar={{ atlas_monitor_calendar }}
|
||||
Persistent=true
|
||||
RandomizedDelaySec=5min
|
||||
Unit=atlas-health-monitor.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
14
ansible/roles/profile_atlas/templates/atlas-icloudpd.conf.j2
Normal file
14
ansible/roles/profile_atlas/templates/atlas-icloudpd.conf.j2
Normal file
@@ -0,0 +1,14 @@
|
||||
# Managed by Ansible. Password, keyring and MFA cookies are stored separately in /config.
|
||||
apple_id={{ vault_atlas_icloudpd_apple_id }}
|
||||
authentication_type=MFA
|
||||
user=user
|
||||
user_id=1000
|
||||
group=group
|
||||
group_id=1000
|
||||
download_path=/home/user/iCloud
|
||||
folder_structure={:%Y/%m/%d}
|
||||
directory_permissions=750
|
||||
file_permissions=640
|
||||
download_interval=86400
|
||||
auto_delete=false
|
||||
delete_after_download=false
|
||||
@@ -0,0 +1,25 @@
|
||||
# Managed by Ansible. Start automatically with the lingering admin user manager.
|
||||
[Unit]
|
||||
Description=Atlas rootless iCloud Photos Downloader
|
||||
RequiresMountsFor={{ atlas_icloudpd_state_dir }} {{ atlas_icloudpd_photos_dir }}
|
||||
|
||||
[Container]
|
||||
ContainerName=atlas-icloudpd
|
||||
Image={{ atlas_icloudpd_image }}
|
||||
UserNS=keep-id:uid=1000,gid=1000
|
||||
# The image initialises its unprivileged UID 1000 account as container root.
|
||||
User=0
|
||||
# Upstream launcher requires traceroute for its iCloud reachability check.
|
||||
AddCapability=NET_RAW
|
||||
Environment=TZ={{ atlas_icloudpd_timezone }}
|
||||
Volume={{ atlas_icloudpd_photos_dir }}:/home/user/iCloud:z
|
||||
Volume={{ atlas_icloudpd_config_dir }}:/config:Z
|
||||
NoNewPrivileges=true
|
||||
|
||||
[Service]
|
||||
Restart=on-failure
|
||||
RestartSec=300
|
||||
TimeoutStartSec=900
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -0,0 +1,2 @@
|
||||
[Unit]
|
||||
OnFailure=atlas-monitor-failure@%n.service
|
||||
@@ -0,0 +1,19 @@
|
||||
[Unit]
|
||||
Description=Submit a 45Drives Alert for failed Atlas job %i
|
||||
Requires=houston-dbus.service
|
||||
After=houston-dbus.service
|
||||
ConditionFileIsExecutable=/usr/local/libexec/atlas-health-monitor
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/libexec/atlas-health-monitor --job-failed %i
|
||||
User=root
|
||||
Group=root
|
||||
UMask=0077
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectHome=true
|
||||
ProtectSystem=strict
|
||||
RestrictAddressFamilies=AF_UNIX
|
||||
RestrictRealtime=true
|
||||
LockPersonality=true
|
||||
@@ -0,0 +1,19 @@
|
||||
[Unit]
|
||||
Description=Pull a prepared read-only Prometheus backup to Atlas
|
||||
RequiresMountsFor={{ atlas_backup_prometheus_mountpoint }}
|
||||
Wants=network-online.target
|
||||
After=network-online.target zfs.target
|
||||
ConditionFileIsExecutable=/usr/local/sbin/atlas-prometheus-pull
|
||||
ConditionPathExists={{ atlas_prometheus_pull_private_key_path }}
|
||||
ConditionPathExists={{ atlas_prometheus_pull_known_hosts_path }}
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/atlas-prometheus-pull
|
||||
User=root
|
||||
Group=root
|
||||
UMask=0077
|
||||
TimeoutStartSec=infinity
|
||||
Nice=15
|
||||
IOSchedulingClass=best-effort
|
||||
IOSchedulingPriority=7
|
||||
@@ -0,0 +1,75 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
|
||||
backup_root={{ atlas_backup_prometheus_mountpoint | quote }}
|
||||
snapshots="$backup_root/snapshots"
|
||||
stage=''
|
||||
exec 9>/run/lock/atlas-prometheus-pull.lock
|
||||
flock -n 9 || { echo 'A Prometheus pull is already running' >&2; exit 1; }
|
||||
|
||||
cleanup() {
|
||||
local rc=$?
|
||||
trap - EXIT
|
||||
if (( rc != 0 )) && [[ -n "$stage" && -d "$stage" ]]; then
|
||||
rm -rf -- "$stage"
|
||||
fi
|
||||
exit "$rc"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
zpool list -H -o name {{ atlas_zfs_pool | quote }} >/dev/null
|
||||
findmnt -rn --mountpoint "$backup_root" >/dev/null
|
||||
stage=$(mktemp -d "$backup_root/.staging.XXXXXXXX")
|
||||
ssh_cmd='/usr/bin/ssh -F /dev/null -o BatchMode=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile={{ atlas_prometheus_pull_known_hosts_path }} -o IdentitiesOnly=yes -i {{ atlas_prometheus_pull_private_key_path }} -p {{ atlas_prometheus_pull_source_port }}'
|
||||
rsync -a --partial --delay-updates -e "$ssh_cmd" \
|
||||
{{ (atlas_prometheus_pull_source_user ~ '@' ~ hostvars['prometheus'].ansible_host ~ ':current/') | quote }} \
|
||||
"$stage/"
|
||||
|
||||
test -s "$stage/payload.tar"
|
||||
test -s "$stage/payload.sha256"
|
||||
test -s "$stage/metadata.json"
|
||||
(cd "$stage" && sha256sum -c payload.sha256)
|
||||
tar -tf "$stage/payload.tar" >/dev/null
|
||||
stamp=$(python3 - "$stage/metadata.json" <<'PY'
|
||||
import json
|
||||
import datetime as dt
|
||||
import re
|
||||
import sys
|
||||
|
||||
with open(sys.argv[1], encoding="utf-8") as stream:
|
||||
metadata = json.load(stream)
|
||||
stamp = metadata.get("created_utc", "")
|
||||
if metadata.get("schema") != 1 or metadata.get("host") != "prometheus":
|
||||
raise SystemExit("Unexpected Prometheus backup metadata")
|
||||
if not re.fullmatch(r"[0-9]{8}T[0-9]{6}Z", stamp):
|
||||
raise SystemExit("Invalid Prometheus backup timestamp")
|
||||
created = dt.datetime.strptime(stamp, "%Y%m%dT%H%M%SZ").replace(tzinfo=dt.timezone.utc)
|
||||
age = dt.datetime.now(dt.timezone.utc) - created
|
||||
if age.total_seconds() < -300 or age > dt.timedelta(hours={{ atlas_prometheus_pull_max_age_hours }}):
|
||||
raise SystemExit("Prometheus backup is outside the configured freshness window")
|
||||
print(stamp)
|
||||
PY
|
||||
)
|
||||
if [[ -e "$snapshots/$stamp" ]]; then
|
||||
cmp "$stage/payload.sha256" "$snapshots/$stamp/payload.sha256"
|
||||
cmp "$stage/metadata.json" "$snapshots/$stamp/metadata.json"
|
||||
(cd "$snapshots/$stamp" && sha256sum -c payload.sha256)
|
||||
rm -rf -- "${stage:?}"
|
||||
stage=''
|
||||
else
|
||||
chown -R root:root "$stage"
|
||||
chmod 0700 "$stage"
|
||||
chmod 0600 "$stage/payload.tar" "$stage/payload.sha256" "$stage/metadata.json"
|
||||
mv -- "$stage" "$snapshots/$stamp"
|
||||
stage=''
|
||||
fi
|
||||
latest_link=$(readlink "$backup_root/latest" 2>/dev/null || true)
|
||||
latest_stamp=${latest_link##*/}
|
||||
if [[ -z "$latest_stamp" || "$stamp" > "$latest_stamp" ]]; then
|
||||
ln -s "snapshots/$stamp" "$backup_root/.latest.new"
|
||||
mv -Tf -- "$backup_root/.latest.new" "$backup_root/latest"
|
||||
fi
|
||||
python3 /usr/local/libexec/atlas-prometheus-prune "$snapshots" \
|
||||
{{ atlas_prometheus_pull_keep_daily }} {{ atlas_prometheus_pull_keep_weekly }} {{ atlas_prometheus_pull_keep_monthly }}
|
||||
echo "Verified and published Prometheus backup $stamp"
|
||||
@@ -0,0 +1,10 @@
|
||||
[Unit]
|
||||
Description=Schedule Atlas pull of prepared Prometheus backups
|
||||
|
||||
[Timer]
|
||||
OnCalendar={{ atlas_prometheus_pull_calendar }}
|
||||
Persistent=true
|
||||
Unit=atlas-prometheus-pull.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,30 @@
|
||||
[Unit]
|
||||
Description=Run a manual, UUID-bound offline USB backup of Atlas ZFS datasets
|
||||
Requires=zfs.target
|
||||
After=zfs.target
|
||||
ConditionFileIsExecutable=/usr/local/sbin/atlas-usb-backup
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/atlas-usb-backup
|
||||
ExecStopPost=+/usr/local/sbin/atlas-usb-snapshot-cleanup
|
||||
User=root
|
||||
Group=root
|
||||
UMask=0077
|
||||
TimeoutStartSec=infinity
|
||||
RuntimeDirectory=atlas-usb-backup
|
||||
RuntimeDirectoryMode=0700
|
||||
Nice=15
|
||||
IOSchedulingClass=best-effort
|
||||
IOSchedulingPriority=7
|
||||
PrivateMounts=true
|
||||
PrivateTmp=true
|
||||
ProtectHome=true
|
||||
ProtectSystem=strict
|
||||
ReadWritePaths=/run/atlas-usb-backup /run/lock
|
||||
ProtectKernelTunables=true
|
||||
ProtectKernelModules=true
|
||||
ProtectControlGroups=true
|
||||
RestrictAddressFamilies=AF_UNIX
|
||||
RestrictRealtime=true
|
||||
LockPersonality=true
|
||||
242
ansible/roles/profile_atlas/templates/atlas-usb-backup.sh.j2
Normal file
242
ansible/roles/profile_atlas/templates/atlas-usb-backup.sh.j2
Normal file
@@ -0,0 +1,242 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
export LC_ALL=C.utf8
|
||||
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
|
||||
|
||||
readonly pool={{ atlas_zfs_pool | quote }}
|
||||
readonly mount_root={{ atlas_mount_root | quote }}
|
||||
readonly luks_uuid={{ atlas_usb_backup_luks_uuid | quote }}
|
||||
readonly fs_uuid={{ atlas_usb_backup_fs_uuid | quote }}
|
||||
readonly mapper_name={{ atlas_usb_backup_mapper_name | quote }}
|
||||
readonly snapshot_prefix={{ atlas_usb_backup_snapshot_prefix | quote }}
|
||||
readonly min_free_bytes={{ atlas_usb_backup_min_free_bytes | int }}
|
||||
readonly mapper="/dev/mapper/${mapper_name}"
|
||||
readonly outer="/dev/disk/by-uuid/${luks_uuid}"
|
||||
readonly runtime_dir=/run/atlas-usb-backup
|
||||
readonly snapshot_marker="${runtime_dir}/snapshot-name"
|
||||
readonly source_dir="${runtime_dir}/source"
|
||||
readonly usb_mount="${runtime_dir}/target"
|
||||
readonly backup_root="${usb_mount}/atlas"
|
||||
|
||||
snapshot_name=""
|
||||
mapper_opened_by_script=false
|
||||
usb_mounted=false
|
||||
published=false
|
||||
partial=""
|
||||
mounted_targets=()
|
||||
|
||||
# shellcheck disable=SC2329
|
||||
cleanup() {
|
||||
local status=$?
|
||||
local cleanup_status=0
|
||||
local index
|
||||
local source_mount_failed=false
|
||||
trap - EXIT HUP INT TERM
|
||||
set +e
|
||||
|
||||
if [[ -n "$partial" && "$published" == false && "$usb_mounted" == true ]]; then
|
||||
rm -rf -- "$partial" || cleanup_status=2
|
||||
fi
|
||||
if [[ "$usb_mounted" == true ]]; then
|
||||
umount "$usb_mount" || cleanup_status=2
|
||||
fi
|
||||
|
||||
{% raw %}
|
||||
for ((index = ${#mounted_targets[@]} - 1; index >= 0; index--)); do
|
||||
{% endraw %}
|
||||
if mountpoint -q "${mounted_targets[$index]}" && ! umount -R "${mounted_targets[$index]}"; then
|
||||
printf 'Source snapshot mount cleanup failed: %s\n' "${mounted_targets[$index]}" >&2
|
||||
source_mount_failed=true
|
||||
fi
|
||||
if mountpoint -q "${mounted_targets[$index]}"; then
|
||||
printf 'Source snapshot mount is still active: %s\n' "${mounted_targets[$index]}" >&2
|
||||
source_mount_failed=true
|
||||
else
|
||||
rmdir -- "${mounted_targets[$index]}" 2>/dev/null || true
|
||||
fi
|
||||
done
|
||||
if [[ "$source_mount_failed" == false ]]; then
|
||||
rmdir -- "$source_dir" 2>/dev/null || true
|
||||
else
|
||||
cleanup_status=2
|
||||
printf 'Source bind mount cleanup failed; keeping the snapshot for recovery\n' >&2
|
||||
fi
|
||||
|
||||
if [[ "$usb_mounted" == true || "$mapper_opened_by_script" == true ]] &&
|
||||
! mountpoint -q "$usb_mount" &&
|
||||
! findmnt -rn -S "$mapper" >/dev/null; then
|
||||
cryptsetup close "$mapper_name" || cleanup_status=2
|
||||
fi
|
||||
rmdir -- "$usb_mount" 2>/dev/null || true
|
||||
|
||||
if ((status == 0 && cleanup_status != 0)); then
|
||||
status=$cleanup_status
|
||||
fi
|
||||
exit "$status"
|
||||
}
|
||||
|
||||
trap cleanup EXIT
|
||||
trap 'exit 143' HUP INT TERM
|
||||
|
||||
exec 8>/run/lock/atlas-usb-backup.lock
|
||||
flock -n 8 || { printf 'Atlas USB backup is already running\n' >&2; exit 75; }
|
||||
exec 9>/run/lock/atlas-zfs-snapshot.lock
|
||||
|
||||
zpool list -H -o name "$pool" >/dev/null
|
||||
[[ -b "$outer" ]] || { printf 'Configured LUKS UUID is not connected\n' >&2; exit 66; }
|
||||
[[ "$(blkid -s TYPE -o value "$outer")" == crypto_LUKS ]] || {
|
||||
printf 'Configured outer UUID is not a LUKS container\n' >&2
|
||||
exit 65
|
||||
}
|
||||
[[ "$(blkid -s UUID -o value "$outer")" == "$luks_uuid" ]] || exit 65
|
||||
if ! cryptsetup status "$mapper_name" >/dev/null; then
|
||||
printf 'Requesting the LUKS passphrase for the configured USB disk\n'
|
||||
systemd-ask-password -n --no-tty --timeout=300 \
|
||||
--id="atlas-usb-backup:${luks_uuid}" \
|
||||
'Atlas offline USB backup LUKS passphrase:' |
|
||||
cryptsetup open --type luks2 --key-file - "$outer" "$mapper_name"
|
||||
mapper_opened_by_script=true
|
||||
fi
|
||||
backing_device="$(cryptsetup status "$mapper_name" | awk '$1 == "device:" { print $2 }')"
|
||||
[[ -n "$backing_device" && "$(readlink -f "$backing_device")" == "$(readlink -f "$outer")" ]] || {
|
||||
printf 'The unlocked mapper does not belong to the configured LUKS UUID\n' >&2
|
||||
exit 65
|
||||
}
|
||||
[[ "$(blkid -s TYPE -o value "$mapper")" == ext4 ]] || {
|
||||
printf 'The unlocked USB filesystem is not ext4\n' >&2
|
||||
exit 65
|
||||
}
|
||||
[[ "$(blkid -s UUID -o value "$mapper")" == "$fs_uuid" ]] || {
|
||||
printf 'The unlocked USB filesystem UUID does not match\n' >&2
|
||||
exit 65
|
||||
}
|
||||
if findmnt -rn -S "$mapper" >/dev/null; then
|
||||
printf 'The USB filesystem is already mounted elsewhere\n' >&2
|
||||
exit 65
|
||||
fi
|
||||
[[ ! -e "$source_dir" && ! -e "$usb_mount" ]] || {
|
||||
printf 'USB backup staging directories already exist; inspect them manually\n' >&2
|
||||
exit 65
|
||||
}
|
||||
|
||||
mkdir -m 0700 "$usb_mount"
|
||||
mount -t ext4 -o nodev,nosuid,noexec "$mapper" "$usb_mount"
|
||||
usb_mounted=true
|
||||
[[ "$(readlink -f "$(findmnt -nro SOURCE --target "$usb_mount")")" == "$(readlink -f "$mapper")" ]] || {
|
||||
printf 'Mounted USB source does not match the verified mapper\n' >&2
|
||||
exit 65
|
||||
}
|
||||
|
||||
for path in "$backup_root" "$backup_root/snapshots"; do
|
||||
[[ ! -L "$path" ]] || { printf 'Unsafe symlink in USB backup destination\n' >&2; exit 65; }
|
||||
mkdir -p -- "$path"
|
||||
[[ -d "$path" ]] || exit 65
|
||||
chown root:root -- "$path"
|
||||
chmod 0700 -- "$path"
|
||||
done
|
||||
|
||||
free_bytes="$(df -B1 --output=avail "$usb_mount" | tail -n 1 | tr -d ' ')"
|
||||
if ((free_bytes < min_free_bytes)); then
|
||||
printf 'USB free space (%s bytes) is below the required reserve (%s bytes)\n' \
|
||||
"$free_bytes" "$min_free_bytes" >&2
|
||||
exit 73
|
||||
fi
|
||||
|
||||
mkdir -m 0700 "$source_dir"
|
||||
flock 9
|
||||
timestamp="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
snapshot_name="${snapshot_prefix}-${timestamp}-$$"
|
||||
printf '%s\n' "$snapshot_name" >"$snapshot_marker"
|
||||
zfs snapshot -r "${pool}@${snapshot_name}"
|
||||
flock -u 9
|
||||
printf 'Created recursive USB source snapshot %s@%s\n' "$pool" "$snapshot_name"
|
||||
|
||||
while IFS=$'\t' read -r dataset dataset_mountpoint mounted; do
|
||||
if [[ "$mounted" != yes ]]; then
|
||||
printf 'Dataset %s is not mounted; refusing an incomplete backup\n' "$dataset" >&2
|
||||
exit 65
|
||||
fi
|
||||
if [[ "$dataset_mountpoint" != "$mount_root" && "$dataset_mountpoint" != "$mount_root/"* ]]; then
|
||||
printf 'Dataset %s has unexpected mountpoint %s\n' "$dataset" "$dataset_mountpoint" >&2
|
||||
exit 65
|
||||
fi
|
||||
dataset_suffix="${dataset#"$pool"}"
|
||||
source_path="${dataset_mountpoint}/.zfs/snapshot/${snapshot_name}"
|
||||
target_path="${source_dir}${dataset_suffix}"
|
||||
mkdir -p "$target_path"
|
||||
mount --bind "$source_path" "$target_path"
|
||||
mounted_targets+=("$target_path")
|
||||
mount -o remount,bind,ro "$target_path"
|
||||
done < <(zfs list -H -o name,mountpoint,mounted -s name -r "$pool")
|
||||
|
||||
previous=""
|
||||
if [[ -e "$backup_root/latest" || -L "$backup_root/latest" ]]; then
|
||||
[[ -L "$backup_root/latest" ]] || { printf 'latest is not a symlink\n' >&2; exit 65; }
|
||||
previous="$(readlink -e "$backup_root/latest")"
|
||||
[[ -n "$previous" && "$previous" == "$backup_root/snapshots/"* && -d "$previous" ]] || {
|
||||
printf 'latest does not point to a complete snapshot on the USB disk\n' >&2
|
||||
exit 65
|
||||
}
|
||||
fi
|
||||
|
||||
backup_name="${timestamp}-$$"
|
||||
candidate_partial="${backup_root}/snapshots/.incomplete-${backup_name}"
|
||||
complete="${backup_root}/snapshots/${backup_name}"
|
||||
[[ ! -e "$candidate_partial" && ! -L "$candidate_partial" && ! -e "$complete" && ! -L "$complete" ]] || exit 65
|
||||
mkdir -m 0700 "$candidate_partial"
|
||||
partial="$candidate_partial"
|
||||
|
||||
printf 'Copying the consistent pool tree to USB backup %s\n' "$backup_name"
|
||||
# Preserve POSIX ACLs, ownership, modes, timestamps, hard links, and sparse
|
||||
# files. Do not preserve generic xattrs: Rocky 9's rsync 3.2.7 fails when
|
||||
# combining xattrs with --link-dest, while SELinux labels were intentionally
|
||||
# excluded because restores must relabel for their destination host.
|
||||
rsync_args=(-aHAS --numeric-ids "--info=progress2,stats2")
|
||||
estimate_args=(-aHAS --numeric-ids --dry-run --stats)
|
||||
if [[ -n "$previous" ]]; then
|
||||
rsync_args+=("--link-dest=$previous")
|
||||
estimate_args+=("--link-dest=$previous")
|
||||
fi
|
||||
|
||||
# The rsync dry run estimates changed file bytes after link-dest deduplication.
|
||||
# Metadata and filesystem allocation still require the separate free-space reserve.
|
||||
estimate="$(rsync "${estimate_args[@]}" "${source_dir}/" "${partial}/")"
|
||||
transfer_bytes="$(printf '%s\n' "$estimate" | awk -F: \
|
||||
'/^Total transferred file size:/ { gsub(/[^0-9]/, "", $2); print $2 }')"
|
||||
[[ "$transfer_bytes" =~ ^[0-9]+$ ]] || {
|
||||
printf 'Could not determine the USB transfer size\n' >&2
|
||||
exit 74
|
||||
}
|
||||
if ((free_bytes - transfer_bytes < min_free_bytes)); then
|
||||
printf 'Insufficient USB space: %s bytes free, %s estimated transfer, %s reserved\n' \
|
||||
"$free_bytes" "$transfer_bytes" "$min_free_bytes" >&2
|
||||
exit 73
|
||||
fi
|
||||
|
||||
rsync "${rsync_args[@]}" "${source_dir}/" "${partial}/"
|
||||
|
||||
printf 'Verifying USB backup %s with a checksum-based dry run\n' "$backup_name"
|
||||
verification="${runtime_dir}/verification.out"
|
||||
rsync -aHAS --numeric-ids \
|
||||
--checksum --dry-run --delete --itemize-changes \
|
||||
"${source_dir}/" "${partial}/" >"$verification"
|
||||
if [[ -s "$verification" ]]; then
|
||||
printf 'USB verification found mismatches; refusing to publish the backup\n' >&2
|
||||
exit 74
|
||||
fi
|
||||
|
||||
free_bytes="$(df -B1 --output=avail "$usb_mount" | tail -n 1 | tr -d ' ')"
|
||||
if ((free_bytes < min_free_bytes)); then
|
||||
printf 'USB backup completed below the free-space reserve; refusing to publish it\n' >&2
|
||||
exit 73
|
||||
fi
|
||||
|
||||
mv -- "$partial" "$complete"
|
||||
partial=""
|
||||
ln -s "snapshots/${backup_name}" "${backup_root}/.latest-${backup_name}"
|
||||
mv -Tf -- "${backup_root}/.latest-${backup_name}" "${backup_root}/latest"
|
||||
published=true
|
||||
sync -f "$complete"
|
||||
sync -f "$backup_root"
|
||||
printf 'USB backup %s verified and published; unmounting and closing LUKS\n' "$backup_name"
|
||||
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/python3
|
||||
"""Submit a manual-backup reminder through Atlas' existing Houston notifier."""
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
message = {
|
||||
"timestamp": now.isoformat(timespec="seconds"),
|
||||
"unixtime": int(now.timestamp()),
|
||||
"event": "atlas_usb_backup_reminder",
|
||||
"severity": "warning",
|
||||
"subject": "Promemoria backup USB offline Atlas",
|
||||
"email_message": (
|
||||
"Collega il disco USB di backup ad Atlas ed esegui manualmente il backup offline.\n"
|
||||
"Il promemoria non avvia il backup. Controlla che il disco non sia\n"
|
||||
"montato; poi esegui:\n\n"
|
||||
" sudo systemctl start atlas-usb-backup.service\n\n"
|
||||
"Verifica l'esito con:\n"
|
||||
" sudo journalctl -u atlas-usb-backup.service -n 100 --no-pager\n\n"
|
||||
"Dopo la riuscita, scollega fisicamente il disco."
|
||||
),
|
||||
}
|
||||
|
||||
subprocess.run(
|
||||
[{{ atlas_usb_reminder_notifier | to_json }}, json.dumps(message)],
|
||||
check=True,
|
||||
)
|
||||
print("Atlas USB backup reminder submitted to 45Drives Alerts; email delivery is not verified.", flush=True)
|
||||
@@ -0,0 +1,22 @@
|
||||
[Unit]
|
||||
Description=45Drives Alerts reminder to run the manual Atlas offline USB backup
|
||||
Requires=houston-dbus.service
|
||||
After=houston-dbus.service
|
||||
ConditionFileIsExecutable=/usr/local/libexec/atlas-usb-reminder
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/libexec/atlas-usb-reminder
|
||||
User=root
|
||||
Group=root
|
||||
UMask=0077
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectHome=true
|
||||
ProtectSystem=strict
|
||||
ProtectKernelTunables=true
|
||||
ProtectKernelModules=true
|
||||
ProtectControlGroups=true
|
||||
RestrictAddressFamilies=AF_UNIX
|
||||
RestrictRealtime=true
|
||||
LockPersonality=true
|
||||
@@ -0,0 +1,10 @@
|
||||
[Unit]
|
||||
Description=Remind the administrator to run the manual Atlas offline USB backup
|
||||
|
||||
[Timer]
|
||||
OnCalendar={{ atlas_usb_reminder_calendar }}
|
||||
Persistent=true
|
||||
Unit=atlas-usb-reminder.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,51 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
|
||||
|
||||
readonly pool={{ atlas_zfs_pool | quote }}
|
||||
readonly mount_root={{ atlas_mount_root | quote }}
|
||||
readonly snapshot_prefix={{ atlas_usb_backup_snapshot_prefix | quote }}
|
||||
readonly marker=/run/atlas-usb-backup/snapshot-name
|
||||
|
||||
[[ -e "$marker" ]] || exit 0
|
||||
[[ -f "$marker" && ! -L "$marker" ]] || {
|
||||
printf 'Unsafe Atlas USB snapshot marker; leaving snapshots unchanged\n' >&2
|
||||
exit 2
|
||||
}
|
||||
IFS= read -r snapshot_name <"$marker"
|
||||
[[ "$snapshot_name" =~ ^${snapshot_prefix}-[0-9]{8}T[0-9]{6}Z-[0-9]+$ ]] || {
|
||||
printf 'Invalid Atlas USB snapshot marker; leaving snapshots unchanged\n' >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
exec 9>/run/lock/atlas-zfs-snapshot.lock
|
||||
flock 9
|
||||
if zfs list -H -t snapshot -o name "${pool}@${snapshot_name}" >/dev/null 2>&1; then
|
||||
# ZFS can leave its on-demand .zfs/snapshot mounts in the host namespace
|
||||
# even after the backup's private bind mounts and process have exited.
|
||||
snapshot_mounts=()
|
||||
snapshot_sources=()
|
||||
while IFS=$'\t' read -r dataset dataset_mountpoint; do
|
||||
[[ "$dataset_mountpoint" == "$mount_root" || "$dataset_mountpoint" == "$mount_root/"* ]] || continue
|
||||
snapshot_mounts+=("${dataset_mountpoint}/.zfs/snapshot/${snapshot_name}")
|
||||
snapshot_sources+=("${dataset}@${snapshot_name}")
|
||||
done < <(zfs list -H -o name,mountpoint -s name -r "$pool")
|
||||
|
||||
{% raw %}
|
||||
for ((index = ${#snapshot_mounts[@]} - 1; index >= 0; index--)); do
|
||||
{% endraw %}
|
||||
mounted_source="$(findmnt -rn -M "${snapshot_mounts[$index]}" -o SOURCE || true)"
|
||||
[[ -n "$mounted_source" ]] || continue
|
||||
[[ "$mounted_source" == "${snapshot_sources[$index]}" ]] || {
|
||||
printf 'Unexpected source on Atlas USB snapshot mount: %s\n' \
|
||||
"${snapshot_mounts[$index]}" >&2
|
||||
exit 2
|
||||
}
|
||||
umount "${snapshot_mounts[$index]}"
|
||||
done
|
||||
|
||||
zfs destroy -r "${pool}@${snapshot_name}"
|
||||
printf 'Removed recursive Atlas USB source snapshot %s@%s after backup exit\n' \
|
||||
"$pool" "$snapshot_name"
|
||||
fi
|
||||
@@ -0,0 +1,6 @@
|
||||
[Timer]
|
||||
OnCalendar=
|
||||
OnCalendar={{ atlas_zfs_scrub_calendar }}
|
||||
Persistent=true
|
||||
RandomizedDelaySec=0
|
||||
AccuracySec=1min
|
||||
@@ -0,0 +1,72 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
export LC_ALL=C
|
||||
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
|
||||
|
||||
if [[ $# -ne 1 ]]; then
|
||||
printf 'Usage: %s <policy>\n' "$0" >&2
|
||||
exit 64
|
||||
fi
|
||||
|
||||
readonly pool={{ atlas_zfs_pool | quote }}
|
||||
readonly prefix={{ atlas_zfs_snapshot_prefix | quote }}
|
||||
readonly period="$1"
|
||||
|
||||
case "$period" in
|
||||
{% for policy in atlas_zfs_snapshot_policies %}
|
||||
{{ policy.name | quote }})
|
||||
keep={{ policy.keep | int }}
|
||||
;;
|
||||
{% endfor %}
|
||||
*)
|
||||
printf 'Unknown Atlas ZFS snapshot policy: %s\n' "$period" >&2
|
||||
exit 64
|
||||
;;
|
||||
esac
|
||||
readonly keep
|
||||
|
||||
zpool list -H -o name "$pool" >/dev/null
|
||||
|
||||
exec 9>/run/lock/atlas-zfs-snapshot.lock
|
||||
flock 9
|
||||
|
||||
timestamp="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
readonly timestamp
|
||||
readonly snapshot_prefix="${pool}@${prefix}-${period}-"
|
||||
readonly snapshot="${snapshot_prefix}${timestamp}"
|
||||
|
||||
zfs snapshot -r "$snapshot"
|
||||
printf 'Created recursive ZFS snapshot %s\n' "$snapshot"
|
||||
|
||||
snapshot_listing="$(zfs list -H -t snapshot -o name -s creation -r "$pool")"
|
||||
managed_snapshots=()
|
||||
while IFS= read -r snapshot_name; do
|
||||
if [[ "$snapshot_name" == "$snapshot_prefix"* ]]; then
|
||||
snapshot_suffix="${snapshot_name#"$snapshot_prefix"}"
|
||||
if [[ "$snapshot_suffix" =~ ^[0-9]{8}T[0-9]{6}Z$ ]]; then
|
||||
managed_snapshots+=("$snapshot_name")
|
||||
fi
|
||||
fi
|
||||
done <<< "$snapshot_listing"
|
||||
|
||||
{% raw %}
|
||||
managed_snapshot_count="${#managed_snapshots[@]}"
|
||||
{% endraw %}
|
||||
prune_count=$((managed_snapshot_count - keep))
|
||||
if ((prune_count <= 0)); then
|
||||
printf 'Retaining %d of %d managed %s snapshots\n' \
|
||||
"$managed_snapshot_count" "$keep" "$period"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
for ((index = 0; index < prune_count; index++)); do
|
||||
candidate="${managed_snapshots[$index]}"
|
||||
if [[ "$candidate" != "$snapshot_prefix"* ]]; then
|
||||
printf 'Refusing to destroy unexpected snapshot: %s\n' "$candidate" >&2
|
||||
exit 65
|
||||
fi
|
||||
|
||||
zfs destroy -r "$candidate"
|
||||
printf 'Pruned recursive ZFS snapshot %s\n' "$candidate"
|
||||
done
|
||||
@@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=Schedule {{ item.name }} ZFS snapshots for {{ atlas_zfs_pool }}
|
||||
|
||||
[Timer]
|
||||
OnCalendar={{ item.calendar }}
|
||||
Persistent=true
|
||||
AccuracySec=1min
|
||||
Unit=atlas-zfs-snapshot@{{ item.name }}.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,25 @@
|
||||
[Unit]
|
||||
Description=Create and retain %i ZFS snapshots for {{ atlas_zfs_pool }}
|
||||
Documentation=man:zfs-snapshot(8) man:zfs-destroy(8)
|
||||
Requires=zfs.target
|
||||
After=zfs.target
|
||||
ConditionFileIsExecutable=/usr/local/sbin/atlas-zfs-snapshot
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/atlas-zfs-snapshot %i
|
||||
User=root
|
||||
Group=root
|
||||
UMask=0077
|
||||
Nice=10
|
||||
IOSchedulingClass=best-effort
|
||||
IOSchedulingPriority=7
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectHome=true
|
||||
ProtectSystem=strict
|
||||
ProtectKernelTunables=true
|
||||
ProtectKernelModules=true
|
||||
ProtectControlGroups=true
|
||||
RestrictRealtime=true
|
||||
LockPersonality=true
|
||||
@@ -5,14 +5,18 @@ backend_phase1_username: "{{ atlas_admin_username }}"
|
||||
backend_phase1_user_group: "{{ atlas_admin_group }}"
|
||||
backend_phase1_user_home: "{{ atlas_admin_home }}"
|
||||
backend_phase1_quadlet_dir: "{{ backend_phase1_user_home }}/.config/containers/systemd"
|
||||
backend_phase1_wireguard_interface: wg0
|
||||
backend_phase1_wireguard_address: CHANGEME_ATLAS_WIREGUARD_ADDRESS
|
||||
backend_phase1_wireguard_firewalld_zone: wireguard
|
||||
backend_phase1_bind_address: CHANGEME_ATLAS_BIND_ADDRESS
|
||||
backend_phase1_firewalld_zone: public
|
||||
backend_phase1_npm_source_ip: CHANGEME_AEGIS_IP
|
||||
backend_phase1_syncthing_native_subnet: CHANGEME_LAN_SUBNET
|
||||
backend_phase1_music_dir: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_music }}"
|
||||
backend_phase1_archive_dir: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_archive }}"
|
||||
backend_phase1_syncthing_folder_path: "{{ backend_phase1_archive_dir }}/Org"
|
||||
backend_phase1_app_data_root: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_app_data }}"
|
||||
backend_phase1_navidrome_data_dir: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_navidrome }}"
|
||||
backend_phase1_syncthing_root: "{{ atlas_mount_root }}/{{ atlas_zfs_dataset_syncthing }}"
|
||||
backend_phase1_music_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_music }}"
|
||||
backend_phase1_archive_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_archive }}"
|
||||
backend_phase1_app_data_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_app_data }}"
|
||||
backend_phase1_navidrome_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_navidrome }}"
|
||||
backend_phase1_syncthing_dataset: "{{ atlas_zfs_pool }}/{{ atlas_zfs_dataset_syncthing }}"
|
||||
@@ -26,3 +30,7 @@ backend_phase1_timezone: Europe/Rome
|
||||
backend_phase1_services:
|
||||
- atlas-navidrome.service
|
||||
- atlas-syncthing.service
|
||||
backend_phase1_music_sync_enabled: false
|
||||
backend_phase1_music_source_dir: "{{ backend_phase1_archive_dir }}/Music"
|
||||
backend_phase1_music_sync_calendar: "*-*-* 00:45:00 Europe/Rome"
|
||||
backend_phase1_user_systemd_dir: "{{ backend_phase1_user_home }}/.config/systemd/user"
|
||||
|
||||
@@ -7,27 +7,39 @@
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- not (atlas_manage_media_stack | bool)
|
||||
- backend_phase1_wireguard_interface in ansible_facts.interfaces
|
||||
- backend_phase1_wireguard_address != 'CHANGEME_ATLAS_WIREGUARD_ADDRESS'
|
||||
- backend_phase1_bind_address != 'CHANGEME_ATLAS_BIND_ADDRESS'
|
||||
- backend_phase1_bind_address is match('^[0-9]{1,3}(\.[0-9]{1,3}){3}$')
|
||||
- backend_phase1_firewalld_zone | length > 0
|
||||
- backend_phase1_npm_source_ip != 'CHANGEME_AEGIS_IP'
|
||||
- backend_phase1_syncthing_native_subnet != 'CHANGEME_LAN_SUBNET'
|
||||
- backend_phase1_music_dir.startswith('/')
|
||||
- backend_phase1_archive_dir.startswith('/')
|
||||
- backend_phase1_syncthing_folder_path.startswith(backend_phase1_archive_dir + '/')
|
||||
- backend_phase1_app_data_root.startswith('/')
|
||||
- backend_phase1_navidrome_data_dir.startswith(backend_phase1_app_data_root + '/')
|
||||
- backend_phase1_syncthing_root.startswith(backend_phase1_app_data_root + '/')
|
||||
- >-
|
||||
not (backend_phase1_music_sync_enabled | bool) or
|
||||
(backend_phase1_music_source_dir.startswith(backend_phase1_archive_dir + '/')
|
||||
and backend_phase1_music_sync_calendar | length > 0)
|
||||
fail_msg: >-
|
||||
Disable the rootful media-stack gate and provide the active
|
||||
WireGuard interface/address and absolute ZFS-backed paths before
|
||||
Disable the rootful media-stack gate and provide the Atlas LAN bind
|
||||
address, firewall sources, and absolute ZFS-backed paths before
|
||||
enabling phase one. This role does not manage Prometheus or migrate
|
||||
Navidrome application data.
|
||||
application data.
|
||||
tags: [music_sync]
|
||||
|
||||
- name: Read the rootless service account
|
||||
ansible.builtin.getent:
|
||||
database: passwd
|
||||
key: "{{ backend_phase1_username }}"
|
||||
tags: [music_sync]
|
||||
|
||||
- name: Record rootless service account IDs
|
||||
ansible.builtin.set_fact:
|
||||
backend_phase1_uid: "{{ ansible_facts['getent_passwd'][backend_phase1_username][1] }}"
|
||||
backend_phase1_gid: "{{ ansible_facts['getent_passwd'][backend_phase1_username][2] }}"
|
||||
tags: [music_sync]
|
||||
|
||||
- name: Read system service state before starting rootless Syncthing
|
||||
ansible.builtin.service_facts:
|
||||
@@ -49,6 +61,8 @@
|
||||
loop:
|
||||
- dataset: "{{ backend_phase1_music_dataset }}"
|
||||
mountpoint: "{{ backend_phase1_music_dir }}"
|
||||
- dataset: "{{ backend_phase1_archive_dataset }}"
|
||||
mountpoint: "{{ backend_phase1_archive_dir }}"
|
||||
- dataset: "{{ backend_phase1_app_data_dataset }}"
|
||||
mountpoint: "{{ backend_phase1_app_data_root }}"
|
||||
- dataset: "{{ backend_phase1_navidrome_dataset }}"
|
||||
@@ -58,6 +72,7 @@
|
||||
loop_control:
|
||||
label: "{{ item.dataset }}"
|
||||
register: backend_phase1_zfs_facts
|
||||
tags: [music_sync]
|
||||
|
||||
- name: Require mounted datasets at the declared paths
|
||||
ansible.builtin.assert:
|
||||
@@ -72,6 +87,23 @@
|
||||
loop: "{{ backend_phase1_zfs_facts.results }}"
|
||||
loop_control:
|
||||
label: "{{ item.item.dataset }}"
|
||||
tags: [music_sync]
|
||||
|
||||
- name: Inspect the music copy source
|
||||
ansible.builtin.stat:
|
||||
path: "{{ backend_phase1_music_source_dir }}"
|
||||
register: backend_phase1_music_source_stat
|
||||
when: backend_phase1_music_sync_enabled | bool
|
||||
tags: [music_sync]
|
||||
|
||||
- name: Require an existing music source directory
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- backend_phase1_music_source_stat.stat.isdir | default(false)
|
||||
fail_msg: >-
|
||||
{{ backend_phase1_music_source_dir }} must exist before enabling the daily music copy.
|
||||
when: backend_phase1_music_sync_enabled | bool
|
||||
tags: [music_sync]
|
||||
|
||||
- name: Enable lingering for the rootless service account
|
||||
ansible.builtin.command:
|
||||
@@ -80,12 +112,14 @@
|
||||
- enable-linger
|
||||
- "{{ backend_phase1_username }}"
|
||||
creates: "/var/lib/systemd/linger/{{ backend_phase1_username }}"
|
||||
tags: [music_sync]
|
||||
|
||||
- name: Start the rootless user systemd manager
|
||||
ansible.builtin.systemd:
|
||||
name: "user@{{ backend_phase1_uid }}.service"
|
||||
state: started
|
||||
when: not ansible_check_mode
|
||||
tags: [music_sync]
|
||||
|
||||
- name: Create rootless Quadlet and application directories
|
||||
ansible.builtin.file:
|
||||
@@ -101,9 +135,48 @@
|
||||
mode: "0750"
|
||||
- path: "{{ backend_phase1_syncthing_root }}"
|
||||
mode: "0750"
|
||||
- path: "{{ backend_phase1_syncthing_folder_path }}"
|
||||
mode: "0770"
|
||||
loop_control:
|
||||
label: "{{ item.path }}"
|
||||
|
||||
- name: Install rsync for the daily music copy
|
||||
ansible.builtin.dnf:
|
||||
name: rsync
|
||||
state: present
|
||||
when: backend_phase1_music_sync_enabled | bool
|
||||
tags: [music_sync]
|
||||
|
||||
- name: Create the rootless user systemd directory
|
||||
ansible.builtin.file:
|
||||
path: "{{ backend_phase1_user_systemd_dir }}"
|
||||
state: directory
|
||||
owner: "{{ backend_phase1_username }}"
|
||||
group: "{{ backend_phase1_user_group }}"
|
||||
mode: "0700"
|
||||
when: backend_phase1_music_sync_enabled | bool
|
||||
tags: [music_sync]
|
||||
|
||||
- name: Install the daily music copy service
|
||||
ansible.builtin.template:
|
||||
src: atlas-music-sync.service.j2
|
||||
dest: "{{ backend_phase1_user_systemd_dir }}/atlas-music-sync.service"
|
||||
owner: "{{ backend_phase1_username }}"
|
||||
group: "{{ backend_phase1_user_group }}"
|
||||
mode: "0644"
|
||||
when: backend_phase1_music_sync_enabled | bool
|
||||
tags: [music_sync]
|
||||
|
||||
- name: Install the daily music copy timer
|
||||
ansible.builtin.template:
|
||||
src: atlas-music-sync.timer.j2
|
||||
dest: "{{ backend_phase1_user_systemd_dir }}/atlas-music-sync.timer"
|
||||
owner: "{{ backend_phase1_username }}"
|
||||
group: "{{ backend_phase1_user_group }}"
|
||||
mode: "0644"
|
||||
when: backend_phase1_music_sync_enabled | bool
|
||||
tags: [music_sync]
|
||||
|
||||
- name: Render the rootless Navidrome Quadlet
|
||||
ansible.builtin.template:
|
||||
src: atlas-navidrome.container.j2
|
||||
@@ -131,20 +204,37 @@
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ backend_phase1_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ backend_phase1_uid }}/bus"
|
||||
when: not ansible_check_mode
|
||||
tags: [music_sync]
|
||||
|
||||
- name: Permit phase-one services only through the WireGuard zone
|
||||
- name: Permit NPM access to phase-one web interfaces through Aegis
|
||||
ansible.posix.firewalld:
|
||||
port: "{{ item }}"
|
||||
zone: "{{ backend_phase1_wireguard_firewalld_zone }}"
|
||||
rich_rule: >-
|
||||
rule family="ipv4" source address="{{ backend_phase1_npm_source_ip }}"
|
||||
port port="{{ item }}" protocol="tcp" accept
|
||||
zone: "{{ backend_phase1_firewalld_zone }}"
|
||||
state: enabled
|
||||
permanent: true
|
||||
immediate: true
|
||||
loop:
|
||||
- "{{ backend_phase1_navidrome_port }}/tcp"
|
||||
- "{{ backend_phase1_syncthing_gui_port }}/tcp"
|
||||
- "{{ backend_phase1_syncthing_transfer_port }}/tcp"
|
||||
- "{{ backend_phase1_syncthing_transfer_port }}/udp"
|
||||
- "{{ backend_phase1_syncthing_discovery_port }}/udp"
|
||||
- "{{ backend_phase1_navidrome_port }}"
|
||||
- "{{ backend_phase1_syncthing_gui_port }}"
|
||||
|
||||
- name: Permit native Syncthing traffic from the LAN
|
||||
ansible.posix.firewalld:
|
||||
rich_rule: >-
|
||||
rule family="ipv4" source address="{{ backend_phase1_syncthing_native_subnet }}"
|
||||
port port="{{ item.port }}" protocol="{{ item.protocol }}" accept
|
||||
zone: "{{ backend_phase1_firewalld_zone }}"
|
||||
state: enabled
|
||||
permanent: true
|
||||
immediate: true
|
||||
loop:
|
||||
- port: "{{ backend_phase1_syncthing_transfer_port }}"
|
||||
protocol: tcp
|
||||
- port: "{{ backend_phase1_syncthing_transfer_port }}"
|
||||
protocol: udp
|
||||
- port: "{{ backend_phase1_syncthing_discovery_port }}"
|
||||
protocol: udp
|
||||
|
||||
- name: Start rootless phase-one Quadlets
|
||||
become_user: "{{ backend_phase1_username }}"
|
||||
@@ -161,3 +251,19 @@
|
||||
when:
|
||||
- backend_phase1_start_services | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Enable the daily music copy timer
|
||||
become_user: "{{ backend_phase1_username }}"
|
||||
ansible.builtin.systemd:
|
||||
name: atlas-music-sync.timer
|
||||
scope: user
|
||||
state: started
|
||||
enabled: true
|
||||
daemon_reload: true
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ backend_phase1_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ backend_phase1_uid }}/bus"
|
||||
when:
|
||||
- backend_phase1_music_sync_enabled | bool
|
||||
- not ansible_check_mode
|
||||
tags: [music_sync]
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
# Managed by Ansible. Do not edit manually.
|
||||
[Unit]
|
||||
Description=Copy Atlas Archive music to the Navidrome library
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStartPre=/usr/bin/mountpoint -q {{ backend_phase1_archive_dir }}
|
||||
ExecStartPre=/usr/bin/mountpoint -q {{ backend_phase1_music_dir }}
|
||||
ExecStartPre=/usr/bin/test -d {{ backend_phase1_music_source_dir }}
|
||||
ExecStart=/usr/bin/rsync -aH --no-perms --no-owner --no-group --delay-updates --stats -- {{ backend_phase1_music_source_dir }}/ {{ backend_phase1_music_dir }}/
|
||||
@@ -0,0 +1,11 @@
|
||||
# Managed by Ansible. Do not edit manually.
|
||||
[Unit]
|
||||
Description=Schedule the daily Atlas Navidrome music copy
|
||||
|
||||
[Timer]
|
||||
OnCalendar={{ backend_phase1_music_sync_calendar }}
|
||||
Persistent=true
|
||||
Unit=atlas-music-sync.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -8,7 +8,7 @@ Image={{ backend_phase1_navidrome_image }}
|
||||
UserNS=keep-id
|
||||
User={{ backend_phase1_uid }}
|
||||
Group={{ backend_phase1_gid }}
|
||||
PublishPort={{ backend_phase1_wireguard_address }}:{{ backend_phase1_navidrome_port }}:4533
|
||||
PublishPort={{ backend_phase1_bind_address }}:{{ backend_phase1_navidrome_port }}:4533
|
||||
Environment=ND_LOGLEVEL=info
|
||||
Environment=ND_SCANSCHEDULE=1h
|
||||
Environment=ND_SESSIONTIMEOUT=24h
|
||||
|
||||
@@ -9,15 +9,17 @@ HostName=atlas-syncthing
|
||||
UserNS=keep-id
|
||||
User={{ backend_phase1_uid }}
|
||||
Group={{ backend_phase1_gid }}
|
||||
PublishPort={{ backend_phase1_wireguard_address }}:{{ backend_phase1_syncthing_gui_port }}:{{ backend_phase1_syncthing_gui_port }}
|
||||
PublishPort={{ backend_phase1_wireguard_address }}:{{ backend_phase1_syncthing_transfer_port }}:{{ backend_phase1_syncthing_transfer_port }}
|
||||
PublishPort={{ backend_phase1_wireguard_address }}:{{ backend_phase1_syncthing_transfer_port }}:{{ backend_phase1_syncthing_transfer_port }}/udp
|
||||
PublishPort={{ backend_phase1_wireguard_address }}:{{ backend_phase1_syncthing_discovery_port }}:{{ backend_phase1_syncthing_discovery_port }}/udp
|
||||
PublishPort={{ backend_phase1_bind_address }}:{{ backend_phase1_syncthing_gui_port }}:{{ backend_phase1_syncthing_gui_port }}
|
||||
PublishPort={{ backend_phase1_bind_address }}:{{ backend_phase1_syncthing_transfer_port }}:{{ backend_phase1_syncthing_transfer_port }}
|
||||
PublishPort={{ backend_phase1_bind_address }}:{{ backend_phase1_syncthing_transfer_port }}:{{ backend_phase1_syncthing_transfer_port }}/udp
|
||||
PublishPort={{ backend_phase1_bind_address }}:{{ backend_phase1_syncthing_discovery_port }}:{{ backend_phase1_syncthing_discovery_port }}/udp
|
||||
Environment=HOME=/var/syncthing
|
||||
Environment=STHOMEDIR=/var/syncthing/config
|
||||
Environment=STGUIADDRESS=0.0.0.0:{{ backend_phase1_syncthing_gui_port }}
|
||||
Environment=TZ={{ backend_phase1_timezone }}
|
||||
Volume={{ backend_phase1_syncthing_root }}:/var/syncthing:Z
|
||||
# Archive is intentionally shared with Samba; use a shared SELinux container label.
|
||||
Volume={{ backend_phase1_archive_dir }}:/data:z
|
||||
NoNewPrivileges=true
|
||||
DropCapability=all
|
||||
|
||||
|
||||
103
ansible/roles/profile_server/tasks/backup_export_identity.yml
Normal file
103
ansible/roles/profile_server/tasks/backup_export_identity.yml
Normal file
@@ -0,0 +1,103 @@
|
||||
---
|
||||
- name: Validate Prometheus backup export identity inputs
|
||||
tags: [services, backup, prometheus_backup]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- inventory_hostname == 'prometheus'
|
||||
- server_backup_username is match('^[a-z_][a-z0-9_-]*$')
|
||||
- server_backup_username not in ['root', server_username]
|
||||
- server_backup_export_root.startswith('/var/lib/')
|
||||
- server_backup_public_key_name is match('^[a-z0-9_-]+$')
|
||||
- hostvars['atlas'].atlas_manage_prometheus_backup_pull | default(false) | bool
|
||||
fail_msg: Enable Atlas and Prometheus backup roles together with dedicated identity settings.
|
||||
when: server_backup_export_enabled | bool
|
||||
|
||||
- name: Create dedicated Prometheus backup export group
|
||||
tags: [services, backup, prometheus_backup]
|
||||
ansible.builtin.group:
|
||||
name: "{{ server_backup_username }}"
|
||||
system: true
|
||||
state: present
|
||||
when: server_backup_export_enabled | bool
|
||||
|
||||
- name: Create locked Prometheus backup export account
|
||||
tags: [services, backup, prometheus_backup]
|
||||
ansible.builtin.user:
|
||||
name: "{{ server_backup_username }}"
|
||||
group: "{{ server_backup_username }}"
|
||||
groups: []
|
||||
append: false
|
||||
comment: Read-only prepared backup export for Atlas
|
||||
home: "{{ server_backup_export_root }}"
|
||||
create_home: false
|
||||
shell: /bin/bash
|
||||
password_lock: true
|
||||
system: true
|
||||
state: present
|
||||
when: server_backup_export_enabled | bool
|
||||
|
||||
- name: Require restricted rrsync helper on Prometheus
|
||||
tags: [services, backup, prometheus_backup]
|
||||
ansible.builtin.stat:
|
||||
path: "{{ server_backup_rrsync_path }}"
|
||||
register: server_backup_rrsync_file
|
||||
when: server_backup_export_enabled | bool
|
||||
|
||||
- name: Validate restricted rrsync helper
|
||||
tags: [services, backup, prometheus_backup]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- server_backup_rrsync_file.stat.exists
|
||||
- server_backup_rrsync_file.stat.isreg
|
||||
- server_backup_rrsync_file.stat.pw_name == 'root'
|
||||
fail_msg: Rocky rsync must provide the root-owned rrsync support script.
|
||||
when: server_backup_export_enabled | bool
|
||||
|
||||
- name: Create prepared backup export root
|
||||
tags: [services, backup, prometheus_backup]
|
||||
ansible.builtin.file:
|
||||
path: "{{ server_backup_export_root }}"
|
||||
state: directory
|
||||
owner: root
|
||||
group: "{{ server_backup_username }}"
|
||||
mode: "0750"
|
||||
when: server_backup_export_enabled | bool
|
||||
|
||||
- name: Create restricted Prometheus backup SSH directories
|
||||
tags: [services, backup, prometheus_backup]
|
||||
ansible.builtin.file:
|
||||
path: "{{ item }}"
|
||||
state: directory
|
||||
owner: root
|
||||
group: "{{ server_backup_username }}"
|
||||
mode: "0750"
|
||||
loop:
|
||||
- "{{ server_backup_export_root }}/.ssh"
|
||||
- "{{ server_backup_export_root }}/.ssh/authorized_keys.d"
|
||||
when: server_backup_export_enabled | bool
|
||||
|
||||
- name: Read Atlas public key for Prometheus backup pull
|
||||
tags: [services, backup, prometheus_backup]
|
||||
ansible.builtin.slurp:
|
||||
src: "{{ hostvars['atlas'].atlas_prometheus_pull_private_key_path | default('/etc/atlas-prometheus-pull/id_ed25519') }}.pub"
|
||||
delegate_to: atlas
|
||||
become: true
|
||||
register: server_backup_atlas_public_key
|
||||
when:
|
||||
- server_backup_export_enabled | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Authorize only restricted read-only backup access from Atlas
|
||||
tags: [services, backup, prometheus_backup]
|
||||
ansible.builtin.copy:
|
||||
content: >-
|
||||
{{ 'command="/usr/bin/python3 ' ~ server_backup_rrsync_path ~ ' -ro '
|
||||
~ server_backup_export_root ~ '/versions",restrict '
|
||||
~ (server_backup_atlas_public_key.content | b64decode | trim) ~ '\n' }}
|
||||
dest: "{{ server_backup_export_root }}/.ssh/authorized_keys.d/{{ server_backup_public_key_name }}"
|
||||
owner: root
|
||||
group: "{{ server_backup_username }}"
|
||||
mode: "0640"
|
||||
when:
|
||||
- server_backup_export_enabled | bool
|
||||
- not ansible_check_mode
|
||||
91
ansible/roles/profile_server/tasks/backup_export_job.yml
Normal file
91
ansible/roles/profile_server/tasks/backup_export_job.yml
Normal file
@@ -0,0 +1,91 @@
|
||||
---
|
||||
- name: Validate Prometheus backup export job inputs
|
||||
tags: [services, backup, prometheus_backup]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- server_backup_export_source_keep | int >= 2
|
||||
- server_backup_export_paths | length > 0
|
||||
- server_backup_export_paths | unique | length == server_backup_export_paths | length
|
||||
- >-
|
||||
server_backup_export_paths
|
||||
| select('match', '^[a-zA-Z0-9][a-zA-Z0-9._/-]*$') | list | length
|
||||
== server_backup_export_paths | length
|
||||
- >-
|
||||
server_backup_export_paths
|
||||
| reject('search', '(^|/)\.\.(/|$)') | list | length
|
||||
== server_backup_export_paths | length
|
||||
- >-
|
||||
server_backup_export_excludes
|
||||
| select('match', '^[a-zA-Z0-9][a-zA-Z0-9._/-]*$') | list | length
|
||||
== server_backup_export_excludes | length
|
||||
- >-
|
||||
server_backup_export_excludes
|
||||
| reject('search', '(^|/)\.\.(/|$)') | list | length
|
||||
== server_backup_export_excludes | length
|
||||
fail_msg: Define safe relative paths and at least two prepared export versions.
|
||||
when: server_backup_export_enabled | bool
|
||||
|
||||
- name: Validate Prometheus backup export calendar
|
||||
tags: [services, backup, prometheus_backup]
|
||||
ansible.builtin.command:
|
||||
argv: [systemd-analyze, calendar, "{{ server_backup_export_calendar }}"]
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
when: server_backup_export_enabled | bool
|
||||
|
||||
- name: Ensure prepared Prometheus backup versions directory exists
|
||||
tags: [services, backup, prometheus_backup]
|
||||
ansible.builtin.file:
|
||||
path: "{{ server_backup_export_root }}/versions"
|
||||
state: directory
|
||||
owner: root
|
||||
group: "{{ server_backup_username }}"
|
||||
mode: "0750"
|
||||
when: server_backup_export_enabled | bool
|
||||
|
||||
- name: Install Prometheus backup export helper
|
||||
tags: [services, backup, prometheus_backup, gitea_cutover, npm_quadlet_backup]
|
||||
ansible.builtin.template:
|
||||
src: prometheus-backup-export.sh.j2
|
||||
dest: /usr/local/sbin/prometheus-backup-export
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0750"
|
||||
validate: "bash -n %s"
|
||||
when: server_backup_export_enabled | bool
|
||||
|
||||
- name: Install Prometheus backup export systemd units
|
||||
tags: [services, backup, prometheus_backup]
|
||||
ansible.builtin.template:
|
||||
src: "{{ item }}.j2"
|
||||
dest: "/etc/systemd/system/{{ item }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
loop:
|
||||
- prometheus-backup-export.service
|
||||
- prometheus-backup-export.timer
|
||||
loop_control:
|
||||
label: "{{ item }}"
|
||||
register: server_backup_export_units
|
||||
when: server_backup_export_enabled | bool
|
||||
|
||||
- name: Reload systemd after Prometheus backup export unit changes
|
||||
tags: [services, backup, prometheus_backup]
|
||||
ansible.builtin.systemd:
|
||||
daemon_reload: true
|
||||
when:
|
||||
- server_backup_export_enabled | bool
|
||||
- server_backup_export_units is changed
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Enable Prometheus backup export timer only after explicit activation
|
||||
tags: [services, backup, prometheus_backup]
|
||||
ansible.builtin.systemd:
|
||||
name: prometheus-backup-export.timer
|
||||
enabled: true
|
||||
state: started
|
||||
when:
|
||||
- server_backup_export_enabled | bool
|
||||
- server_backup_export_start_timer | bool
|
||||
- not ansible_check_mode
|
||||
38
ansible/roles/profile_server/tasks/gitea_final_export.yml
Normal file
38
ansible/roles/profile_server/tasks/gitea_final_export.yml
Normal file
@@ -0,0 +1,38 @@
|
||||
---
|
||||
- name: Install the explicit Gitea final-export helper
|
||||
tags: [services, gitea_final_export]
|
||||
ansible.builtin.template:
|
||||
src: prometheus-gitea-final-export.sh.j2
|
||||
dest: /usr/local/sbin/prometheus-gitea-final-export
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0750"
|
||||
when:
|
||||
- server_gitea_cutover_tools_enabled | bool
|
||||
- not server_legacy_stack_retired | bool
|
||||
|
||||
- name: Require the prepared source and explicit final-export approval
|
||||
tags: [services, gitea_final_export]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- server_gitea_cutover_tools_enabled | bool
|
||||
- server_backup_export_enabled | bool
|
||||
- not server_legacy_stack_retired | bool
|
||||
- not ansible_check_mode
|
||||
fail_msg: >-
|
||||
Install the cutover helper and perform an explicit non-check-mode run
|
||||
only after the Gitea outage gate has been approved.
|
||||
when: server_gitea_final_export | bool
|
||||
|
||||
- name: Stop source Gitea and publish the final consistent export
|
||||
tags: [services, gitea_final_export]
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- /usr/local/sbin/prometheus-gitea-final-export
|
||||
register: server_gitea_final_export_result
|
||||
changed_when: server_gitea_final_export_result.rc == 0
|
||||
no_log: true
|
||||
when:
|
||||
- server_gitea_final_export | bool
|
||||
- not server_legacy_stack_retired | bool
|
||||
- not ansible_check_mode
|
||||
59
ansible/roles/profile_server/tasks/gitea_npm_proxy.yml
Normal file
59
ansible/roles/profile_server/tasks/gitea_npm_proxy.yml
Normal file
@@ -0,0 +1,59 @@
|
||||
---
|
||||
- name: Validate the NPM Gitea cutover override
|
||||
tags: [services, gitea_cutover]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- server_gitea_cutover_tools_enabled | bool
|
||||
- server_gitea_npm_domains | length > 0
|
||||
- server_gitea_npm_domains | select('match', '^[a-zA-Z0-9.-]+$') | list | length == server_gitea_npm_domains | length
|
||||
fail_msg: Declare the exact NPM Gitea hostnames before enabling the Atlas upstream.
|
||||
when: server_gitea_on_atlas | bool
|
||||
|
||||
- name: Ensure the NPM custom configuration directory exists
|
||||
tags: [services, gitea_cutover]
|
||||
ansible.builtin.file:
|
||||
path: /opt/npm/data/nginx/custom
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
when: server_gitea_cutover_tools_enabled | bool
|
||||
|
||||
- name: Render the Gitea-only NPM runtime upstream override
|
||||
tags: [services, gitea_cutover]
|
||||
ansible.builtin.template:
|
||||
src: prometheus-gitea-npm-proxy.conf.j2
|
||||
dest: /opt/npm/data/nginx/custom/server_proxy.conf
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
register: server_gitea_npm_override
|
||||
when: server_gitea_on_atlas | bool
|
||||
|
||||
- name: Remove the Gitea NPM override when source routing is selected
|
||||
tags: [services, gitea_cutover]
|
||||
ansible.builtin.file:
|
||||
path: /opt/npm/data/nginx/custom/server_proxy.conf
|
||||
state: absent
|
||||
when:
|
||||
- server_gitea_cutover_tools_enabled | bool
|
||||
- not server_gitea_on_atlas | bool
|
||||
|
||||
- name: Validate NPM configuration after a Gitea upstream change
|
||||
tags: [services, gitea_cutover]
|
||||
ansible.builtin.command:
|
||||
argv: [podman, exec, nginx-proxy-manager, nginx, -t]
|
||||
changed_when: false
|
||||
when:
|
||||
- server_gitea_on_atlas | bool
|
||||
- server_gitea_npm_override is changed
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Reload NPM after validating the Gitea upstream change
|
||||
tags: [services, gitea_cutover]
|
||||
ansible.builtin.command:
|
||||
argv: [podman, exec, nginx-proxy-manager, nginx, -s, reload]
|
||||
when:
|
||||
- server_gitea_on_atlas | bool
|
||||
- server_gitea_npm_override is changed
|
||||
- not ansible_check_mode
|
||||
61
ansible/roles/profile_server/tasks/gitea_ssh_proxy.yml
Normal file
61
ansible/roles/profile_server/tasks/gitea_ssh_proxy.yml
Normal file
@@ -0,0 +1,61 @@
|
||||
---
|
||||
- name: Validate the Prometheus Gitea SSH cutover inputs
|
||||
tags: [services, gitea_cutover]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- server_gitea_cutover_tools_enabled | bool
|
||||
- server_gitea_atlas_address is match('^[0-9]{1,3}(\.[0-9]{1,3}){3}$')
|
||||
- server_gitea_ssh_public_port | int > 1024
|
||||
- server_gitea_ssh_public_port | int < 65536
|
||||
- server_gitea_ssh_target_port | int > 1024
|
||||
- server_gitea_ssh_target_port | int < 65536
|
||||
- server_gitea_ssh_public_port | int != 22
|
||||
fail_msg: Keep administrative SSH on 22 and provide the Atlas rootless Gitea SSH endpoint.
|
||||
when: server_gitea_on_atlas | bool
|
||||
|
||||
- name: Install the Gitea SSH socket proxy units without activating them
|
||||
tags: [services, gitea_cutover]
|
||||
ansible.builtin.template:
|
||||
src: "{{ item }}.j2"
|
||||
dest: "/etc/systemd/system/{{ item }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
loop:
|
||||
- prometheus-gitea-ssh-proxy.socket
|
||||
- prometheus-gitea-ssh-proxy.service
|
||||
loop_control:
|
||||
label: "{{ item }}"
|
||||
register: server_gitea_ssh_proxy_units
|
||||
when: server_gitea_cutover_tools_enabled | bool
|
||||
|
||||
- name: Reload systemd after Gitea SSH proxy unit changes
|
||||
tags: [services, gitea_cutover]
|
||||
ansible.builtin.systemd:
|
||||
daemon_reload: true
|
||||
when:
|
||||
- server_gitea_cutover_tools_enabled | bool
|
||||
- server_gitea_ssh_proxy_units is changed
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Manage the public Gitea SSH socket separately from administrative SSH
|
||||
tags: [services, gitea_cutover]
|
||||
ansible.builtin.systemd:
|
||||
name: prometheus-gitea-ssh-proxy.socket
|
||||
state: "{{ 'started' if server_gitea_on_atlas | bool else 'stopped' }}"
|
||||
enabled: "{{ server_gitea_on_atlas | bool }}"
|
||||
when:
|
||||
- server_gitea_cutover_tools_enabled | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Open only the public Gitea SSH port after cutover
|
||||
tags: [services, gitea_cutover]
|
||||
ansible.posix.firewalld:
|
||||
port: "{{ server_gitea_ssh_public_port }}/tcp"
|
||||
zone: "{{ server_firewalld_zone }}"
|
||||
state: "{{ 'enabled' if server_gitea_on_atlas | bool else 'disabled' }}"
|
||||
permanent: true
|
||||
immediate: true
|
||||
when:
|
||||
- server_gitea_cutover_tools_enabled | bool
|
||||
- server_firewall_backend == 'firewalld'
|
||||
112
ansible/roles/profile_server/tasks/legacy_cleanup.yml
Normal file
112
ansible/roles/profile_server/tasks/legacy_cleanup.yml
Normal file
@@ -0,0 +1,112 @@
|
||||
---
|
||||
- name: Require explicit retirement of the migrated Prometheus source
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- inventory_hostname == 'prometheus'
|
||||
- server_legacy_stack_retired | bool
|
||||
- server_gitea_on_atlas | bool
|
||||
- server_npm_quadlet_cutover | bool
|
||||
- server_backup_export_enabled | bool
|
||||
|
||||
- name: Verify legacy paths have no mounts or container users
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- python3
|
||||
- -c
|
||||
- |
|
||||
import json, os, pathlib, subprocess
|
||||
def run(*args):
|
||||
return subprocess.check_output(args, text=True).strip()
|
||||
paths = ['/opt/gitea', '/home/git/.ssh', '/opt/navidrome',
|
||||
'/opt/postgres', '/opt/music', '/opt/containerd', '/opt/docker']
|
||||
mounts = json.loads(run('findmnt', '--json', '--list', '-o', 'TARGET'))['filesystems']
|
||||
for path in paths:
|
||||
assert os.path.realpath(path) == path, 'Symlink in cleanup path: ' + path
|
||||
for mount in mounts:
|
||||
target = mount['target']
|
||||
assert target != path and not target.startswith(path + '/'), 'Mounted cleanup path: ' + path
|
||||
ids = run('podman', 'ps', '-aq').split()
|
||||
containers = json.loads(run('podman', 'inspect', *ids)) if ids else []
|
||||
for container in containers:
|
||||
assert container['Name'].lstrip('/') == 'nginx-proxy-manager', 'Unexpected container; review before cleanup'
|
||||
for mount in container.get('Mounts', []):
|
||||
source = os.path.realpath(mount['Source'])
|
||||
for path in paths:
|
||||
assert source != path and not source.startswith(path + '/'), 'Container uses cleanup path: ' + path
|
||||
for path in ['/opt/music', '/opt/containerd']:
|
||||
if os.path.isdir(path):
|
||||
for entry in pathlib.Path(path).rglob('*'):
|
||||
assert entry.is_dir() and not entry.is_symlink(), 'Unexpected file in empty legacy path: ' + str(entry)
|
||||
if os.path.isdir('/opt/docker'):
|
||||
allowed = {'/opt/docker/server', '/opt/docker/server/docker-compose.yml'}
|
||||
for entry in pathlib.Path('/opt/docker').rglob('*'):
|
||||
assert str(entry) in allowed and not entry.is_symlink(), 'Unexpected legacy Docker content: ' + str(entry)
|
||||
assert run('systemctl', 'is-active', 'prometheus-npm.service') == 'active'
|
||||
assert subprocess.run(['systemctl', 'is-active', '--quiet', 'podman-compose-server.service']).returncode != 0
|
||||
assert subprocess.run(['systemctl', 'is-active', '--quiet', 'prometheus-backup-export.service']).returncode != 0
|
||||
print('Legacy cleanup preflight passed')
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
|
||||
- name: Require the updated backup configuration before deleting fallback files
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- python3
|
||||
- -c
|
||||
- |
|
||||
import pathlib, subprocess
|
||||
unit = subprocess.check_output(['systemctl', 'show', 'prometheus-backup-export.service',
|
||||
'-p', 'RequiresMountsFor', '--value'], text=True)
|
||||
assert '/opt/gitea' not in unit, 'Backup unit still depends on legacy Gitea'
|
||||
helper = pathlib.Path('/usr/local/sbin/prometheus-backup-export').read_text()
|
||||
assert 'podman-compose-server' not in helper and 'opt/docker/server' not in helper
|
||||
subprocess.run(['bash', '-n', '/usr/local/sbin/prometheus-backup-export'], check=True)
|
||||
changed_when: false
|
||||
when: not ansible_check_mode
|
||||
|
||||
- name: Delete only the explicitly approved legacy data and fallback files
|
||||
ansible.builtin.file:
|
||||
path: "{{ item }}"
|
||||
state: absent
|
||||
loop:
|
||||
- /opt/gitea
|
||||
- /home/git/.ssh
|
||||
- /opt/navidrome
|
||||
- /opt/postgres
|
||||
- /opt/music
|
||||
- /opt/containerd
|
||||
- /opt/docker
|
||||
- /usr/local/sbin/prometheus-gitea-final-export
|
||||
- /etc/systemd/system/podman-compose-server.service
|
||||
register: server_legacy_deleted
|
||||
diff: false
|
||||
|
||||
- name: Reload systemd after removing the inactive legacy unit
|
||||
ansible.builtin.systemd:
|
||||
daemon_reload: true
|
||||
when:
|
||||
- server_legacy_deleted is changed
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Inspect the obsolete Git home without following symlinks
|
||||
ansible.builtin.stat:
|
||||
path: /home/git
|
||||
follow: false
|
||||
register: server_legacy_git_home
|
||||
|
||||
- name: Require the obsolete Git account to be absent before removing its empty home
|
||||
ansible.builtin.command:
|
||||
argv: [getent, passwd, git]
|
||||
register: server_legacy_git_account
|
||||
changed_when: false
|
||||
failed_when: server_legacy_git_account.rc != 2
|
||||
check_mode: false
|
||||
when: server_legacy_git_home.stat.exists
|
||||
|
||||
# rmdir refuses any nonempty directory; never recursively delete this parent.
|
||||
- name: Remove only the empty obsolete Git home
|
||||
ansible.builtin.command:
|
||||
argv: [rmdir, /home/git]
|
||||
register: server_legacy_git_home_removed
|
||||
changed_when: server_legacy_git_home_removed.rc == 0
|
||||
when: server_legacy_git_home.stat.exists
|
||||
33
ansible/roles/profile_server/tasks/legacy_image_cleanup.yml
Normal file
33
ansible/roles/profile_server/tasks/legacy_image_cleanup.yml
Normal file
@@ -0,0 +1,33 @@
|
||||
---
|
||||
- name: Require the migrated Prometheus topology for image cleanup
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- inventory_hostname == 'prometheus'
|
||||
- server_gitea_on_atlas | bool
|
||||
- server_npm_quadlet_cutover | bool
|
||||
- server_legacy_images | default([]) | length > 0
|
||||
- >-
|
||||
server_legacy_images | difference([
|
||||
'docker.gitea.com/gitea:1.25.2',
|
||||
'docker.io/deluan/navidrome:latest',
|
||||
'docker.io/library/postgres:13']) | length == 0
|
||||
|
||||
- name: Check whether the explicitly selected legacy images exist
|
||||
ansible.builtin.command:
|
||||
argv: [podman, image, exists, "{{ item }}"]
|
||||
loop: "{{ server_legacy_images }}"
|
||||
register: server_legacy_image_presence
|
||||
changed_when: false
|
||||
failed_when: server_legacy_image_presence.rc not in [0, 1]
|
||||
check_mode: false
|
||||
|
||||
# No --force: Podman must refuse images referenced by any existing container.
|
||||
- name: Remove only unused explicitly selected legacy images
|
||||
ansible.builtin.command:
|
||||
argv: [podman, image, rm, "{{ item.item }}"]
|
||||
loop: "{{ server_legacy_image_presence.results }}"
|
||||
loop_control:
|
||||
label: "{{ item.item }}"
|
||||
when: item.rc == 0
|
||||
register: server_legacy_image_removal
|
||||
changed_when: server_legacy_image_removal.rc == 0
|
||||
@@ -11,6 +11,7 @@
|
||||
- name: Configure DuckDNS updater
|
||||
tags: [dotfiles, dotfiles:server, duckdns]
|
||||
ansible.builtin.import_tasks: duckdns.yml
|
||||
when: server_duckdns_enabled | bool
|
||||
|
||||
- name: Ensure server directories exist
|
||||
tags: [dotfiles, services]
|
||||
@@ -20,17 +21,12 @@
|
||||
owner: "{{ item.owner }}"
|
||||
group: "{{ item.group }}"
|
||||
mode: "{{ item.mode }}"
|
||||
loop: >-
|
||||
{{
|
||||
(server_directories | default([]))
|
||||
+ (
|
||||
server_syncthing_directories | default([])
|
||||
if server_syncthing_enabled | default(true) | bool
|
||||
else []
|
||||
)
|
||||
}}
|
||||
loop: "{{ server_directories | default([]) }}"
|
||||
loop_control:
|
||||
label: "{{ item.path }}"
|
||||
when:
|
||||
- item.path != '/opt/gitea/data' or not server_gitea_on_atlas | bool
|
||||
- item.path != server_container_stack_dir or not server_legacy_stack_retired | bool
|
||||
|
||||
- name: Copy server dotfiles
|
||||
tags: [dotfiles, dotfiles:server]
|
||||
@@ -45,7 +41,7 @@
|
||||
label: "{{ item.dest }}"
|
||||
|
||||
- name: Render server templates
|
||||
tags: [dotfiles, dotfiles:server]
|
||||
tags: [dotfiles, dotfiles:server, gitea_cutover]
|
||||
ansible.builtin.template:
|
||||
src: "{{ item.src }}"
|
||||
dest: "{{ item.dest if item.dest.startswith('/') else server_user_home ~ '/' ~ item.dest }}"
|
||||
@@ -56,10 +52,41 @@
|
||||
loop_control:
|
||||
label: "{{ item.dest }}"
|
||||
no_log: "{{ item.no_log | default(false) }}"
|
||||
when: item.src != 'server/docker-compose.yml.j2' or not server_legacy_stack_retired | bool
|
||||
|
||||
- name: Manage Podman Compose stack
|
||||
tags: [services, podman]
|
||||
ansible.builtin.include_tasks: podman-compose.yml
|
||||
when: not server_legacy_stack_retired | bool
|
||||
|
||||
- name: Import staged NPM Quadlet tasks
|
||||
ansible.builtin.import_tasks: npm_quadlet.yml
|
||||
|
||||
- name: Import explicit legacy server image cleanup
|
||||
ansible.builtin.import_tasks: legacy_image_cleanup.yml
|
||||
tags: [never, server_image_cleanup]
|
||||
when: server_legacy_image_cleanup | default(false) | bool
|
||||
|
||||
- name: Import Prometheus backup export identity tasks
|
||||
ansible.builtin.import_tasks: backup_export_identity.yml
|
||||
|
||||
- name: Import Prometheus backup export job tasks
|
||||
ansible.builtin.import_tasks: backup_export_job.yml
|
||||
tags: [server_legacy_cleanup]
|
||||
|
||||
- name: Import explicitly approved legacy server data cleanup
|
||||
ansible.builtin.import_tasks: legacy_cleanup.yml
|
||||
tags: [never, server_legacy_cleanup]
|
||||
when: server_legacy_cleanup | bool
|
||||
|
||||
- name: Import explicit Prometheus Gitea final-export tasks
|
||||
ansible.builtin.import_tasks: gitea_final_export.yml
|
||||
|
||||
- name: Import Prometheus Gitea SSH proxy tasks
|
||||
ansible.builtin.import_tasks: gitea_ssh_proxy.yml
|
||||
|
||||
- name: Import Prometheus Gitea NPM proxy override tasks
|
||||
ansible.builtin.import_tasks: gitea_npm_proxy.yml
|
||||
|
||||
- name: Ensure server SSH authorized key fragments directory exists
|
||||
tags: [services, ssh]
|
||||
@@ -85,13 +112,17 @@
|
||||
when: server_ssh_authorized_keys | length > 0
|
||||
|
||||
- name: Configure server SSH authorized key fragments
|
||||
tags: [services, ssh]
|
||||
tags: [services, ssh, prometheus_backup]
|
||||
ansible.builtin.lineinfile:
|
||||
path: /etc/ssh/sshd_config
|
||||
regexp: '^\s*AuthorizedKeysFile\s+'
|
||||
line: >-
|
||||
AuthorizedKeysFile {{ server_ssh_authorized_keys | map(attribute='name')
|
||||
| map('regex_replace', '^', '%h/.ssh/authorized_keys.d/') | join(' ') }}
|
||||
AuthorizedKeysFile {{
|
||||
((server_ssh_authorized_keys | map(attribute='name')
|
||||
| map('regex_replace', '^', '%h/.ssh/authorized_keys.d/') | list)
|
||||
+ (['%h/.ssh/authorized_keys.d/' ~ server_backup_public_key_name]
|
||||
if server_backup_export_enabled | bool else [])) | join(' ')
|
||||
}}
|
||||
state: present
|
||||
validate: "sshd -t -f %s"
|
||||
notify: Reload SSH service
|
||||
@@ -108,11 +139,13 @@
|
||||
notify: Reload SSH service
|
||||
|
||||
- name: Restrict SSH login to allowed users on server
|
||||
tags: [services]
|
||||
tags: [services, prometheus_backup]
|
||||
ansible.builtin.lineinfile:
|
||||
path: /etc/ssh/sshd_config
|
||||
regexp: '^\s*AllowUsers\s+'
|
||||
line: "AllowUsers {{ server_sshd_allow_users | join(' ') }}"
|
||||
line: >-
|
||||
AllowUsers {{ (server_sshd_allow_users
|
||||
+ ([server_backup_username] if server_backup_export_enabled | bool else [])) | join(' ') }}
|
||||
state: present
|
||||
validate: "sshd -t -f %s"
|
||||
notify: Reload SSH service
|
||||
|
||||
71
ansible/roles/profile_server/tasks/npm_quadlet.yml
Normal file
71
ansible/roles/profile_server/tasks/npm_quadlet.yml
Normal file
@@ -0,0 +1,71 @@
|
||||
---
|
||||
- name: Require staged NPM Quadlet for an active cutover
|
||||
tags: [services, npm_quadlet]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- not server_npm_quadlet_cutover | bool or server_npm_quadlet_stage | bool
|
||||
fail_msg: The NPM Quadlet cutover requires the staged container and network.
|
||||
|
||||
- name: Validate staged NPM Quadlet inputs
|
||||
tags: [services, npm_quadlet]
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- server_npm_quadlet_image is defined
|
||||
- server_npm_quadlet_image is match('^docker\.io/jc21/nginx-proxy-manager@sha256:[a-f0-9]{64}$')
|
||||
- server_gitea_on_atlas | bool
|
||||
fail_msg: Stage the exact running NPM image only after Gitea has left Compose.
|
||||
when: server_npm_quadlet_stage | bool
|
||||
|
||||
- name: Ensure rootful Quadlet directory exists for NPM
|
||||
tags: [services, npm_quadlet]
|
||||
ansible.builtin.file:
|
||||
path: /etc/containers/systemd
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
when: server_npm_quadlet_stage | bool
|
||||
|
||||
- name: Render staged NPM container and network Quadlets
|
||||
tags: [services, npm_quadlet]
|
||||
ansible.builtin.template:
|
||||
src: "{{ item }}.j2"
|
||||
dest: "/etc/containers/systemd/{{ item }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
loop:
|
||||
- prometheus-npm.container
|
||||
- server-web.network
|
||||
loop_control:
|
||||
label: "{{ item }}"
|
||||
register: server_npm_quadlet_units
|
||||
when: server_npm_quadlet_stage | bool
|
||||
|
||||
- name: Reload systemd after staging NPM Quadlets
|
||||
tags: [services, npm_quadlet]
|
||||
ansible.builtin.systemd:
|
||||
daemon_reload: true
|
||||
when:
|
||||
- server_npm_quadlet_stage | bool
|
||||
- server_npm_quadlet_units is changed
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Verify the staged NPM Quadlet was generated
|
||||
tags: [services, npm_quadlet]
|
||||
ansible.builtin.command:
|
||||
argv: [systemctl, show, prometheus-npm.service, --property=LoadState, --value]
|
||||
register: server_npm_quadlet_load_state
|
||||
changed_when: false
|
||||
when:
|
||||
- server_npm_quadlet_stage | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Reject an invalid staged NPM Quadlet
|
||||
tags: [services, npm_quadlet]
|
||||
ansible.builtin.assert:
|
||||
that: server_npm_quadlet_load_state.stdout == 'loaded'
|
||||
fail_msg: Quadlet generator did not produce prometheus-npm.service.
|
||||
when:
|
||||
- server_npm_quadlet_stage | bool
|
||||
- not ansible_check_mode
|
||||
@@ -0,0 +1,15 @@
|
||||
[Unit]
|
||||
Description=Prepare a read-only Prometheus application backup for Atlas
|
||||
RequiresMountsFor=/opt/npm {% if not server_gitea_on_atlas | bool %}/opt/gitea {% endif %}{{ server_backup_export_root }}
|
||||
ConditionFileIsExecutable=/usr/local/sbin/prometheus-backup-export
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/prometheus-backup-export
|
||||
User=root
|
||||
Group=root
|
||||
UMask=0077
|
||||
TimeoutStartSec=infinity
|
||||
Nice=10
|
||||
IOSchedulingClass=best-effort
|
||||
IOSchedulingPriority=7
|
||||
@@ -0,0 +1,118 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
|
||||
export_root={{ server_backup_export_root | quote }}
|
||||
versions="$export_root/versions"
|
||||
{% if server_legacy_stack_retired | bool %}
|
||||
stack_unit=prometheus-npm.service
|
||||
{% else %}
|
||||
stack_unit=''
|
||||
compose_active=false
|
||||
quadlet_active=false
|
||||
systemctl is-active --quiet podman-compose-server.service && compose_active=true
|
||||
systemctl is-active --quiet prometheus-npm.service && quadlet_active=true
|
||||
if [[ "$compose_active" == "$quadlet_active" ]]; then
|
||||
echo 'Expected exactly one active NPM service (Compose or Quadlet)' >&2
|
||||
exit 1
|
||||
fi
|
||||
if "$quadlet_active"; then
|
||||
stack_unit=prometheus-npm.service
|
||||
else
|
||||
stack_unit=podman-compose-server.service
|
||||
fi
|
||||
{% endif %}
|
||||
stamp=$(date -u +%Y%m%dT%H%M%SZ)
|
||||
stage=''
|
||||
stack_stopped=false
|
||||
|
||||
exec 9>/run/lock/prometheus-backup-export.lock
|
||||
flock -n 9 || { echo 'A backup export is already running' >&2; exit 1; }
|
||||
|
||||
cleanup() {
|
||||
local rc=$?
|
||||
trap - EXIT
|
||||
if "$stack_stopped"; then
|
||||
if systemctl is-active --quiet "$stack_unit"; then
|
||||
systemctl restart "$stack_unit" || rc=1
|
||||
else
|
||||
systemctl start "$stack_unit" || rc=1
|
||||
fi
|
||||
fi
|
||||
if (( rc != 0 )) && [[ -n "$stage" && -d "$stage" ]]; then
|
||||
rm -rf -- "$stage"
|
||||
fi
|
||||
exit "$rc"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
trap 'exit 129' HUP
|
||||
trap 'exit 130' INT
|
||||
trap 'exit 143' TERM
|
||||
|
||||
systemctl is-active --quiet "$stack_unit" || {
|
||||
echo "The managed NPM unit $stack_unit must be active before preparing a backup" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
paths=(
|
||||
{% for path in server_backup_export_paths %}
|
||||
{{ path | quote }}
|
||||
{% endfor %}
|
||||
)
|
||||
excludes=(
|
||||
{% for path in server_backup_export_excludes %}
|
||||
--exclude={{ path | quote }}
|
||||
{% endfor %}
|
||||
)
|
||||
for path in "${paths[@]}"; do
|
||||
[[ -e "/$path" ]] || { echo "Required backup path missing: /$path" >&2; exit 1; }
|
||||
done
|
||||
[[ ! -e "$versions/$stamp" ]] || { echo "Export version already exists: $stamp" >&2; exit 1; }
|
||||
stage=$(mktemp -d "$export_root/.staging.XXXXXXXX")
|
||||
|
||||
# SQLite databases and their accompanying files are copied while both
|
||||
# managed containers are stopped. The EXIT trap restarts the stack on error.
|
||||
stack_stopped=true
|
||||
systemctl stop "$stack_unit"
|
||||
tar --acls --xattrs --selinux "${excludes[@]}" -C / -cf "$stage/payload.tar" "${paths[@]}"
|
||||
systemctl start "$stack_unit"
|
||||
for container in nginx-proxy-manager{% if not server_gitea_on_atlas | bool %} gitea{% endif %}; do
|
||||
running=false
|
||||
for _ in {1..30}; do
|
||||
if [[ $(podman inspect --format '{{ '{{.State.Running}}' }}' "$container" 2>/dev/null) == true ]]; then
|
||||
running=true
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
"$running" || { echo "Container did not restart: $container" >&2; exit 1; }
|
||||
done
|
||||
ready=false
|
||||
for _ in {1..60}; do
|
||||
if curl -fsS --connect-timeout 2 --max-time 3 -o /dev/null http://127.0.0.1:81/; then
|
||||
ready=true
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
"$ready" || { echo 'NPM administration did not become ready after backup' >&2; exit 1; }
|
||||
stack_stopped=false
|
||||
|
||||
tar -tf "$stage/payload.tar" >/dev/null
|
||||
(cd "$stage" && sha256sum payload.tar >payload.sha256)
|
||||
printf '{"schema":1,"host":"prometheus","created_utc":"%s"}\n' "$stamp" >"$stage/metadata.json"
|
||||
chown root:{{ server_backup_username }} "$stage" "$stage/payload.tar" "$stage/payload.sha256" "$stage/metadata.json"
|
||||
chmod 0750 "$stage"
|
||||
chmod 0640 "$stage/payload.tar" "$stage/payload.sha256" "$stage/metadata.json"
|
||||
mv -- "$stage" "$versions/$stamp"
|
||||
stage=''
|
||||
ln -s "$stamp" "$versions/.current.new"
|
||||
mv -Tf -- "$versions/.current.new" "$versions/current"
|
||||
|
||||
# Keep a small source-side safety window; Atlas owns long-term retention.
|
||||
mapfile -t old_versions < <(find "$versions" -mindepth 1 -maxdepth 1 -type d \
|
||||
-printf '%f\n' | grep -E '^[0-9]{8}T[0-9]{6}Z$' | sort -r | tail -n +{{ server_backup_export_source_keep + 1 }})
|
||||
for old in "${old_versions[@]}"; do
|
||||
rm -rf -- "${versions:?}/$old"
|
||||
done
|
||||
echo "Prepared Prometheus backup export $stamp"
|
||||
@@ -0,0 +1,10 @@
|
||||
[Unit]
|
||||
Description=Prepare daily Prometheus application backup for Atlas
|
||||
|
||||
[Timer]
|
||||
OnCalendar={{ server_backup_export_calendar }}
|
||||
Persistent=false
|
||||
Unit=prometheus-backup-export.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
|
||||
export_root={{ server_backup_export_root | quote }}
|
||||
versions="$export_root/versions"
|
||||
stamp=$(date -u +%Y%m%dT%H%M%SZ)
|
||||
stage=''
|
||||
gitea_stopped=false
|
||||
|
||||
exec 9>/run/lock/prometheus-backup-export.lock
|
||||
flock -n 9 || { echo 'A Prometheus backup export is already running' >&2; exit 1; }
|
||||
|
||||
cleanup() {
|
||||
local rc=$?
|
||||
trap - EXIT
|
||||
if (( rc != 0 )) && "$gitea_stopped"; then
|
||||
podman start gitea >/dev/null || rc=1
|
||||
fi
|
||||
if (( rc != 0 )) && [[ -n "$stage" && -d "$stage" ]]; then
|
||||
rm -rf -- "$stage"
|
||||
fi
|
||||
exit "$rc"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
trap 'exit 129' HUP
|
||||
trap 'exit 130' INT
|
||||
trap 'exit 143' TERM
|
||||
|
||||
systemctl is-active --quiet podman-compose-server.service || {
|
||||
echo 'Prometheus Compose stack is not active' >&2; exit 1;
|
||||
}
|
||||
if systemctl is-active --quiet prometheus-backup-export.timer; then
|
||||
echo 'Stop the scheduled export timer for the cutover first' >&2
|
||||
exit 1
|
||||
fi
|
||||
[[ $(podman inspect --format '{{ '{{.State.Running}}' }}' gitea) == true ]] || {
|
||||
echo 'Source Gitea must be running before the final export' >&2; exit 1;
|
||||
}
|
||||
[[ -d /opt/gitea/data && -d /home/git/.ssh ]] || {
|
||||
echo 'Required source Gitea paths are missing' >&2; exit 1;
|
||||
}
|
||||
[[ ! -e "$versions/$stamp" ]] || {
|
||||
echo 'Final export timestamp already exists' >&2; exit 1;
|
||||
}
|
||||
|
||||
gitea_stopped=true
|
||||
podman stop --time 30 gitea >/dev/null
|
||||
[[ $(podman inspect --format '{{ '{{.State.Running}}' }}' gitea) == false ]] || {
|
||||
echo 'Source Gitea did not stop' >&2; exit 1;
|
||||
}
|
||||
python3 - <<'PY'
|
||||
import sqlite3
|
||||
path = '/opt/gitea/data/gitea/gitea.db'
|
||||
with sqlite3.connect(f'file:{path}?mode=ro', uri=True) as database:
|
||||
if database.execute('PRAGMA quick_check').fetchone()[0] != 'ok':
|
||||
raise SystemExit('Source Gitea SQLite quick_check failed')
|
||||
PY
|
||||
|
||||
stage=$(mktemp -d "$export_root/.staging.XXXXXXXX")
|
||||
tar --acls --xattrs --selinux -C / -cf "$stage/payload.tar" \
|
||||
opt/gitea/data home/git/.ssh
|
||||
tar -tf "$stage/payload.tar" >/dev/null
|
||||
(cd "$stage" && sha256sum payload.tar >payload.sha256)
|
||||
printf '{"schema":1,"host":"prometheus","purpose":"gitea-cutover","created_utc":"%s"}\n' \
|
||||
"$stamp" >"$stage/metadata.json"
|
||||
|
||||
[[ $(podman inspect --format '{{ '{{.State.Running}}' }}' gitea) == false ]] || {
|
||||
echo 'Source Gitea restarted during final export' >&2; exit 1;
|
||||
}
|
||||
chown root:{{ server_backup_username }} "$stage" "$stage/payload.tar" \
|
||||
"$stage/payload.sha256" "$stage/metadata.json"
|
||||
chmod 0750 "$stage"
|
||||
chmod 0640 "$stage/payload.tar" "$stage/payload.sha256" "$stage/metadata.json"
|
||||
mv -- "$stage" "$versions/$stamp"
|
||||
stage=''
|
||||
ln -s "$stamp" "$versions/.current.new"
|
||||
mv -Tf -- "$versions/.current.new" "$versions/current"
|
||||
|
||||
echo "Prepared final Gitea export $stamp; source Gitea remains stopped"
|
||||
@@ -0,0 +1,6 @@
|
||||
# Managed by Ansible. NPM's variable proxy upstream uses Nginx DNS, not /etc/hosts.
|
||||
{% for domain in server_gitea_npm_domains %}
|
||||
if ($host = {{ domain }}) {
|
||||
set $server {{ server_gitea_atlas_address }};
|
||||
}
|
||||
{% endfor %}
|
||||
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Forward public Gitea SSH to Atlas through Aegis
|
||||
Requires=prometheus-gitea-ssh-proxy.socket
|
||||
After=network-online.target wg-quick@wg0.service
|
||||
|
||||
[Service]
|
||||
ExecStart=/usr/lib/systemd/systemd-socket-proxyd {{ server_gitea_atlas_address }}:{{ server_gitea_ssh_target_port }}
|
||||
DynamicUser=true
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
@@ -0,0 +1,9 @@
|
||||
[Unit]
|
||||
Description=Public Gitea SSH socket on Prometheus
|
||||
|
||||
[Socket]
|
||||
ListenStream=0.0.0.0:{{ server_gitea_ssh_public_port }}
|
||||
NoDelay=true
|
||||
|
||||
[Install]
|
||||
WantedBy=sockets.target
|
||||
@@ -0,0 +1,26 @@
|
||||
[Unit]
|
||||
Description=Nginx Proxy Manager on Prometheus
|
||||
RequiresMountsFor=/opt/npm/data /opt/npm/letsencrypt
|
||||
|
||||
[Container]
|
||||
Image={{ server_npm_quadlet_image }}
|
||||
ContainerName=nginx-proxy-manager
|
||||
Network=server-web.network
|
||||
NetworkAlias=nginx-proxy-manager
|
||||
AddHost=host.containers.internal:host-gateway
|
||||
PublishPort=80:80
|
||||
PublishPort=443:443
|
||||
PublishPort=127.0.0.1:81:81
|
||||
Volume=/opt/npm/data:/data
|
||||
Volume=/opt/npm/letsencrypt:/etc/letsencrypt
|
||||
Pull=missing
|
||||
|
||||
[Service]
|
||||
Restart=always
|
||||
TimeoutStartSec=180
|
||||
TimeoutStopSec=120
|
||||
|
||||
{% if server_npm_quadlet_cutover | bool %}
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
{% endif %}
|
||||
@@ -0,0 +1,5 @@
|
||||
[Network]
|
||||
NetworkName=server_web
|
||||
Driver=bridge
|
||||
Subnet=10.89.0.0/24
|
||||
Gateway=10.89.0.1
|
||||
@@ -66,6 +66,17 @@
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
|
||||
- name: Check whether the pinned Java 25 version is installed with Mise
|
||||
tags: [packages, mise, java, wsl]
|
||||
ansible.builtin.command:
|
||||
cmd: "mise where java@{{ workstation_mise_java_25_version }}"
|
||||
become_user: "{{ username }}"
|
||||
environment:
|
||||
HOME: "{{ user_home }}"
|
||||
register: workstation_mise_java_25_where
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
|
||||
- name: Check whether the pinned Maven version is installed with Mise
|
||||
tags: [packages, mise, maven, wsl]
|
||||
ansible.builtin.command:
|
||||
@@ -86,6 +97,7 @@
|
||||
HOME: "{{ user_home }}"
|
||||
when: >-
|
||||
workstation_mise_java_where.rc != 0 or
|
||||
workstation_mise_java_25_where.rc != 0 or
|
||||
workstation_mise_maven_where.rc != 0
|
||||
|
||||
- name: Ensure WSL boot configuration file exists
|
||||
|
||||
@@ -10,6 +10,12 @@ wireguard_mtu: 1420
|
||||
wireguard_firewalld_zone: wireguard
|
||||
wireguard_public_firewalld_zone: public
|
||||
wireguard_enable_ipv4_forwarding: false
|
||||
# Legacy zone-level masquerading; policy-level masquerading is required for inter-zone forwarding.
|
||||
wireguard_enable_masquerade: false
|
||||
# A list of narrowly scoped inter-zone forwarding rules, rendered as persistent
|
||||
# firewalld policies. Each item requires name, ingress_zone, egress_zone,
|
||||
# source, and destination.
|
||||
wireguard_forwarding_policies: []
|
||||
wireguard_reload_rootful_podman_networks: false
|
||||
wireguard_handshake_retries: 12
|
||||
wireguard_handshake_delay: 5
|
||||
|
||||
@@ -14,10 +14,44 @@
|
||||
Configure this host's WireGuard address and peers, and run the first
|
||||
key bootstrap against every peer in the same play.
|
||||
|
||||
- name: Install WireGuard userspace tools
|
||||
- name: Validate WireGuard forwarding policies
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- item.name is defined
|
||||
- item.ingress_zone is defined
|
||||
- item.egress_zone is defined
|
||||
- item.source is defined
|
||||
- item.destination is defined
|
||||
fail_msg: >-
|
||||
Every WireGuard forwarding policy requires name, ingress_zone,
|
||||
egress_zone, source, and destination.
|
||||
loop: "{{ wireguard_forwarding_policies }}"
|
||||
loop_control:
|
||||
label: "{{ item.name | default('unnamed policy') }}"
|
||||
|
||||
- name: Install WireGuard userspace tools on mutable hosts
|
||||
ansible.builtin.dnf:
|
||||
name: "{{ wireguard_packages }}"
|
||||
state: present
|
||||
when: "'platform_fedora_iot' not in group_names"
|
||||
|
||||
- name: Require WireGuard userspace tools in the booted deployment
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- wg
|
||||
- --version
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
register: wireguard_userspace_tools
|
||||
|
||||
- name: Require active WireGuard userspace tools
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- wireguard_userspace_tools.rc == 0
|
||||
fail_msg: >-
|
||||
WireGuard userspace tools are not present in the booted deployment.
|
||||
On Fedora IoT, reboot after rpm-ostree layers wireguard-tools, then
|
||||
rerun the WireGuard play.
|
||||
|
||||
- name: Create private WireGuard configuration directory
|
||||
ansible.builtin.file:
|
||||
@@ -124,12 +158,33 @@
|
||||
permanent: true
|
||||
register: wireguard_firewalld_zone_result
|
||||
|
||||
- name: Reload firewalld after creating the WireGuard zone
|
||||
- name: Create the firewalld policy directory
|
||||
ansible.builtin.file:
|
||||
path: /etc/firewalld/policies
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
when: wireguard_forwarding_policies | length > 0
|
||||
|
||||
- name: Render WireGuard forwarding policies
|
||||
ansible.builtin.template:
|
||||
src: wireguard-forwarding-policy.xml.j2
|
||||
dest: "/etc/firewalld/policies/{{ item.name }}.xml"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
loop: "{{ wireguard_forwarding_policies }}"
|
||||
loop_control:
|
||||
label: "{{ item.name }}"
|
||||
register: wireguard_forwarding_policy_result
|
||||
|
||||
- name: Reload firewalld after WireGuard firewall changes
|
||||
ansible.builtin.systemd:
|
||||
name: firewalld.service
|
||||
state: reloaded
|
||||
when:
|
||||
- wireguard_firewalld_zone_result is changed
|
||||
- wireguard_firewalld_zone_result is changed or (wireguard_forwarding_policy_result | default({})) is changed
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Restore rootful Podman networking after firewalld reload
|
||||
@@ -142,7 +197,7 @@
|
||||
register: wireguard_podman_network_reload
|
||||
changed_when: wireguard_podman_network_reload.stdout_lines | length > 0
|
||||
when:
|
||||
- wireguard_firewalld_zone_result is changed
|
||||
- wireguard_firewalld_zone_result is changed or (wireguard_forwarding_policy_result | default({})) is changed
|
||||
- wireguard_reload_rootful_podman_networks | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
@@ -154,6 +209,14 @@
|
||||
permanent: true
|
||||
immediate: true
|
||||
|
||||
- name: Manage legacy WireGuard zone masquerading
|
||||
ansible.posix.firewalld:
|
||||
zone: "{{ wireguard_firewalld_zone }}"
|
||||
masquerade: true
|
||||
state: "{{ 'enabled' if wireguard_enable_masquerade | bool else 'disabled' }}"
|
||||
permanent: true
|
||||
immediate: true
|
||||
|
||||
- name: Permit this host's public WireGuard listener
|
||||
ansible.posix.firewalld:
|
||||
port: "{{ wireguard_listen_port }}/udp"
|
||||
|
||||
@@ -11,7 +11,8 @@ ListenPort = {{ wireguard_listen_port }}
|
||||
{% for peer in wireguard_peers %}
|
||||
[Peer]
|
||||
# {{ peer.name }}
|
||||
PublicKey = {{ hostvars[peer.host].wireguard_public_key }}
|
||||
{% if peer.comment is defined %}# {{ peer.comment }}
|
||||
{% endif %}PublicKey = {{ hostvars[peer.host].wireguard_public_key }}
|
||||
AllowedIPs = {{ peer.allowed_ips | join(', ') }}
|
||||
{% if peer.endpoint is defined %}
|
||||
Endpoint = {{ peer.endpoint }}
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<policy target="CONTINUE">
|
||||
<short>WireGuard forwarding: {{ item.name }}</short>
|
||||
<description>Managed WireGuard-to-LAN forwarding policy.</description>
|
||||
{% if item.masquerade | default(false) %}
|
||||
<masquerade/>
|
||||
{% endif %}
|
||||
<ingress-zone name="{{ item.ingress_zone }}"/>
|
||||
<egress-zone name="{{ item.egress_zone }}"/>
|
||||
<rule family="ipv4">
|
||||
<source address="{{ item.source }}"/>
|
||||
<destination address="{{ item.destination }}"/>
|
||||
<accept/>
|
||||
</rule>
|
||||
</policy>
|
||||
@@ -73,13 +73,6 @@
|
||||
- packages_rocky
|
||||
- services_systemd
|
||||
|
||||
- name: Configure WireGuard overlay
|
||||
hosts: wireguard_overlay
|
||||
become: true
|
||||
|
||||
roles:
|
||||
- wireguard_overlay
|
||||
|
||||
- name: Configure Aegis Fedora IoT profile
|
||||
hosts: role_aegis
|
||||
become: true
|
||||
@@ -87,6 +80,13 @@
|
||||
roles:
|
||||
- profile_aegis
|
||||
|
||||
- name: Configure WireGuard overlay
|
||||
hosts: wireguard_overlay
|
||||
become: true
|
||||
|
||||
roles:
|
||||
- wireguard_overlay
|
||||
|
||||
- name: Configure Atlas NAS profile
|
||||
hosts: atlas
|
||||
become: true
|
||||
|
||||
@@ -4,7 +4,7 @@ name: server
|
||||
|
||||
services:
|
||||
nginx-proxy-manager:
|
||||
image: docker.io/jc21/nginx-proxy-manager:latest
|
||||
image: {{ server_npm_quadlet_image if server_npm_quadlet_stage | bool else 'docker.io/jc21/nginx-proxy-manager:latest' }}
|
||||
container_name: nginx-proxy-manager
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
@@ -38,6 +38,7 @@ services:
|
||||
# networks:
|
||||
# - web
|
||||
|
||||
{% if not server_gitea_on_atlas | bool %}
|
||||
gitea:
|
||||
image: docker.gitea.com/gitea:1.25.2
|
||||
container_name: gitea
|
||||
@@ -55,26 +56,9 @@ services:
|
||||
ports:
|
||||
- "3000:3000"
|
||||
- "127.0.0.1:222:22"
|
||||
|
||||
{% if server_syncthing_enabled | default(true) | bool %}
|
||||
syncthing:
|
||||
image: docker.io/syncthing/syncthing:2
|
||||
container_name: syncthing
|
||||
hostname: syncthing
|
||||
restart: unless-stopped
|
||||
expose:
|
||||
- "8384"
|
||||
volumes:
|
||||
- "/opt/syncthing/config:/var/syncthing{{ ':' ~ selinux_volume_option if selinux_volume_option else '' }}"
|
||||
- "/srv/syncthing/data:/data{{ ':' ~ selinux_volume_option if selinux_volume_option else '' }}"
|
||||
ports:
|
||||
- "22000:22000/tcp"
|
||||
- "22000:22000/udp"
|
||||
- "21027:21027/udp"
|
||||
networks:
|
||||
- web
|
||||
{% endif %}
|
||||
|
||||
|
||||
networks:
|
||||
web:
|
||||
name: server_web
|
||||
|
||||
74
docs/atlas-dr-lab.md
Normal file
74
docs/atlas-dr-lab.md
Normal file
@@ -0,0 +1,74 @@
|
||||
# Isolated Atlas DR lab
|
||||
|
||||
This is a **scaled rehearsal**, not a substitute for a full-data restore. The
|
||||
`atlas-dr-lab` libvirt VM on Ikaros was left **shut off** on 2026-09-30. Its
|
||||
persistent volumes are in the default libvirt pool: the current 30 GiB OS
|
||||
volume `atlas-dr-lab-os-rebuild2.qcow2`, the pre-rebuild OS volume
|
||||
`atlas-dr-lab-os.qcow2`, and four independent 4 GiB
|
||||
`atlas-dr-lab-data{1,2,3,4}.qcow2` volumes. The VM uses libvirt's `default`
|
||||
NAT network (last DHCP address `192.168.122.168`), 2 vCPU, and 4 GiB RAM.
|
||||
The data disks have `virtio-atlasdrdata{1,2,3,4}` serials. No physical disk or
|
||||
production Atlas storage is attached. The VM has no autostart.
|
||||
|
||||
## Rebuild inputs and isolation
|
||||
|
||||
- Use Rocky's **9.8 GenericCloud Base x86_64** image
|
||||
`Rocky-9-GenericCloud-Base-9.8-20260525.0.x86_64.qcow2` from
|
||||
`https://download.rockylinux.org/pub/rocky/9.8/images/x86_64/`.
|
||||
Verify its `.CHECKSUM` file; the observed SHA-256 was
|
||||
`92c206cc6f790c61583247eefe87890f8828420662c17cacf247cec78ab4eec8`.
|
||||
- Use a dedicated lab-only inventory merged **after** the repository
|
||||
inventory, and always `--limit atlas_dr_lab`. The temporary 2026-09-30
|
||||
inventory/playbook and logs are in `/tmp/atlas-dr-lab-image/`; copy a
|
||||
sanitized inventory to durable private storage before `/tmp` is cleared if
|
||||
the lab will be repeated. Never reuse `host_vars/atlas.yml`, production
|
||||
Vault secrets, or production disk by-id paths for the lab.
|
||||
- The lab host belongs to `platform_rocky` and `atlas`. It uses `dradmin`
|
||||
(UID/GID 1000) with the operator's **public** SSH key and a random,
|
||||
unknown password hash, the libvirt DHCP address, pool `zpool`, mount root
|
||||
`/zpool`, the four `virtio-atlasdrdata*` by-id paths, a 1 GiB backup
|
||||
reservation, and `rocky_manage_openzfs_repo: true` with only `zfs` in
|
||||
`host_packages`. The following gates remain false: sharing, firewall,
|
||||
media stack, ZFS timers, Borg, USB, monitoring, and Prometheus pull.
|
||||
`atlas_manage_storage` is true. Set `atlas_create_pool: true` **only for the
|
||||
first disposable pool creation**, then set it false before any later run.
|
||||
- A minimal lab playbook selects `atlas_dr_lab`, `become: true`, and the
|
||||
existing `packages_rocky` and `profile_atlas` roles. Use a separate
|
||||
`ANSIBLE_CONFIG` without the production Vault password script, and keep
|
||||
host-key checking on with a lab-specific known-hosts file. The 2026-09-30
|
||||
runs used `-i ansible/inventory/hosts.yml -i <lab-inventory.yml>` and
|
||||
`--limit atlas_dr_lab` throughout.
|
||||
|
||||
## Rehearsal and narrow checks
|
||||
|
||||
1. Before any pool operation, compare `virsh -c qemu:///system domblklist
|
||||
atlas-dr-lab` with the four intended qcow2 paths, and in the guest compare
|
||||
`/dev/disk/by-id/virtio-atlasdrdata*` with `lsblk`. Do not proceed if a
|
||||
physical disk or production identity appears.
|
||||
2. For a first-time disposable build only, run the lab playbook with
|
||||
`--tags pool` and `atlas_create_pool: true`, then immediately set the gate
|
||||
false. Run the full lab playbook and check `zpool status -P zpool`,
|
||||
`zfs list -r zpool`, SELinux, and failed systemd units.
|
||||
3. Write a non-sensitive canary under the lab `/zpool/archive` and snapshot
|
||||
it. Record the pool GUID and canary SHA-256. Export the lab pool cleanly,
|
||||
shut down the VM, and replace **only the OS volume** with a fresh verified
|
||||
Rocky image. Preserve all four data volumes. Reconfigure cloud-init for a
|
||||
new instance; the seed CD-ROM must use **SATA**. The SCSI seed attachment
|
||||
tried during this rehearsal was not detected by cloud-init and was
|
||||
replaced with a SATA attachment before proceeding.
|
||||
4. On the new OS, apply `packages_rocky` to reinstall OpenZFS. First run
|
||||
`zpool import -d /dev/disk/by-id` **without importing**, compare GUID and
|
||||
vdev membership, then use ordinary `zpool import -d /dev/disk/by-id zpool`.
|
||||
Do not use `-f`, `-F`, `-X`, rollback, or pool creation.
|
||||
5. Reapply `profile_atlas` with the lab gates and `atlas_create_pool: false`.
|
||||
Verify the canary, restored snapshot file in an empty temporary directory,
|
||||
dataset hierarchy, SELinux, and pool health. A second full playbook run
|
||||
should report `changed=0`. Remove temporary restored files and shut down
|
||||
the VM after testing.
|
||||
|
||||
The observed 2026-09-30 pool GUID was `8880368391795119587`; the canary
|
||||
SHA-256 was `949701c7a95fadae1fddc21abe846c4312212dbfeb7477948f3188fc3ec34a78`.
|
||||
The post-rebuild Ansible run succeeded, a repeat run reported `changed=0`,
|
||||
12 datasets and the original snapshot were present, and the pool was healthy.
|
||||
The snapshot-restored file matched content and basic metadata. See
|
||||
[`atlas-recovery.md`](atlas-recovery.md) for the production runbook and limits.
|
||||
241
docs/atlas-gitea-migration.md
Normal file
241
docs/atlas-gitea-migration.md
Normal file
@@ -0,0 +1,241 @@
|
||||
# Gitea migration from Prometheus to Atlas
|
||||
|
||||
The later 2026-10-03 canonical-domain change to `git.fscotto.co` is recorded
|
||||
in `docs/domain-fscotto-co.md`. Public SSH remains on TCP/2222; the old
|
||||
DuckDNS Proxy Host was observed disabled. Earlier domain references below
|
||||
describe migration evidence, not the current canonical URL.
|
||||
|
||||
This records the staged migration and its observed partial cutover. Gitea is
|
||||
temporary on Atlas until Uranus; NPM remains on Prometheus. On 2026-10-03
|
||||
the operator explicitly approved removal of the old Prometheus Gitea data,
|
||||
SSH fragment and final-export helper. NPM now uses a rootful Quadlet with no
|
||||
installed Compose fallback. The source-retention and rollback steps below
|
||||
are historical migration gates, not current recovery instructions.
|
||||
Existing backup archives were preserved; use current Atlas data and verified
|
||||
backups for recovery. Do not recreate or restart stale source Gitea.
|
||||
|
||||
## Observed source before cutover and chosen topology (2026-10-01)
|
||||
|
||||
- Prometheus runs the rootful `docker.gitea.com/gitea:1.25.2` image in its
|
||||
managed Compose stack. `/opt/gitea/data` is about 280 MiB, uses SQLite,
|
||||
and contains 33 repositories. A live read-only SQLite `quick_check` passed.
|
||||
`/home/git/.ssh` is a separate small bind mount; `/opt/gitea/data/ssh`
|
||||
contains the existing SSH host keys. Neither tree may be discarded.
|
||||
- Gitea answers HTTP 200 on Prometheus port 3000. NPM currently forwards
|
||||
`git.fscotto.duckdns.org` and `git.ov-ad3410.infomaniak.ch` to the Compose
|
||||
hostname `gitea:3000`. Public DNS resolves to Prometheus. The container's
|
||||
SSH port is bound only to `127.0.0.1:222`; this is not a public Gitea SSH
|
||||
listener. Prometheus' public port 22 remains administrative SSH.
|
||||
- Atlas has a healthy pool and a verified, private Prometheus backup under
|
||||
`/zpool/backup/hosts/prometheus/latest`. The 2026-10-01 scheduled export
|
||||
and pull succeeded. The intended target is a separate
|
||||
`/zpool/services/data/gitea` dataset, not `Archive` or the backup dataset.
|
||||
- The approved cutover keeps NPM on Prometheus, changes the two HTTP Proxy
|
||||
Hosts' effective upstream to Atlas over the Prometheus--Aegis gateway, and offers public Gitea
|
||||
SSH on port 2222 via the same gateway. Prometheus port 22 is unchanged.
|
||||
HTTPS and SSH must be validated together before declaring cutover.
|
||||
- The initial staging ran as a **rootless user Quadlet** under a dedicated,
|
||||
non-login Atlas account, using the pinned `1.25.2-rootless` image. This was an explicit
|
||||
rootful-to-rootless **data-layout conversion**, not a drop-in image swap:
|
||||
the target mounts `/var/lib/gitea` and `/etc/gitea`, and uses Gitea's
|
||||
built-in SSH server instead of the source image's OpenSSH daemon. Keep the
|
||||
application version unchanged until the conversion has passed an isolated
|
||||
restore test. The host's rootful Quadlet directory must not be used.
|
||||
|
||||
## Phase 1: prepare without traffic changes
|
||||
|
||||
Preparation completed on 2026-10-01: Ansible created
|
||||
`zpool/services/data/gitea`, a dedicated non-login `gitea` account (UID/GID
|
||||
1101), separate subordinate IDs, parent-dataset traverse ACLs, and an inactive
|
||||
user Quadlet under `/var/lib/atlas-gitea/.config/containers/systemd/`. The
|
||||
Quadlet has no `[Install]` section and, until the final cutover, binds only
|
||||
loopback staging ports 3001/2223 if started manually. A second targeted
|
||||
Ansible run changed nothing; the generated service was inactive and neither
|
||||
staging port listened.
|
||||
|
||||
The explicit rehearsal is managed by:
|
||||
|
||||
```bash
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags gitea_restore \
|
||||
-e atlas_gitea_restore_test=true
|
||||
```
|
||||
|
||||
On 2026-10-01 this selected the latest verified Prometheus backup, checked its
|
||||
SHA-256, extracted only `opt/gitea/data`, moved `app.ini` into the rootless
|
||||
config mount, rewrote `/data/` paths, enabled built-in SSH on internal port
|
||||
2222, and retained the three source SSH host-key pairs. SQLite `quick_check`
|
||||
passed, all 33 restored repositories passed `git fsck`, and each source/target
|
||||
public host-key fingerprint matched. A temporary `1.25.2-rootless` container
|
||||
with `--network none` answered HTTP internally and listened on internal
|
||||
SSH/2222. The container was removed; the user Quadlet remains inactive, with
|
||||
no staging listener. The second restore run changed nothing. This copy is
|
||||
deliberately stale once new source writes occur and **must not** be used as the
|
||||
final cutover copy.
|
||||
|
||||
Target backup checks on 2026-10-01: the managed recursive hourly ZFS snapshot
|
||||
`atlas-auto-hourly-20261001T193401Z` contains the new dataset. The managed
|
||||
Borg service completed archive `atlas-20261001T193420Z`, whose contents list
|
||||
includes the staged Gitea database. A separate one-file restore from each
|
||||
source into private `/var/tmp` directories matched the live staged database
|
||||
and passed SQLite `quick_check`. Temporary files and the on-demand snapshot
|
||||
mount were removed; the Borg temporary snapshot was cleaned up and the pool
|
||||
remained healthy. This is file-level proof, **not** a full Gitea recovery.
|
||||
The operator's UUID-bound offline USB run published version
|
||||
`20261001T201220Z-254397` on 2026-10-02. A separate read-only mount and
|
||||
temporary restore of `services/data/gitea/data/gitea/gitea.db` matched
|
||||
contents, owner, group, mode, size, mtime and POSIX ACL; SQLite
|
||||
`quick_check` returned `ok`. The temporary mount and copy were removed,
|
||||
LUKS was closed, and the pool was healthy. This is a file-level restore test,
|
||||
not a complete Gitea recovery rehearsal from USB.
|
||||
|
||||
1. Provision a dedicated target dataset and non-login service identity via
|
||||
Ansible, keeping UID/GID distinct from Atlas' reserved Immich `1100`.
|
||||
Install the user Quadlet in that identity's
|
||||
`~/.config/containers/systemd/`, **without** an `[Install]` section;
|
||||
do not enable, start, or expose it yet.
|
||||
2. Verify the selected Atlas backup SHA-256 and metadata, then extract **only**
|
||||
`opt/gitea/data` to private staging. Keep `home/git/.ssh` in the source
|
||||
backup for rollback; the rootless image does not consume its OpenSSH mount.
|
||||
Never unpack NPM,
|
||||
WireGuard, or other host configuration from this sensitive tarball into a
|
||||
live namespace. Convert the rootful `/data` tree on a disposable copy:
|
||||
place application data under `/var/lib/gitea`, move `app.ini` to
|
||||
`/etc/gitea`, and rewrite every absolute `/data/...` path for the new
|
||||
layout. Enable `START_SSH_SERVER`, use internal SSH port 2222, and retain
|
||||
the source host-key pairs for the built-in server only after verifying
|
||||
their fingerprints and compatibility. Do not rely on the old
|
||||
`/home/git/.ssh` OpenSSH mount in the rootless image. Set only the target
|
||||
copy's ownership and path-scoped SELinux labels.
|
||||
3. Validate SQLite integrity, repository count and representative `git fsck`,
|
||||
LFS/attachment presence, permissions, and an isolated rootless test
|
||||
container with no production ingress or outbound network. Because the
|
||||
source stays active, this is a rehearsal copy, not the final cutover copy.
|
||||
Regenerate Git hooks if the changed installation path requires it.
|
||||
4. ZFS, Borg and UUID-bound offline USB inclusion and one-file restores have
|
||||
passed. These do not replace the final consistent source copy.
|
||||
|
||||
## Phase 2: explicit final cutover
|
||||
|
||||
The opt-in `/usr/local/sbin/prometheus-gitea-final-export` helper was installed
|
||||
on 2026-10-01 and passed `bash -n`. It refuses to
|
||||
run while the scheduled Prometheus export timer is active. When explicitly
|
||||
triggered, it stops only the source Gitea container, checks SQLite, publishes
|
||||
a checksum-verified Gitea-only version for Atlas' existing pull, and leaves
|
||||
the source stopped on success. NPM remains running. A failure before
|
||||
completion restarts source Gitea. Its Ansible gate is
|
||||
`--tags gitea_final_export -e server_gitea_final_export=true`.
|
||||
After Atlas pulls that version, its separate
|
||||
`--tags gitea_final_restore -e atlas_gitea_final_restore=true` gate accepts
|
||||
only metadata marked `gitea-cutover`, validates a private staged replacement,
|
||||
and swaps it for the marked rehearsal. The swap and its rollback path passed
|
||||
synthetic tests on 2026-10-01; the live gate succeeded on 2026-10-02.
|
||||
|
||||
On 2026-10-02 the operator approved the outage. The final stopped-source
|
||||
export `20261002T071525Z` passed the Atlas pull checksum; the guarded restore
|
||||
replaced the rehearsal. SQLite `quick_check`, all 33 repository `git fsck`
|
||||
checks, and the source/target SSH host-key comparison passed. The rootless
|
||||
Atlas Quadlet serves LAN HTTP/3000 and SSH/2222, reachable from Prometheus
|
||||
through Aegis; its firewall admits only Aegis. The final marker gates startup.
|
||||
|
||||
Prometheus now runs the NPM-only Compose stack. Both NPM database records still
|
||||
say `gitea:3000`, but Nginx evaluates this variable upstream through its
|
||||
runtime DNS resolver, which **does not** use a Compose `extra_hosts` alias.
|
||||
The initial alias attempt returned 502. A managed `server_proxy.conf` override
|
||||
sets `$server` to Atlas' IP for only the two declared Gitea domains; it passed
|
||||
`nginx -t` and primary HTTPS/API returned 200 after a clean NPM restart
|
||||
without the alias; a representative public `git ls-remote` also succeeded.
|
||||
Navidrome and Syncthing Proxy Hosts still responded. No NPM SQLite records
|
||||
or credentials were changed. The
|
||||
secondary hostname `git.ov-ad3410.infomaniak.ch` did not resolve from Ikaros
|
||||
and had no generated NPM config file at the time of inspection.
|
||||
|
||||
On 2026-10-03 the operator retired this unused secondary hostname. Its NPM
|
||||
Proxy Host was already soft-deleted; Ansible now declares only
|
||||
`git.fscotto.duckdns.org` and removes the secondary runtime override.
|
||||
|
||||
Prometheus' public TCP/2222 socket proxies to Atlas without changing admin
|
||||
SSH/22. The local socket presents the preserved Gitea ED25519 host key, but
|
||||
an external TCP/2222 connection from Ikaros initially timed out. During that
|
||||
test no SYN reached Prometheus `eth0`; its socket and firewalld port were active.
|
||||
After the VPS firewall was opened later on 2026-10-02, the public port connected,
|
||||
its ED25519 host-key fingerprint matched Atlas, Gitea authenticated the `ikaros`
|
||||
key as `fscotto`, and a public SSH `git ls-remote` for `fscotto/infra.git`
|
||||
returned HEAD. The operator subsequently reported successful authenticated
|
||||
SSH pull and push; the agent did not perform a write test. HTTPS write/login
|
||||
remain untested. Do not
|
||||
restart the stale source after public HTTPS has accepted target writes.
|
||||
|
||||
The Prometheus export timer resumed with NPM-only paths. A recursive ZFS
|
||||
snapshot at `20261002T073032Z` and encrypted Borg archive
|
||||
`atlas-20261002T073044Z` captured the Atlas target after cutover; Borg exited
|
||||
successfully, cleaned its temporary snapshot, and the pool was healthy.
|
||||
|
||||
## Corrected Atlas service owner (2026-10-02)
|
||||
|
||||
The operator required the host Quadlet to belong to `admin`, while the Unix
|
||||
user **inside** the container must be named `gitea`. The pinned derived
|
||||
`Containerfile.gitea-rootless` changes only the base image's UID/GID 1000
|
||||
passwd/group names from `git` to `gitea`; it retains the rootless image's
|
||||
paths and entrypoint. Gitea's `RUN_USER` is `gitea`, while its built-in SSH
|
||||
user and advertised clone user remain `git`, preserving `git@` URLs. The
|
||||
selective restore helper now generates the same three settings for any future
|
||||
explicit restore, instead of recreating a `RUN_USER = git` target.
|
||||
|
||||
A disposable, loopback-only container using a copy of a Gitea ZFS snapshot
|
||||
passed HTTP, SQLite, internal-user and SSH host-key checks without touching
|
||||
live data. After explicit outage approval, the opt-in
|
||||
`--tags gitea_owner_migration -e atlas_gitea_owner_migration=true` run stopped
|
||||
the old user service, took safety snapshot
|
||||
`zpool/services/data/gitea@gitea-owner-migration-20261002T100104`, transferred
|
||||
only the Gitea dataset to `admin`, tested an `admin` staging Quadlet on
|
||||
loopback, then promoted it to the production LAN ports. The old Atlas Quadlet
|
||||
was removed. The old host `gitea` account and its sub-ID range are retained
|
||||
for a deliberate rollback; they must not restart stale Gitea. The parent
|
||||
traverse ACL is removed by the normal Gitea role once the new owner is live.
|
||||
|
||||
The new service returned HTTP 200 locally and through public primary HTTPS;
|
||||
Navidrome and Syncthing remained active under `admin`, the pool was healthy,
|
||||
and a second normal Gitea Ansible run was idempotent. This does **not** close
|
||||
the separate external TCP/2222 or authenticated clone/push validation gap.
|
||||
|
||||
1. Agree on an outage and record source/target versions, pool health, the
|
||||
latest backups, SSH host-key fingerprints, and both current NPM routes.
|
||||
Stop the Prometheus export timer for the change window so it cannot
|
||||
restart the old Compose stack unexpectedly.
|
||||
2. Quiesce source writes with the final-export helper: it stops Gitea before
|
||||
the consistent export and leaves it stopped after success. Pull that export
|
||||
to Atlas and verify checksum and timestamp. Keep
|
||||
`/opt/gitea/data` and `/home/git/.ssh` intact for rollback. Do not allow
|
||||
source Gitea to restart after accepting writes on Atlas.
|
||||
3. Restore the final Gitea-only payload to the target and repeat integrity
|
||||
checks. Verify its advertised SSH port is 2222, its existing HTTPS
|
||||
`ROOT_URL`, repositories, LFS/attachments, and SSH host-key identity. Enable
|
||||
the production Atlas Quadlet only after the final-restore marker exists;
|
||||
its firewall permits only Aegis to reach HTTP and SSH. Validate local HTTP
|
||||
and the target service before switching NPM.
|
||||
4. Enable the public TCP/2222 socket proxy on Prometheus to Atlas over Aegis
|
||||
without changing administrative TCP/22. Switch Prometheus to the desired
|
||||
NPM-only Compose stack and use the managed Gitea-only NPM runtime upstream
|
||||
override. Do not use Compose `extra_hosts`: Nginx bypasses it for the
|
||||
variable upstream. The old Gitea data stays intact. Do not change public DNS.
|
||||
5. Test HTTPS login, representative clone/push, LFS, and public SSH clone/push
|
||||
on port 2222 from outside the Atlas LAN. Record the last source write and
|
||||
first healthy target service times; do not claim RPO/RTO without measuring.
|
||||
6. Resume the Prometheus NPM-only backup export timer after the desired stack
|
||||
is active and verify its next result. Verify the next Atlas snapshot/Borg
|
||||
run covers Gitea and test a restored target copy. Do not delete old source
|
||||
data.
|
||||
|
||||
## Rollback gate
|
||||
|
||||
Before Atlas accepts writes, restore the old Compose definition and remove the
|
||||
NPM override, disable the public 2222 proxy, and restart the unchanged source
|
||||
Gitea if target validation fails. **After Atlas accepts writes, do not blindly restart the source:** its
|
||||
SQLite database and repositories are stale. Quiesce Atlas, capture its new
|
||||
data, and decide a reverse migration or an extended outage explicitly.
|
||||
|
||||
Upstream references: [rootful container layout](https://docs.gitea.com/1.25/installation/install-with-docker/),
|
||||
[rootless image layout and incompatibility](https://docs.gitea.com/installation/install-with-docker-rootless/),
|
||||
[rootless Podman Quadlet](https://docs.gitea.com/installation/install-with-podman-quadlet/),
|
||||
[standard-image conversion](https://docs.gitea.com/1.24/installation/install-with-docker-rootless/),
|
||||
and [restore and hook regeneration](https://docs.gitea.com/1.26/administration/backup-and-restore/).
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user