diff --git a/AGENTS.md b/AGENTS.md index 629fb06..d3e2348 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -54,8 +54,9 @@ Ansible-driven personal infrastructure repo for Fedora and Void desktops, Fedora - Emacs is disabled by default; temporary Emacs check: `ansible-playbook ansible/site.yml --limit --tags emacs --check --diff -e emacs_enabled=true` - AI coding agents: `ansible-playbook ansible/site.yml --limit --tags ai_agents --check --diff` - Mail bootstrap: `sh -n scripts/bootstrap_mail.sh` and `shellcheck scripts/bootstrap_mail.sh` - - Server NPM Quadlet: `systemctl status prometheus-npm.service`; disabled Compose fallback render: - `podman-compose -f /opt/docker/server/docker-compose.yml config` + - Server NPM Quadlet: `systemctl status prometheus-npm.service`; the Compose fallback is retired. + - Explicit Prometheus legacy cleanup (destructive only without check mode): + `ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true` - Atlas media stack: `ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff` - Atlas rootless Gitea staging (does not start Gitea): @@ -143,12 +144,15 @@ The dotfile vars follow the same split: `desktop_common_dotfiles` carries mode-i and disables diffs. Provisioning does not execute the updater or change its external schedule. - `rocky_server` is a child of both `platform_rocky` and `server`; `prometheus` is its active target. - The target must already provide `server_username` with local sudo access before the profile runs. -- The Rocky profile installs Podman and podman-compose and renders the disabled legacy - `podman-compose-server` unit for rollback. On Prometheus, Nginx Proxy Manager is now the rootful +- The Rocky profile installs Podman and podman-compose. Prometheus explicitly retires the legacy + Compose unit, files and final-export helper with `server_legacy_stack_retired: true`. + Its approved opt-in cleanup removed old application data on 2026-10-03; normal runs do not + delete data or recreate the retired files. On Prometheus, Nginx Proxy Manager is now the rootful `prometheus-npm.service` Quadlet with a pinned image digest and the existing `/opt/npm/data` and `/opt/npm/letsencrypt` bind mounts. The rootful `server_web` bridge remains `10.89.0.0/24`. Gitea runs on Atlas; PostgreSQL and Navidrome are absent from the desired Prometheus stack. - The profile does not delete legacy data, update DNS, or perform an implicit cutover. + Normal runs do not delete legacy data, update DNS, or perform an implicit cutover; + destructive cleanup requires its explicit tag and opt-in extra-var. - Firewalld enables SSH, Cockpit (`9090/tcp`), HTTP and HTTPS. Nginx Proxy Manager publishes `80/tcp` and `443/tcp`; bind its administration interface only to `127.0.0.1:81` and use `npm-tunnel` from Ikaros or Nymph. Nextcloud remains disabled; do not provision `/srv/nextcloud` directories. @@ -413,6 +417,22 @@ successfully. The first monthly scrub remains a runtime check. both Quadlet files were present, and the complete Let's Encrypt tree (70 regular files plus 12 symlinks) matched the live data. A targeted normal Ansible run changed nothing. Details and rollback boundaries are in `docs/prometheus-npm-quadlet.md`. +- [x] Remove only unused Gitea, Navidrome and PostgreSQL images with opt-in + Ansible tasks on 2026-10-03. Second run changed nothing; NPM stayed active + with zero restarts, HTTP/HTTPS passed, backup timer and SSH proxy stayed active. + This image-only step preserved data and fallback; the later approved deletion is tracked below. Validation: + `ansible-playbook ansible/site.yml --limit prometheus --tags server_image_cleanup --check --diff -e server_legacy_image_cleanup=true` +- [x] Complete explicitly approved old-data and Compose fallback removal on 2026-10-03. + Backup paths and mount dependencies were reconciled before deletion; repeat cleanup changed + nothing. The normal Compose/template/helper check did not recreate retired files. + A separately approved manual export/pull published `20261003T112906Z`; checksum and isolated + SQLite restore passed with ten proxy hosts and both Quadlet definitions. NPM, primary HTTPS, + WireGuard, SSH proxy and backup timer remained healthy; existing backup archives were preserved. +- [x] Retire the unused secondary Gitea hostname `git.ov-ad3410.infomaniak.ch` + on 2026-10-03. Its NPM Proxy Host was already soft-deleted and had no + associated certificate. Its Ansible domain and runtime override were removed; + nginx -t and reload passed without restarting NPM. Primary HTTPS returned 200 + with valid TLS; only `git.fscotto.duckdns.org` remains declared for Gitea. - [ ] Observe the first scheduled export and Atlas pull after the cutover; the manual end-to-end cycle passed, but the next unattended cycle has not yet occurred. diff --git a/README.it.md b/README.it.md index e0da59f..e6e5931 100644 --- a/README.it.md +++ b/README.it.md @@ -202,14 +202,17 @@ Lo stato attuale del profilo server include: - installazione di Podman e podman-compose - abilitazione dei servizi systemd dichiarati in inventory/group vars - copia dei dotfiles server e rendering del Quadlet rootful `prometheus-npm.service` per Nginx Proxy - Manager; il vecchio `podman-compose-server` resta disabilitato soltanto per un rollback controllato + Manager; il vecchio fallback Compose è stato rimosso con autorizzazione esplicita - attivazione di firewalld con SSH, Cockpit (`9090/tcp`), HTTP e HTTPS abilitati - Syncthing escluso dal profilo server Rocky -Il Compose desiderato su Prometheus non include piu Gitea, Navidrome ne il database PostgreSQL obsoleto. -Navidrome e Syncthing appartengono ad Atlas; Navidrome ufficiale usa invece SQLite. Il profilo non -arresta o rimuove automaticamente eventuali container legacy e non elimina `/opt/postgres/data`. -Il cutover NPM, le verifiche dei dati e del backup e i limiti del rollback sono documentati in +Il 2026-10-03 la pulizia opt-in autorizzata ha rimosso dati e immagini precedenti di Gitea, +Navidrome e PostgreSQL, directory obsolete vuote, helper finale Gitea e fallback Compose NPM. +I servizi migrati restano su Atlas. `server_legacy_stack_retired: true` evita che i normali task +ricreino i residui; la cancellazione richiede `--tags server_legacy_cleanup` e +`-e server_legacy_cleanup=true`. NPM attivo e archivi di backup restano intatti. +Export, pull Atlas e restore SQLite isolato post-pulizia sono riusciti; il primo ciclo automatico +resta da osservare. Evidenze e confini del recovery: [`docs/prometheus-npm-quadlet.md`](docs/prometheus-npm-quadlet.md). Nginx Proxy Manager pubblica solo `80/tcp` e `443/tcp`; la sua interfaccia di amministrazione e @@ -642,7 +645,7 @@ Questo significa che, allo stato attuale: - il server Rocky (`prometheus`) e gestito con pacchetti, servizi, dotfiles server e firewalld - il NAS Rocky (`atlas`) usa un pool ZFS gia esistente, condivisioni NFSv4/SMB limitate alla LAN e Cockpit/45Drives - NPM è un Quadlet rootful su Prometheus, mentre Gitea, Navidrome e Syncthing sono Quadlet - rootless su Atlas; il Compose server resta disabilitato per rollback + rootless su Atlas; il fallback Compose server è stato rimosso # Dotfiles @@ -748,7 +751,7 @@ ansible-playbook ansible/site.yml --limit --tags , --check -- ansible-playbook ansible/site.yml --limit --start-at-task "" --check --diff ansible-lint ansible/roles/ yamllint ansible/path/to/file.yml -podman-compose -f /opt/docker/server/docker-compose.yml config +ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 --check --diff ansible-playbook ansible/site.yml --limit atlas --tags music_sync --check --diff diff --git a/README.md b/README.md index 314eae7..cbbc7e8 100644 --- a/README.md +++ b/README.md @@ -130,14 +130,18 @@ dotfiles and templates. The profile does not transfer application data, update D implicit service cutover. The server profile installs platform-specific packages, Podman and podman-compose, declared systemd -services, and firewalld. Nginx Proxy Manager now runs as the rootful `prometheus-npm.service` Quadlet; -the old `podman-compose-server` unit is disabled and retained only for controlled rollback. The -Compose file no longer includes Gitea, Navidrome, Syncthing, or the obsolete Navidrome PostgreSQL -database. The temporary Navidrome and Syncthing deployment on Atlas is managed by -`profile_backend_phase1`. Applying the profile does not delete legacy data or `/opt/postgres/data`. -The NPM cutover, data checks, backup evidence, and rollback boundaries are documented in +services, and firewalld. Nginx Proxy Manager runs as the rootful `prometheus-npm.service` Quadlet. +On 2026-10-03 the operator-approved opt-in cleanup removed old Gitea, Navidrome and PostgreSQL +data/images, empty legacy directories, the Gitea final-export helper and the Compose rollback files. +The migrated services stay on Atlas. `server_legacy_stack_retired: true` prevents normal runs from +recreating retired files. Data deletion requires `--tags server_legacy_cleanup` and +`-e server_legacy_cleanup=true`; image-only cleanup has its own `server_image_cleanup` tag and flag. +Active NPM resources and existing backup archives remain preserved. +The post-cleanup export/pull and isolated SQLite restore passed; the first unattended cycle remains +pending. Evidence and recovery boundaries: [`docs/prometheus-npm-quadlet.md`](docs/prometheus-npm-quadlet.md). + Firewalld enables SSH, Cockpit (`9090/tcp`), HTTP and HTTPS. Nginx Proxy Manager publishes only `80/tcp` and `443/tcp`; its administration interface is bound to `127.0.0.1:81` and can be reached from Ikaros or Nymph with the `npm-tunnel` Bash alias. Nextcloud remains disabled and the profile @@ -738,7 +742,7 @@ ansible-playbook ansible/site.yml --limit --tags , --check -- ansible-playbook ansible/site.yml --limit --start-at-task "" --check --diff ansible-lint ansible/roles/ yamllint ansible/path/to/file.yml -podman-compose -f /opt/docker/server/docker-compose.yml config +ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff ansible-playbook ansible/site.yml --limit atlas --tags music_sync --check --diff ``` diff --git a/ansible/inventory/group_vars/server.yml b/ansible/inventory/group_vars/server.yml index 8208d4b..d3f22e6 100644 --- a/ansible/inventory/group_vars/server.yml +++ b/ansible/inventory/group_vars/server.yml @@ -8,6 +8,8 @@ effective_user_home: "{{ server_user_home }}" server_container_stack_dir: /opt/docker/server server_npm_quadlet_stage: false server_npm_quadlet_cutover: false +server_legacy_stack_retired: false +server_legacy_cleanup: false ai_agents: {} vim_plugins_enabled: false @@ -101,8 +103,9 @@ server_backup_export_source_keep: 3 server_backup_export_paths: >- {{ ['opt/npm/data', 'opt/npm/letsencrypt'] + ([] if server_gitea_on_atlas | bool else ['opt/gitea/data', 'home/git/.ssh']) - + ['opt/docker/server/docker-compose.yml', - 'etc/systemd/system/podman-compose-server.service'] + + ([] if server_legacy_stack_retired | bool else + ['opt/docker/server/docker-compose.yml', + 'etc/systemd/system/podman-compose-server.service']) + (['etc/containers/systemd/prometheus-npm.container', 'etc/containers/systemd/server-web.network'] if server_npm_quadlet_stage | bool else []) diff --git a/ansible/inventory/host_vars/prometheus.yml b/ansible/inventory/host_vars/prometheus.yml index 1046f20..e14d525 100644 --- a/ansible/inventory/host_vars/prometheus.yml +++ b/ansible/inventory/host_vars/prometheus.yml @@ -6,6 +6,15 @@ ansible_port: 22 ansible_ssh_private_key_file: /home/fscotto/.ssh/id_ed25519 server_username: rocky +server_legacy_stack_retired: true +# Destructive deletion runs only with an explicit extra-var and cleanup tag. +server_legacy_cleanup: false +# Explicit opt-in cleanup; no data, volumes, networks or NPM images are removed. +server_legacy_image_cleanup: false +server_legacy_images: + - docker.gitea.com/gitea:1.25.2 + - docker.io/deluan/navidrome:latest + - docker.io/library/postgres:13 server_npm_quadlet_stage: true server_npm_quadlet_image: docker.io/jc21/nginx-proxy-manager@sha256:52b2c59994f3d36acfcf70a1626f29734df0ed8c71bacc0269f78b6f939858bb # The stopped-source export and live Quadlet cutover passed on 2026-10-03. @@ -17,7 +26,6 @@ server_gitea_cutover_tools_enabled: true server_gitea_on_atlas: true server_gitea_npm_domains: - git.fscotto.duckdns.org - - git.ov-ad3410.infomaniak.ch server_duckdns_domain: fscotto server_ssh_authorized_keys: - name: ikaros diff --git a/ansible/roles/profile_server/tasks/backup_export_job.yml b/ansible/roles/profile_server/tasks/backup_export_job.yml index 6b55d82..1cf031a 100644 --- a/ansible/roles/profile_server/tasks/backup_export_job.yml +++ b/ansible/roles/profile_server/tasks/backup_export_job.yml @@ -51,6 +51,7 @@ owner: root group: root mode: "0750" + validate: "bash -n %s" when: server_backup_export_enabled | bool - name: Install Prometheus backup export systemd units diff --git a/ansible/roles/profile_server/tasks/gitea_final_export.yml b/ansible/roles/profile_server/tasks/gitea_final_export.yml index 4a5ec81..fb02d79 100644 --- a/ansible/roles/profile_server/tasks/gitea_final_export.yml +++ b/ansible/roles/profile_server/tasks/gitea_final_export.yml @@ -7,7 +7,9 @@ owner: root group: root mode: "0750" - when: server_gitea_cutover_tools_enabled | bool + when: + - server_gitea_cutover_tools_enabled | bool + - not server_legacy_stack_retired | bool - name: Require the prepared source and explicit final-export approval tags: [services, gitea_final_export] @@ -15,6 +17,7 @@ that: - server_gitea_cutover_tools_enabled | bool - server_backup_export_enabled | bool + - not server_legacy_stack_retired | bool - not ansible_check_mode fail_msg: >- Install the cutover helper and perform an explicit non-check-mode run @@ -31,4 +34,5 @@ no_log: true when: - server_gitea_final_export | bool + - not server_legacy_stack_retired | bool - not ansible_check_mode diff --git a/ansible/roles/profile_server/tasks/legacy_cleanup.yml b/ansible/roles/profile_server/tasks/legacy_cleanup.yml new file mode 100644 index 0000000..dc171c4 --- /dev/null +++ b/ansible/roles/profile_server/tasks/legacy_cleanup.yml @@ -0,0 +1,112 @@ +--- +- name: Require explicit retirement of the migrated Prometheus source + ansible.builtin.assert: + that: + - inventory_hostname == 'prometheus' + - server_legacy_stack_retired | bool + - server_gitea_on_atlas | bool + - server_npm_quadlet_cutover | bool + - server_backup_export_enabled | bool + +- name: Verify legacy paths have no mounts or container users + ansible.builtin.command: + argv: + - python3 + - -c + - | + import json, os, pathlib, subprocess + def run(*args): + return subprocess.check_output(args, text=True).strip() + paths = ['/opt/gitea', '/home/git/.ssh', '/opt/navidrome', + '/opt/postgres', '/opt/music', '/opt/containerd', '/opt/docker'] + mounts = json.loads(run('findmnt', '--json', '--list', '-o', 'TARGET'))['filesystems'] + for path in paths: + assert os.path.realpath(path) == path, 'Symlink in cleanup path: ' + path + for mount in mounts: + target = mount['target'] + assert target != path and not target.startswith(path + '/'), 'Mounted cleanup path: ' + path + ids = run('podman', 'ps', '-aq').split() + containers = json.loads(run('podman', 'inspect', *ids)) if ids else [] + for container in containers: + assert container['Name'].lstrip('/') == 'nginx-proxy-manager', 'Unexpected container; review before cleanup' + for mount in container.get('Mounts', []): + source = os.path.realpath(mount['Source']) + for path in paths: + assert source != path and not source.startswith(path + '/'), 'Container uses cleanup path: ' + path + for path in ['/opt/music', '/opt/containerd']: + if os.path.isdir(path): + for entry in pathlib.Path(path).rglob('*'): + assert entry.is_dir() and not entry.is_symlink(), 'Unexpected file in empty legacy path: ' + str(entry) + if os.path.isdir('/opt/docker'): + allowed = {'/opt/docker/server', '/opt/docker/server/docker-compose.yml'} + for entry in pathlib.Path('/opt/docker').rglob('*'): + assert str(entry) in allowed and not entry.is_symlink(), 'Unexpected legacy Docker content: ' + str(entry) + assert run('systemctl', 'is-active', 'prometheus-npm.service') == 'active' + assert subprocess.run(['systemctl', 'is-active', '--quiet', 'podman-compose-server.service']).returncode != 0 + assert subprocess.run(['systemctl', 'is-active', '--quiet', 'prometheus-backup-export.service']).returncode != 0 + print('Legacy cleanup preflight passed') + changed_when: false + check_mode: false + +- name: Require the updated backup configuration before deleting fallback files + ansible.builtin.command: + argv: + - python3 + - -c + - | + import pathlib, subprocess + unit = subprocess.check_output(['systemctl', 'show', 'prometheus-backup-export.service', + '-p', 'RequiresMountsFor', '--value'], text=True) + assert '/opt/gitea' not in unit, 'Backup unit still depends on legacy Gitea' + helper = pathlib.Path('/usr/local/sbin/prometheus-backup-export').read_text() + assert 'podman-compose-server' not in helper and 'opt/docker/server' not in helper + subprocess.run(['bash', '-n', '/usr/local/sbin/prometheus-backup-export'], check=True) + changed_when: false + when: not ansible_check_mode + +- name: Delete only the explicitly approved legacy data and fallback files + ansible.builtin.file: + path: "{{ item }}" + state: absent + loop: + - /opt/gitea + - /home/git/.ssh + - /opt/navidrome + - /opt/postgres + - /opt/music + - /opt/containerd + - /opt/docker + - /usr/local/sbin/prometheus-gitea-final-export + - /etc/systemd/system/podman-compose-server.service + register: server_legacy_deleted + diff: false + +- name: Reload systemd after removing the inactive legacy unit + ansible.builtin.systemd: + daemon_reload: true + when: + - server_legacy_deleted is changed + - not ansible_check_mode + +- name: Inspect the obsolete Git home without following symlinks + ansible.builtin.stat: + path: /home/git + follow: false + register: server_legacy_git_home + +- name: Require the obsolete Git account to be absent before removing its empty home + ansible.builtin.command: + argv: [getent, passwd, git] + register: server_legacy_git_account + changed_when: false + failed_when: server_legacy_git_account.rc != 2 + check_mode: false + when: server_legacy_git_home.stat.exists + +# rmdir refuses any nonempty directory; never recursively delete this parent. +- name: Remove only the empty obsolete Git home + ansible.builtin.command: + argv: [rmdir, /home/git] + register: server_legacy_git_home_removed + changed_when: server_legacy_git_home_removed.rc == 0 + when: server_legacy_git_home.stat.exists diff --git a/ansible/roles/profile_server/tasks/legacy_image_cleanup.yml b/ansible/roles/profile_server/tasks/legacy_image_cleanup.yml new file mode 100644 index 0000000..225ff61 --- /dev/null +++ b/ansible/roles/profile_server/tasks/legacy_image_cleanup.yml @@ -0,0 +1,33 @@ +--- +- name: Require the migrated Prometheus topology for image cleanup + ansible.builtin.assert: + that: + - inventory_hostname == 'prometheus' + - server_gitea_on_atlas | bool + - server_npm_quadlet_cutover | bool + - server_legacy_images | default([]) | length > 0 + - >- + server_legacy_images | difference([ + 'docker.gitea.com/gitea:1.25.2', + 'docker.io/deluan/navidrome:latest', + 'docker.io/library/postgres:13']) | length == 0 + +- name: Check whether the explicitly selected legacy images exist + ansible.builtin.command: + argv: [podman, image, exists, "{{ item }}"] + loop: "{{ server_legacy_images }}" + register: server_legacy_image_presence + changed_when: false + failed_when: server_legacy_image_presence.rc not in [0, 1] + check_mode: false + +# No --force: Podman must refuse images referenced by any existing container. +- name: Remove only unused explicitly selected legacy images + ansible.builtin.command: + argv: [podman, image, rm, "{{ item.item }}"] + loop: "{{ server_legacy_image_presence.results }}" + loop_control: + label: "{{ item.item }}" + when: item.rc == 0 + register: server_legacy_image_removal + changed_when: server_legacy_image_removal.rc == 0 diff --git a/ansible/roles/profile_server/tasks/main.yml b/ansible/roles/profile_server/tasks/main.yml index 09f7a13..fb9450c 100644 --- a/ansible/roles/profile_server/tasks/main.yml +++ b/ansible/roles/profile_server/tasks/main.yml @@ -23,6 +23,9 @@ loop: "{{ server_directories | default([]) }}" loop_control: label: "{{ item.path }}" + when: + - item.path != '/opt/gitea/data' or not server_gitea_on_atlas | bool + - item.path != server_container_stack_dir or not server_legacy_stack_retired | bool - name: Copy server dotfiles tags: [dotfiles, dotfiles:server] @@ -48,19 +51,32 @@ loop_control: label: "{{ item.dest }}" no_log: "{{ item.no_log | default(false) }}" + when: item.src != 'server/docker-compose.yml.j2' or not server_legacy_stack_retired | bool - name: Manage Podman Compose stack tags: [services, podman] ansible.builtin.include_tasks: podman-compose.yml + when: not server_legacy_stack_retired | bool - name: Import staged NPM Quadlet tasks ansible.builtin.import_tasks: npm_quadlet.yml +- name: Import explicit legacy server image cleanup + ansible.builtin.import_tasks: legacy_image_cleanup.yml + tags: [never, server_image_cleanup] + when: server_legacy_image_cleanup | default(false) | bool + - name: Import Prometheus backup export identity tasks ansible.builtin.import_tasks: backup_export_identity.yml - name: Import Prometheus backup export job tasks ansible.builtin.import_tasks: backup_export_job.yml + tags: [server_legacy_cleanup] + +- name: Import explicitly approved legacy server data cleanup + ansible.builtin.import_tasks: legacy_cleanup.yml + tags: [never, server_legacy_cleanup] + when: server_legacy_cleanup | bool - name: Import explicit Prometheus Gitea final-export tasks ansible.builtin.import_tasks: gitea_final_export.yml diff --git a/ansible/roles/profile_server/templates/prometheus-backup-export.service.j2 b/ansible/roles/profile_server/templates/prometheus-backup-export.service.j2 index a12224f..c2a362b 100644 --- a/ansible/roles/profile_server/templates/prometheus-backup-export.service.j2 +++ b/ansible/roles/profile_server/templates/prometheus-backup-export.service.j2 @@ -1,6 +1,6 @@ [Unit] Description=Prepare a read-only Prometheus application backup for Atlas -RequiresMountsFor=/opt/npm /opt/gitea {{ server_backup_export_root }} +RequiresMountsFor=/opt/npm {% if not server_gitea_on_atlas | bool %}/opt/gitea {% endif %}{{ server_backup_export_root }} ConditionFileIsExecutable=/usr/local/sbin/prometheus-backup-export [Service] diff --git a/ansible/roles/profile_server/templates/prometheus-backup-export.sh.j2 b/ansible/roles/profile_server/templates/prometheus-backup-export.sh.j2 index ef5f77d..7002b5f 100644 --- a/ansible/roles/profile_server/templates/prometheus-backup-export.sh.j2 +++ b/ansible/roles/profile_server/templates/prometheus-backup-export.sh.j2 @@ -4,6 +4,9 @@ umask 077 export_root={{ server_backup_export_root | quote }} versions="$export_root/versions" +{% if server_legacy_stack_retired | bool %} +stack_unit=prometheus-npm.service +{% else %} stack_unit='' compose_active=false quadlet_active=false @@ -18,6 +21,7 @@ if "$quadlet_active"; then else stack_unit=podman-compose-server.service fi +{% endif %} stamp=$(date -u +%Y%m%dT%H%M%SZ) stage='' stack_stopped=false diff --git a/docs/atlas-gitea-migration.md b/docs/atlas-gitea-migration.md index 86559b0..25215f6 100644 --- a/docs/atlas-gitea-migration.md +++ b/docs/atlas-gitea-migration.md @@ -1,8 +1,13 @@ # Gitea migration from Prometheus to Atlas This records the staged migration and its observed partial cutover. Gitea is -temporary on Atlas until Uranus; NPM remains on Prometheus. Preserve the old -Prometheus data, but do not restart its stale Gitea after Atlas accepts writes. +temporary on Atlas until Uranus; NPM remains on Prometheus. On 2026-10-03 +the operator explicitly approved removal of the old Prometheus Gitea data, +SSH fragment and final-export helper. NPM now uses a rootful Quadlet with no +installed Compose fallback. The source-retention and rollback steps below +are historical migration gates, not current recovery instructions. +Existing backup archives were preserved; use current Atlas data and verified +backups for recovery. Do not recreate or restart stale source Gitea. ## Observed source before cutover and chosen topology (2026-10-01) @@ -139,6 +144,10 @@ or credentials were changed. The secondary hostname `git.ov-ad3410.infomaniak.ch` did not resolve from Ikaros and had no generated NPM config file at the time of inspection. +On 2026-10-03 the operator retired this unused secondary hostname. Its NPM +Proxy Host was already soft-deleted; Ansible now declares only +`git.fscotto.duckdns.org` and removes the secondary runtime override. + Prometheus' public TCP/2222 socket proxies to Atlas without changing admin SSH/22. The local socket presents the preserved Gitea ED25519 host key, but an external TCP/2222 connection from Ikaros initially timed out. During that diff --git a/docs/prometheus-backup.md b/docs/prometheus-backup.md index 97504c8..b510c1a 100644 --- a/docs/prometheus-backup.md +++ b/docs/prometheus-backup.md @@ -10,15 +10,15 @@ cutover is still pending. See `docs/prometheus-npm-quadlet.md`. ## Declared design - Prometheus prepares a tar archive of Nginx Proxy Manager data and certificates, - its active Quadlet and network definitions, the disabled Compose fallback, + its active Quadlet and network definitions, and SSH/firewalld/WireGuard configuration. Gitea now runs on Atlas and is no longer included in new Prometheus exports. NPM access logs are excluded. The archive contains credentials, certificates, and the WireGuard private key: protect both copies accordingly. -- The approved consistency mode stops the one active NPM service (Quadlet now, - Compose before cutover) for local tar creation at 02:00 Europe/Rome, then - restarts it even if archiving fails. The helper refuses both services active - or both inactive. A manual test outside that window requires separate approval. +- The approved consistency mode stops the NPM Quadlet for local tar creation + at 02:00 Europe/Rome, then restarts it even if archiving fails. After the + approved legacy cleanup, the helper requires the Quadlet active and has + no Compose dependency. A manual test outside that window requires separate approval. - Prometheus publishes the archive with its checksum as a versioned, read-only source under `/var/lib/prometheus-backup-export`. A locked service account has no sudo or supplementary groups. Its only authorized SSH key is forced @@ -119,3 +119,13 @@ both Quadlet definitions. A manifest of all 70 regular Let's Encrypt files and 12 symlinks, including content hashes and link targets, matched the live Prometheus tree. No private key or secret content was printed. The next scheduled export/pull is still pending observation. + +## Post-cleanup validation (2026-10-03) + +The operator-approved removal of legacy data and Compose fallback also +removed those backup input paths and the obsolete Gitea mount dependency. +A separately approved export and Atlas pull published `20261003T112906Z`. +Both SHA-256 checks passed; an isolated SQLite restore passed `quick_check` +and contained ten proxy hosts. Both active Quadlet definitions were present; +retired paths were absent. Existing backup archives were not deleted by cleanup. +The first scheduled cycle after these changes remains unverified. diff --git a/docs/prometheus-npm-quadlet.md b/docs/prometheus-npm-quadlet.md index 61b772c..38fd9f0 100644 --- a/docs/prometheus-npm-quadlet.md +++ b/docs/prometheus-npm-quadlet.md @@ -8,19 +8,20 @@ Nginx Proxy Manager runs as the **rootful** generated `/etc/containers/systemd/server-web.network`; the image is pinned by digest in `ansible/inventory/host_vars/prometheus.yml`. The generated service is wanted by `multi-user.target` and requires the generated network service. -The old `podman-compose-server.service` is inactive and disabled. Its unit -and Compose file remain as a rollback option, not as another active owner. -Do not start both units or run `podman-compose down` while the Quadlet owns -the shared `server_web` network. +The old Compose unit, Compose file and Gitea final-export helper were +removed by the operator-approved cleanup on 2026-10-03. The retired +application data and empty legacy directories were also removed. +Prometheus host vars set `server_legacy_stack_retired: true` so normal runs +do not recreate those files. Destructive deletion still requires a separate +cleanup tag and explicit extra-var. There was **no data copy** in this cutover. The Quadlet reuses the existing `/opt/npm/data:/data` and `/opt/npm/letsencrypt:/etc/letsencrypt` bind mounts with the same container name and `server_web` bridge (`10.89.0.0/24`). Ports 80 and 443 remain public; administration port 81 remains bound to `127.0.0.1`. Gitea stays on Atlas, and NPM remains on Prometheus. The -Prometheus Compose file is retained with the same pinned NPM image for a -controlled fallback. The Quadlet uses `Pull=missing`, not an automatic -floating-tag update. +Compose fallback is no longer installed. The Quadlet uses `Pull=missing`, +not an automatic floating-tag update. ## Cutover and recovery boundaries @@ -36,16 +37,12 @@ then restarted the timer. Its failure trap would have restarted Compose. **Do not rerun that forward-cutover script after success**: its preconditions intentionally reject an active Quadlet. -A future rollback is a separate outage decision, not an ordinary Ansible run. -First verify a usable recent Atlas backup and stop the export timer. Stop -the Quadlet and verify that its container is gone before allowing Compose -to own the same name, mounts, network, and ports; use the pinned Compose -configuration, then validate NPM/HTTPS and restart the timer. Set -`server_npm_quadlet_cutover: false` only as part of that controlled rollback. -Do not run the two owners concurrently, restore an old NPM database over a -live instance, or delete either bind mount. This reverse procedure has not -been exercised on production; the forward script's in-window rollback path -is not evidence of a later reverse cutover. +Recovery is now a Quadlet rebuild and restoration from a verified Atlas +backup, with an explicit outage decision before replacing live NPM state. +The old Compose owner is no longer installed; reintroducing it would require +a separately reviewed configuration and outage plan. The historical +in-window rollback trap is not a supported post-cleanup rollback procedure. +Do not restore an old database over a live instance or remove NPM bind mounts. ## Verified evidence @@ -81,10 +78,68 @@ the new path works but not its next scheduled execution. ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \ ansible-playbook ansible/site.yml --limit prometheus --tags npm_quadlet --check --diff sudo systemctl status prometheus-npm.service prometheus-backup-export.timer -sudo systemctl is-active podman-compose-server.service -sudo systemctl is-enabled podman-compose-server.service +sudo systemctl show podman-compose-server.service -p LoadState # expected: not-found ``` The backup archive includes credentials, certificates, and WireGuard configuration. Do not publish it or print its contents in diagnostics; see `docs/prometheus-backup.md` for the restricted pull and restore procedure. + +## Selective legacy image cleanup + +On 2026-10-03 opt-in Ansible tasks removed only the unused Gitea 1.25.2, +Navidrome latest and PostgreSQL 13 rootful images, without force or global +prune. Podman refuses images referenced by existing containers. The second +run changed nothing. NPM remained active with zero restarts; local admin +and public Gitea HTTPS returned 200. Backup timer and SSH proxy stayed active. + +Validation: +```bash +ansible-playbook ansible/site.yml --limit prometheus --tags server_image_cleanup --check --diff -e server_legacy_image_cleanup=true +``` + +The image cleanup defaults to disabled and carries the `never` tag. +Check mode probes image presence but skips removal; it does not prove +Podman would accept deletion. It never removes NPM resources. + +## Approved legacy data and fallback cleanup + +The operator explicitly approved deletion on 2026-10-03. The separate +`server_legacy_cleanup` tasks removed `/opt/gitea`, `/home/git/.ssh`, +`/opt/navidrome`, `/opt/postgres`, `/opt/music`, `/opt/containerd`, +`/opt/docker`, the old Compose unit and the final Gitea export helper. +The empty `/home/git` parent is removed only with `rmdir`, after confirming +the Git account is absent. Guards reject symlinked paths, nested mounts, +unexpected containers, container users of these paths, unexpected content +in the empty legacy trees, and an active Compose or export service. +The second cleanup run changed nothing. + +Before deletion, Ansible removed obsolete backup input paths and the +Gitea mount dependency. Normal Compose/template/final-export task checks +changed nothing and did not recreate the retired files. Deletion is opt-in: + +```bash +ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true +``` + +Remove check mode only for approved deletion. No active NPM data, certificate, +image, network, volume, SSH proxy, WireGuard configuration or backup archive +is removed. No services were restarted by the cleanup. + +After separate approval for the brief managed NPM pause, the new export +`20261003T112906Z` completed successfully and was pulled to Atlas. SHA-256 +passed on both hosts; an isolated SQLite restore passed `quick_check` and +contained ten proxy hosts. Both Quadlet definitions were present, and +retired paths were absent. Temporary restore files were removed. +NPM was active with zero automatic restarts; primary public Gitea HTTPS +returned 200 with valid TLS. Backup timer, SSH proxy and WireGuard stayed active. +The first scheduled post-cleanup cycle remains unverified. + +After separate operator approval on 2026-10-03, the unused secondary hostname +`git.ov-ad3410.infomaniak.ch` was removed from the declared domains and +the managed NPM runtime override. Its Proxy Host (id 10) was already +soft-deleted, with no generated config or associated certificate. Historical +deleted records and backup archives are preserved; no DNS changes were made. +Only `git.fscotto.duckdns.org` remains declared for the Gitea override. +Nginx validation and reload passed without restarting NPM; the primary +public HTTPS endpoint returned 200 with valid TLS.