Retire Prometheus Compose stack and document cleanup

This commit is contained in:
Fabio Scotto di Santolo
2026-10-03 13:44:57 +02:00
parent 7bc7f0e645
commit 755f24bc72
15 changed files with 332 additions and 50 deletions

View File

@@ -54,8 +54,9 @@ Ansible-driven personal infrastructure repo for Fedora and Void desktops, Fedora
- Emacs is disabled by default; temporary Emacs check: `ansible-playbook ansible/site.yml --limit <host> --tags emacs --check --diff -e emacs_enabled=true`
- AI coding agents: `ansible-playbook ansible/site.yml --limit <host> --tags ai_agents --check --diff`
- Mail bootstrap: `sh -n scripts/bootstrap_mail.sh` and `shellcheck scripts/bootstrap_mail.sh`
- Server NPM Quadlet: `systemctl status prometheus-npm.service`; disabled Compose fallback render:
`podman-compose -f /opt/docker/server/docker-compose.yml config`
- Server NPM Quadlet: `systemctl status prometheus-npm.service`; the Compose fallback is retired.
- Explicit Prometheus legacy cleanup (destructive only without check mode):
`ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true`
- Atlas media stack:
`ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff`
- Atlas rootless Gitea staging (does not start Gitea):
@@ -143,12 +144,15 @@ The dotfile vars follow the same split: `desktop_common_dotfiles` carries mode-i
and disables diffs. Provisioning does not execute the updater or change its external schedule.
- `rocky_server` is a child of both `platform_rocky` and `server`; `prometheus` is its active target.
- The target must already provide `server_username` with local sudo access before the profile runs.
- The Rocky profile installs Podman and podman-compose and renders the disabled legacy
`podman-compose-server` unit for rollback. On Prometheus, Nginx Proxy Manager is now the rootful
- The Rocky profile installs Podman and podman-compose. Prometheus explicitly retires the legacy
Compose unit, files and final-export helper with `server_legacy_stack_retired: true`.
Its approved opt-in cleanup removed old application data on 2026-10-03; normal runs do not
delete data or recreate the retired files. On Prometheus, Nginx Proxy Manager is now the rootful
`prometheus-npm.service` Quadlet with a pinned image digest and the existing `/opt/npm/data` and
`/opt/npm/letsencrypt` bind mounts. The rootful `server_web` bridge remains `10.89.0.0/24`.
Gitea runs on Atlas; PostgreSQL and Navidrome are absent from the desired Prometheus stack.
The profile does not delete legacy data, update DNS, or perform an implicit cutover.
Normal runs do not delete legacy data, update DNS, or perform an implicit cutover;
destructive cleanup requires its explicit tag and opt-in extra-var.
- Firewalld enables SSH, Cockpit (`9090/tcp`), HTTP and HTTPS. Nginx Proxy Manager publishes `80/tcp` and
`443/tcp`; bind its administration interface only to `127.0.0.1:81` and use `npm-tunnel` from Ikaros or Nymph.
Nextcloud remains disabled; do not provision `/srv/nextcloud` directories.
@@ -413,6 +417,22 @@ successfully. The first monthly scrub remains a runtime check.
both Quadlet files were present, and the complete Let's Encrypt tree (70 regular files plus
12 symlinks) matched the live data. A targeted normal Ansible run changed nothing. Details and rollback
boundaries are in `docs/prometheus-npm-quadlet.md`.
- [x] Remove only unused Gitea, Navidrome and PostgreSQL images with opt-in
Ansible tasks on 2026-10-03. Second run changed nothing; NPM stayed active
with zero restarts, HTTP/HTTPS passed, backup timer and SSH proxy stayed active.
This image-only step preserved data and fallback; the later approved deletion is tracked below. Validation:
`ansible-playbook ansible/site.yml --limit prometheus --tags server_image_cleanup --check --diff -e server_legacy_image_cleanup=true`
- [x] Complete explicitly approved old-data and Compose fallback removal on 2026-10-03.
Backup paths and mount dependencies were reconciled before deletion; repeat cleanup changed
nothing. The normal Compose/template/helper check did not recreate retired files.
A separately approved manual export/pull published `20261003T112906Z`; checksum and isolated
SQLite restore passed with ten proxy hosts and both Quadlet definitions. NPM, primary HTTPS,
WireGuard, SSH proxy and backup timer remained healthy; existing backup archives were preserved.
- [x] Retire the unused secondary Gitea hostname `git.ov-ad3410.infomaniak.ch`
on 2026-10-03. Its NPM Proxy Host was already soft-deleted and had no
associated certificate. Its Ansible domain and runtime override were removed;
nginx -t and reload passed without restarting NPM. Primary HTTPS returned 200
with valid TLS; only `git.fscotto.duckdns.org` remains declared for Gitea.
- [ ] Observe the first scheduled export and Atlas pull after the cutover; the manual end-to-end
cycle passed, but the next unattended cycle has not yet occurred.

View File

@@ -202,14 +202,17 @@ Lo stato attuale del profilo server include:
- installazione di Podman e podman-compose
- abilitazione dei servizi systemd dichiarati in inventory/group vars
- copia dei dotfiles server e rendering del Quadlet rootful `prometheus-npm.service` per Nginx Proxy
Manager; il vecchio `podman-compose-server` resta disabilitato soltanto per un rollback controllato
Manager; il vecchio fallback Compose è stato rimosso con autorizzazione esplicita
- attivazione di firewalld con SSH, Cockpit (`9090/tcp`), HTTP e HTTPS abilitati
- Syncthing escluso dal profilo server Rocky
Il Compose desiderato su Prometheus non include piu Gitea, Navidrome ne il database PostgreSQL obsoleto.
Navidrome e Syncthing appartengono ad Atlas; Navidrome ufficiale usa invece SQLite. Il profilo non
arresta o rimuove automaticamente eventuali container legacy e non elimina `/opt/postgres/data`.
Il cutover NPM, le verifiche dei dati e del backup e i limiti del rollback sono documentati in
Il 2026-10-03 la pulizia opt-in autorizzata ha rimosso dati e immagini precedenti di Gitea,
Navidrome e PostgreSQL, directory obsolete vuote, helper finale Gitea e fallback Compose NPM.
I servizi migrati restano su Atlas. `server_legacy_stack_retired: true` evita che i normali task
ricreino i residui; la cancellazione richiede `--tags server_legacy_cleanup` e
`-e server_legacy_cleanup=true`. NPM attivo e archivi di backup restano intatti.
Export, pull Atlas e restore SQLite isolato post-pulizia sono riusciti; il primo ciclo automatico
resta da osservare. Evidenze e confini del recovery:
[`docs/prometheus-npm-quadlet.md`](docs/prometheus-npm-quadlet.md).
Nginx Proxy Manager pubblica solo `80/tcp` e `443/tcp`; la sua interfaccia di amministrazione e
@@ -642,7 +645,7 @@ Questo significa che, allo stato attuale:
- il server Rocky (`prometheus`) e gestito con pacchetti, servizi, dotfiles server e firewalld
- il NAS Rocky (`atlas`) usa un pool ZFS gia esistente, condivisioni NFSv4/SMB limitate alla LAN e Cockpit/45Drives
- NPM è un Quadlet rootful su Prometheus, mentre Gitea, Navidrome e Syncthing sono Quadlet
rootless su Atlas; il Compose server resta disabilitato per rollback
rootless su Atlas; il fallback Compose server è stato rimosso
# Dotfiles
@@ -748,7 +751,7 @@ ansible-playbook ansible/site.yml --limit <host> --tags <tag1>,<tag2> --check --
ansible-playbook ansible/site.yml --limit <host> --start-at-task "<task name>" --check --diff
ansible-lint ansible/roles/<role>
yamllint ansible/path/to/file.yml
podman-compose -f /opt/docker/server/docker-compose.yml config
ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true
ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff
ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 --check --diff
ansible-playbook ansible/site.yml --limit atlas --tags music_sync --check --diff

View File

@@ -130,14 +130,18 @@ dotfiles and templates. The profile does not transfer application data, update D
implicit service cutover.
The server profile installs platform-specific packages, Podman and podman-compose, declared systemd
services, and firewalld. Nginx Proxy Manager now runs as the rootful `prometheus-npm.service` Quadlet;
the old `podman-compose-server` unit is disabled and retained only for controlled rollback. The
Compose file no longer includes Gitea, Navidrome, Syncthing, or the obsolete Navidrome PostgreSQL
database. The temporary Navidrome and Syncthing deployment on Atlas is managed by
`profile_backend_phase1`. Applying the profile does not delete legacy data or `/opt/postgres/data`.
The NPM cutover, data checks, backup evidence, and rollback boundaries are documented in
services, and firewalld. Nginx Proxy Manager runs as the rootful `prometheus-npm.service` Quadlet.
On 2026-10-03 the operator-approved opt-in cleanup removed old Gitea, Navidrome and PostgreSQL
data/images, empty legacy directories, the Gitea final-export helper and the Compose rollback files.
The migrated services stay on Atlas. `server_legacy_stack_retired: true` prevents normal runs from
recreating retired files. Data deletion requires `--tags server_legacy_cleanup` and
`-e server_legacy_cleanup=true`; image-only cleanup has its own `server_image_cleanup` tag and flag.
Active NPM resources and existing backup archives remain preserved.
The post-cleanup export/pull and isolated SQLite restore passed; the first unattended cycle remains
pending. Evidence and recovery boundaries:
[`docs/prometheus-npm-quadlet.md`](docs/prometheus-npm-quadlet.md).
Firewalld enables SSH, Cockpit (`9090/tcp`), HTTP and HTTPS. Nginx Proxy Manager publishes only
`80/tcp` and `443/tcp`; its administration interface is bound to `127.0.0.1:81` and can be reached
from Ikaros or Nymph with the `npm-tunnel` Bash alias. Nextcloud remains disabled and the profile
@@ -738,7 +742,7 @@ ansible-playbook ansible/site.yml --limit <host> --tags <tag1>,<tag2> --check --
ansible-playbook ansible/site.yml --limit <host> --start-at-task "<task name>" --check --diff
ansible-lint ansible/roles/<role>
yamllint ansible/path/to/file.yml
podman-compose -f /opt/docker/server/docker-compose.yml config
ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true
ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff
ansible-playbook ansible/site.yml --limit atlas --tags music_sync --check --diff
```

View File

@@ -8,6 +8,8 @@ effective_user_home: "{{ server_user_home }}"
server_container_stack_dir: /opt/docker/server
server_npm_quadlet_stage: false
server_npm_quadlet_cutover: false
server_legacy_stack_retired: false
server_legacy_cleanup: false
ai_agents: {}
vim_plugins_enabled: false
@@ -101,8 +103,9 @@ server_backup_export_source_keep: 3
server_backup_export_paths: >-
{{ ['opt/npm/data', 'opt/npm/letsencrypt']
+ ([] if server_gitea_on_atlas | bool else ['opt/gitea/data', 'home/git/.ssh'])
+ ['opt/docker/server/docker-compose.yml',
'etc/systemd/system/podman-compose-server.service']
+ ([] if server_legacy_stack_retired | bool else
['opt/docker/server/docker-compose.yml',
'etc/systemd/system/podman-compose-server.service'])
+ (['etc/containers/systemd/prometheus-npm.container',
'etc/containers/systemd/server-web.network']
if server_npm_quadlet_stage | bool else [])

View File

@@ -6,6 +6,15 @@ ansible_port: 22
ansible_ssh_private_key_file: /home/fscotto/.ssh/id_ed25519
server_username: rocky
server_legacy_stack_retired: true
# Destructive deletion runs only with an explicit extra-var and cleanup tag.
server_legacy_cleanup: false
# Explicit opt-in cleanup; no data, volumes, networks or NPM images are removed.
server_legacy_image_cleanup: false
server_legacy_images:
- docker.gitea.com/gitea:1.25.2
- docker.io/deluan/navidrome:latest
- docker.io/library/postgres:13
server_npm_quadlet_stage: true
server_npm_quadlet_image: docker.io/jc21/nginx-proxy-manager@sha256:52b2c59994f3d36acfcf70a1626f29734df0ed8c71bacc0269f78b6f939858bb
# The stopped-source export and live Quadlet cutover passed on 2026-10-03.
@@ -17,7 +26,6 @@ server_gitea_cutover_tools_enabled: true
server_gitea_on_atlas: true
server_gitea_npm_domains:
- git.fscotto.duckdns.org
- git.ov-ad3410.infomaniak.ch
server_duckdns_domain: fscotto
server_ssh_authorized_keys:
- name: ikaros

View File

@@ -51,6 +51,7 @@
owner: root
group: root
mode: "0750"
validate: "bash -n %s"
when: server_backup_export_enabled | bool
- name: Install Prometheus backup export systemd units

View File

@@ -7,7 +7,9 @@
owner: root
group: root
mode: "0750"
when: server_gitea_cutover_tools_enabled | bool
when:
- server_gitea_cutover_tools_enabled | bool
- not server_legacy_stack_retired | bool
- name: Require the prepared source and explicit final-export approval
tags: [services, gitea_final_export]
@@ -15,6 +17,7 @@
that:
- server_gitea_cutover_tools_enabled | bool
- server_backup_export_enabled | bool
- not server_legacy_stack_retired | bool
- not ansible_check_mode
fail_msg: >-
Install the cutover helper and perform an explicit non-check-mode run
@@ -31,4 +34,5 @@
no_log: true
when:
- server_gitea_final_export | bool
- not server_legacy_stack_retired | bool
- not ansible_check_mode

View File

@@ -0,0 +1,112 @@
---
- name: Require explicit retirement of the migrated Prometheus source
ansible.builtin.assert:
that:
- inventory_hostname == 'prometheus'
- server_legacy_stack_retired | bool
- server_gitea_on_atlas | bool
- server_npm_quadlet_cutover | bool
- server_backup_export_enabled | bool
- name: Verify legacy paths have no mounts or container users
ansible.builtin.command:
argv:
- python3
- -c
- |
import json, os, pathlib, subprocess
def run(*args):
return subprocess.check_output(args, text=True).strip()
paths = ['/opt/gitea', '/home/git/.ssh', '/opt/navidrome',
'/opt/postgres', '/opt/music', '/opt/containerd', '/opt/docker']
mounts = json.loads(run('findmnt', '--json', '--list', '-o', 'TARGET'))['filesystems']
for path in paths:
assert os.path.realpath(path) == path, 'Symlink in cleanup path: ' + path
for mount in mounts:
target = mount['target']
assert target != path and not target.startswith(path + '/'), 'Mounted cleanup path: ' + path
ids = run('podman', 'ps', '-aq').split()
containers = json.loads(run('podman', 'inspect', *ids)) if ids else []
for container in containers:
assert container['Name'].lstrip('/') == 'nginx-proxy-manager', 'Unexpected container; review before cleanup'
for mount in container.get('Mounts', []):
source = os.path.realpath(mount['Source'])
for path in paths:
assert source != path and not source.startswith(path + '/'), 'Container uses cleanup path: ' + path
for path in ['/opt/music', '/opt/containerd']:
if os.path.isdir(path):
for entry in pathlib.Path(path).rglob('*'):
assert entry.is_dir() and not entry.is_symlink(), 'Unexpected file in empty legacy path: ' + str(entry)
if os.path.isdir('/opt/docker'):
allowed = {'/opt/docker/server', '/opt/docker/server/docker-compose.yml'}
for entry in pathlib.Path('/opt/docker').rglob('*'):
assert str(entry) in allowed and not entry.is_symlink(), 'Unexpected legacy Docker content: ' + str(entry)
assert run('systemctl', 'is-active', 'prometheus-npm.service') == 'active'
assert subprocess.run(['systemctl', 'is-active', '--quiet', 'podman-compose-server.service']).returncode != 0
assert subprocess.run(['systemctl', 'is-active', '--quiet', 'prometheus-backup-export.service']).returncode != 0
print('Legacy cleanup preflight passed')
changed_when: false
check_mode: false
- name: Require the updated backup configuration before deleting fallback files
ansible.builtin.command:
argv:
- python3
- -c
- |
import pathlib, subprocess
unit = subprocess.check_output(['systemctl', 'show', 'prometheus-backup-export.service',
'-p', 'RequiresMountsFor', '--value'], text=True)
assert '/opt/gitea' not in unit, 'Backup unit still depends on legacy Gitea'
helper = pathlib.Path('/usr/local/sbin/prometheus-backup-export').read_text()
assert 'podman-compose-server' not in helper and 'opt/docker/server' not in helper
subprocess.run(['bash', '-n', '/usr/local/sbin/prometheus-backup-export'], check=True)
changed_when: false
when: not ansible_check_mode
- name: Delete only the explicitly approved legacy data and fallback files
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop:
- /opt/gitea
- /home/git/.ssh
- /opt/navidrome
- /opt/postgres
- /opt/music
- /opt/containerd
- /opt/docker
- /usr/local/sbin/prometheus-gitea-final-export
- /etc/systemd/system/podman-compose-server.service
register: server_legacy_deleted
diff: false
- name: Reload systemd after removing the inactive legacy unit
ansible.builtin.systemd:
daemon_reload: true
when:
- server_legacy_deleted is changed
- not ansible_check_mode
- name: Inspect the obsolete Git home without following symlinks
ansible.builtin.stat:
path: /home/git
follow: false
register: server_legacy_git_home
- name: Require the obsolete Git account to be absent before removing its empty home
ansible.builtin.command:
argv: [getent, passwd, git]
register: server_legacy_git_account
changed_when: false
failed_when: server_legacy_git_account.rc != 2
check_mode: false
when: server_legacy_git_home.stat.exists
# rmdir refuses any nonempty directory; never recursively delete this parent.
- name: Remove only the empty obsolete Git home
ansible.builtin.command:
argv: [rmdir, /home/git]
register: server_legacy_git_home_removed
changed_when: server_legacy_git_home_removed.rc == 0
when: server_legacy_git_home.stat.exists

View File

@@ -0,0 +1,33 @@
---
- name: Require the migrated Prometheus topology for image cleanup
ansible.builtin.assert:
that:
- inventory_hostname == 'prometheus'
- server_gitea_on_atlas | bool
- server_npm_quadlet_cutover | bool
- server_legacy_images | default([]) | length > 0
- >-
server_legacy_images | difference([
'docker.gitea.com/gitea:1.25.2',
'docker.io/deluan/navidrome:latest',
'docker.io/library/postgres:13']) | length == 0
- name: Check whether the explicitly selected legacy images exist
ansible.builtin.command:
argv: [podman, image, exists, "{{ item }}"]
loop: "{{ server_legacy_images }}"
register: server_legacy_image_presence
changed_when: false
failed_when: server_legacy_image_presence.rc not in [0, 1]
check_mode: false
# No --force: Podman must refuse images referenced by any existing container.
- name: Remove only unused explicitly selected legacy images
ansible.builtin.command:
argv: [podman, image, rm, "{{ item.item }}"]
loop: "{{ server_legacy_image_presence.results }}"
loop_control:
label: "{{ item.item }}"
when: item.rc == 0
register: server_legacy_image_removal
changed_when: server_legacy_image_removal.rc == 0

View File

@@ -23,6 +23,9 @@
loop: "{{ server_directories | default([]) }}"
loop_control:
label: "{{ item.path }}"
when:
- item.path != '/opt/gitea/data' or not server_gitea_on_atlas | bool
- item.path != server_container_stack_dir or not server_legacy_stack_retired | bool
- name: Copy server dotfiles
tags: [dotfiles, dotfiles:server]
@@ -48,19 +51,32 @@
loop_control:
label: "{{ item.dest }}"
no_log: "{{ item.no_log | default(false) }}"
when: item.src != 'server/docker-compose.yml.j2' or not server_legacy_stack_retired | bool
- name: Manage Podman Compose stack
tags: [services, podman]
ansible.builtin.include_tasks: podman-compose.yml
when: not server_legacy_stack_retired | bool
- name: Import staged NPM Quadlet tasks
ansible.builtin.import_tasks: npm_quadlet.yml
- name: Import explicit legacy server image cleanup
ansible.builtin.import_tasks: legacy_image_cleanup.yml
tags: [never, server_image_cleanup]
when: server_legacy_image_cleanup | default(false) | bool
- name: Import Prometheus backup export identity tasks
ansible.builtin.import_tasks: backup_export_identity.yml
- name: Import Prometheus backup export job tasks
ansible.builtin.import_tasks: backup_export_job.yml
tags: [server_legacy_cleanup]
- name: Import explicitly approved legacy server data cleanup
ansible.builtin.import_tasks: legacy_cleanup.yml
tags: [never, server_legacy_cleanup]
when: server_legacy_cleanup | bool
- name: Import explicit Prometheus Gitea final-export tasks
ansible.builtin.import_tasks: gitea_final_export.yml

View File

@@ -1,6 +1,6 @@
[Unit]
Description=Prepare a read-only Prometheus application backup for Atlas
RequiresMountsFor=/opt/npm /opt/gitea {{ server_backup_export_root }}
RequiresMountsFor=/opt/npm {% if not server_gitea_on_atlas | bool %}/opt/gitea {% endif %}{{ server_backup_export_root }}
ConditionFileIsExecutable=/usr/local/sbin/prometheus-backup-export
[Service]

View File

@@ -4,6 +4,9 @@ umask 077
export_root={{ server_backup_export_root | quote }}
versions="$export_root/versions"
{% if server_legacy_stack_retired | bool %}
stack_unit=prometheus-npm.service
{% else %}
stack_unit=''
compose_active=false
quadlet_active=false
@@ -18,6 +21,7 @@ if "$quadlet_active"; then
else
stack_unit=podman-compose-server.service
fi
{% endif %}
stamp=$(date -u +%Y%m%dT%H%M%SZ)
stage=''
stack_stopped=false

View File

@@ -1,8 +1,13 @@
# Gitea migration from Prometheus to Atlas
This records the staged migration and its observed partial cutover. Gitea is
temporary on Atlas until Uranus; NPM remains on Prometheus. Preserve the old
Prometheus data, but do not restart its stale Gitea after Atlas accepts writes.
temporary on Atlas until Uranus; NPM remains on Prometheus. On 2026-10-03
the operator explicitly approved removal of the old Prometheus Gitea data,
SSH fragment and final-export helper. NPM now uses a rootful Quadlet with no
installed Compose fallback. The source-retention and rollback steps below
are historical migration gates, not current recovery instructions.
Existing backup archives were preserved; use current Atlas data and verified
backups for recovery. Do not recreate or restart stale source Gitea.
## Observed source before cutover and chosen topology (2026-10-01)
@@ -139,6 +144,10 @@ or credentials were changed. The
secondary hostname `git.ov-ad3410.infomaniak.ch` did not resolve from Ikaros
and had no generated NPM config file at the time of inspection.
On 2026-10-03 the operator retired this unused secondary hostname. Its NPM
Proxy Host was already soft-deleted; Ansible now declares only
`git.fscotto.duckdns.org` and removes the secondary runtime override.
Prometheus' public TCP/2222 socket proxies to Atlas without changing admin
SSH/22. The local socket presents the preserved Gitea ED25519 host key, but
an external TCP/2222 connection from Ikaros initially timed out. During that

View File

@@ -10,15 +10,15 @@ cutover is still pending. See `docs/prometheus-npm-quadlet.md`.
## Declared design
- Prometheus prepares a tar archive of Nginx Proxy Manager data and certificates,
its active Quadlet and network definitions, the disabled Compose fallback,
its active Quadlet and network definitions,
and SSH/firewalld/WireGuard configuration. Gitea now runs on Atlas and is no
longer included in new Prometheus exports. NPM access logs are excluded.
The archive contains credentials, certificates, and the WireGuard private
key: protect both copies accordingly.
- The approved consistency mode stops the one active NPM service (Quadlet now,
Compose before cutover) for local tar creation at 02:00 Europe/Rome, then
restarts it even if archiving fails. The helper refuses both services active
or both inactive. A manual test outside that window requires separate approval.
- The approved consistency mode stops the NPM Quadlet for local tar creation
at 02:00 Europe/Rome, then restarts it even if archiving fails. After the
approved legacy cleanup, the helper requires the Quadlet active and has
no Compose dependency. A manual test outside that window requires separate approval.
- Prometheus publishes the archive with its checksum as a versioned, read-only
source under `/var/lib/prometheus-backup-export`. A locked service account
has no sudo or supplementary groups. Its only authorized SSH key is forced
@@ -119,3 +119,13 @@ both Quadlet definitions. A manifest of all 70 regular Let's Encrypt files
and 12 symlinks, including content hashes and link targets, matched the live
Prometheus tree. No private key or secret content was printed. The next
scheduled export/pull is still pending observation.
## Post-cleanup validation (2026-10-03)
The operator-approved removal of legacy data and Compose fallback also
removed those backup input paths and the obsolete Gitea mount dependency.
A separately approved export and Atlas pull published `20261003T112906Z`.
Both SHA-256 checks passed; an isolated SQLite restore passed `quick_check`
and contained ten proxy hosts. Both active Quadlet definitions were present;
retired paths were absent. Existing backup archives were not deleted by cleanup.
The first scheduled cycle after these changes remains unverified.

View File

@@ -8,19 +8,20 @@ Nginx Proxy Manager runs as the **rootful** generated
`/etc/containers/systemd/server-web.network`; the image is pinned by digest
in `ansible/inventory/host_vars/prometheus.yml`. The generated service is
wanted by `multi-user.target` and requires the generated network service.
The old `podman-compose-server.service` is inactive and disabled. Its unit
and Compose file remain as a rollback option, not as another active owner.
Do not start both units or run `podman-compose down` while the Quadlet owns
the shared `server_web` network.
The old Compose unit, Compose file and Gitea final-export helper were
removed by the operator-approved cleanup on 2026-10-03. The retired
application data and empty legacy directories were also removed.
Prometheus host vars set `server_legacy_stack_retired: true` so normal runs
do not recreate those files. Destructive deletion still requires a separate
cleanup tag and explicit extra-var.
There was **no data copy** in this cutover. The Quadlet reuses the existing
`/opt/npm/data:/data` and `/opt/npm/letsencrypt:/etc/letsencrypt` bind mounts
with the same container name and `server_web` bridge (`10.89.0.0/24`). Ports
80 and 443 remain public; administration port 81 remains bound to
`127.0.0.1`. Gitea stays on Atlas, and NPM remains on Prometheus. The
Prometheus Compose file is retained with the same pinned NPM image for a
controlled fallback. The Quadlet uses `Pull=missing`, not an automatic
floating-tag update.
Compose fallback is no longer installed. The Quadlet uses `Pull=missing`,
not an automatic floating-tag update.
## Cutover and recovery boundaries
@@ -36,16 +37,12 @@ then restarted the timer. Its failure trap would have restarted Compose.
**Do not rerun that forward-cutover script after success**: its preconditions
intentionally reject an active Quadlet.
A future rollback is a separate outage decision, not an ordinary Ansible run.
First verify a usable recent Atlas backup and stop the export timer. Stop
the Quadlet and verify that its container is gone before allowing Compose
to own the same name, mounts, network, and ports; use the pinned Compose
configuration, then validate NPM/HTTPS and restart the timer. Set
`server_npm_quadlet_cutover: false` only as part of that controlled rollback.
Do not run the two owners concurrently, restore an old NPM database over a
live instance, or delete either bind mount. This reverse procedure has not
been exercised on production; the forward script's in-window rollback path
is not evidence of a later reverse cutover.
Recovery is now a Quadlet rebuild and restoration from a verified Atlas
backup, with an explicit outage decision before replacing live NPM state.
The old Compose owner is no longer installed; reintroducing it would require
a separately reviewed configuration and outage plan. The historical
in-window rollback trap is not a supported post-cleanup rollback procedure.
Do not restore an old database over a live instance or remove NPM bind mounts.
## Verified evidence
@@ -81,10 +78,68 @@ the new path works but not its next scheduled execution.
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit prometheus --tags npm_quadlet --check --diff
sudo systemctl status prometheus-npm.service prometheus-backup-export.timer
sudo systemctl is-active podman-compose-server.service
sudo systemctl is-enabled podman-compose-server.service
sudo systemctl show podman-compose-server.service -p LoadState # expected: not-found
```
The backup archive includes credentials, certificates, and WireGuard
configuration. Do not publish it or print its contents in diagnostics; see
`docs/prometheus-backup.md` for the restricted pull and restore procedure.
## Selective legacy image cleanup
On 2026-10-03 opt-in Ansible tasks removed only the unused Gitea 1.25.2,
Navidrome latest and PostgreSQL 13 rootful images, without force or global
prune. Podman refuses images referenced by existing containers. The second
run changed nothing. NPM remained active with zero restarts; local admin
and public Gitea HTTPS returned 200. Backup timer and SSH proxy stayed active.
Validation:
```bash
ansible-playbook ansible/site.yml --limit prometheus --tags server_image_cleanup --check --diff -e server_legacy_image_cleanup=true
```
The image cleanup defaults to disabled and carries the `never` tag.
Check mode probes image presence but skips removal; it does not prove
Podman would accept deletion. It never removes NPM resources.
## Approved legacy data and fallback cleanup
The operator explicitly approved deletion on 2026-10-03. The separate
`server_legacy_cleanup` tasks removed `/opt/gitea`, `/home/git/.ssh`,
`/opt/navidrome`, `/opt/postgres`, `/opt/music`, `/opt/containerd`,
`/opt/docker`, the old Compose unit and the final Gitea export helper.
The empty `/home/git` parent is removed only with `rmdir`, after confirming
the Git account is absent. Guards reject symlinked paths, nested mounts,
unexpected containers, container users of these paths, unexpected content
in the empty legacy trees, and an active Compose or export service.
The second cleanup run changed nothing.
Before deletion, Ansible removed obsolete backup input paths and the
Gitea mount dependency. Normal Compose/template/final-export task checks
changed nothing and did not recreate the retired files. Deletion is opt-in:
```bash
ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true
```
Remove check mode only for approved deletion. No active NPM data, certificate,
image, network, volume, SSH proxy, WireGuard configuration or backup archive
is removed. No services were restarted by the cleanup.
After separate approval for the brief managed NPM pause, the new export
`20261003T112906Z` completed successfully and was pulled to Atlas. SHA-256
passed on both hosts; an isolated SQLite restore passed `quick_check` and
contained ten proxy hosts. Both Quadlet definitions were present, and
retired paths were absent. Temporary restore files were removed.
NPM was active with zero automatic restarts; primary public Gitea HTTPS
returned 200 with valid TLS. Backup timer, SSH proxy and WireGuard stayed active.
The first scheduled post-cleanup cycle remains unverified.
After separate operator approval on 2026-10-03, the unused secondary hostname
`git.ov-ad3410.infomaniak.ch` was removed from the declared domains and
the managed NPM runtime override. Its Proxy Host (id 10) was already
soft-deleted, with no generated config or associated certificate. Historical
deleted records and backup archives are preserved; no DNS changes were made.
Only `git.fscotto.duckdns.org` remains declared for the Gitea override.
Nginx validation and reload passed without restarting NPM; the primary
public HTTPS endpoint returned 200 with valid TLS.