Retire Prometheus Compose stack and document cleanup

This commit is contained in:
Fabio Scotto di Santolo
2026-10-03 13:44:57 +02:00
parent 7bc7f0e645
commit 755f24bc72
15 changed files with 332 additions and 50 deletions

View File

@@ -54,8 +54,9 @@ Ansible-driven personal infrastructure repo for Fedora and Void desktops, Fedora
- Emacs is disabled by default; temporary Emacs check: `ansible-playbook ansible/site.yml --limit <host> --tags emacs --check --diff -e emacs_enabled=true` - Emacs is disabled by default; temporary Emacs check: `ansible-playbook ansible/site.yml --limit <host> --tags emacs --check --diff -e emacs_enabled=true`
- AI coding agents: `ansible-playbook ansible/site.yml --limit <host> --tags ai_agents --check --diff` - AI coding agents: `ansible-playbook ansible/site.yml --limit <host> --tags ai_agents --check --diff`
- Mail bootstrap: `sh -n scripts/bootstrap_mail.sh` and `shellcheck scripts/bootstrap_mail.sh` - Mail bootstrap: `sh -n scripts/bootstrap_mail.sh` and `shellcheck scripts/bootstrap_mail.sh`
- Server NPM Quadlet: `systemctl status prometheus-npm.service`; disabled Compose fallback render: - Server NPM Quadlet: `systemctl status prometheus-npm.service`; the Compose fallback is retired.
`podman-compose -f /opt/docker/server/docker-compose.yml config` - Explicit Prometheus legacy cleanup (destructive only without check mode):
`ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true`
- Atlas media stack: - Atlas media stack:
`ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff` `ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff`
- Atlas rootless Gitea staging (does not start Gitea): - Atlas rootless Gitea staging (does not start Gitea):
@@ -143,12 +144,15 @@ The dotfile vars follow the same split: `desktop_common_dotfiles` carries mode-i
and disables diffs. Provisioning does not execute the updater or change its external schedule. and disables diffs. Provisioning does not execute the updater or change its external schedule.
- `rocky_server` is a child of both `platform_rocky` and `server`; `prometheus` is its active target. - `rocky_server` is a child of both `platform_rocky` and `server`; `prometheus` is its active target.
- The target must already provide `server_username` with local sudo access before the profile runs. - The target must already provide `server_username` with local sudo access before the profile runs.
- The Rocky profile installs Podman and podman-compose and renders the disabled legacy - The Rocky profile installs Podman and podman-compose. Prometheus explicitly retires the legacy
`podman-compose-server` unit for rollback. On Prometheus, Nginx Proxy Manager is now the rootful Compose unit, files and final-export helper with `server_legacy_stack_retired: true`.
Its approved opt-in cleanup removed old application data on 2026-10-03; normal runs do not
delete data or recreate the retired files. On Prometheus, Nginx Proxy Manager is now the rootful
`prometheus-npm.service` Quadlet with a pinned image digest and the existing `/opt/npm/data` and `prometheus-npm.service` Quadlet with a pinned image digest and the existing `/opt/npm/data` and
`/opt/npm/letsencrypt` bind mounts. The rootful `server_web` bridge remains `10.89.0.0/24`. `/opt/npm/letsencrypt` bind mounts. The rootful `server_web` bridge remains `10.89.0.0/24`.
Gitea runs on Atlas; PostgreSQL and Navidrome are absent from the desired Prometheus stack. Gitea runs on Atlas; PostgreSQL and Navidrome are absent from the desired Prometheus stack.
The profile does not delete legacy data, update DNS, or perform an implicit cutover. Normal runs do not delete legacy data, update DNS, or perform an implicit cutover;
destructive cleanup requires its explicit tag and opt-in extra-var.
- Firewalld enables SSH, Cockpit (`9090/tcp`), HTTP and HTTPS. Nginx Proxy Manager publishes `80/tcp` and - Firewalld enables SSH, Cockpit (`9090/tcp`), HTTP and HTTPS. Nginx Proxy Manager publishes `80/tcp` and
`443/tcp`; bind its administration interface only to `127.0.0.1:81` and use `npm-tunnel` from Ikaros or Nymph. `443/tcp`; bind its administration interface only to `127.0.0.1:81` and use `npm-tunnel` from Ikaros or Nymph.
Nextcloud remains disabled; do not provision `/srv/nextcloud` directories. Nextcloud remains disabled; do not provision `/srv/nextcloud` directories.
@@ -413,6 +417,22 @@ successfully. The first monthly scrub remains a runtime check.
both Quadlet files were present, and the complete Let's Encrypt tree (70 regular files plus both Quadlet files were present, and the complete Let's Encrypt tree (70 regular files plus
12 symlinks) matched the live data. A targeted normal Ansible run changed nothing. Details and rollback 12 symlinks) matched the live data. A targeted normal Ansible run changed nothing. Details and rollback
boundaries are in `docs/prometheus-npm-quadlet.md`. boundaries are in `docs/prometheus-npm-quadlet.md`.
- [x] Remove only unused Gitea, Navidrome and PostgreSQL images with opt-in
Ansible tasks on 2026-10-03. Second run changed nothing; NPM stayed active
with zero restarts, HTTP/HTTPS passed, backup timer and SSH proxy stayed active.
This image-only step preserved data and fallback; the later approved deletion is tracked below. Validation:
`ansible-playbook ansible/site.yml --limit prometheus --tags server_image_cleanup --check --diff -e server_legacy_image_cleanup=true`
- [x] Complete explicitly approved old-data and Compose fallback removal on 2026-10-03.
Backup paths and mount dependencies were reconciled before deletion; repeat cleanup changed
nothing. The normal Compose/template/helper check did not recreate retired files.
A separately approved manual export/pull published `20261003T112906Z`; checksum and isolated
SQLite restore passed with ten proxy hosts and both Quadlet definitions. NPM, primary HTTPS,
WireGuard, SSH proxy and backup timer remained healthy; existing backup archives were preserved.
- [x] Retire the unused secondary Gitea hostname `git.ov-ad3410.infomaniak.ch`
on 2026-10-03. Its NPM Proxy Host was already soft-deleted and had no
associated certificate. Its Ansible domain and runtime override were removed;
nginx -t and reload passed without restarting NPM. Primary HTTPS returned 200
with valid TLS; only `git.fscotto.duckdns.org` remains declared for Gitea.
- [ ] Observe the first scheduled export and Atlas pull after the cutover; the manual end-to-end - [ ] Observe the first scheduled export and Atlas pull after the cutover; the manual end-to-end
cycle passed, but the next unattended cycle has not yet occurred. cycle passed, but the next unattended cycle has not yet occurred.

View File

@@ -202,14 +202,17 @@ Lo stato attuale del profilo server include:
- installazione di Podman e podman-compose - installazione di Podman e podman-compose
- abilitazione dei servizi systemd dichiarati in inventory/group vars - abilitazione dei servizi systemd dichiarati in inventory/group vars
- copia dei dotfiles server e rendering del Quadlet rootful `prometheus-npm.service` per Nginx Proxy - copia dei dotfiles server e rendering del Quadlet rootful `prometheus-npm.service` per Nginx Proxy
Manager; il vecchio `podman-compose-server` resta disabilitato soltanto per un rollback controllato Manager; il vecchio fallback Compose è stato rimosso con autorizzazione esplicita
- attivazione di firewalld con SSH, Cockpit (`9090/tcp`), HTTP e HTTPS abilitati - attivazione di firewalld con SSH, Cockpit (`9090/tcp`), HTTP e HTTPS abilitati
- Syncthing escluso dal profilo server Rocky - Syncthing escluso dal profilo server Rocky
Il Compose desiderato su Prometheus non include piu Gitea, Navidrome ne il database PostgreSQL obsoleto. Il 2026-10-03 la pulizia opt-in autorizzata ha rimosso dati e immagini precedenti di Gitea,
Navidrome e Syncthing appartengono ad Atlas; Navidrome ufficiale usa invece SQLite. Il profilo non Navidrome e PostgreSQL, directory obsolete vuote, helper finale Gitea e fallback Compose NPM.
arresta o rimuove automaticamente eventuali container legacy e non elimina `/opt/postgres/data`. I servizi migrati restano su Atlas. `server_legacy_stack_retired: true` evita che i normali task
Il cutover NPM, le verifiche dei dati e del backup e i limiti del rollback sono documentati in ricreino i residui; la cancellazione richiede `--tags server_legacy_cleanup` e
`-e server_legacy_cleanup=true`. NPM attivo e archivi di backup restano intatti.
Export, pull Atlas e restore SQLite isolato post-pulizia sono riusciti; il primo ciclo automatico
resta da osservare. Evidenze e confini del recovery:
[`docs/prometheus-npm-quadlet.md`](docs/prometheus-npm-quadlet.md). [`docs/prometheus-npm-quadlet.md`](docs/prometheus-npm-quadlet.md).
Nginx Proxy Manager pubblica solo `80/tcp` e `443/tcp`; la sua interfaccia di amministrazione e Nginx Proxy Manager pubblica solo `80/tcp` e `443/tcp`; la sua interfaccia di amministrazione e
@@ -642,7 +645,7 @@ Questo significa che, allo stato attuale:
- il server Rocky (`prometheus`) e gestito con pacchetti, servizi, dotfiles server e firewalld - il server Rocky (`prometheus`) e gestito con pacchetti, servizi, dotfiles server e firewalld
- il NAS Rocky (`atlas`) usa un pool ZFS gia esistente, condivisioni NFSv4/SMB limitate alla LAN e Cockpit/45Drives - il NAS Rocky (`atlas`) usa un pool ZFS gia esistente, condivisioni NFSv4/SMB limitate alla LAN e Cockpit/45Drives
- NPM è un Quadlet rootful su Prometheus, mentre Gitea, Navidrome e Syncthing sono Quadlet - NPM è un Quadlet rootful su Prometheus, mentre Gitea, Navidrome e Syncthing sono Quadlet
rootless su Atlas; il Compose server resta disabilitato per rollback rootless su Atlas; il fallback Compose server è stato rimosso
# Dotfiles # Dotfiles
@@ -748,7 +751,7 @@ ansible-playbook ansible/site.yml --limit <host> --tags <tag1>,<tag2> --check --
ansible-playbook ansible/site.yml --limit <host> --start-at-task "<task name>" --check --diff ansible-playbook ansible/site.yml --limit <host> --start-at-task "<task name>" --check --diff
ansible-lint ansible/roles/<role> ansible-lint ansible/roles/<role>
yamllint ansible/path/to/file.yml yamllint ansible/path/to/file.yml
podman-compose -f /opt/docker/server/docker-compose.yml config ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true
ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff
ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 --check --diff ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 --check --diff
ansible-playbook ansible/site.yml --limit atlas --tags music_sync --check --diff ansible-playbook ansible/site.yml --limit atlas --tags music_sync --check --diff

View File

@@ -130,14 +130,18 @@ dotfiles and templates. The profile does not transfer application data, update D
implicit service cutover. implicit service cutover.
The server profile installs platform-specific packages, Podman and podman-compose, declared systemd The server profile installs platform-specific packages, Podman and podman-compose, declared systemd
services, and firewalld. Nginx Proxy Manager now runs as the rootful `prometheus-npm.service` Quadlet; services, and firewalld. Nginx Proxy Manager runs as the rootful `prometheus-npm.service` Quadlet.
the old `podman-compose-server` unit is disabled and retained only for controlled rollback. The On 2026-10-03 the operator-approved opt-in cleanup removed old Gitea, Navidrome and PostgreSQL
Compose file no longer includes Gitea, Navidrome, Syncthing, or the obsolete Navidrome PostgreSQL data/images, empty legacy directories, the Gitea final-export helper and the Compose rollback files.
database. The temporary Navidrome and Syncthing deployment on Atlas is managed by The migrated services stay on Atlas. `server_legacy_stack_retired: true` prevents normal runs from
`profile_backend_phase1`. Applying the profile does not delete legacy data or `/opt/postgres/data`. recreating retired files. Data deletion requires `--tags server_legacy_cleanup` and
The NPM cutover, data checks, backup evidence, and rollback boundaries are documented in `-e server_legacy_cleanup=true`; image-only cleanup has its own `server_image_cleanup` tag and flag.
Active NPM resources and existing backup archives remain preserved.
The post-cleanup export/pull and isolated SQLite restore passed; the first unattended cycle remains
pending. Evidence and recovery boundaries:
[`docs/prometheus-npm-quadlet.md`](docs/prometheus-npm-quadlet.md). [`docs/prometheus-npm-quadlet.md`](docs/prometheus-npm-quadlet.md).
Firewalld enables SSH, Cockpit (`9090/tcp`), HTTP and HTTPS. Nginx Proxy Manager publishes only Firewalld enables SSH, Cockpit (`9090/tcp`), HTTP and HTTPS. Nginx Proxy Manager publishes only
`80/tcp` and `443/tcp`; its administration interface is bound to `127.0.0.1:81` and can be reached `80/tcp` and `443/tcp`; its administration interface is bound to `127.0.0.1:81` and can be reached
from Ikaros or Nymph with the `npm-tunnel` Bash alias. Nextcloud remains disabled and the profile from Ikaros or Nymph with the `npm-tunnel` Bash alias. Nextcloud remains disabled and the profile
@@ -738,7 +742,7 @@ ansible-playbook ansible/site.yml --limit <host> --tags <tag1>,<tag2> --check --
ansible-playbook ansible/site.yml --limit <host> --start-at-task "<task name>" --check --diff ansible-playbook ansible/site.yml --limit <host> --start-at-task "<task name>" --check --diff
ansible-lint ansible/roles/<role> ansible-lint ansible/roles/<role>
yamllint ansible/path/to/file.yml yamllint ansible/path/to/file.yml
podman-compose -f /opt/docker/server/docker-compose.yml config ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true
ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff
ansible-playbook ansible/site.yml --limit atlas --tags music_sync --check --diff ansible-playbook ansible/site.yml --limit atlas --tags music_sync --check --diff
``` ```

View File

@@ -8,6 +8,8 @@ effective_user_home: "{{ server_user_home }}"
server_container_stack_dir: /opt/docker/server server_container_stack_dir: /opt/docker/server
server_npm_quadlet_stage: false server_npm_quadlet_stage: false
server_npm_quadlet_cutover: false server_npm_quadlet_cutover: false
server_legacy_stack_retired: false
server_legacy_cleanup: false
ai_agents: {} ai_agents: {}
vim_plugins_enabled: false vim_plugins_enabled: false
@@ -101,8 +103,9 @@ server_backup_export_source_keep: 3
server_backup_export_paths: >- server_backup_export_paths: >-
{{ ['opt/npm/data', 'opt/npm/letsencrypt'] {{ ['opt/npm/data', 'opt/npm/letsencrypt']
+ ([] if server_gitea_on_atlas | bool else ['opt/gitea/data', 'home/git/.ssh']) + ([] if server_gitea_on_atlas | bool else ['opt/gitea/data', 'home/git/.ssh'])
+ ['opt/docker/server/docker-compose.yml', + ([] if server_legacy_stack_retired | bool else
'etc/systemd/system/podman-compose-server.service'] ['opt/docker/server/docker-compose.yml',
'etc/systemd/system/podman-compose-server.service'])
+ (['etc/containers/systemd/prometheus-npm.container', + (['etc/containers/systemd/prometheus-npm.container',
'etc/containers/systemd/server-web.network'] 'etc/containers/systemd/server-web.network']
if server_npm_quadlet_stage | bool else []) if server_npm_quadlet_stage | bool else [])

View File

@@ -6,6 +6,15 @@ ansible_port: 22
ansible_ssh_private_key_file: /home/fscotto/.ssh/id_ed25519 ansible_ssh_private_key_file: /home/fscotto/.ssh/id_ed25519
server_username: rocky server_username: rocky
server_legacy_stack_retired: true
# Destructive deletion runs only with an explicit extra-var and cleanup tag.
server_legacy_cleanup: false
# Explicit opt-in cleanup; no data, volumes, networks or NPM images are removed.
server_legacy_image_cleanup: false
server_legacy_images:
- docker.gitea.com/gitea:1.25.2
- docker.io/deluan/navidrome:latest
- docker.io/library/postgres:13
server_npm_quadlet_stage: true server_npm_quadlet_stage: true
server_npm_quadlet_image: docker.io/jc21/nginx-proxy-manager@sha256:52b2c59994f3d36acfcf70a1626f29734df0ed8c71bacc0269f78b6f939858bb server_npm_quadlet_image: docker.io/jc21/nginx-proxy-manager@sha256:52b2c59994f3d36acfcf70a1626f29734df0ed8c71bacc0269f78b6f939858bb
# The stopped-source export and live Quadlet cutover passed on 2026-10-03. # The stopped-source export and live Quadlet cutover passed on 2026-10-03.
@@ -17,7 +26,6 @@ server_gitea_cutover_tools_enabled: true
server_gitea_on_atlas: true server_gitea_on_atlas: true
server_gitea_npm_domains: server_gitea_npm_domains:
- git.fscotto.duckdns.org - git.fscotto.duckdns.org
- git.ov-ad3410.infomaniak.ch
server_duckdns_domain: fscotto server_duckdns_domain: fscotto
server_ssh_authorized_keys: server_ssh_authorized_keys:
- name: ikaros - name: ikaros

View File

@@ -51,6 +51,7 @@
owner: root owner: root
group: root group: root
mode: "0750" mode: "0750"
validate: "bash -n %s"
when: server_backup_export_enabled | bool when: server_backup_export_enabled | bool
- name: Install Prometheus backup export systemd units - name: Install Prometheus backup export systemd units

View File

@@ -7,7 +7,9 @@
owner: root owner: root
group: root group: root
mode: "0750" mode: "0750"
when: server_gitea_cutover_tools_enabled | bool when:
- server_gitea_cutover_tools_enabled | bool
- not server_legacy_stack_retired | bool
- name: Require the prepared source and explicit final-export approval - name: Require the prepared source and explicit final-export approval
tags: [services, gitea_final_export] tags: [services, gitea_final_export]
@@ -15,6 +17,7 @@
that: that:
- server_gitea_cutover_tools_enabled | bool - server_gitea_cutover_tools_enabled | bool
- server_backup_export_enabled | bool - server_backup_export_enabled | bool
- not server_legacy_stack_retired | bool
- not ansible_check_mode - not ansible_check_mode
fail_msg: >- fail_msg: >-
Install the cutover helper and perform an explicit non-check-mode run Install the cutover helper and perform an explicit non-check-mode run
@@ -31,4 +34,5 @@
no_log: true no_log: true
when: when:
- server_gitea_final_export | bool - server_gitea_final_export | bool
- not server_legacy_stack_retired | bool
- not ansible_check_mode - not ansible_check_mode

View File

@@ -0,0 +1,112 @@
---
- name: Require explicit retirement of the migrated Prometheus source
ansible.builtin.assert:
that:
- inventory_hostname == 'prometheus'
- server_legacy_stack_retired | bool
- server_gitea_on_atlas | bool
- server_npm_quadlet_cutover | bool
- server_backup_export_enabled | bool
- name: Verify legacy paths have no mounts or container users
ansible.builtin.command:
argv:
- python3
- -c
- |
import json, os, pathlib, subprocess
def run(*args):
return subprocess.check_output(args, text=True).strip()
paths = ['/opt/gitea', '/home/git/.ssh', '/opt/navidrome',
'/opt/postgres', '/opt/music', '/opt/containerd', '/opt/docker']
mounts = json.loads(run('findmnt', '--json', '--list', '-o', 'TARGET'))['filesystems']
for path in paths:
assert os.path.realpath(path) == path, 'Symlink in cleanup path: ' + path
for mount in mounts:
target = mount['target']
assert target != path and not target.startswith(path + '/'), 'Mounted cleanup path: ' + path
ids = run('podman', 'ps', '-aq').split()
containers = json.loads(run('podman', 'inspect', *ids)) if ids else []
for container in containers:
assert container['Name'].lstrip('/') == 'nginx-proxy-manager', 'Unexpected container; review before cleanup'
for mount in container.get('Mounts', []):
source = os.path.realpath(mount['Source'])
for path in paths:
assert source != path and not source.startswith(path + '/'), 'Container uses cleanup path: ' + path
for path in ['/opt/music', '/opt/containerd']:
if os.path.isdir(path):
for entry in pathlib.Path(path).rglob('*'):
assert entry.is_dir() and not entry.is_symlink(), 'Unexpected file in empty legacy path: ' + str(entry)
if os.path.isdir('/opt/docker'):
allowed = {'/opt/docker/server', '/opt/docker/server/docker-compose.yml'}
for entry in pathlib.Path('/opt/docker').rglob('*'):
assert str(entry) in allowed and not entry.is_symlink(), 'Unexpected legacy Docker content: ' + str(entry)
assert run('systemctl', 'is-active', 'prometheus-npm.service') == 'active'
assert subprocess.run(['systemctl', 'is-active', '--quiet', 'podman-compose-server.service']).returncode != 0
assert subprocess.run(['systemctl', 'is-active', '--quiet', 'prometheus-backup-export.service']).returncode != 0
print('Legacy cleanup preflight passed')
changed_when: false
check_mode: false
- name: Require the updated backup configuration before deleting fallback files
ansible.builtin.command:
argv:
- python3
- -c
- |
import pathlib, subprocess
unit = subprocess.check_output(['systemctl', 'show', 'prometheus-backup-export.service',
'-p', 'RequiresMountsFor', '--value'], text=True)
assert '/opt/gitea' not in unit, 'Backup unit still depends on legacy Gitea'
helper = pathlib.Path('/usr/local/sbin/prometheus-backup-export').read_text()
assert 'podman-compose-server' not in helper and 'opt/docker/server' not in helper
subprocess.run(['bash', '-n', '/usr/local/sbin/prometheus-backup-export'], check=True)
changed_when: false
when: not ansible_check_mode
- name: Delete only the explicitly approved legacy data and fallback files
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop:
- /opt/gitea
- /home/git/.ssh
- /opt/navidrome
- /opt/postgres
- /opt/music
- /opt/containerd
- /opt/docker
- /usr/local/sbin/prometheus-gitea-final-export
- /etc/systemd/system/podman-compose-server.service
register: server_legacy_deleted
diff: false
- name: Reload systemd after removing the inactive legacy unit
ansible.builtin.systemd:
daemon_reload: true
when:
- server_legacy_deleted is changed
- not ansible_check_mode
- name: Inspect the obsolete Git home without following symlinks
ansible.builtin.stat:
path: /home/git
follow: false
register: server_legacy_git_home
- name: Require the obsolete Git account to be absent before removing its empty home
ansible.builtin.command:
argv: [getent, passwd, git]
register: server_legacy_git_account
changed_when: false
failed_when: server_legacy_git_account.rc != 2
check_mode: false
when: server_legacy_git_home.stat.exists
# rmdir refuses any nonempty directory; never recursively delete this parent.
- name: Remove only the empty obsolete Git home
ansible.builtin.command:
argv: [rmdir, /home/git]
register: server_legacy_git_home_removed
changed_when: server_legacy_git_home_removed.rc == 0
when: server_legacy_git_home.stat.exists

View File

@@ -0,0 +1,33 @@
---
- name: Require the migrated Prometheus topology for image cleanup
ansible.builtin.assert:
that:
- inventory_hostname == 'prometheus'
- server_gitea_on_atlas | bool
- server_npm_quadlet_cutover | bool
- server_legacy_images | default([]) | length > 0
- >-
server_legacy_images | difference([
'docker.gitea.com/gitea:1.25.2',
'docker.io/deluan/navidrome:latest',
'docker.io/library/postgres:13']) | length == 0
- name: Check whether the explicitly selected legacy images exist
ansible.builtin.command:
argv: [podman, image, exists, "{{ item }}"]
loop: "{{ server_legacy_images }}"
register: server_legacy_image_presence
changed_when: false
failed_when: server_legacy_image_presence.rc not in [0, 1]
check_mode: false
# No --force: Podman must refuse images referenced by any existing container.
- name: Remove only unused explicitly selected legacy images
ansible.builtin.command:
argv: [podman, image, rm, "{{ item.item }}"]
loop: "{{ server_legacy_image_presence.results }}"
loop_control:
label: "{{ item.item }}"
when: item.rc == 0
register: server_legacy_image_removal
changed_when: server_legacy_image_removal.rc == 0

View File

@@ -23,6 +23,9 @@
loop: "{{ server_directories | default([]) }}" loop: "{{ server_directories | default([]) }}"
loop_control: loop_control:
label: "{{ item.path }}" label: "{{ item.path }}"
when:
- item.path != '/opt/gitea/data' or not server_gitea_on_atlas | bool
- item.path != server_container_stack_dir or not server_legacy_stack_retired | bool
- name: Copy server dotfiles - name: Copy server dotfiles
tags: [dotfiles, dotfiles:server] tags: [dotfiles, dotfiles:server]
@@ -48,19 +51,32 @@
loop_control: loop_control:
label: "{{ item.dest }}" label: "{{ item.dest }}"
no_log: "{{ item.no_log | default(false) }}" no_log: "{{ item.no_log | default(false) }}"
when: item.src != 'server/docker-compose.yml.j2' or not server_legacy_stack_retired | bool
- name: Manage Podman Compose stack - name: Manage Podman Compose stack
tags: [services, podman] tags: [services, podman]
ansible.builtin.include_tasks: podman-compose.yml ansible.builtin.include_tasks: podman-compose.yml
when: not server_legacy_stack_retired | bool
- name: Import staged NPM Quadlet tasks - name: Import staged NPM Quadlet tasks
ansible.builtin.import_tasks: npm_quadlet.yml ansible.builtin.import_tasks: npm_quadlet.yml
- name: Import explicit legacy server image cleanup
ansible.builtin.import_tasks: legacy_image_cleanup.yml
tags: [never, server_image_cleanup]
when: server_legacy_image_cleanup | default(false) | bool
- name: Import Prometheus backup export identity tasks - name: Import Prometheus backup export identity tasks
ansible.builtin.import_tasks: backup_export_identity.yml ansible.builtin.import_tasks: backup_export_identity.yml
- name: Import Prometheus backup export job tasks - name: Import Prometheus backup export job tasks
ansible.builtin.import_tasks: backup_export_job.yml ansible.builtin.import_tasks: backup_export_job.yml
tags: [server_legacy_cleanup]
- name: Import explicitly approved legacy server data cleanup
ansible.builtin.import_tasks: legacy_cleanup.yml
tags: [never, server_legacy_cleanup]
when: server_legacy_cleanup | bool
- name: Import explicit Prometheus Gitea final-export tasks - name: Import explicit Prometheus Gitea final-export tasks
ansible.builtin.import_tasks: gitea_final_export.yml ansible.builtin.import_tasks: gitea_final_export.yml

View File

@@ -1,6 +1,6 @@
[Unit] [Unit]
Description=Prepare a read-only Prometheus application backup for Atlas Description=Prepare a read-only Prometheus application backup for Atlas
RequiresMountsFor=/opt/npm /opt/gitea {{ server_backup_export_root }} RequiresMountsFor=/opt/npm {% if not server_gitea_on_atlas | bool %}/opt/gitea {% endif %}{{ server_backup_export_root }}
ConditionFileIsExecutable=/usr/local/sbin/prometheus-backup-export ConditionFileIsExecutable=/usr/local/sbin/prometheus-backup-export
[Service] [Service]

View File

@@ -4,6 +4,9 @@ umask 077
export_root={{ server_backup_export_root | quote }} export_root={{ server_backup_export_root | quote }}
versions="$export_root/versions" versions="$export_root/versions"
{% if server_legacy_stack_retired | bool %}
stack_unit=prometheus-npm.service
{% else %}
stack_unit='' stack_unit=''
compose_active=false compose_active=false
quadlet_active=false quadlet_active=false
@@ -18,6 +21,7 @@ if "$quadlet_active"; then
else else
stack_unit=podman-compose-server.service stack_unit=podman-compose-server.service
fi fi
{% endif %}
stamp=$(date -u +%Y%m%dT%H%M%SZ) stamp=$(date -u +%Y%m%dT%H%M%SZ)
stage='' stage=''
stack_stopped=false stack_stopped=false

View File

@@ -1,8 +1,13 @@
# Gitea migration from Prometheus to Atlas # Gitea migration from Prometheus to Atlas
This records the staged migration and its observed partial cutover. Gitea is This records the staged migration and its observed partial cutover. Gitea is
temporary on Atlas until Uranus; NPM remains on Prometheus. Preserve the old temporary on Atlas until Uranus; NPM remains on Prometheus. On 2026-10-03
Prometheus data, but do not restart its stale Gitea after Atlas accepts writes. the operator explicitly approved removal of the old Prometheus Gitea data,
SSH fragment and final-export helper. NPM now uses a rootful Quadlet with no
installed Compose fallback. The source-retention and rollback steps below
are historical migration gates, not current recovery instructions.
Existing backup archives were preserved; use current Atlas data and verified
backups for recovery. Do not recreate or restart stale source Gitea.
## Observed source before cutover and chosen topology (2026-10-01) ## Observed source before cutover and chosen topology (2026-10-01)
@@ -139,6 +144,10 @@ or credentials were changed. The
secondary hostname `git.ov-ad3410.infomaniak.ch` did not resolve from Ikaros secondary hostname `git.ov-ad3410.infomaniak.ch` did not resolve from Ikaros
and had no generated NPM config file at the time of inspection. and had no generated NPM config file at the time of inspection.
On 2026-10-03 the operator retired this unused secondary hostname. Its NPM
Proxy Host was already soft-deleted; Ansible now declares only
`git.fscotto.duckdns.org` and removes the secondary runtime override.
Prometheus' public TCP/2222 socket proxies to Atlas without changing admin Prometheus' public TCP/2222 socket proxies to Atlas without changing admin
SSH/22. The local socket presents the preserved Gitea ED25519 host key, but SSH/22. The local socket presents the preserved Gitea ED25519 host key, but
an external TCP/2222 connection from Ikaros initially timed out. During that an external TCP/2222 connection from Ikaros initially timed out. During that

View File

@@ -10,15 +10,15 @@ cutover is still pending. See `docs/prometheus-npm-quadlet.md`.
## Declared design ## Declared design
- Prometheus prepares a tar archive of Nginx Proxy Manager data and certificates, - Prometheus prepares a tar archive of Nginx Proxy Manager data and certificates,
its active Quadlet and network definitions, the disabled Compose fallback, its active Quadlet and network definitions,
and SSH/firewalld/WireGuard configuration. Gitea now runs on Atlas and is no and SSH/firewalld/WireGuard configuration. Gitea now runs on Atlas and is no
longer included in new Prometheus exports. NPM access logs are excluded. longer included in new Prometheus exports. NPM access logs are excluded.
The archive contains credentials, certificates, and the WireGuard private The archive contains credentials, certificates, and the WireGuard private
key: protect both copies accordingly. key: protect both copies accordingly.
- The approved consistency mode stops the one active NPM service (Quadlet now, - The approved consistency mode stops the NPM Quadlet for local tar creation
Compose before cutover) for local tar creation at 02:00 Europe/Rome, then at 02:00 Europe/Rome, then restarts it even if archiving fails. After the
restarts it even if archiving fails. The helper refuses both services active approved legacy cleanup, the helper requires the Quadlet active and has
or both inactive. A manual test outside that window requires separate approval. no Compose dependency. A manual test outside that window requires separate approval.
- Prometheus publishes the archive with its checksum as a versioned, read-only - Prometheus publishes the archive with its checksum as a versioned, read-only
source under `/var/lib/prometheus-backup-export`. A locked service account source under `/var/lib/prometheus-backup-export`. A locked service account
has no sudo or supplementary groups. Its only authorized SSH key is forced has no sudo or supplementary groups. Its only authorized SSH key is forced
@@ -119,3 +119,13 @@ both Quadlet definitions. A manifest of all 70 regular Let's Encrypt files
and 12 symlinks, including content hashes and link targets, matched the live and 12 symlinks, including content hashes and link targets, matched the live
Prometheus tree. No private key or secret content was printed. The next Prometheus tree. No private key or secret content was printed. The next
scheduled export/pull is still pending observation. scheduled export/pull is still pending observation.
## Post-cleanup validation (2026-10-03)
The operator-approved removal of legacy data and Compose fallback also
removed those backup input paths and the obsolete Gitea mount dependency.
A separately approved export and Atlas pull published `20261003T112906Z`.
Both SHA-256 checks passed; an isolated SQLite restore passed `quick_check`
and contained ten proxy hosts. Both active Quadlet definitions were present;
retired paths were absent. Existing backup archives were not deleted by cleanup.
The first scheduled cycle after these changes remains unverified.

View File

@@ -8,19 +8,20 @@ Nginx Proxy Manager runs as the **rootful** generated
`/etc/containers/systemd/server-web.network`; the image is pinned by digest `/etc/containers/systemd/server-web.network`; the image is pinned by digest
in `ansible/inventory/host_vars/prometheus.yml`. The generated service is in `ansible/inventory/host_vars/prometheus.yml`. The generated service is
wanted by `multi-user.target` and requires the generated network service. wanted by `multi-user.target` and requires the generated network service.
The old `podman-compose-server.service` is inactive and disabled. Its unit The old Compose unit, Compose file and Gitea final-export helper were
and Compose file remain as a rollback option, not as another active owner. removed by the operator-approved cleanup on 2026-10-03. The retired
Do not start both units or run `podman-compose down` while the Quadlet owns application data and empty legacy directories were also removed.
the shared `server_web` network. Prometheus host vars set `server_legacy_stack_retired: true` so normal runs
do not recreate those files. Destructive deletion still requires a separate
cleanup tag and explicit extra-var.
There was **no data copy** in this cutover. The Quadlet reuses the existing There was **no data copy** in this cutover. The Quadlet reuses the existing
`/opt/npm/data:/data` and `/opt/npm/letsencrypt:/etc/letsencrypt` bind mounts `/opt/npm/data:/data` and `/opt/npm/letsencrypt:/etc/letsencrypt` bind mounts
with the same container name and `server_web` bridge (`10.89.0.0/24`). Ports with the same container name and `server_web` bridge (`10.89.0.0/24`). Ports
80 and 443 remain public; administration port 81 remains bound to 80 and 443 remain public; administration port 81 remains bound to
`127.0.0.1`. Gitea stays on Atlas, and NPM remains on Prometheus. The `127.0.0.1`. Gitea stays on Atlas, and NPM remains on Prometheus. The
Prometheus Compose file is retained with the same pinned NPM image for a Compose fallback is no longer installed. The Quadlet uses `Pull=missing`,
controlled fallback. The Quadlet uses `Pull=missing`, not an automatic not an automatic floating-tag update.
floating-tag update.
## Cutover and recovery boundaries ## Cutover and recovery boundaries
@@ -36,16 +37,12 @@ then restarted the timer. Its failure trap would have restarted Compose.
**Do not rerun that forward-cutover script after success**: its preconditions **Do not rerun that forward-cutover script after success**: its preconditions
intentionally reject an active Quadlet. intentionally reject an active Quadlet.
A future rollback is a separate outage decision, not an ordinary Ansible run. Recovery is now a Quadlet rebuild and restoration from a verified Atlas
First verify a usable recent Atlas backup and stop the export timer. Stop backup, with an explicit outage decision before replacing live NPM state.
the Quadlet and verify that its container is gone before allowing Compose The old Compose owner is no longer installed; reintroducing it would require
to own the same name, mounts, network, and ports; use the pinned Compose a separately reviewed configuration and outage plan. The historical
configuration, then validate NPM/HTTPS and restart the timer. Set in-window rollback trap is not a supported post-cleanup rollback procedure.
`server_npm_quadlet_cutover: false` only as part of that controlled rollback. Do not restore an old database over a live instance or remove NPM bind mounts.
Do not run the two owners concurrently, restore an old NPM database over a
live instance, or delete either bind mount. This reverse procedure has not
been exercised on production; the forward script's in-window rollback path
is not evidence of a later reverse cutover.
## Verified evidence ## Verified evidence
@@ -81,10 +78,68 @@ the new path works but not its next scheduled execution.
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \ ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
ansible-playbook ansible/site.yml --limit prometheus --tags npm_quadlet --check --diff ansible-playbook ansible/site.yml --limit prometheus --tags npm_quadlet --check --diff
sudo systemctl status prometheus-npm.service prometheus-backup-export.timer sudo systemctl status prometheus-npm.service prometheus-backup-export.timer
sudo systemctl is-active podman-compose-server.service sudo systemctl show podman-compose-server.service -p LoadState # expected: not-found
sudo systemctl is-enabled podman-compose-server.service
``` ```
The backup archive includes credentials, certificates, and WireGuard The backup archive includes credentials, certificates, and WireGuard
configuration. Do not publish it or print its contents in diagnostics; see configuration. Do not publish it or print its contents in diagnostics; see
`docs/prometheus-backup.md` for the restricted pull and restore procedure. `docs/prometheus-backup.md` for the restricted pull and restore procedure.
## Selective legacy image cleanup
On 2026-10-03 opt-in Ansible tasks removed only the unused Gitea 1.25.2,
Navidrome latest and PostgreSQL 13 rootful images, without force or global
prune. Podman refuses images referenced by existing containers. The second
run changed nothing. NPM remained active with zero restarts; local admin
and public Gitea HTTPS returned 200. Backup timer and SSH proxy stayed active.
Validation:
```bash
ansible-playbook ansible/site.yml --limit prometheus --tags server_image_cleanup --check --diff -e server_legacy_image_cleanup=true
```
The image cleanup defaults to disabled and carries the `never` tag.
Check mode probes image presence but skips removal; it does not prove
Podman would accept deletion. It never removes NPM resources.
## Approved legacy data and fallback cleanup
The operator explicitly approved deletion on 2026-10-03. The separate
`server_legacy_cleanup` tasks removed `/opt/gitea`, `/home/git/.ssh`,
`/opt/navidrome`, `/opt/postgres`, `/opt/music`, `/opt/containerd`,
`/opt/docker`, the old Compose unit and the final Gitea export helper.
The empty `/home/git` parent is removed only with `rmdir`, after confirming
the Git account is absent. Guards reject symlinked paths, nested mounts,
unexpected containers, container users of these paths, unexpected content
in the empty legacy trees, and an active Compose or export service.
The second cleanup run changed nothing.
Before deletion, Ansible removed obsolete backup input paths and the
Gitea mount dependency. Normal Compose/template/final-export task checks
changed nothing and did not recreate the retired files. Deletion is opt-in:
```bash
ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true
```
Remove check mode only for approved deletion. No active NPM data, certificate,
image, network, volume, SSH proxy, WireGuard configuration or backup archive
is removed. No services were restarted by the cleanup.
After separate approval for the brief managed NPM pause, the new export
`20261003T112906Z` completed successfully and was pulled to Atlas. SHA-256
passed on both hosts; an isolated SQLite restore passed `quick_check` and
contained ten proxy hosts. Both Quadlet definitions were present, and
retired paths were absent. Temporary restore files were removed.
NPM was active with zero automatic restarts; primary public Gitea HTTPS
returned 200 with valid TLS. Backup timer, SSH proxy and WireGuard stayed active.
The first scheduled post-cleanup cycle remains unverified.
After separate operator approval on 2026-10-03, the unused secondary hostname
`git.ov-ad3410.infomaniak.ch` was removed from the declared domains and
the managed NPM runtime override. Its Proxy Host (id 10) was already
soft-deleted, with no generated config or associated certificate. Historical
deleted records and backup archives are preserved; no DNS changes were made.
Only `git.fscotto.duckdns.org` remains declared for the Gitea override.
Nginx validation and reload passed without restarting NPM; the primary
public HTTPS endpoint returned 200 with valid TLS.