mirror of
https://github.com/fscotto/infra.git
synced 2026-10-03 13:29:58 +00:00
Retire Prometheus Compose stack and document cleanup
This commit is contained in:
30
AGENTS.md
30
AGENTS.md
@@ -54,8 +54,9 @@ Ansible-driven personal infrastructure repo for Fedora and Void desktops, Fedora
|
||||
- Emacs is disabled by default; temporary Emacs check: `ansible-playbook ansible/site.yml --limit <host> --tags emacs --check --diff -e emacs_enabled=true`
|
||||
- AI coding agents: `ansible-playbook ansible/site.yml --limit <host> --tags ai_agents --check --diff`
|
||||
- Mail bootstrap: `sh -n scripts/bootstrap_mail.sh` and `shellcheck scripts/bootstrap_mail.sh`
|
||||
- Server NPM Quadlet: `systemctl status prometheus-npm.service`; disabled Compose fallback render:
|
||||
`podman-compose -f /opt/docker/server/docker-compose.yml config`
|
||||
- Server NPM Quadlet: `systemctl status prometheus-npm.service`; the Compose fallback is retired.
|
||||
- Explicit Prometheus legacy cleanup (destructive only without check mode):
|
||||
`ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true`
|
||||
- Atlas media stack:
|
||||
`ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff`
|
||||
- Atlas rootless Gitea staging (does not start Gitea):
|
||||
@@ -143,12 +144,15 @@ The dotfile vars follow the same split: `desktop_common_dotfiles` carries mode-i
|
||||
and disables diffs. Provisioning does not execute the updater or change its external schedule.
|
||||
- `rocky_server` is a child of both `platform_rocky` and `server`; `prometheus` is its active target.
|
||||
- The target must already provide `server_username` with local sudo access before the profile runs.
|
||||
- The Rocky profile installs Podman and podman-compose and renders the disabled legacy
|
||||
`podman-compose-server` unit for rollback. On Prometheus, Nginx Proxy Manager is now the rootful
|
||||
- The Rocky profile installs Podman and podman-compose. Prometheus explicitly retires the legacy
|
||||
Compose unit, files and final-export helper with `server_legacy_stack_retired: true`.
|
||||
Its approved opt-in cleanup removed old application data on 2026-10-03; normal runs do not
|
||||
delete data or recreate the retired files. On Prometheus, Nginx Proxy Manager is now the rootful
|
||||
`prometheus-npm.service` Quadlet with a pinned image digest and the existing `/opt/npm/data` and
|
||||
`/opt/npm/letsencrypt` bind mounts. The rootful `server_web` bridge remains `10.89.0.0/24`.
|
||||
Gitea runs on Atlas; PostgreSQL and Navidrome are absent from the desired Prometheus stack.
|
||||
The profile does not delete legacy data, update DNS, or perform an implicit cutover.
|
||||
Normal runs do not delete legacy data, update DNS, or perform an implicit cutover;
|
||||
destructive cleanup requires its explicit tag and opt-in extra-var.
|
||||
- Firewalld enables SSH, Cockpit (`9090/tcp`), HTTP and HTTPS. Nginx Proxy Manager publishes `80/tcp` and
|
||||
`443/tcp`; bind its administration interface only to `127.0.0.1:81` and use `npm-tunnel` from Ikaros or Nymph.
|
||||
Nextcloud remains disabled; do not provision `/srv/nextcloud` directories.
|
||||
@@ -413,6 +417,22 @@ successfully. The first monthly scrub remains a runtime check.
|
||||
both Quadlet files were present, and the complete Let's Encrypt tree (70 regular files plus
|
||||
12 symlinks) matched the live data. A targeted normal Ansible run changed nothing. Details and rollback
|
||||
boundaries are in `docs/prometheus-npm-quadlet.md`.
|
||||
- [x] Remove only unused Gitea, Navidrome and PostgreSQL images with opt-in
|
||||
Ansible tasks on 2026-10-03. Second run changed nothing; NPM stayed active
|
||||
with zero restarts, HTTP/HTTPS passed, backup timer and SSH proxy stayed active.
|
||||
This image-only step preserved data and fallback; the later approved deletion is tracked below. Validation:
|
||||
`ansible-playbook ansible/site.yml --limit prometheus --tags server_image_cleanup --check --diff -e server_legacy_image_cleanup=true`
|
||||
- [x] Complete explicitly approved old-data and Compose fallback removal on 2026-10-03.
|
||||
Backup paths and mount dependencies were reconciled before deletion; repeat cleanup changed
|
||||
nothing. The normal Compose/template/helper check did not recreate retired files.
|
||||
A separately approved manual export/pull published `20261003T112906Z`; checksum and isolated
|
||||
SQLite restore passed with ten proxy hosts and both Quadlet definitions. NPM, primary HTTPS,
|
||||
WireGuard, SSH proxy and backup timer remained healthy; existing backup archives were preserved.
|
||||
- [x] Retire the unused secondary Gitea hostname `git.ov-ad3410.infomaniak.ch`
|
||||
on 2026-10-03. Its NPM Proxy Host was already soft-deleted and had no
|
||||
associated certificate. Its Ansible domain and runtime override were removed;
|
||||
nginx -t and reload passed without restarting NPM. Primary HTTPS returned 200
|
||||
with valid TLS; only `git.fscotto.duckdns.org` remains declared for Gitea.
|
||||
- [ ] Observe the first scheduled export and Atlas pull after the cutover; the manual end-to-end
|
||||
cycle passed, but the next unattended cycle has not yet occurred.
|
||||
|
||||
|
||||
17
README.it.md
17
README.it.md
@@ -202,14 +202,17 @@ Lo stato attuale del profilo server include:
|
||||
- installazione di Podman e podman-compose
|
||||
- abilitazione dei servizi systemd dichiarati in inventory/group vars
|
||||
- copia dei dotfiles server e rendering del Quadlet rootful `prometheus-npm.service` per Nginx Proxy
|
||||
Manager; il vecchio `podman-compose-server` resta disabilitato soltanto per un rollback controllato
|
||||
Manager; il vecchio fallback Compose è stato rimosso con autorizzazione esplicita
|
||||
- attivazione di firewalld con SSH, Cockpit (`9090/tcp`), HTTP e HTTPS abilitati
|
||||
- Syncthing escluso dal profilo server Rocky
|
||||
|
||||
Il Compose desiderato su Prometheus non include piu Gitea, Navidrome ne il database PostgreSQL obsoleto.
|
||||
Navidrome e Syncthing appartengono ad Atlas; Navidrome ufficiale usa invece SQLite. Il profilo non
|
||||
arresta o rimuove automaticamente eventuali container legacy e non elimina `/opt/postgres/data`.
|
||||
Il cutover NPM, le verifiche dei dati e del backup e i limiti del rollback sono documentati in
|
||||
Il 2026-10-03 la pulizia opt-in autorizzata ha rimosso dati e immagini precedenti di Gitea,
|
||||
Navidrome e PostgreSQL, directory obsolete vuote, helper finale Gitea e fallback Compose NPM.
|
||||
I servizi migrati restano su Atlas. `server_legacy_stack_retired: true` evita che i normali task
|
||||
ricreino i residui; la cancellazione richiede `--tags server_legacy_cleanup` e
|
||||
`-e server_legacy_cleanup=true`. NPM attivo e archivi di backup restano intatti.
|
||||
Export, pull Atlas e restore SQLite isolato post-pulizia sono riusciti; il primo ciclo automatico
|
||||
resta da osservare. Evidenze e confini del recovery:
|
||||
[`docs/prometheus-npm-quadlet.md`](docs/prometheus-npm-quadlet.md).
|
||||
|
||||
Nginx Proxy Manager pubblica solo `80/tcp` e `443/tcp`; la sua interfaccia di amministrazione e
|
||||
@@ -642,7 +645,7 @@ Questo significa che, allo stato attuale:
|
||||
- il server Rocky (`prometheus`) e gestito con pacchetti, servizi, dotfiles server e firewalld
|
||||
- il NAS Rocky (`atlas`) usa un pool ZFS gia esistente, condivisioni NFSv4/SMB limitate alla LAN e Cockpit/45Drives
|
||||
- NPM è un Quadlet rootful su Prometheus, mentre Gitea, Navidrome e Syncthing sono Quadlet
|
||||
rootless su Atlas; il Compose server resta disabilitato per rollback
|
||||
rootless su Atlas; il fallback Compose server è stato rimosso
|
||||
|
||||
# Dotfiles
|
||||
|
||||
@@ -748,7 +751,7 @@ ansible-playbook ansible/site.yml --limit <host> --tags <tag1>,<tag2> --check --
|
||||
ansible-playbook ansible/site.yml --limit <host> --start-at-task "<task name>" --check --diff
|
||||
ansible-lint ansible/roles/<role>
|
||||
yamllint ansible/path/to/file.yml
|
||||
podman-compose -f /opt/docker/server/docker-compose.yml config
|
||||
ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags backend_phase1 --check --diff
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags music_sync --check --diff
|
||||
|
||||
18
README.md
18
README.md
@@ -130,14 +130,18 @@ dotfiles and templates. The profile does not transfer application data, update D
|
||||
implicit service cutover.
|
||||
|
||||
The server profile installs platform-specific packages, Podman and podman-compose, declared systemd
|
||||
services, and firewalld. Nginx Proxy Manager now runs as the rootful `prometheus-npm.service` Quadlet;
|
||||
the old `podman-compose-server` unit is disabled and retained only for controlled rollback. The
|
||||
Compose file no longer includes Gitea, Navidrome, Syncthing, or the obsolete Navidrome PostgreSQL
|
||||
database. The temporary Navidrome and Syncthing deployment on Atlas is managed by
|
||||
`profile_backend_phase1`. Applying the profile does not delete legacy data or `/opt/postgres/data`.
|
||||
The NPM cutover, data checks, backup evidence, and rollback boundaries are documented in
|
||||
services, and firewalld. Nginx Proxy Manager runs as the rootful `prometheus-npm.service` Quadlet.
|
||||
On 2026-10-03 the operator-approved opt-in cleanup removed old Gitea, Navidrome and PostgreSQL
|
||||
data/images, empty legacy directories, the Gitea final-export helper and the Compose rollback files.
|
||||
The migrated services stay on Atlas. `server_legacy_stack_retired: true` prevents normal runs from
|
||||
recreating retired files. Data deletion requires `--tags server_legacy_cleanup` and
|
||||
`-e server_legacy_cleanup=true`; image-only cleanup has its own `server_image_cleanup` tag and flag.
|
||||
Active NPM resources and existing backup archives remain preserved.
|
||||
The post-cleanup export/pull and isolated SQLite restore passed; the first unattended cycle remains
|
||||
pending. Evidence and recovery boundaries:
|
||||
[`docs/prometheus-npm-quadlet.md`](docs/prometheus-npm-quadlet.md).
|
||||
|
||||
|
||||
Firewalld enables SSH, Cockpit (`9090/tcp`), HTTP and HTTPS. Nginx Proxy Manager publishes only
|
||||
`80/tcp` and `443/tcp`; its administration interface is bound to `127.0.0.1:81` and can be reached
|
||||
from Ikaros or Nymph with the `npm-tunnel` Bash alias. Nextcloud remains disabled and the profile
|
||||
@@ -738,7 +742,7 @@ ansible-playbook ansible/site.yml --limit <host> --tags <tag1>,<tag2> --check --
|
||||
ansible-playbook ansible/site.yml --limit <host> --start-at-task "<task name>" --check --diff
|
||||
ansible-lint ansible/roles/<role>
|
||||
yamllint ansible/path/to/file.yml
|
||||
podman-compose -f /opt/docker/server/docker-compose.yml config
|
||||
ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags storage,sharing,containers --check --diff
|
||||
ansible-playbook ansible/site.yml --limit atlas --tags music_sync --check --diff
|
||||
```
|
||||
|
||||
@@ -8,6 +8,8 @@ effective_user_home: "{{ server_user_home }}"
|
||||
server_container_stack_dir: /opt/docker/server
|
||||
server_npm_quadlet_stage: false
|
||||
server_npm_quadlet_cutover: false
|
||||
server_legacy_stack_retired: false
|
||||
server_legacy_cleanup: false
|
||||
ai_agents: {}
|
||||
vim_plugins_enabled: false
|
||||
|
||||
@@ -101,8 +103,9 @@ server_backup_export_source_keep: 3
|
||||
server_backup_export_paths: >-
|
||||
{{ ['opt/npm/data', 'opt/npm/letsencrypt']
|
||||
+ ([] if server_gitea_on_atlas | bool else ['opt/gitea/data', 'home/git/.ssh'])
|
||||
+ ['opt/docker/server/docker-compose.yml',
|
||||
'etc/systemd/system/podman-compose-server.service']
|
||||
+ ([] if server_legacy_stack_retired | bool else
|
||||
['opt/docker/server/docker-compose.yml',
|
||||
'etc/systemd/system/podman-compose-server.service'])
|
||||
+ (['etc/containers/systemd/prometheus-npm.container',
|
||||
'etc/containers/systemd/server-web.network']
|
||||
if server_npm_quadlet_stage | bool else [])
|
||||
|
||||
@@ -6,6 +6,15 @@ ansible_port: 22
|
||||
ansible_ssh_private_key_file: /home/fscotto/.ssh/id_ed25519
|
||||
|
||||
server_username: rocky
|
||||
server_legacy_stack_retired: true
|
||||
# Destructive deletion runs only with an explicit extra-var and cleanup tag.
|
||||
server_legacy_cleanup: false
|
||||
# Explicit opt-in cleanup; no data, volumes, networks or NPM images are removed.
|
||||
server_legacy_image_cleanup: false
|
||||
server_legacy_images:
|
||||
- docker.gitea.com/gitea:1.25.2
|
||||
- docker.io/deluan/navidrome:latest
|
||||
- docker.io/library/postgres:13
|
||||
server_npm_quadlet_stage: true
|
||||
server_npm_quadlet_image: docker.io/jc21/nginx-proxy-manager@sha256:52b2c59994f3d36acfcf70a1626f29734df0ed8c71bacc0269f78b6f939858bb
|
||||
# The stopped-source export and live Quadlet cutover passed on 2026-10-03.
|
||||
@@ -17,7 +26,6 @@ server_gitea_cutover_tools_enabled: true
|
||||
server_gitea_on_atlas: true
|
||||
server_gitea_npm_domains:
|
||||
- git.fscotto.duckdns.org
|
||||
- git.ov-ad3410.infomaniak.ch
|
||||
server_duckdns_domain: fscotto
|
||||
server_ssh_authorized_keys:
|
||||
- name: ikaros
|
||||
|
||||
@@ -51,6 +51,7 @@
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0750"
|
||||
validate: "bash -n %s"
|
||||
when: server_backup_export_enabled | bool
|
||||
|
||||
- name: Install Prometheus backup export systemd units
|
||||
|
||||
@@ -7,7 +7,9 @@
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0750"
|
||||
when: server_gitea_cutover_tools_enabled | bool
|
||||
when:
|
||||
- server_gitea_cutover_tools_enabled | bool
|
||||
- not server_legacy_stack_retired | bool
|
||||
|
||||
- name: Require the prepared source and explicit final-export approval
|
||||
tags: [services, gitea_final_export]
|
||||
@@ -15,6 +17,7 @@
|
||||
that:
|
||||
- server_gitea_cutover_tools_enabled | bool
|
||||
- server_backup_export_enabled | bool
|
||||
- not server_legacy_stack_retired | bool
|
||||
- not ansible_check_mode
|
||||
fail_msg: >-
|
||||
Install the cutover helper and perform an explicit non-check-mode run
|
||||
@@ -31,4 +34,5 @@
|
||||
no_log: true
|
||||
when:
|
||||
- server_gitea_final_export | bool
|
||||
- not server_legacy_stack_retired | bool
|
||||
- not ansible_check_mode
|
||||
|
||||
112
ansible/roles/profile_server/tasks/legacy_cleanup.yml
Normal file
112
ansible/roles/profile_server/tasks/legacy_cleanup.yml
Normal file
@@ -0,0 +1,112 @@
|
||||
---
|
||||
- name: Require explicit retirement of the migrated Prometheus source
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- inventory_hostname == 'prometheus'
|
||||
- server_legacy_stack_retired | bool
|
||||
- server_gitea_on_atlas | bool
|
||||
- server_npm_quadlet_cutover | bool
|
||||
- server_backup_export_enabled | bool
|
||||
|
||||
- name: Verify legacy paths have no mounts or container users
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- python3
|
||||
- -c
|
||||
- |
|
||||
import json, os, pathlib, subprocess
|
||||
def run(*args):
|
||||
return subprocess.check_output(args, text=True).strip()
|
||||
paths = ['/opt/gitea', '/home/git/.ssh', '/opt/navidrome',
|
||||
'/opt/postgres', '/opt/music', '/opt/containerd', '/opt/docker']
|
||||
mounts = json.loads(run('findmnt', '--json', '--list', '-o', 'TARGET'))['filesystems']
|
||||
for path in paths:
|
||||
assert os.path.realpath(path) == path, 'Symlink in cleanup path: ' + path
|
||||
for mount in mounts:
|
||||
target = mount['target']
|
||||
assert target != path and not target.startswith(path + '/'), 'Mounted cleanup path: ' + path
|
||||
ids = run('podman', 'ps', '-aq').split()
|
||||
containers = json.loads(run('podman', 'inspect', *ids)) if ids else []
|
||||
for container in containers:
|
||||
assert container['Name'].lstrip('/') == 'nginx-proxy-manager', 'Unexpected container; review before cleanup'
|
||||
for mount in container.get('Mounts', []):
|
||||
source = os.path.realpath(mount['Source'])
|
||||
for path in paths:
|
||||
assert source != path and not source.startswith(path + '/'), 'Container uses cleanup path: ' + path
|
||||
for path in ['/opt/music', '/opt/containerd']:
|
||||
if os.path.isdir(path):
|
||||
for entry in pathlib.Path(path).rglob('*'):
|
||||
assert entry.is_dir() and not entry.is_symlink(), 'Unexpected file in empty legacy path: ' + str(entry)
|
||||
if os.path.isdir('/opt/docker'):
|
||||
allowed = {'/opt/docker/server', '/opt/docker/server/docker-compose.yml'}
|
||||
for entry in pathlib.Path('/opt/docker').rglob('*'):
|
||||
assert str(entry) in allowed and not entry.is_symlink(), 'Unexpected legacy Docker content: ' + str(entry)
|
||||
assert run('systemctl', 'is-active', 'prometheus-npm.service') == 'active'
|
||||
assert subprocess.run(['systemctl', 'is-active', '--quiet', 'podman-compose-server.service']).returncode != 0
|
||||
assert subprocess.run(['systemctl', 'is-active', '--quiet', 'prometheus-backup-export.service']).returncode != 0
|
||||
print('Legacy cleanup preflight passed')
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
|
||||
- name: Require the updated backup configuration before deleting fallback files
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- python3
|
||||
- -c
|
||||
- |
|
||||
import pathlib, subprocess
|
||||
unit = subprocess.check_output(['systemctl', 'show', 'prometheus-backup-export.service',
|
||||
'-p', 'RequiresMountsFor', '--value'], text=True)
|
||||
assert '/opt/gitea' not in unit, 'Backup unit still depends on legacy Gitea'
|
||||
helper = pathlib.Path('/usr/local/sbin/prometheus-backup-export').read_text()
|
||||
assert 'podman-compose-server' not in helper and 'opt/docker/server' not in helper
|
||||
subprocess.run(['bash', '-n', '/usr/local/sbin/prometheus-backup-export'], check=True)
|
||||
changed_when: false
|
||||
when: not ansible_check_mode
|
||||
|
||||
- name: Delete only the explicitly approved legacy data and fallback files
|
||||
ansible.builtin.file:
|
||||
path: "{{ item }}"
|
||||
state: absent
|
||||
loop:
|
||||
- /opt/gitea
|
||||
- /home/git/.ssh
|
||||
- /opt/navidrome
|
||||
- /opt/postgres
|
||||
- /opt/music
|
||||
- /opt/containerd
|
||||
- /opt/docker
|
||||
- /usr/local/sbin/prometheus-gitea-final-export
|
||||
- /etc/systemd/system/podman-compose-server.service
|
||||
register: server_legacy_deleted
|
||||
diff: false
|
||||
|
||||
- name: Reload systemd after removing the inactive legacy unit
|
||||
ansible.builtin.systemd:
|
||||
daemon_reload: true
|
||||
when:
|
||||
- server_legacy_deleted is changed
|
||||
- not ansible_check_mode
|
||||
|
||||
- name: Inspect the obsolete Git home without following symlinks
|
||||
ansible.builtin.stat:
|
||||
path: /home/git
|
||||
follow: false
|
||||
register: server_legacy_git_home
|
||||
|
||||
- name: Require the obsolete Git account to be absent before removing its empty home
|
||||
ansible.builtin.command:
|
||||
argv: [getent, passwd, git]
|
||||
register: server_legacy_git_account
|
||||
changed_when: false
|
||||
failed_when: server_legacy_git_account.rc != 2
|
||||
check_mode: false
|
||||
when: server_legacy_git_home.stat.exists
|
||||
|
||||
# rmdir refuses any nonempty directory; never recursively delete this parent.
|
||||
- name: Remove only the empty obsolete Git home
|
||||
ansible.builtin.command:
|
||||
argv: [rmdir, /home/git]
|
||||
register: server_legacy_git_home_removed
|
||||
changed_when: server_legacy_git_home_removed.rc == 0
|
||||
when: server_legacy_git_home.stat.exists
|
||||
33
ansible/roles/profile_server/tasks/legacy_image_cleanup.yml
Normal file
33
ansible/roles/profile_server/tasks/legacy_image_cleanup.yml
Normal file
@@ -0,0 +1,33 @@
|
||||
---
|
||||
- name: Require the migrated Prometheus topology for image cleanup
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- inventory_hostname == 'prometheus'
|
||||
- server_gitea_on_atlas | bool
|
||||
- server_npm_quadlet_cutover | bool
|
||||
- server_legacy_images | default([]) | length > 0
|
||||
- >-
|
||||
server_legacy_images | difference([
|
||||
'docker.gitea.com/gitea:1.25.2',
|
||||
'docker.io/deluan/navidrome:latest',
|
||||
'docker.io/library/postgres:13']) | length == 0
|
||||
|
||||
- name: Check whether the explicitly selected legacy images exist
|
||||
ansible.builtin.command:
|
||||
argv: [podman, image, exists, "{{ item }}"]
|
||||
loop: "{{ server_legacy_images }}"
|
||||
register: server_legacy_image_presence
|
||||
changed_when: false
|
||||
failed_when: server_legacy_image_presence.rc not in [0, 1]
|
||||
check_mode: false
|
||||
|
||||
# No --force: Podman must refuse images referenced by any existing container.
|
||||
- name: Remove only unused explicitly selected legacy images
|
||||
ansible.builtin.command:
|
||||
argv: [podman, image, rm, "{{ item.item }}"]
|
||||
loop: "{{ server_legacy_image_presence.results }}"
|
||||
loop_control:
|
||||
label: "{{ item.item }}"
|
||||
when: item.rc == 0
|
||||
register: server_legacy_image_removal
|
||||
changed_when: server_legacy_image_removal.rc == 0
|
||||
@@ -23,6 +23,9 @@
|
||||
loop: "{{ server_directories | default([]) }}"
|
||||
loop_control:
|
||||
label: "{{ item.path }}"
|
||||
when:
|
||||
- item.path != '/opt/gitea/data' or not server_gitea_on_atlas | bool
|
||||
- item.path != server_container_stack_dir or not server_legacy_stack_retired | bool
|
||||
|
||||
- name: Copy server dotfiles
|
||||
tags: [dotfiles, dotfiles:server]
|
||||
@@ -48,19 +51,32 @@
|
||||
loop_control:
|
||||
label: "{{ item.dest }}"
|
||||
no_log: "{{ item.no_log | default(false) }}"
|
||||
when: item.src != 'server/docker-compose.yml.j2' or not server_legacy_stack_retired | bool
|
||||
|
||||
- name: Manage Podman Compose stack
|
||||
tags: [services, podman]
|
||||
ansible.builtin.include_tasks: podman-compose.yml
|
||||
when: not server_legacy_stack_retired | bool
|
||||
|
||||
- name: Import staged NPM Quadlet tasks
|
||||
ansible.builtin.import_tasks: npm_quadlet.yml
|
||||
|
||||
- name: Import explicit legacy server image cleanup
|
||||
ansible.builtin.import_tasks: legacy_image_cleanup.yml
|
||||
tags: [never, server_image_cleanup]
|
||||
when: server_legacy_image_cleanup | default(false) | bool
|
||||
|
||||
- name: Import Prometheus backup export identity tasks
|
||||
ansible.builtin.import_tasks: backup_export_identity.yml
|
||||
|
||||
- name: Import Prometheus backup export job tasks
|
||||
ansible.builtin.import_tasks: backup_export_job.yml
|
||||
tags: [server_legacy_cleanup]
|
||||
|
||||
- name: Import explicitly approved legacy server data cleanup
|
||||
ansible.builtin.import_tasks: legacy_cleanup.yml
|
||||
tags: [never, server_legacy_cleanup]
|
||||
when: server_legacy_cleanup | bool
|
||||
|
||||
- name: Import explicit Prometheus Gitea final-export tasks
|
||||
ansible.builtin.import_tasks: gitea_final_export.yml
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[Unit]
|
||||
Description=Prepare a read-only Prometheus application backup for Atlas
|
||||
RequiresMountsFor=/opt/npm /opt/gitea {{ server_backup_export_root }}
|
||||
RequiresMountsFor=/opt/npm {% if not server_gitea_on_atlas | bool %}/opt/gitea {% endif %}{{ server_backup_export_root }}
|
||||
ConditionFileIsExecutable=/usr/local/sbin/prometheus-backup-export
|
||||
|
||||
[Service]
|
||||
|
||||
@@ -4,6 +4,9 @@ umask 077
|
||||
|
||||
export_root={{ server_backup_export_root | quote }}
|
||||
versions="$export_root/versions"
|
||||
{% if server_legacy_stack_retired | bool %}
|
||||
stack_unit=prometheus-npm.service
|
||||
{% else %}
|
||||
stack_unit=''
|
||||
compose_active=false
|
||||
quadlet_active=false
|
||||
@@ -18,6 +21,7 @@ if "$quadlet_active"; then
|
||||
else
|
||||
stack_unit=podman-compose-server.service
|
||||
fi
|
||||
{% endif %}
|
||||
stamp=$(date -u +%Y%m%dT%H%M%SZ)
|
||||
stage=''
|
||||
stack_stopped=false
|
||||
|
||||
@@ -1,8 +1,13 @@
|
||||
# Gitea migration from Prometheus to Atlas
|
||||
|
||||
This records the staged migration and its observed partial cutover. Gitea is
|
||||
temporary on Atlas until Uranus; NPM remains on Prometheus. Preserve the old
|
||||
Prometheus data, but do not restart its stale Gitea after Atlas accepts writes.
|
||||
temporary on Atlas until Uranus; NPM remains on Prometheus. On 2026-10-03
|
||||
the operator explicitly approved removal of the old Prometheus Gitea data,
|
||||
SSH fragment and final-export helper. NPM now uses a rootful Quadlet with no
|
||||
installed Compose fallback. The source-retention and rollback steps below
|
||||
are historical migration gates, not current recovery instructions.
|
||||
Existing backup archives were preserved; use current Atlas data and verified
|
||||
backups for recovery. Do not recreate or restart stale source Gitea.
|
||||
|
||||
## Observed source before cutover and chosen topology (2026-10-01)
|
||||
|
||||
@@ -139,6 +144,10 @@ or credentials were changed. The
|
||||
secondary hostname `git.ov-ad3410.infomaniak.ch` did not resolve from Ikaros
|
||||
and had no generated NPM config file at the time of inspection.
|
||||
|
||||
On 2026-10-03 the operator retired this unused secondary hostname. Its NPM
|
||||
Proxy Host was already soft-deleted; Ansible now declares only
|
||||
`git.fscotto.duckdns.org` and removes the secondary runtime override.
|
||||
|
||||
Prometheus' public TCP/2222 socket proxies to Atlas without changing admin
|
||||
SSH/22. The local socket presents the preserved Gitea ED25519 host key, but
|
||||
an external TCP/2222 connection from Ikaros initially timed out. During that
|
||||
|
||||
@@ -10,15 +10,15 @@ cutover is still pending. See `docs/prometheus-npm-quadlet.md`.
|
||||
## Declared design
|
||||
|
||||
- Prometheus prepares a tar archive of Nginx Proxy Manager data and certificates,
|
||||
its active Quadlet and network definitions, the disabled Compose fallback,
|
||||
its active Quadlet and network definitions,
|
||||
and SSH/firewalld/WireGuard configuration. Gitea now runs on Atlas and is no
|
||||
longer included in new Prometheus exports. NPM access logs are excluded.
|
||||
The archive contains credentials, certificates, and the WireGuard private
|
||||
key: protect both copies accordingly.
|
||||
- The approved consistency mode stops the one active NPM service (Quadlet now,
|
||||
Compose before cutover) for local tar creation at 02:00 Europe/Rome, then
|
||||
restarts it even if archiving fails. The helper refuses both services active
|
||||
or both inactive. A manual test outside that window requires separate approval.
|
||||
- The approved consistency mode stops the NPM Quadlet for local tar creation
|
||||
at 02:00 Europe/Rome, then restarts it even if archiving fails. After the
|
||||
approved legacy cleanup, the helper requires the Quadlet active and has
|
||||
no Compose dependency. A manual test outside that window requires separate approval.
|
||||
- Prometheus publishes the archive with its checksum as a versioned, read-only
|
||||
source under `/var/lib/prometheus-backup-export`. A locked service account
|
||||
has no sudo or supplementary groups. Its only authorized SSH key is forced
|
||||
@@ -119,3 +119,13 @@ both Quadlet definitions. A manifest of all 70 regular Let's Encrypt files
|
||||
and 12 symlinks, including content hashes and link targets, matched the live
|
||||
Prometheus tree. No private key or secret content was printed. The next
|
||||
scheduled export/pull is still pending observation.
|
||||
|
||||
## Post-cleanup validation (2026-10-03)
|
||||
|
||||
The operator-approved removal of legacy data and Compose fallback also
|
||||
removed those backup input paths and the obsolete Gitea mount dependency.
|
||||
A separately approved export and Atlas pull published `20261003T112906Z`.
|
||||
Both SHA-256 checks passed; an isolated SQLite restore passed `quick_check`
|
||||
and contained ten proxy hosts. Both active Quadlet definitions were present;
|
||||
retired paths were absent. Existing backup archives were not deleted by cleanup.
|
||||
The first scheduled cycle after these changes remains unverified.
|
||||
|
||||
@@ -8,19 +8,20 @@ Nginx Proxy Manager runs as the **rootful** generated
|
||||
`/etc/containers/systemd/server-web.network`; the image is pinned by digest
|
||||
in `ansible/inventory/host_vars/prometheus.yml`. The generated service is
|
||||
wanted by `multi-user.target` and requires the generated network service.
|
||||
The old `podman-compose-server.service` is inactive and disabled. Its unit
|
||||
and Compose file remain as a rollback option, not as another active owner.
|
||||
Do not start both units or run `podman-compose down` while the Quadlet owns
|
||||
the shared `server_web` network.
|
||||
The old Compose unit, Compose file and Gitea final-export helper were
|
||||
removed by the operator-approved cleanup on 2026-10-03. The retired
|
||||
application data and empty legacy directories were also removed.
|
||||
Prometheus host vars set `server_legacy_stack_retired: true` so normal runs
|
||||
do not recreate those files. Destructive deletion still requires a separate
|
||||
cleanup tag and explicit extra-var.
|
||||
|
||||
There was **no data copy** in this cutover. The Quadlet reuses the existing
|
||||
`/opt/npm/data:/data` and `/opt/npm/letsencrypt:/etc/letsencrypt` bind mounts
|
||||
with the same container name and `server_web` bridge (`10.89.0.0/24`). Ports
|
||||
80 and 443 remain public; administration port 81 remains bound to
|
||||
`127.0.0.1`. Gitea stays on Atlas, and NPM remains on Prometheus. The
|
||||
Prometheus Compose file is retained with the same pinned NPM image for a
|
||||
controlled fallback. The Quadlet uses `Pull=missing`, not an automatic
|
||||
floating-tag update.
|
||||
Compose fallback is no longer installed. The Quadlet uses `Pull=missing`,
|
||||
not an automatic floating-tag update.
|
||||
|
||||
## Cutover and recovery boundaries
|
||||
|
||||
@@ -36,16 +37,12 @@ then restarted the timer. Its failure trap would have restarted Compose.
|
||||
**Do not rerun that forward-cutover script after success**: its preconditions
|
||||
intentionally reject an active Quadlet.
|
||||
|
||||
A future rollback is a separate outage decision, not an ordinary Ansible run.
|
||||
First verify a usable recent Atlas backup and stop the export timer. Stop
|
||||
the Quadlet and verify that its container is gone before allowing Compose
|
||||
to own the same name, mounts, network, and ports; use the pinned Compose
|
||||
configuration, then validate NPM/HTTPS and restart the timer. Set
|
||||
`server_npm_quadlet_cutover: false` only as part of that controlled rollback.
|
||||
Do not run the two owners concurrently, restore an old NPM database over a
|
||||
live instance, or delete either bind mount. This reverse procedure has not
|
||||
been exercised on production; the forward script's in-window rollback path
|
||||
is not evidence of a later reverse cutover.
|
||||
Recovery is now a Quadlet rebuild and restoration from a verified Atlas
|
||||
backup, with an explicit outage decision before replacing live NPM state.
|
||||
The old Compose owner is no longer installed; reintroducing it would require
|
||||
a separately reviewed configuration and outage plan. The historical
|
||||
in-window rollback trap is not a supported post-cleanup rollback procedure.
|
||||
Do not restore an old database over a live instance or remove NPM bind mounts.
|
||||
|
||||
## Verified evidence
|
||||
|
||||
@@ -81,10 +78,68 @@ the new path works but not its next scheduled execution.
|
||||
ANSIBLE_LOCAL_TEMP=/tmp/ansible-local \
|
||||
ansible-playbook ansible/site.yml --limit prometheus --tags npm_quadlet --check --diff
|
||||
sudo systemctl status prometheus-npm.service prometheus-backup-export.timer
|
||||
sudo systemctl is-active podman-compose-server.service
|
||||
sudo systemctl is-enabled podman-compose-server.service
|
||||
sudo systemctl show podman-compose-server.service -p LoadState # expected: not-found
|
||||
```
|
||||
|
||||
The backup archive includes credentials, certificates, and WireGuard
|
||||
configuration. Do not publish it or print its contents in diagnostics; see
|
||||
`docs/prometheus-backup.md` for the restricted pull and restore procedure.
|
||||
|
||||
## Selective legacy image cleanup
|
||||
|
||||
On 2026-10-03 opt-in Ansible tasks removed only the unused Gitea 1.25.2,
|
||||
Navidrome latest and PostgreSQL 13 rootful images, without force or global
|
||||
prune. Podman refuses images referenced by existing containers. The second
|
||||
run changed nothing. NPM remained active with zero restarts; local admin
|
||||
and public Gitea HTTPS returned 200. Backup timer and SSH proxy stayed active.
|
||||
|
||||
Validation:
|
||||
```bash
|
||||
ansible-playbook ansible/site.yml --limit prometheus --tags server_image_cleanup --check --diff -e server_legacy_image_cleanup=true
|
||||
```
|
||||
|
||||
The image cleanup defaults to disabled and carries the `never` tag.
|
||||
Check mode probes image presence but skips removal; it does not prove
|
||||
Podman would accept deletion. It never removes NPM resources.
|
||||
|
||||
## Approved legacy data and fallback cleanup
|
||||
|
||||
The operator explicitly approved deletion on 2026-10-03. The separate
|
||||
`server_legacy_cleanup` tasks removed `/opt/gitea`, `/home/git/.ssh`,
|
||||
`/opt/navidrome`, `/opt/postgres`, `/opt/music`, `/opt/containerd`,
|
||||
`/opt/docker`, the old Compose unit and the final Gitea export helper.
|
||||
The empty `/home/git` parent is removed only with `rmdir`, after confirming
|
||||
the Git account is absent. Guards reject symlinked paths, nested mounts,
|
||||
unexpected containers, container users of these paths, unexpected content
|
||||
in the empty legacy trees, and an active Compose or export service.
|
||||
The second cleanup run changed nothing.
|
||||
|
||||
Before deletion, Ansible removed obsolete backup input paths and the
|
||||
Gitea mount dependency. Normal Compose/template/final-export task checks
|
||||
changed nothing and did not recreate the retired files. Deletion is opt-in:
|
||||
|
||||
```bash
|
||||
ansible-playbook ansible/site.yml --limit prometheus --tags server_legacy_cleanup --check --diff -e server_legacy_cleanup=true
|
||||
```
|
||||
|
||||
Remove check mode only for approved deletion. No active NPM data, certificate,
|
||||
image, network, volume, SSH proxy, WireGuard configuration or backup archive
|
||||
is removed. No services were restarted by the cleanup.
|
||||
|
||||
After separate approval for the brief managed NPM pause, the new export
|
||||
`20261003T112906Z` completed successfully and was pulled to Atlas. SHA-256
|
||||
passed on both hosts; an isolated SQLite restore passed `quick_check` and
|
||||
contained ten proxy hosts. Both Quadlet definitions were present, and
|
||||
retired paths were absent. Temporary restore files were removed.
|
||||
NPM was active with zero automatic restarts; primary public Gitea HTTPS
|
||||
returned 200 with valid TLS. Backup timer, SSH proxy and WireGuard stayed active.
|
||||
The first scheduled post-cleanup cycle remains unverified.
|
||||
|
||||
After separate operator approval on 2026-10-03, the unused secondary hostname
|
||||
`git.ov-ad3410.infomaniak.ch` was removed from the declared domains and
|
||||
the managed NPM runtime override. Its Proxy Host (id 10) was already
|
||||
soft-deleted, with no generated config or associated certificate. Historical
|
||||
deleted records and backup archives are preserved; no DNS changes were made.
|
||||
Only `git.fscotto.duckdns.org` remains declared for the Gitea override.
|
||||
Nginx validation and reload passed without restarting NPM; the primary
|
||||
public HTTPS endpoint returned 200 with valid TLS.
|
||||
|
||||
Reference in New Issue
Block a user