diff --git a/AGENTS.md b/AGENTS.md index fb9cfc9..627a45e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -274,7 +274,7 @@ successfully. The first monthly scrub remains a runtime check. - [x] Design the staged Prometheus-to-Atlas Gitea migration in `docs/atlas-gitea-migration.md`. The approved topology keeps NPM on Prometheus and moves HTTPS and public SSH (TCP/2222) together; Gitea must run as a dedicated rootless user Quadlet on Atlas. The rootful-to-rootless data-layout - conversion passed an isolated restore rehearsal. No production traffic has changed. + conversion passed an isolated restore rehearsal. The later partial cutover is tracked below. - [x] Prepare the dedicated Atlas Gitea dataset, non-login UID/GID 1101 with a separate rootless Podman sub-ID range, and disabled user Quadlet. On 2026-10-01 the targeted Ansible run and a second idempotent run passed; the generated unit was inactive, with no staging HTTP/SSH listener. POSIX ACLs on only the @@ -300,20 +300,29 @@ successfully. The first monthly scrub remains a runtime check. - [x] Install a separate opt-in final Gitea export helper on Prometheus. Its 2026-10-01 targeted deployment and `bash -n` passed while Gitea and NPM stayed running. It refuses an active export timer, stops only Gitea, verifies SQLite, publishes a checksum-verified Gitea-only version for - Atlas' existing pull, and leaves the source stopped on success; it has **not** been invoked. + Atlas' existing pull, and leaves the source stopped on success. It was invoked on 2026-10-02 + after the export timer was stopped; version `20261002T071525Z` was pulled and verified on Atlas. - [x] Prepare the Atlas final-restore gate without replacing the rehearsal: it accepts only a checksum-verified `gitea-cutover` export, refuses a running target, stages and validates the new layout before replacing the marked rehearsal, and rolls back a failed swap. Synthetic success - and rollback tests and a second idempotent rehearsal run passed on 2026-10-01; the final gate - has **not** been invoked. -- [x] Prepare, but do not activate, the Atlas LAN rootless Quadlet and Prometheus TCP/2222 socket - proxy. NPM's two existing `gitea:3000` Proxy Hosts will resolve that name to Atlas through a - managed Compose `extra_hosts` entry after the source container is removed; no NPM database edit - is needed. The future-mode Prometheus check-run passed, both current-mode runs were idempotent, - the new systemd units passed verification, and source HTTP remained 200 on 2026-10-01. Public - 2222 is closed and the target remains inactive until the explicit cutover flags are enabled. -- [ ] After an explicit outage approval, perform the final consistent copy and HTTPS/SSH cutover, - then remove Gitea from Prometheus' desired stack and backup export without deleting source data. + and rollback tests passed on 2026-10-01. On 2026-10-02 the final gate replaced the rehearsal; + SQLite `quick_check`, all 33 repository `git fsck` checks, checksum and SSH host-key comparison passed. +- [x] Start the rootless Atlas Gitea Quadlet and move the primary HTTPS route. On 2026-10-02 Atlas + answered HTTP 200 through the Aegis gateway. NPM stayed on Prometheus; its variable upstream + required a managed Nginx `server_proxy.conf` override because runtime DNS ignores Compose + `extra_hosts`. The primary public HTTPS page and API returned 200, and `git ls-remote` succeeded + for a representative repository after NPM restart; the Navidrome and Syncthing Proxy Hosts also + responded. The source + Gitea container was removed from the desired Compose stack without deleting its data; the + Prometheus backup export timer resumed for NPM only. A post-cutover recursive ZFS snapshot and + encrypted Borg archive `atlas-20261002T073044Z` completed successfully. +- [ ] Complete public SSH/2222 and representative authenticated HTTPS/SSH clone/push validation. + Prometheus' TCP/2222 socket and firewalld rule are active and the local proxy presents the + matching Atlas host key, but Ikaros' external TCP connection timed out and no SYN reached + Prometheus `eth0` during the test. Investigate upstream/provider filtering; do not claim the + approved simultaneous HTTPS+SSH cutover complete. The secondary NPM hostname + `git.ov-ad3410.infomaniak.ch` did not resolve from Ikaros and had no generated NPM config file. + Do not restart the stale source Gitea after Atlas has accepted writes. - [ ] Design and deploy Nextcloud as another explicitly temporary Atlas service before Uranus. Give it separate persistent application, database, and cache storage; keep credentials in Vault; publish it only through NPM over the Prometheus--Aegis gateway; and define backup, upgrade, and eventual Uranus-migration diff --git a/README.md b/README.md index 04453db..76e50c7 100644 --- a/README.md +++ b/README.md @@ -298,12 +298,13 @@ and Aegis (`10.0.0.2`). Their state is initialized ex novo in `/zpool/services/d `/zpool/services/data/syncthing`; no source application state is migrated. The music library at `/zpool/media/music` is populated separately. -The Gitea move from Prometheus to Atlas is staged in -[`docs/atlas-gitea-migration.md`](docs/atlas-gitea-migration.md). Atlas has a dedicated dataset and -non-login account with an inactive rootless user Quadlet. An isolated copy from the verified -Prometheus backup passed SQLite, Git, and network-disabled -rootless-container checks; it is not the final cutover copy. NPM remains on Prometheus; the source -stack and public routes stay unchanged until a separately validated HTTPS and SSH cutover. +The Gitea move from Prometheus to Atlas is tracked in +[`docs/atlas-gitea-migration.md`](docs/atlas-gitea-migration.md). The final consistent copy runs in +Atlas' dedicated dataset under a rootless user Quadlet. NPM remains on Prometheus and the primary +public HTTPS route serves Atlas. The public SSH/2222 socket works locally on Prometheus, but an +external connection did not reach its interface on 2026-10-02; check upstream filtering before +declaring the HTTPS+SSH cutover complete. The old Gitea data remains on Prometheus, but its container +is absent from the desired stack. The separate `wireguard_overlay` role manages `wg0` between Prometheus (`10.0.0.1`) and Aegis (`10.0.0.2`), generating private keys once on their respective hosts and exchanging only public keys diff --git a/ansible/inventory/group_vars/server.yml b/ansible/inventory/group_vars/server.yml index 75aa3d8..360acdd 100644 --- a/ansible/inventory/group_vars/server.yml +++ b/ansible/inventory/group_vars/server.yml @@ -92,6 +92,7 @@ server_gitea_cutover_tools_enabled: false server_gitea_final_export: false server_gitea_on_atlas: false server_gitea_atlas_address: "{{ hostvars['atlas'].ansible_host }}" +server_gitea_npm_domains: [] server_gitea_ssh_public_port: 2222 server_gitea_ssh_target_port: 2222 server_backup_export_source_keep: 3 diff --git a/ansible/inventory/host_vars/prometheus.yml b/ansible/inventory/host_vars/prometheus.yml index 538af8d..a5c8590 100644 --- a/ansible/inventory/host_vars/prometheus.yml +++ b/ansible/inventory/host_vars/prometheus.yml @@ -10,6 +10,10 @@ server_backup_export_enabled: true server_backup_export_start_timer: true # Install the final-copy helper only; it is never run by a normal playbook invocation. server_gitea_cutover_tools_enabled: true +server_gitea_on_atlas: true +server_gitea_npm_domains: + - git.fscotto.duckdns.org + - git.ov-ad3410.infomaniak.ch server_duckdns_domain: fscotto server_ssh_authorized_keys: - name: ikaros diff --git a/ansible/roles/profile_server/tasks/gitea_npm_proxy.yml b/ansible/roles/profile_server/tasks/gitea_npm_proxy.yml new file mode 100644 index 0000000..3466d60 --- /dev/null +++ b/ansible/roles/profile_server/tasks/gitea_npm_proxy.yml @@ -0,0 +1,59 @@ +--- +- name: Validate the NPM Gitea cutover override + tags: [services, gitea_cutover] + ansible.builtin.assert: + that: + - server_gitea_cutover_tools_enabled | bool + - server_gitea_npm_domains | length > 0 + - server_gitea_npm_domains | select('match', '^[a-zA-Z0-9.-]+$') | list | length == server_gitea_npm_domains | length + fail_msg: Declare the exact NPM Gitea hostnames before enabling the Atlas upstream. + when: server_gitea_on_atlas | bool + +- name: Ensure the NPM custom configuration directory exists + tags: [services, gitea_cutover] + ansible.builtin.file: + path: /opt/npm/data/nginx/custom + state: directory + owner: root + group: root + mode: "0755" + when: server_gitea_cutover_tools_enabled | bool + +- name: Render the Gitea-only NPM runtime upstream override + tags: [services, gitea_cutover] + ansible.builtin.template: + src: prometheus-gitea-npm-proxy.conf.j2 + dest: /opt/npm/data/nginx/custom/server_proxy.conf + owner: root + group: root + mode: "0644" + register: server_gitea_npm_override + when: server_gitea_on_atlas | bool + +- name: Remove the Gitea NPM override when source routing is selected + tags: [services, gitea_cutover] + ansible.builtin.file: + path: /opt/npm/data/nginx/custom/server_proxy.conf + state: absent + when: + - server_gitea_cutover_tools_enabled | bool + - not server_gitea_on_atlas | bool + +- name: Validate NPM configuration after a Gitea upstream change + tags: [services, gitea_cutover] + ansible.builtin.command: + argv: [podman, exec, nginx-proxy-manager, nginx, -t] + changed_when: false + when: + - server_gitea_on_atlas | bool + - server_gitea_npm_override is changed + - not ansible_check_mode + +- name: Reload NPM after validating the Gitea upstream change + tags: [services, gitea_cutover] + ansible.builtin.command: + argv: [podman, exec, nginx-proxy-manager, nginx, -s, reload] + when: + - server_gitea_on_atlas | bool + - server_gitea_npm_override is changed + - not ansible_check_mode diff --git a/ansible/roles/profile_server/tasks/main.yml b/ansible/roles/profile_server/tasks/main.yml index 9163fcf..59e7c1b 100644 --- a/ansible/roles/profile_server/tasks/main.yml +++ b/ansible/roles/profile_server/tasks/main.yml @@ -65,6 +65,9 @@ - name: Import Prometheus Gitea SSH proxy tasks ansible.builtin.import_tasks: gitea_ssh_proxy.yml +- name: Import Prometheus Gitea NPM proxy override tasks + ansible.builtin.import_tasks: gitea_npm_proxy.yml + - name: Ensure server SSH authorized key fragments directory exists tags: [services, ssh] ansible.builtin.file: diff --git a/ansible/roles/profile_server/templates/prometheus-gitea-npm-proxy.conf.j2 b/ansible/roles/profile_server/templates/prometheus-gitea-npm-proxy.conf.j2 new file mode 100644 index 0000000..754fc27 --- /dev/null +++ b/ansible/roles/profile_server/templates/prometheus-gitea-npm-proxy.conf.j2 @@ -0,0 +1,6 @@ +# Managed by Ansible. NPM's variable proxy upstream uses Nginx DNS, not /etc/hosts. +{% for domain in server_gitea_npm_domains %} +if ($host = {{ domain }}) { + set $server {{ server_gitea_atlas_address }}; +} +{% endfor %} diff --git a/ansible/templates/server/docker-compose.yml.j2 b/ansible/templates/server/docker-compose.yml.j2 index 69003ba..45055b1 100644 --- a/ansible/templates/server/docker-compose.yml.j2 +++ b/ansible/templates/server/docker-compose.yml.j2 @@ -13,10 +13,6 @@ services: - "127.0.0.1:81:81" extra_hosts: - "host.containers.internal:host-gateway" -{% if server_gitea_on_atlas | bool %} - # Keep both existing NPM Proxy Hosts unchanged; their gitea name now resolves to Atlas. - - "gitea:{{ server_gitea_atlas_address }}" -{% endif %} volumes: - "/opt/npm/data:/data{{ ':' ~ selinux_volume_option if selinux_volume_option else '' }}" - "/opt/npm/letsencrypt:/etc/letsencrypt{{ ':' ~ selinux_volume_option if selinux_volume_option else '' }}" diff --git a/docs/atlas-gitea-migration.md b/docs/atlas-gitea-migration.md index 72dfd87..13d4d97 100644 --- a/docs/atlas-gitea-migration.md +++ b/docs/atlas-gitea-migration.md @@ -1,11 +1,10 @@ # Gitea migration from Prometheus to Atlas -This is a staged migration plan, not a cutover authorization. Keep the source -Gitea, its data, both NPM Proxy Hosts, and public DNS unchanged until the -target and rollback have been tested. Gitea is temporary on Atlas until -Uranus; NPM remains on Prometheus. +This records the staged migration and its observed partial cutover. Gitea is +temporary on Atlas until Uranus; NPM remains on Prometheus. Preserve the old +Prometheus data, but do not restart its stale Gitea after Atlas accepts writes. -## Observed source and chosen topology (2026-10-01) +## Observed source before cutover and chosen topology (2026-10-01) - Prometheus runs the rootful `docker.gitea.com/gitea:1.25.2` image in its managed Compose stack. `/opt/gitea/data` is about 280 MiB, uses SQLite, @@ -108,7 +107,7 @@ not a complete Gitea recovery rehearsal from USB. ## Phase 2: explicit final cutover The opt-in `/usr/local/sbin/prometheus-gitea-final-export` helper was installed -on 2026-10-01 and passed `bash -n`; it has **not** been invoked. It refuses to +on 2026-10-01 and passed `bash -n`. It refuses to run while the scheduled Prometheus export timer is active. When explicitly triggered, it stops only the source Gitea container, checks SQLite, publishes a checksum-verified Gitea-only version for Atlas' existing pull, and leaves @@ -119,26 +118,38 @@ After Atlas pulls that version, its separate `--tags gitea_final_restore -e atlas_gitea_final_restore=true` gate accepts only metadata marked `gitea-cutover`, validates a private staged replacement, and swaps it for the marked rehearsal. The swap and its rollback path passed -synthetic tests on 2026-10-01; the gate has not been used on live Gitea data. +synthetic tests on 2026-10-01; the live gate succeeded on 2026-10-02. -The network change is also prepared but inactive. `server_gitea_on_atlas=true` -removes the rootful Gitea service from the desired Prometheus Compose stack, -adds `gitea:192.168.178.55` to NPM's container hosts file, and removes Gitea -from future Prometheus backup exports. Both existing NPM Proxy Host records -remain at `gitea:3000`, but that name then resolves to Atlas; no direct SQLite -edit or NPM login is required. A separate systemd socket on public TCP/2222 -proxies SSH to Atlas TCP/2222 over the gateway, leaving administrative TCP/22 -unchanged. Atlas' production flag changes the user Quadlet from loopback -staging ports to LAN ports 3000/2222, grants only Aegis access in firewalld, -and starts it **only** after the `.final-sha256` marker exists. Neither flag -is enabled yet. The future Prometheus configuration passed a check-run; the -installed socket units passed `systemd-analyze verify` while remaining -inactive. Source Gitea still answered HTTP 200 after preparation. +On 2026-10-02 the operator approved the outage. The final stopped-source +export `20261002T071525Z` passed the Atlas pull checksum; the guarded restore +replaced the rehearsal. SQLite `quick_check`, all 33 repository `git fsck` +checks, and the source/target SSH host-key comparison passed. The rootless +Atlas Quadlet serves LAN HTTP/3000 and SSH/2222, reachable from Prometheus +through Aegis; its firewall admits only Aegis. The final marker gates startup. -The previously agreed USB prerequisite is now met, but the final export, -production flags, and public cutover remain uninvoked. The prepared -configuration alone does not constitute a migration; confirm a fresh outage -window before stopping the source or switching traffic. +Prometheus now runs the NPM-only Compose stack. Both NPM database records still +say `gitea:3000`, but Nginx evaluates this variable upstream through its +runtime DNS resolver, which **does not** use a Compose `extra_hosts` alias. +The initial alias attempt returned 502. A managed `server_proxy.conf` override +sets `$server` to Atlas' IP for only the two declared Gitea domains; it passed +`nginx -t` and primary HTTPS/API returned 200 after a clean NPM restart +without the alias; a representative public `git ls-remote` also succeeded. +Navidrome and Syncthing Proxy Hosts still responded. No NPM SQLite records +or credentials were changed. The +secondary hostname `git.ov-ad3410.infomaniak.ch` did not resolve from Ikaros +and had no generated NPM config file at the time of inspection. + +Prometheus' public TCP/2222 socket proxies to Atlas without changing admin +SSH/22. The local socket presents the preserved Gitea ED25519 host key, but +an external TCP/2222 connection from Ikaros timed out. During the test no SYN +reached Prometheus `eth0`; its socket and firewalld port were active. Check +upstream/provider filtering before declaring public SSH complete. Do not +restart the stale source after public HTTPS has accepted target writes. + +The Prometheus export timer resumed with NPM-only paths. A recursive ZFS +snapshot at `20261002T073032Z` and encrypted Borg archive +`atlas-20261002T073044Z` captured the Atlas target after cutover; Borg exited +successfully, cleaned its temporary snapshot, and the pool was healthy. 1. Agree on an outage and record source/target versions, pool health, the latest backups, SSH host-key fingerprints, and both current NPM routes. @@ -157,23 +168,22 @@ window before stopping the source or switching traffic. and the target service before switching NPM. 4. Enable the public TCP/2222 socket proxy on Prometheus to Atlas over Aegis without changing administrative TCP/22. Switch Prometheus to the desired - NPM-only Compose stack and recreate NPM with the managed `gitea` host alias - so **both** existing Proxy Hosts reach Atlas without changing their database - records. The old Gitea data stays intact. Do not change public DNS. + NPM-only Compose stack and use the managed Gitea-only NPM runtime upstream + override. Do not use Compose `extra_hosts`: Nginx bypasses it for the + variable upstream. The old Gitea data stays intact. Do not change public DNS. 5. Test HTTPS login, representative clone/push, LFS, and public SSH clone/push on port 2222 from outside the Atlas LAN. Record the last source write and first healthy target service times; do not claim RPO/RTO without measuring. -6. After successful traffic validation, resume the Prometheus NPM-only backup - export timer and verify its next result. Verify the next Atlas snapshot/Borg +6. Resume the Prometheus NPM-only backup export timer after the desired stack + is active and verify its next result. Verify the next Atlas snapshot/Borg run covers Gitea and test a restored target copy. Do not delete old source data. ## Rollback gate -Before Atlas accepts writes, restore the old Compose definition (removing the -NPM `gitea` host alias), disable the public 2222 proxy, and restart the -unchanged source Gitea if target validation -fails. **After Atlas accepts writes, do not blindly restart the source:** its +Before Atlas accepts writes, restore the old Compose definition and remove the +NPM override, disable the public 2222 proxy, and restart the unchanged source +Gitea if target validation fails. **After Atlas accepts writes, do not blindly restart the source:** its SQLite database and repositories are stale. Quiesce Atlas, capture its new data, and decide a reverse migration or an extended outage explicitly.