Cut over Gitea HTTPS to Atlas with managed NPM upstream

AGENTS.md

@@ -274,7 +274,7 @@ successfully. The first monthly scrub remains a runtime check.
 - [x] Design the staged Prometheus-to-Atlas Gitea migration in `docs/atlas-gitea-migration.md`.
   The approved topology keeps NPM on Prometheus and moves HTTPS and public SSH (TCP/2222) together;
   Gitea must run as a dedicated rootless user Quadlet on Atlas. The rootful-to-rootless data-layout
-  conversion passed an isolated restore rehearsal. No production traffic has changed.
+  conversion passed an isolated restore rehearsal. The later partial cutover is tracked below.
 - [x] Prepare the dedicated Atlas Gitea dataset, non-login UID/GID 1101 with a separate rootless Podman
   sub-ID range, and disabled user Quadlet. On 2026-10-01 the targeted Ansible run and a second idempotent
   run passed; the generated unit was inactive, with no staging HTTP/SSH listener. POSIX ACLs on only the
@@ -300,20 +300,29 @@ successfully. The first monthly scrub remains a runtime check.
 - [x] Install a separate opt-in final Gitea export helper on Prometheus. Its 2026-10-01 targeted
   deployment and `bash -n` passed while Gitea and NPM stayed running. It refuses an active export
   timer, stops only Gitea, verifies SQLite, publishes a checksum-verified Gitea-only version for
-  Atlas' existing pull, and leaves the source stopped on success; it has **not** been invoked.
+  Atlas' existing pull, and leaves the source stopped on success. It was invoked on 2026-10-02
+  after the export timer was stopped; version `20261002T071525Z` was pulled and verified on Atlas.
 - [x] Prepare the Atlas final-restore gate without replacing the rehearsal: it accepts only a
   checksum-verified `gitea-cutover` export, refuses a running target, stages and validates the new
   layout before replacing the marked rehearsal, and rolls back a failed swap. Synthetic success
-  and rollback tests and a second idempotent rehearsal run passed on 2026-10-01; the final gate
-  has **not** been invoked.
-- [x] Prepare, but do not activate, the Atlas LAN rootless Quadlet and Prometheus TCP/2222 socket
-  proxy. NPM's two existing `gitea:3000` Proxy Hosts will resolve that name to Atlas through a
-  managed Compose `extra_hosts` entry after the source container is removed; no NPM database edit
-  is needed. The future-mode Prometheus check-run passed, both current-mode runs were idempotent,
-  the new systemd units passed verification, and source HTTP remained 200 on 2026-10-01. Public
-  2222 is closed and the target remains inactive until the explicit cutover flags are enabled.
-- [ ] After an explicit outage approval, perform the final consistent copy and HTTPS/SSH cutover,
-  then remove Gitea from Prometheus' desired stack and backup export without deleting source data.
+  and rollback tests passed on 2026-10-01. On 2026-10-02 the final gate replaced the rehearsal;
+  SQLite `quick_check`, all 33 repository `git fsck` checks, checksum and SSH host-key comparison passed.
+- [x] Start the rootless Atlas Gitea Quadlet and move the primary HTTPS route. On 2026-10-02 Atlas
+  answered HTTP 200 through the Aegis gateway. NPM stayed on Prometheus; its variable upstream
+  required a managed Nginx `server_proxy.conf` override because runtime DNS ignores Compose
+  `extra_hosts`. The primary public HTTPS page and API returned 200, and `git ls-remote` succeeded
+  for a representative repository after NPM restart; the Navidrome and Syncthing Proxy Hosts also
+  responded. The source
+  Gitea container was removed from the desired Compose stack without deleting its data; the
+  Prometheus backup export timer resumed for NPM only. A post-cutover recursive ZFS snapshot and
+  encrypted Borg archive `atlas-20261002T073044Z` completed successfully.
+- [ ] Complete public SSH/2222 and representative authenticated HTTPS/SSH clone/push validation.
+  Prometheus' TCP/2222 socket and firewalld rule are active and the local proxy presents the
+  matching Atlas host key, but Ikaros' external TCP connection timed out and no SYN reached
+  Prometheus `eth0` during the test. Investigate upstream/provider filtering; do not claim the
+  approved simultaneous HTTPS+SSH cutover complete. The secondary NPM hostname
+  `git.ov-ad3410.infomaniak.ch` did not resolve from Ikaros and had no generated NPM config file.
+  Do not restart the stale source Gitea after Atlas has accepted writes.
 - [ ] Design and deploy Nextcloud as another explicitly temporary Atlas service before Uranus. Give it
   separate persistent application, database, and cache storage; keep credentials in Vault; publish it only
   through NPM over the Prometheus--Aegis gateway; and define backup, upgrade, and eventual Uranus-migration

README.md

@@ -298,12 +298,13 @@ and Aegis (`10.0.0.2`). Their state is initialized ex novo in `/zpool/services/d
 `/zpool/services/data/syncthing`; no source application state is migrated. The music library at
 `/zpool/media/music` is populated separately.
 
-The Gitea move from Prometheus to Atlas is staged in
-[`docs/atlas-gitea-migration.md`](docs/atlas-gitea-migration.md). Atlas has a dedicated dataset and
-non-login account with an inactive rootless user Quadlet. An isolated copy from the verified
-Prometheus backup passed SQLite, Git, and network-disabled
-rootless-container checks; it is not the final cutover copy. NPM remains on Prometheus; the source
-stack and public routes stay unchanged until a separately validated HTTPS and SSH cutover.
+The Gitea move from Prometheus to Atlas is tracked in
+[`docs/atlas-gitea-migration.md`](docs/atlas-gitea-migration.md). The final consistent copy runs in
+Atlas' dedicated dataset under a rootless user Quadlet. NPM remains on Prometheus and the primary
+public HTTPS route serves Atlas. The public SSH/2222 socket works locally on Prometheus, but an
+external connection did not reach its interface on 2026-10-02; check upstream filtering before
+declaring the HTTPS+SSH cutover complete. The old Gitea data remains on Prometheus, but its container
+is absent from the desired stack.
 
 The separate `wireguard_overlay` role manages `wg0` between Prometheus (`10.0.0.1`) and Aegis
 (`10.0.0.2`), generating private keys once on their respective hosts and exchanging only public keys

ansible/inventory/group_vars/server.yml

@@ -92,6 +92,7 @@ server_gitea_cutover_tools_enabled: false
 server_gitea_final_export: false
 server_gitea_on_atlas: false
 server_gitea_atlas_address: "{{ hostvars['atlas'].ansible_host }}"
+server_gitea_npm_domains: []
 server_gitea_ssh_public_port: 2222
 server_gitea_ssh_target_port: 2222
 server_backup_export_source_keep: 3

ansible/inventory/host_vars/prometheus.yml

@@ -10,6 +10,10 @@ server_backup_export_enabled: true
 server_backup_export_start_timer: true
 # Install the final-copy helper only; it is never run by a normal playbook invocation.
 server_gitea_cutover_tools_enabled: true
+server_gitea_on_atlas: true
+server_gitea_npm_domains:
+  - git.fscotto.duckdns.org
+  - git.ov-ad3410.infomaniak.ch
 server_duckdns_domain: fscotto
 server_ssh_authorized_keys:
   - name: ikaros

ansible/roles/profile_server/tasks/gitea_npm_proxy.yml

@@ -0,0 +1,59 @@
+---
+- name: Validate the NPM Gitea cutover override
+  tags: [services, gitea_cutover]
+  ansible.builtin.assert:
+    that:
+      - server_gitea_cutover_tools_enabled | bool
+      - server_gitea_npm_domains | length > 0
+      - server_gitea_npm_domains | select('match', '^[a-zA-Z0-9.-]+$') | list | length == server_gitea_npm_domains | length
+    fail_msg: Declare the exact NPM Gitea hostnames before enabling the Atlas upstream.
+  when: server_gitea_on_atlas | bool
+
+- name: Ensure the NPM custom configuration directory exists
+  tags: [services, gitea_cutover]
+  ansible.builtin.file:
+    path: /opt/npm/data/nginx/custom
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+  when: server_gitea_cutover_tools_enabled | bool
+
+- name: Render the Gitea-only NPM runtime upstream override
+  tags: [services, gitea_cutover]
+  ansible.builtin.template:
+    src: prometheus-gitea-npm-proxy.conf.j2
+    dest: /opt/npm/data/nginx/custom/server_proxy.conf
+    owner: root
+    group: root
+    mode: "0644"
+  register: server_gitea_npm_override
+  when: server_gitea_on_atlas | bool
+
+- name: Remove the Gitea NPM override when source routing is selected
+  tags: [services, gitea_cutover]
+  ansible.builtin.file:
+    path: /opt/npm/data/nginx/custom/server_proxy.conf
+    state: absent
+  when:
+    - server_gitea_cutover_tools_enabled | bool
+    - not server_gitea_on_atlas | bool
+
+- name: Validate NPM configuration after a Gitea upstream change
+  tags: [services, gitea_cutover]
+  ansible.builtin.command:
+    argv: [podman, exec, nginx-proxy-manager, nginx, -t]
+  changed_when: false
+  when:
+    - server_gitea_on_atlas | bool
+    - server_gitea_npm_override is changed
+    - not ansible_check_mode
+
+- name: Reload NPM after validating the Gitea upstream change
+  tags: [services, gitea_cutover]
+  ansible.builtin.command:
+    argv: [podman, exec, nginx-proxy-manager, nginx, -s, reload]
+  when:
+    - server_gitea_on_atlas | bool
+    - server_gitea_npm_override is changed
+    - not ansible_check_mode

ansible/roles/profile_server/tasks/main.yml

@@ -65,6 +65,9 @@
 - name: Import Prometheus Gitea SSH proxy tasks
   ansible.builtin.import_tasks: gitea_ssh_proxy.yml
 
+- name: Import Prometheus Gitea NPM proxy override tasks
+  ansible.builtin.import_tasks: gitea_npm_proxy.yml
+
 - name: Ensure server SSH authorized key fragments directory exists
   tags: [services, ssh]
   ansible.builtin.file:

ansible/roles/profile_server/templates/prometheus-gitea-npm-proxy.conf.j2

@@ -0,0 +1,6 @@
+# Managed by Ansible. NPM's variable proxy upstream uses Nginx DNS, not /etc/hosts.
+{% for domain in server_gitea_npm_domains %}
+if ($host = {{ domain }}) {
+  set $server {{ server_gitea_atlas_address }};
+}
+{% endfor %}

ansible/templates/server/docker-compose.yml.j2

@@ -13,10 +13,6 @@ services:
       - "127.0.0.1:81:81"
     extra_hosts:
       - "host.containers.internal:host-gateway"
-{% if server_gitea_on_atlas | bool %}
-      # Keep both existing NPM Proxy Hosts unchanged; their gitea name now resolves to Atlas.
-      - "gitea:{{ server_gitea_atlas_address }}"
-{% endif %}
     volumes:
       - "/opt/npm/data:/data{{ ':' ~ selinux_volume_option if selinux_volume_option else '' }}"
       - "/opt/npm/letsencrypt:/etc/letsencrypt{{ ':' ~ selinux_volume_option if selinux_volume_option else '' }}"

docs/atlas-gitea-migration.md

@@ -1,11 +1,10 @@
 # Gitea migration from Prometheus to Atlas
 
-This is a staged migration plan, not a cutover authorization. Keep the source
-Gitea, its data, both NPM Proxy Hosts, and public DNS unchanged until the
-target and rollback have been tested. Gitea is temporary on Atlas until
-Uranus; NPM remains on Prometheus.
+This records the staged migration and its observed partial cutover. Gitea is
+temporary on Atlas until Uranus; NPM remains on Prometheus. Preserve the old
+Prometheus data, but do not restart its stale Gitea after Atlas accepts writes.
 
-## Observed source and chosen topology (2026-10-01)
+## Observed source before cutover and chosen topology (2026-10-01)
 
 - Prometheus runs the rootful `docker.gitea.com/gitea:1.25.2` image in its
   managed Compose stack. `/opt/gitea/data` is about 280 MiB, uses SQLite,
@@ -108,7 +107,7 @@ not a complete Gitea recovery rehearsal from USB.
 ## Phase 2: explicit final cutover
 
 The opt-in `/usr/local/sbin/prometheus-gitea-final-export` helper was installed
-on 2026-10-01 and passed `bash -n`; it has **not** been invoked. It refuses to
+on 2026-10-01 and passed `bash -n`. It refuses to
 run while the scheduled Prometheus export timer is active. When explicitly
 triggered, it stops only the source Gitea container, checks SQLite, publishes
 a checksum-verified Gitea-only version for Atlas' existing pull, and leaves
@@ -119,26 +118,38 @@ After Atlas pulls that version, its separate
 `--tags gitea_final_restore -e atlas_gitea_final_restore=true` gate accepts
 only metadata marked `gitea-cutover`, validates a private staged replacement,
 and swaps it for the marked rehearsal. The swap and its rollback path passed
-synthetic tests on 2026-10-01; the gate has not been used on live Gitea data.
+synthetic tests on 2026-10-01; the live gate succeeded on 2026-10-02.
 
-The network change is also prepared but inactive. `server_gitea_on_atlas=true`
-removes the rootful Gitea service from the desired Prometheus Compose stack,
-adds `gitea:192.168.178.55` to NPM's container hosts file, and removes Gitea
-from future Prometheus backup exports. Both existing NPM Proxy Host records
-remain at `gitea:3000`, but that name then resolves to Atlas; no direct SQLite
-edit or NPM login is required. A separate systemd socket on public TCP/2222
-proxies SSH to Atlas TCP/2222 over the gateway, leaving administrative TCP/22
-unchanged. Atlas' production flag changes the user Quadlet from loopback
-staging ports to LAN ports 3000/2222, grants only Aegis access in firewalld,
-and starts it **only** after the `.final-sha256` marker exists. Neither flag
-is enabled yet. The future Prometheus configuration passed a check-run; the
-installed socket units passed `systemd-analyze verify` while remaining
-inactive. Source Gitea still answered HTTP 200 after preparation.
+On 2026-10-02 the operator approved the outage. The final stopped-source
+export `20261002T071525Z` passed the Atlas pull checksum; the guarded restore
+replaced the rehearsal. SQLite `quick_check`, all 33 repository `git fsck`
+checks, and the source/target SSH host-key comparison passed. The rootless
+Atlas Quadlet serves LAN HTTP/3000 and SSH/2222, reachable from Prometheus
+through Aegis; its firewall admits only Aegis. The final marker gates startup.
 
-The previously agreed USB prerequisite is now met, but the final export,
-production flags, and public cutover remain uninvoked. The prepared
-configuration alone does not constitute a migration; confirm a fresh outage
-window before stopping the source or switching traffic.
+Prometheus now runs the NPM-only Compose stack. Both NPM database records still
+say `gitea:3000`, but Nginx evaluates this variable upstream through its
+runtime DNS resolver, which **does not** use a Compose `extra_hosts` alias.
+The initial alias attempt returned 502. A managed `server_proxy.conf` override
+sets `$server` to Atlas' IP for only the two declared Gitea domains; it passed
+`nginx -t` and primary HTTPS/API returned 200 after a clean NPM restart
+without the alias; a representative public `git ls-remote` also succeeded.
+Navidrome and Syncthing Proxy Hosts still responded. No NPM SQLite records
+or credentials were changed. The
+secondary hostname `git.ov-ad3410.infomaniak.ch` did not resolve from Ikaros
+and had no generated NPM config file at the time of inspection.
+
+Prometheus' public TCP/2222 socket proxies to Atlas without changing admin
+SSH/22. The local socket presents the preserved Gitea ED25519 host key, but
+an external TCP/2222 connection from Ikaros timed out. During the test no SYN
+reached Prometheus `eth0`; its socket and firewalld port were active. Check
+upstream/provider filtering before declaring public SSH complete. Do not
+restart the stale source after public HTTPS has accepted target writes.
+
+The Prometheus export timer resumed with NPM-only paths. A recursive ZFS
+snapshot at `20261002T073032Z` and encrypted Borg archive
+`atlas-20261002T073044Z` captured the Atlas target after cutover; Borg exited
+successfully, cleaned its temporary snapshot, and the pool was healthy.
 
 1. Agree on an outage and record source/target versions, pool health, the
    latest backups, SSH host-key fingerprints, and both current NPM routes.
@@ -157,23 +168,22 @@ window before stopping the source or switching traffic.
    and the target service before switching NPM.
 4. Enable the public TCP/2222 socket proxy on Prometheus to Atlas over Aegis
    without changing administrative TCP/22. Switch Prometheus to the desired
-   NPM-only Compose stack and recreate NPM with the managed `gitea` host alias
-   so **both** existing Proxy Hosts reach Atlas without changing their database
-   records. The old Gitea data stays intact. Do not change public DNS.
+   NPM-only Compose stack and use the managed Gitea-only NPM runtime upstream
+   override. Do not use Compose `extra_hosts`: Nginx bypasses it for the
+   variable upstream. The old Gitea data stays intact. Do not change public DNS.
 5. Test HTTPS login, representative clone/push, LFS, and public SSH clone/push
    on port 2222 from outside the Atlas LAN. Record the last source write and
    first healthy target service times; do not claim RPO/RTO without measuring.
-6. After successful traffic validation, resume the Prometheus NPM-only backup
-   export timer and verify its next result. Verify the next Atlas snapshot/Borg
+6. Resume the Prometheus NPM-only backup export timer after the desired stack
+   is active and verify its next result. Verify the next Atlas snapshot/Borg
    run covers Gitea and test a restored target copy. Do not delete old source
    data.
 
 ## Rollback gate
 
-Before Atlas accepts writes, restore the old Compose definition (removing the
-NPM `gitea` host alias), disable the public 2222 proxy, and restart the
-unchanged source Gitea if target validation
-fails. **After Atlas accepts writes, do not blindly restart the source:** its
+Before Atlas accepts writes, restore the old Compose definition and remove the
+NPM override, disable the public 2222 proxy, and restart the unchanged source
+Gitea if target validation fails. **After Atlas accepts writes, do not blindly restart the source:** its
 SQLite database and repositories are stale. Quiesce Atlas, capture its new
 data, and decide a reverse migration or an extended outage explicitly.