Restore after a live cluster

Standing up a new node is not this page. Use Provision from scratch for k3s, Helm, Argo, Infisical GUI, and Keycloak first-login. That playbook is the same on every server.

This page is after a cluster existed: Git still rebuilds objects; data and out-of-band credentials do not come back from a push. Example app names (Penvoice, Talon) are illustrations.

What’s in Git vs by hand

In Git (Argo CD)	By hand
Operators, Infisical server CRs, ApplicationSet overlays	k3s + Helm — Provision
Rollout, Ingress, Redis, CNPG Cluster shape	Out-of-band Secrets — Reference
Keycloak CR + realm YAML (keycloak-config-cli)	keycloak-config-cli-sa per Keycloak namespace (first time: Provision §8)
Infisical operator CRs (projectId, env slug)	Infisical values (re-type in the GUI if the Infisical DB was empty)

Postgres data is restored from Oracle Object Storage — Backup & restore. Do not restore a CNPG cluster in place.

After a wipe or empty node

Follow Provision end to end. Do not apply bootstrap/root-app.yaml from a main that still has the old single-namespace layout. Then restore each Postgres from object storage if you need the old data.

Locked out of Keycloak (no data loss)

Use this when keycloak-config-cli cannot authenticate or you cannot open the admin console. It does not wipe realm data. Repeat per env (keycloak-staging or keycloak-prod). Substitute that namespace below.

If a permanent admin from Provision §8 still works, stop here.

bootstrapAdmin is a no-op once a master realm exists. Recreating a temp-admin Secret and restarting does nothing. Use the bootstrap-admin CLI against the DB. Every kubectl below is on the node or locally — Access & consoles → kubectl.

1. Stop Keycloak (CLI needs exclusive DB access):

   kubectl -n keycloak-<env> patch keycloak keycloak --type merge -p '{"spec":{"instances":0}}'

2. Store a recovery password (never print it):

   kubectl -n keycloak-<env> create secret generic keycloak-recovery-admin-creds \
     --from-literal=password="$(openssl rand -base64 24)"

3. One-off Pod — same image and DB as the instance, command bootstrap-admin:

   apiVersion: v1
   kind: Pod
   metadata: { name: keycloak-recovery-admin, namespace: keycloak-<env> }
   spec:
     restartPolicy: Never
     containers:
       - name: kc-bootstrap
         image: quay.io/keycloak/keycloak:26.6.3   # match the live instance
         command: [/opt/keycloak/bin/kc.sh, bootstrap-admin, user, --username, recovery-admin, --password:env, RECOVERY_PASSWORD]
         env:
           - { name: KC_DB, value: postgres }
           - { name: KC_DB_URL_DATABASE, value: keycloak }
           - { name: KC_DB_URL_HOST, value: keycloak-pg-rw }
           - name: KC_DB_USERNAME
             valueFrom: { secretKeyRef: { name: keycloak-pg-app, key: username } }
           - name: KC_DB_PASSWORD
             valueFrom: { secretKeyRef: { name: keycloak-pg-app, key: password } }
           - name: RECOVERY_PASSWORD
             valueFrom: { secretKeyRef: { name: keycloak-recovery-admin-creds, key: password } }

   kubectl apply -f recovery-pod.yaml
   kubectl -n keycloak-<env> logs keycloak-recovery-admin | grep "Created temporary admin"
   kubectl -n keycloak-<env> delete pod keycloak-recovery-admin

4. Bring Keycloak back:

   kubectl -n keycloak-<env> patch keycloak keycloak --type merge -p '{"spec":{"instances":1}}'

5. Log in as recovery-admin (also ~2h). Immediately recreate your permanent admin and, if needed, keycloak-config-cli — same GUI steps as Provision §8. Delete keycloak-recovery-admin-creds when done.

Wiping one Keycloak (deliberate empty realm)

Only when you want to lose that env’s realm data and re-apply workloads/keycloak/overlays/<env>/realms/ from nothing.

On the node or locally.

1. Stop Keycloak:

   kubectl -n keycloak-<env> patch keycloak keycloak --type merge -p '{"spec":{"instances":0}}'

2. Delete the CNPG cluster:

   kubectl -n keycloak-<env> delete cluster keycloak-pg

3. Check for leftover PVCs, then delete any that remain:

   kubectl -n keycloak-<env> get pvc
   kubectl -n keycloak-<env> delete pvc -l cnpg.io/cluster=keycloak-pg

4. Delete leftover Secrets — the label below usually covers it, or delete the specific initial-admin name you see:

   kubectl -n keycloak-<env> delete secret -l app=keycloak

5. Bring Keycloak back up:

   kubectl -n keycloak-<env> patch keycloak keycloak --type merge -p '{"spec":{"instances":1}}'

Argo recreates an empty Cluster. bootstrapAdmin fires. Redo Provision §8, then let the Sync Job apply realms. Copy new confidential client secrets into the Infisical GUI for the apps that use them.