Troubleshooting

The non-obvious failures Project Loom actually hit while standing it up and onboarding its first example app (Penvoice), each as Symptom → Cause → Fix. These are reproduced from the build notes; trust them over intuition.

Keycloak unreachable / serving Traefik’s default cert

Symptom. A host service (e.g. host-Docker Keycloak fronted by host nginx) is silently bypassed; the browser gets Traefik’s default self-signed cert instead of the expected backend.

Cause. k3s ServiceLB (klipper) hijacks host ports 80/443 via iptables DNAT — it intercepts public traffic before it reaches a host nginx socket. So nginx never sees the request and Traefik answers instead.

Fix. Don’t run host services on 80/443 alongside k3s. Make Traefik the sole edge and route everything through Kubernetes Ingress (this is why the host Keycloak container + host nginx were retired).

CNPG backups to Oracle fail with NotImplemented

Symptom. Backups/WAL uploads to Oracle Object Storage via CNPG fail; the error mentions NotImplemented for a checksum.

Cause. botocore ≥ 1.36 sends checksum trailers that Oracle’s S3-compat API returns NotImplemented for. Region is also required.

Fix. On the CNPG cluster, set:

AWS_REQUEST_CHECKSUM_CALCULATION=when_required
AWS_RESPONSE_CHECKSUM_VALIDATION=when_required
AWS_DEFAULT_REGION=af-johannesburg-1

CNPG backups fail with a signature / header error

Symptom. Backups fail SigV4 signing or header parsing, even though the keys look correct.

Cause. The Oracle Customer Secret Key halves were swapped. The access key is clean hex; the secret key contains +/=. Putting the secret (with a /) into the access-key slot inserts a / into the SigV4 Credential and breaks header parsing.

Fix. Access key = the clean hex string; secret key = the one with +/=. Don’t swap them.

Keycloak automation silently stops authenticating

Symptom. keycloak-config-cli’s Sync-hook Job fails with HTTP 400/403; realm YAML edits stop landing in live Keycloak with no obvious error surfaced elsewhere.

Cause. The Keycloak Operator’s bootstrapAdmin mints a temp-admin account once, on first ever start, and it’s genuinely temporary (2h default). Once a master realm exists, bootstrapAdmin is permanently ignored — even across restarts — so nothing ever refreshes it. Anything authenticating as temp-admin (including a first-draft config-cli-job.yaml) works for exactly one bootstrap window and then fails forever.

Fix. Give automation a non-expiring service account instead of a human-style temp-admin: a dedicated client in the master realm with the composite admin realm role, authenticating via client_credentials. See Example: Keycloak for the exact setup. If you’re currently locked out and need to get back in first, see Disaster recovery → Locked out of Keycloak — not the full-wipe section; recovering an expired admin doesn’t require losing any realm data.

ArgoCD shows a stale spec after a targeted fix

Symptom. You push a fix, kubectl -n argocd get application <app> still shows Synced, but the live resource’s actual spec (an env var, an image, an annotation) doesn’t reflect the new commit.

Cause. ArgoCD’s comparison cache doesn’t always refresh on its own poll interval alone, especially right after you’ve been manually deleting/patching the same resource out-of-band (e.g. clearing a stuck hook Job).

Fix. Force it:

kubectl -n argocd annotate application <app> argocd.argoproj.io/refresh=hard --overwrite

If a Sync-hook Job is involved, Kubernetes Jobs are immutable — ArgoCD can’t patch a running/completed Job’s spec.template in place. Delete the Job and let a fresh sync recreate it; if it’s stuck Terminating (an argocd.argoproj.io/hook-finalizer that doesn’t clear on its own after manual intervention), remove the finalizer directly:

kubectl -n <ns> delete job <name> --wait=false
kubectl -n <ns> patch job <name> --type json -p '[{"op":"remove","path":"/metadata/finalizers"}]'

Talon ImagePullBackOff

Symptom. kubectl -n talon get pods shows ImagePullBackOff; pod events say 403 Forbidden fetching an anonymous pull token for ghcr.io/webb1es/talon-core.

Cause. Unlike Penvoice’s GHCR package, Talon’s is private, and no imagePullSecret is wired up yet — the same choice documented in Onboard → Secrets, just not made for this app yet.

Fix. Not yet applied on this instance — either make the package public (matches Penvoice’s approach) or add a docker-registry imagePullSecret with a GHCR PAT. See Onboard → Secrets for both options.

InfisicalAuth/InfisicalStaticSecret rejected: namespace: Required value

Symptom. Argo’s sync fails with InfisicalAuth.secrets.infisical.com "<app>-auth" is invalid: [spec.infisicalConnectionRef.namespace: Required value, ...] (and the same for InfisicalStaticSecret’s infisicalAuthRef/targets[0]). The objects never get created — kubectl get infisicalstaticsecret in the app namespace returns NotFound, even though everything else in the same kustomization (Rollout, CNPG Cluster, Redis) deployed fine.

Cause. The infisical-secrets-operator CRDs require an explicit namespace: on every cross-reference field (infisicalConnectionRef, clientIdRef, clientSecretRef, infisicalAuthRef, targets[]). Kustomize’s namespace: transformer only sets metadata.namespace — it doesn’t know about arbitrary CRD spec fields, so a base manifest that omits these never passes validation, in any environment.

Fix. Add namespace: <app>-<env> to every ref, in the overlay’s infisical-patch.yaml (not base, since the value differs per environment). Kustomize’s merge patch replaces arrays wholesale rather than merging by key, so targets needs its complete original content repeated with namespace added, not just a fragment — see workloads/penvoice/overlays/*/infisical-patch.yaml for the working version.

InfisicalStaticSecret reconcile fails with Unauthorized access: status 403

Symptom. kubectl describe infisicalstaticsecret shows LastReconcileAuthMethod: OK (the Machine Identity logs in fine) but LastReconcileStatus fails with unable to fetch all secret sources: Unauthorized access: status 403.

Cause. The Project Role assigned to the identity only had Read Value checked. Infisical’s permission model requires Describe Secret granted alongside Read Value for a bulk fetch — Read Value alone isn’t enough, even though it sounds like the only permission that should matter.

Fix. On the role (Access Control → Roles → role → Policies → Secrets), check both Describe Secret and Read Value, Save. The operator picks it up on its next reconcile — Provision § Machine Identity already reflects this.

Certificate stuck READY=False

Symptom. A cert-manager Certificate never becomes ready:

kubectl get certificate -A
# READY=False, stuck on the HTTP-01 challenge

Cause. Let’s Encrypt’s HTTP-01 challenge can’t reach the host. The public DNS A record is not the node’s public IP, or port 80 is closed (host firewall / VCN).

Fix. Point the ingress hostname’s A record at the node’s public IP, and open 80/443 to the internet. Re-check:

kubectl describe certificate <name> -n <ns>
kubectl get challenges -A

CNPG barmanObjectStore deprecation

Symptom. Deprecation warnings on the in-tree backup config; backups break after a CNPG operator upgrade.

Cause. The in-tree barmanObjectStore is deprecated — it works through CNPG 1.29 and is removed in 1.30.

Fix. Stay on ≤ 1.29 for now; migrate to the Barman Cloud plugin before upgrading the operator past 1.29.

busybox nslookup shows NXDOMAIN (red herring)

Symptom. A debug nslookup from a busybox pod returns NXDOMAIN for a service, suggesting broken DNS.

Cause. busybox’s nslookup mishandles short names / search-domain expansion. The DNS is usually fine.

Fix. Query the FQDN instead (e.g. <service>.<namespace>.svc.cluster.local), or use a different tool. Don’t chase a DNS outage off the busybox result alone.

OIDC redirect breaks on Vercel previews

Symptom. The web app’s login (OIDC redirect to Keycloak) fails on a Vercel preview deployment — stuck behind an auth wall.

Cause. Vercel Deployment Protection puts a preview auth wall in front of the deployment, which intercepts the OIDC redirect flow.

Fix. Disable Deployment Protection for the environment so the OIDC redirect can complete.

Canary stalls on pod readiness

Symptom. A canary rollout sits at step 1 indefinitely; the new ReplicaSet shows 0 Ready pods. There is no Prometheus gate — this is readiness.

Cause. The new pods are failing readiness (/readyz) — usually a bad config repoint, a missing/wrong Secret, or an unreachable dependency.

Fix. This is the canary working as designed — stable pods keep serving. Diagnose the new pods, fix the config, push again:

kubectl argo rollouts get rollout penvoice-api -n penvoice-staging --watch
kubectl describe pod -n penvoice-staging -l app=penvoice-api
kubectl logs -n penvoice-staging -l app=penvoice-api --tail=50

See Operate rollouts for promote/abort/retry.