Troubleshooting

The non-obvious failures Project Loom actually hit while standing it up and onboarding its first example app (Penvoice), each as Symptom → Cause → Fix. These are reproduced from the build notes; trust them over intuition.

Keycloak unreachable / serving Traefik’s default cert

Symptom. A host service (e.g. host-Docker Keycloak fronted by host nginx) is silently bypassed; the browser gets Traefik’s default self-signed cert instead of the expected backend.

Cause. k3s ServiceLB (klipper) hijacks host ports 80/443 via iptables DNAT: it intercepts public traffic before it reaches a host nginx socket. So nginx never sees the request and Traefik answers instead.

Fix. Don’t run host services on 80/443 alongside k3s. Make Traefik the sole edge and route everything through Kubernetes Ingress (this is why the host Keycloak container + host nginx were retired).

CNPG backups to Oracle fail with NotImplemented

Symptom. Backups/WAL uploads to Oracle Object Storage via CNPG fail; the error mentions NotImplemented for a checksum.

Cause. botocore ≥ 1.36 sends checksum trailers that Oracle’s S3-compat API returns NotImplemented for. Region is also required.

Fix. On the CNPG cluster, set:

AWS_REQUEST_CHECKSUM_CALCULATION=when_required
AWS_RESPONSE_CHECKSUM_VALIDATION=when_required
AWS_DEFAULT_REGION=af-johannesburg-1

CNPG backups fail with a signature / header error

Symptom. Backups fail SigV4 signing or header parsing, even though the keys look correct.

Cause. The Oracle Customer Secret Key halves were swapped. The access key is clean hex; the secret key contains +/=. Putting the secret (with a /) into the access-key slot inserts a / into the SigV4 Credential and breaks header parsing.

Fix. Access key = the clean hex string; secret key = the one with +/=. Don’t swap them.

Keycloak automation silently stops authenticating

Symptom. keycloak-config-cli’s Sync-hook Job fails with HTTP 400/403; realm YAML edits stop landing in live Keycloak with no obvious error surfaced elsewhere.

Cause. The Keycloak Operator’s bootstrapAdmin mints a temp-admin account once, on first ever start, and it’s genuinely temporary (2h default). Once a master realm exists, bootstrapAdmin is permanently ignored. Even across restarts. So nothing ever refreshes it. Anything authenticating as temp-admin (including a first-draft config-cli-job.yaml) works for exactly one bootstrap window and then fails forever.

Fix. Give automation a non-expiring service account instead of a human-style temp-admin: a dedicated client in the master realm with the composite admin realm role, authenticating via client_credentials. See Example: Keycloak for the exact setup. If you’re currently locked out and need to get back in first, see Disaster recovery → Locked out of Keycloak . not the full-wipe section; recovering an expired admin doesn’t require losing any realm data.

ArgoCD shows a stale spec after a targeted fix

Symptom. You push a fix, kubectl -n argocd get application <app> still shows Synced, but the live resource’s actual spec (an env var, an image, an annotation) doesn’t reflect the new commit.

Cause. ArgoCD’s comparison cache doesn’t always refresh on its own poll interval alone, especially right after you’ve been manually deleting/patching the same resource out-of-band (e.g. clearing a stuck hook Job).

Fix. Force it:

kubectl -n argocd annotate application <app> argocd.argoproj.io/refresh=hard --overwrite

If a Sync-hook Job is involved, Kubernetes Jobs are immutable: ArgoCD can’t patch a running/completed Job’s spec.template in place. Delete the Job and let a fresh sync recreate it; if it’s stuck Terminating (an argocd.argoproj.io/hook-finalizer that doesn’t clear on its own after manual intervention), remove the finalizer directly:

kubectl -n <ns> delete job <name> --wait=false
kubectl -n <ns> patch job <name> --type json -p '[{"op":"remove","path":"/metadata/finalizers"}]'

ImagePullBackOff / 401 Unauthorized on a brand-new GHCR package

Symptom. kubectl -n <app>-<env> get pods shows ImagePullBackOff right after an app’s first-ever image push; pod events show a 401/anonymous-pull-token failure for ghcr.io/webb1es/<app>.

Cause. A newly created GHCR package inherits its source repo’s visibility. If the repo is private, the package is private too, and the cluster has no imagePullSecret. See Onboard → Secrets for both options (public package vs imagePullSecret). This platform defaults to public packages, so a new app’s first real build will hit this unless the package is made public right away.

Fix. Make the package public (matches Penvoice’s approach), or add a docker-registry imagePullSecret with a GHCR PAT. See Onboard → Secrets for both options.

CreateContainerConfigError: image has non-numeric user

Symptom. The image pulls fine but the pod never starts: CreateContainerConfigError, and kubectl describe pod shows container has runAsNonRoot and image has non-numeric user (nonroot), cannot verify user is non-root.

Cause. The pod sets runAsNonRoot: true and the image declares its user by name (USER nonroot:nonroot). The kubelet can only check a numeric user against root (0), so it refuses to start the container.

Fix. Declare the user by number in the Dockerfile, USER 65532:65532 (distroless’s nonroot, see Build & CI). To fix running pods without a rebuild, also set runAsUser: 65532 and runAsGroup: 65532 in the pod’s securityContext. Don’t drop runAsNonRoot to make the error go away: it is what stops a root image from ever running.

Prod ImagePullBackOff after the very first push: tag not found

Symptom. Right after onboarding an app, the prod overlay’s newTag names an image that doesn’t exist yet, and prod pods sit in ImagePullBackOff with manifest unknown.

Cause. main was pushed before staging had built that commit. promote.yml only re-tags. It never builds, so it wrote a tag for an image nobody had pushed. See Staging-to-prod promotion.

Fix. If main and staging point at the same commit, it resolves itself once the staging build finishes: the kubelet retries the pull. Otherwise push that commit to staging first. On a new app, always push staging first and only move main by fast-forward (git push origin staging:main or a rebase merge).

Traefik answers no available server for a few seconds on each deploy

Symptom. A single-pod app (for example SQLite on a volume, like Invoicer) returns no available server briefly during a rollout, then recovers.

Cause. The workload is a Deployment with strategy: Recreate: the old pod stops before the new one starts, because two pods can’t share the volume. There is no Ready endpoint in between.

Fix. Nothing, if it clears within the image pull plus startup time. If it persists, the new pod is failing readiness: kubectl describe pod -n <app>-<env> -l app=<app> and check /readyz.

InfisicalAuth/InfisicalStaticSecret rejected: namespace: Required value

Symptom. Argo’s sync fails with InfisicalAuth.secrets.infisical.com "<app>-auth" is invalid: [spec.infisicalConnectionRef.namespace: Required value, ...] (and the same for InfisicalStaticSecret’s infisicalAuthRef/targets[0]). The objects never get created. kubectl get infisicalstaticsecret in the app namespace returns NotFound, even though everything else in the same kustomization (Rollout, CNPG Cluster, Redis) deployed fine.

Cause. The infisical-secrets-operator CRDs require an explicit namespace: on every cross-reference field (infisicalConnectionRef, clientIdRef, clientSecretRef, infisicalAuthRef, targets[]). Kustomize’s namespace: transformer only sets metadata.namespace. It doesn’t know about arbitrary CRD spec fields, so a base manifest that omits these never passes validation, in any environment.

Fix. Add namespace: <app>-<env> to every ref, in the overlay’s infisical-patch.yaml (not base, since the value differs per environment). Kustomize’s merge patch replaces arrays wholesale rather than merging by key, so targets needs its complete original content repeated with namespace added, not just a fragment. See workloads/penvoice/overlays/*/infisical-patch.yaml for the working version.

InfisicalStaticSecret reconcile fails with Unauthorized access: status 403

Symptom. kubectl describe infisicalstaticsecret shows LastReconcileAuthMethod: OK (the Machine Identity logs in fine) but LastReconcileStatus fails with unable to fetch all secret sources: Unauthorized access: status 403.

Cause. The Project Role assigned to the identity only had Read Value checked. Infisical’s permission model requires Describe Secret granted alongside Read Value for a bulk fetch. Read Value alone isn’t enough, even though it sounds like the only permission that should matter.

Fix. On the role (Access Control → Roles → role → Policies → Secrets), check both Describe Secret and Read Value, Save. The operator picks it up on its next reconcile. Provision § Machine Identity already reflects this.

InfisicalStaticSecret reconciles “successfully” but the Secret has no data

Symptom. kubectl describe infisicalstaticsecret shows LastReconcileStatus: OK, LastReconcileAuthMethod: OK, and a recent LastSuccessfulReconcileAt. Everything looks healthy. But kubectl get secret <app>-api-kc -o json | jq '.data' returns null. The app crash-loops on an unresolved placeholder for a variable you can see sitting in Infisical with a value.

Cause. The Machine Identity’s Project Role is scoped to the wrong environment (a typo in the environment slug it’s restricted to, or reusing one identity’s staging-scoped role for prod). Infisical treats “no secrets visible in the scoped environment” as a normal, empty result. Not a permission error. So the reconcile genuinely succeeds, it just has nothing to return. This is easy to hit when each environment gets its own Machine Identity (the recommended setup, see Provision § Machine Identity): projectId, environmentSlug, and secretPath on the InfisicalStaticSecret can all be correct and this still happens.

Fix. In Infisical, Access Control → Identities → find the Machine Identity behind the app-env’s infisical-<app>-identity Secret → check its Project Role’s environment scope actually includes the environment you’re deploying to. Correct the scope (or the slug, if it was a typo) and save. The operator picks it up on its next reconcile, no restart needed on the Infisical side. If the pod itself is stably Running rather than crash-looping when this happens, it’ll need a manual restart to pick up the newly-populated Secret. See Operate rollouts → Force a restart.

Certificate stuck READY=False

Symptom. A cert-manager Certificate never becomes ready:

kubectl get certificate -A
# READY=False, stuck on the HTTP-01 challenge

Cause. Let’s Encrypt’s HTTP-01 challenge can’t reach the host. The public DNS A record is not the node’s public IP, or port 80 is closed (host firewall / VCN).

Fix. Point the ingress hostname’s A record at the node’s public IP, and open 80/443 to the internet. Re-check:

kubectl describe certificate <name> -n <ns>
kubectl get challenges -A

CNPG barmanObjectStore deprecation

Symptom. Deprecation warnings on the in-tree backup config; backups break after a CNPG operator upgrade.

Cause. The in-tree barmanObjectStore is deprecated: it works through CNPG 1.29 and is removed in 1.30.

Fix. Stay on ≤ 1.29 for now; migrate to the Barman Cloud plugin before upgrading the operator past 1.29.

busybox nslookup shows NXDOMAIN (red herring)

Symptom. A debug nslookup from a busybox pod returns NXDOMAIN for a service, suggesting broken DNS.

Cause. busybox’s nslookup mishandles short names / search-domain expansion. The DNS is usually fine.

Fix. Query the FQDN instead (e.g. <service>.<namespace>.svc.cluster.local), or use a different tool. Don’t chase a DNS outage off the busybox result alone.

OIDC redirect breaks on Vercel previews

Symptom. The web app’s login (OIDC redirect to Keycloak) fails on a Vercel preview deployment. Stuck behind an auth wall.

Cause. Vercel Deployment Protection puts a preview auth wall in front of the deployment, which intercepts the OIDC redirect flow.

Fix. Disable Deployment Protection for the environment so the OIDC redirect can complete.

Canary stalls on pod readiness

Symptom. A canary rollout sits at step 1 indefinitely; the new ReplicaSet shows 0 Ready pods. There is no Prometheus gate. This is readiness.

Cause. The new pods are failing readiness (/readyz). Usually a bad config repoint, a missing/wrong Secret, or an unreachable dependency.

Fix. This is the canary working as designed. Stable pods keep serving. Diagnose the new pods, fix the config, push again:

kubectl argo rollouts get rollout penvoice-api -n penvoice-staging --watch
kubectl describe pod -n penvoice-staging -l app=penvoice-api
kubectl logs -n penvoice-staging -l app=penvoice-api --tail=50

See Operate rollouts for promote/abort/retry.

OCI email: 535 authentication failed

Symptom. The app logs a 535 when signing in to smtp.email.af-johannesburg-1.oci.oraclecloud.com (Invoicer: smtp send: smtp auth LOGIN failed: 535 …), and no mail is sent.

Cause. OCI rejected the login: the SMTP username or password is wrong (the username is long and easy to truncate, and a pasted line break counts), the user isn’t in its email-<app> group, the email-apps-send policy line doesn’t match the group or compartment, or the credentials and policy are newer than about 5 minutes.

Fix. Run the login part of the Cloud Shell test. If it fails, fix OCI: re-copy or regenerate the credentials, check group membership and the policy line. If it passes, the pod has different values: see the next entry.

OCI email: settings fixed but the app still fails

Symptom. The Cloud Shell test sends, but the app keeps logging the old error.

Cause. The pod reads its environment only at start. The Infisical operator refreshes the app’s Secret every 60s, and nothing restarts the pod when it changes. A value saved with a trailing space or line break also differs from what you typed into Cloud Shell.

Fix. Re-save the value in vault.webbies.dev with nothing after the last character. Wait a minute (or restart the infisical-secrets-operator Deployment in Argo CD to sync at once), then restart the app pod (Force a restart). Check the pod’s image tag in Argo CD too: a deploy that hasn’t rolled yet still runs the old code.

OCI email: 535 Envelope From address not authorized

Symptom. Login succeeds, then sending fails with 535 Authorization failed: Envelope From address <x@mail.visarend.systems> not authorized (an app may log only server replied 535).

Cause. No approved sender matches that address in a compartment the user’s group may use. Usually the approved sender was created as the notices alias (oci-<app>@visarend.systems) instead of <app>@mail.visarend.systems, or in the wrong compartment, or the app’s MAIL_FROM differs from it.

Fix. Email Delivery → Approved Senders, compartment email → email-<app>: create <app>@mail.visarend.systems there and delete any wrong one. Set MAIL_FROM to exactly that. No restart is needed for an OCI-side fix. Concepts § Email.

OCI email: DKIM stuck or failing

Symptom. The email domain’s DKIM stays Creating, or received mail shows DKIM: FAIL.

Cause. The DKIM CNAME’s name or target is mistyped, or (on Cloudflare) it’s proxied (orange cloud). Hosts that take relative names want <selector>._domainkey.mail, not the full host.

Fix. On Cloudflare, set the record to DNS only. Compare it character by character with OCI’s DKIM record. SPF: FAIL instead means the TXT record isn’t on mail or lacks the include: values.

OCI email: message too large

Symptom. Small emails send, but one with a big attachment fails.

Cause. OCI’s max-message-size is about 2 MB after base64 encoding, so attachments over about 1.4 MB exceed it.

Fix. Request a max-message-size increase (Limits, Quotas and Usage → Request a service limit increase), or send a link instead of the attachment. OCI bills each started 2 MB per recipient as one email.