Provision a cluster from scratch

This is how you stand up Project Loom, not how you recover a broken cluster. Use Restore only after a cluster already existed and you are restoring data.

The playbook is the same on every server. Penvoice, Ayde, and Talon are examples of apps on one node. The next box gets the same install; you change overlay hosts and ApplicationSet entries in Git, not the install steps.

Monitoring (Prometheus / Grafana) is not part of this install.

Every block says where to run it:

Where	Meaning
On the node	SSH into the VPS. Paste in that shell. Required — the step installs something onto that host or reads its local logs.
On the node or locally	Pure `kubectl`/`helm` against the API server. SSH into the VPS, or run from your laptop once you’ve set up local `kubectl` access — Access & consoles → kubectl. Same cluster either way.
In the browser	The URL or GUI named in that step.
In Git	Your laptop (or anywhere you edit and push `webb1es/loom`).
In DNS / OCI	Your DNS host or cloud console.

Tokens are always uppercase inside <ANGLE_BRACKETS> — yours to replace, and easy to spot by the format alone. Replace every <TOKEN> in your editor before you paste. A leftover < is a shell redirect. Expected results are under each block.

This company’s platform hosts (Argo CD, Infisical, Keycloak) and Let’s Encrypt account are already filled in below — see Reference → Live endpoints if you need the list. Only replace these:

Replace these

Token	What it is	Where you get it
`<NODE_PUBLIC_IP>`	Cloud public IPv4	In OCI — Compute → Instances → (your instance) → Public IP address
`<GITHUB_PAT>`	PAT, Contents: read	In the browser — GitHub → Settings → Developer settings → PAT
`<S3_ACCESS_KEY>` / `<S3_SECRET_KEY>`	Oracle Object Storage key pair, for CNPG backups	In OCI — profile icon (top right) → My Profile → Customer Secret Keys → Generate Secret Key. Both values show once — copy them immediately. Access key is clean hex; secret key contains `+/=` — don’t swap them, or backups fail with a signature error.
`<APP>` / `<ENV>`	Product slug + `staging`\|`prod`	you choose — must match ApplicationSet + overlay dir
`<INFISICAL_PROJECT_ID>`	Infisical project UUID	In the browser — Infisical → project → Project Settings
`<IDENTITY_CLIENT_ID>` / `<IDENTITY_CLIENT_SECRET>`	Machine Identity	In the browser — Infisical → project → Project Settings → Machine Identities → Create (secret shown once)

Create every Ingress A record before Helm/Argo request certificates. Point them at <NODE_PUBLIC_IP>.

In the browser (GitHub): Settings → Rename the repo to webb1es/loom if it is still gitops.

What you install

What	How	Pin
k3s + bundled Traefik	install script	keep Traefik; no host nginx
Helm	official script	—
cert-manager + ClusterIssuers	Helm + YAML	`v1.20.2`
Argo CD / Rollouts / Workflows / Events	Helm	`9.5.21` / `2.41.0` / `1.0.15` / `2.4.21`
`webb1es/loom`	Argo pulls GitHub	`bootstrap/root-app.yaml`
CNPG + Keycloak operators, Infisical	Argo (from Git)	pinned in `apps/`

Not installed here: kube-prometheus-stack, Docker, a second ingress.

Before you start

Node is a clean Linux (Ubuntu aarch64 or amd64).
You can SSH to the node.
In OCI: firewall 80/443 from the internet. You do not need 6443 open.
In DNS: A records for control.webbies.dev, vault.webbies.dev, uat.auth.webbies.dev, auth.webbies.dev, and each app host → <NODE_PUBLIC_IP>.
In the browser (GitHub): <GITHUB_PAT> on webb1es/loom, Contents: read.
In OCI: Object Storage keys (<S3_ACCESS_KEY> / <S3_SECRET_KEY>).

1. k3s

On the node.

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--write-kubeconfig-mode 644 --tls-san $(hostname)" sh -
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
mkdir -p ~/.kube && sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && sudo chown "$USER" ~/.kube/config
kubectl get nodes

Success: one node, Ready. The node can show NotReady for up to ~30 seconds right after install while kubelet and the CNI start — re-run kubectl get nodes before troubleshooting. If it’s still not Ready after that: journalctl -u k3s -e (on the node).

Optional, now: set up local kubectl access from your laptop over Tailscale — Access & consoles → kubectl. Everything from here on is pure kubectl/ helm, so once that’s done the remaining steps run on the node or locally, your choice. (Running locally still needs Helm installed there too — step 2.)

2. Helm

On the node or locally.

curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm version
helm repo add jetstack https://charts.jetstack.io
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

Success: helm version prints a version.

3. cert-manager + issuers

On the node or locally.

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --version v1.20.2 \
  --set crds.enabled=true

kubectl -n cert-manager wait --for=condition=Available deploy --all --timeout=180s

Success: wait returns; three cert-manager deploys Available. This can take up to the 180s timeout on a slow pull — the command blocks until then, so just let it sit.

On the node or locally.

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: projectloom@webbies.dev
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
      - http01:
          ingress:
            class: traefik
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: projectloom@webbies.dev
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            class: traefik
EOF
kubectl get clusterissuer

Success: letsencrypt-staging and letsencrypt-prod are Ready.

4. Argo CD, Rollouts, Workflows, Events

The Kubernetes namespace stays argocd (in-cluster name of the software).

On the node or locally.

kubectl create namespace argocd --dry-run=client -o yaml | kubectl apply -f -

helm install argo-cd argo/argo-cd \
  --namespace argocd \
  --version 9.5.21 \
  --set configs.params."server\.insecure"=true \
  --set server.ingress.enabled=true \
  --set server.ingress.ingressClassName=traefik \
  --set server.ingress.hostname='control.webbies.dev' \
  --set 'server.ingress.annotations.cert-manager\.io/cluster-issuer=letsencrypt-prod' \
  --set 'server.ingress.annotations.traefik\.ingress\.kubernetes\.io/router\.entrypoints=websecure' \
  --set server.ingress.tls=true

helm install argo-rollouts argo/argo-rollouts --namespace argocd --version 2.41.0
helm install argo-workflows argo/argo-workflows --namespace argocd --version 1.0.15
helm install argo-events argo/argo-events --namespace argocd --version 2.4.21

kubectl -n argocd get pods
kubectl -n argocd get ingress
kubectl -n argocd get certificate
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo

Success: Argo pods Running. Pods can take a minute or two to pull images and start — re-run the get pods check if they’re still Pending/ContainerCreating. Ingress host is control.webbies.dev. Certificate becomes READY=True after DNS + port 80 work — this can also take a minute or two, separately from the pods.

On the node or locally (if the cert is not Ready):

kubectl -n argocd describe certificate

GUI — Argo CD

In the browser.

Open https://control.webbies.dev.
Username admin, password from the command above.
Change that password (User info → Update password).
Confirm you can open Settings → Repositories.

5. Register this repo and apply the root app

The cluster pulls Git. Git never pushes to the node.

Register the repo

Replace <GITHUB_PAT> (table). Pick one of the two options below — you don’t need both.

Option A — kubectl, on the node or locally:

# <GITHUB_PAT> = PAT, Contents: read
kubectl -n argocd create secret generic loom-repo \
  --from-literal=type=git \
  --from-literal=url='https://github.com/webb1es/loom.git' \
  --from-literal=username=git \
  --from-literal=password='<GITHUB_PAT>'
kubectl -n argocd label secret loom-repo argocd.argoproj.io/secret-type=repository

Option B — in the browser (Argo CD): Settings → Repositories → Connect repo → HTTPS → https://github.com/webb1es/loom.git → username git → PAT as password → Connect.

Success (either option): in Argo CD, Settings → Repositories shows the repo as Successful / connection OK.

Apply the root app

bootstrap/root-app.yaml and apps/workloads.yaml both default to targetRevision: main. Before you apply: if you are still on a feature branch, change both files to that branch in Git first, or merge to main first. Applying an old main installs the old layout.

On the node or locally, from a clone of webb1es/loom (clone it wherever you’re running these steps from, if it is not there):

# Skip clone if you already have the tree
git clone 'https://github.com/webb1es/loom.git'
cd loom
kubectl apply -f bootstrap/root-app.yaml

Check the result:

kubectl -n argocd get applications
kubectl -n argocd get applicationsets

Success: root Synced. Give it up to a minute for Argo to pick up and cascade the Applications if the first check looks empty. Platform Applications appear (cnpg-operator, keycloak-operator, infisical*). ApplicationSet creates one Application per app×env. Many workloads stay Degraded until secrets and Infisical exist — that is expected.

6. Out-of-band Kubernetes Secrets

These are Secrets you create by hand, directly with kubectl — not the app’s own business env vars (those come later, via the Infisical GUI in step 7).

Namespaces

On the node or locally.

for ns in infisical keycloak-staging keycloak-prod; do
  kubectl create namespace "$ns" --dry-run=client -o yaml | kubectl apply -f -
done

for app in penvoice ayde talon; do
  for env in staging prod; do
    kubectl create namespace "${app}-${env}" --dry-run=client -o yaml | kubectl apply -f -
  done
done

Add or remove names from the app in ... list as you onboard more apps.

kubectl get namespaces

Success: every namespace above is listed with STATUS Active.

CNPG backup credentials

Every CNPG Cluster reads its Oracle Object Storage keys from a Secret you create — Argo can’t bring the Cluster up healthy without it. Every cluster shares the same Customer Secret Key pair, so replace <S3_ACCESS_KEY> and <S3_SECRET_KEY> (table) once below.

On the node or locally.

for app in penvoice ayde talon; do
  for env in staging prod; do
    kubectl -n "${app}-${env}" create secret generic "${app}-pg-backup-creds" \
      --dry-run=client -o yaml \
      --from-literal=ACCESS_KEY_ID='<S3_ACCESS_KEY>' \
      --from-literal=ACCESS_SECRET_KEY='<S3_SECRET_KEY>' \
      | kubectl apply -f -
  done
done

for env in staging prod; do
  kubectl -n "keycloak-${env}" create secret generic keycloak-pg-backup-creds \
    --dry-run=client -o yaml \
    --from-literal=ACCESS_KEY_ID='<S3_ACCESS_KEY>' \
    --from-literal=ACCESS_SECRET_KEY='<S3_SECRET_KEY>' \
    | kubectl apply -f -
done

kubectl -n infisical create secret generic infisical-pg-backup-creds \
  --dry-run=client -o yaml \
  --from-literal=ACCESS_KEY_ID='<S3_ACCESS_KEY>' \
  --from-literal=ACCESS_SECRET_KEY='<S3_SECRET_KEY>' \
  | kubectl apply -f -

Add or remove names from the app in ... list as you onboard more apps. infisical is a single instance, not split by env, so it only needs one Secret.

Once Argo has synced the Cluster CRs from step 5, check each namespace:

kubectl -n <NAMESPACE> get cluster

Success: each shows Cluster in healthy state. This can take a minute after both the Secret and the CR exist.

Infisical server bootstrap

Infisical needs its own bootstrap Secret before its pods can start. Wait until infisical-pg (previous sub-step) is healthy, then run this.

Check readiness — on the node or locally:

kubectl -n infisical get cluster infisical-pg

Wait for Cluster in healthy state before continuing.

Create the secret — on the node or locally:

kubectl create secret generic infisical-secrets --namespace infisical \
  --from-literal=AUTH_SECRET="$(openssl rand -base64 32)" \
  --from-literal=ENCRYPTION_KEY="$(openssl rand -hex 16)" \
  --from-literal=SITE_URL="https://vault.webbies.dev" \
  --from-literal=DB_CONNECTION_URI="postgresql://infisical:$(kubectl -n infisical get secret infisical-pg-app -o jsonpath='{.data.password}' | base64 -d)@infisical-pg-rw.infisical:5432/infisical" \
  --from-literal=REDIS_URL="redis://redis-master.infisical:6379"

Check the result:

kubectl -n infisical get pods
curl -sI https://vault.webbies.dev

Success: Infisical pods Running; the curl response starts with HTTP/2 200 (or another 2xx/3xx — anything but a connection error). Pods that were crash-looping on the missing Secret can take up to a minute to restart and stabilize — re-check if they’re still not Running yet.

In Git: apps/infisical.yaml must use the same host (vault.webbies.dev). Push if you changed it.

7. Infisical — GUI only for values

Do not put app business keys in Git. You type values in the GUI.

In the browser.

Open https://vault.webbies.dev.
Create the first account — that user is org admin.

Per app (repeat for every product)

In the browser (Infisical).

Organization → Projects → Create — name it <APP>.
Environments staging and prod.
Create each infra key once; fill both cells. Typical patterns (names differ per codebase):
- Keycloak issuer for this env (https://uat.auth.webbies.dev/realms/… or prod)
- REDIS_URL = redis://<APP>-redis:6379
- Object storage connection settings if the app talks to S3
Do not copy CNPG DB_URL / JDBC. Do not put the container port here.
Product env vars — here, when you decide. Not listed in this guide.

In Git: copy the Project ID from Infisical Project Settings (same page as step 1). It’s one Project ID shared by both environments, so paste it into both overlay files for this app:

workloads/<APP>/overlays/staging/infisical-patch.yaml
workloads/<APP>/overlays/prod/infisical-patch.yaml

Set spec.sources[0].projectId in each:

spec:
   sources:
      - projectId: "<INFISICAL_PROJECT_ID>"   # same value in both overlays — one Project, two environments
        environmentSlug: staging              # the prod overlay already has this set to prod — leave it alone
        secretPath: "/"

Commit both files and push to main.

Machine Identity + Secret (repeat per environment)

A Machine Identity has no environment field of its own — access is scoped entirely by the Project Role assigned to it. Do the steps below once for staging, once for prod.

In the browser (Infisical).

Access Control → Roles → Create Role. Fill Name/Slug/Description (skip if this env’s role already exists). Open it → Policies → Secrets: Permission Allow, check both Describe Secret and Read Value (Read Value alone fails with a 403 — Infisical requires Describe Secret too for a bulk fetch), + Add Condition → Environment Slug → Equal → staging (or prod) → Save.
Access Control → Machine Identities → Create new. Name it <APP>-staging / <APP>-prod, assign the role from step 1.
Open the identity — the Client ID is shown under Authentication → Universal Auth. Click + Add Client Secret to generate the Client Secret. Copy both (secret shown once).

On the node or locally.

kubectl -n <APP>-staging create secret generic infisical-<APP>-identity \
  --dry-run=client -o yaml \
  --from-literal=clientId='<STAGING_CLIENT_ID>' \
  --from-literal=clientSecret='<STAGING_CLIENT_SECRET>' \
  | kubectl apply -f -

kubectl -n <APP>-prod create secret generic infisical-<APP>-identity \
  --dry-run=client -o yaml \
  --from-literal=clientId='<PROD_CLIENT_ID>' \
  --from-literal=clientSecret='<PROD_CLIENT_SECRET>' \
  | kubectl apply -f -

Check the result:

kubectl -n <APP>-<ENV> describe infisicalstaticsecret <APP>-synced-secrets
kubectl -n <APP>-<ENV> get secret <APP>-api-kc
kubectl -n <APP>-<ENV> get pods

Success: the describe output’s Status/Conditions show a recent successful sync (a clear auth or lookup error there means the Role, Identity, or Secret values don’t line up); the <APP>-api-kc Secret exists; pods leave CreateContainerConfigError once required infra keys exist. The sync itself can take up to a minute — the operator’s refresh interval — so re-run describe before assuming something’s wrong.

8. Keycloak config-cli — GUI + one Secret, per Keycloak

The operator mints temp-admin for ~2 hours once per Keycloak, on a fresh master realm. Do everything below for both staging and prod.

On the node or locally.

kubectl -n keycloak-staging get secret keycloak-initial-admin -o jsonpath='{.data.username}' | base64 -d; echo
kubectl -n keycloak-staging get secret keycloak-initial-admin -o jsonpath='{.data.password}' | base64 -d; echo

kubectl -n keycloak-prod get secret keycloak-initial-admin -o jsonpath='{.data.username}' | base64 -d; echo
kubectl -n keycloak-prod get secret keycloak-initial-admin -o jsonpath='{.data.password}' | base64 -d; echo

If the operator ever names this secret differently, find it with kubectl -n keycloak-<ENV> get secret | grep -i admin.

GUI (repeat for each Keycloak host)

In the browser.

Open https://uat.auth.webbies.dev/admin or https://auth.webbies.dev/admin.
Log in with temp-admin.
Realm master → Clients → Create client → ID keycloak-config-cli.
Client authentication on. Uncheck Standard flow and Direct access grants. Check Service accounts roles. Save.
Service accounts roles → Assign role → filter realm roles → admin → Assign.
Credentials → copy Client secret.
Users → Add user — your permanent admin, password not temporary, realm role admin.

On the node or locally, once per environment with that environment’s own Client secret:

kubectl -n keycloak-staging create secret generic keycloak-config-cli-sa \
  --dry-run=client -o yaml \
  --from-literal=clientId=keycloak-config-cli \
  --from-literal=clientSecret='<STAGING_KEYCLOAK_CLIENT_SECRET>' \
  | kubectl apply -f -

kubectl -n keycloak-prod create secret generic keycloak-config-cli-sa \
  --dry-run=client -o yaml \
  --from-literal=clientId=keycloak-config-cli \
  --from-literal=clientSecret='<PROD_KEYCLOAK_CLIENT_SECRET>' \
  | kubectl apply -f -

Check the result:

kubectl -n keycloak-staging get jobs
kubectl -n keycloak-prod get jobs

Success: the keycloak-config-cli Sync Job shows COMPLETIONS 1/1; realms from workloads/keycloak/overlays/<ENV>/realms/ exist. The Job can take up to a minute to start and run — re-check if it’s still 0/1. Confidential client secrets go into the Infisical GUI (in the browser) for the app that uses them.

9. Done

On the node or locally.

kubectl get nodes
kubectl -n argocd get applications
kubectl get pods -A

Success: node Ready; Applications Synced or Healthy (or waiting on a secret you have not created); no CrashLoop on argocd, cert-manager, cnpg-system, infisical.

Onboard the next app with Onboard an app.

Next server

Same sections 1–8. Adding capacity to this deployment reuses the same control.webbies.dev / vault.webbies.dev — Argo CD and Infisical are singleton platform services, not per-node. A genuinely separate, independent deployment can’t reuse those hosts: edit them in Git (overlay hosts, ApplicationSet entries, and every hardcoded host in this guide) to that deployment’s own DNS first. Do not fork a second install style.

Provision a cluster from scratch

Replace these

What you install

Before you start

1. k3s

2. Helm

3. cert-manager + issuers

4. Argo CD, Rollouts, Workflows, Events

GUI — Argo CD

5. Register this repo and apply the root app

Register the repo

Apply the root app

6. Out-of-band Kubernetes Secrets

Namespaces

CNPG backup credentials

Infisical server bootstrap

7. Infisical — GUI only for values

First login

Per app (repeat for every product)

Machine Identity + Secret (repeat per environment)

8. Keycloak config-cli — GUI + one Secret, per Keycloak

GUI (repeat for each Keycloak host)

9. Done

Next server