Staging-to-prod promotion

Two branches, two triggers, one artifact: staging builds and deploys itself automatically; main is what’s live in prod, and reaching it never rebuilds. It just re-tags the image that already soaked. This replaces the manual Promote dispatch in Deploy an app change once an app’s CI and merge settings are updated to match it. Penvoice and Talon both run on it end to end; the design is app-agnostic and meant for every app in workloads/.

flowchart TD
    A([push → staging]) --> B[CI: build & push sha-XXXXXXX]
B --> C[overlays/staging newTag]
C --> D[Argo syncs staging]
D --> E([test.app.example, soaking])
E -. fast - forward merge only .-> F([main tip == staging tip])
F --> G[CI: push → main]
G --> H[sha-XXXXXXX already in GHCR, skip build]
H --> I[overlays/prod newTag]
I --> J[Argo syncs prod]
J --> K([canary 25→50→75 → app.example, live])

Why fast-forward, specifically

Git’s fast-forward merge doesn’t create a new commit. It just moves main’s pointer up to staging’s tip. That means the commit SHA on main after the merge is identical to the one already built and tagged sha-<sha> on staging. Because CI only knows an image doesn’t exist yet by its SHA, a push: main trigger can look up that tag, find it already in GHCR, and skip the rebuild entirely. No drift between what tested on staging and what runs in prod.

A regular merge commit or a squash merge stamps a new SHA on main, even when the file tree is identical. CI has no way to know that tree already has an image, so it rebuilds. Same source, different artifact, and the soak on staging no longer says anything about what’s shipping.

The two triggers

Branch push	CI does	Writes
staging	build linux/arm64, push sha-<sha> to GHCR	workloads/<app>/overlays/staging/kustomization.yaml → images.newTag
main (post fast-forward)	look up sha-<sha>, already in GHCR, skip build	workloads/<app>/overlays/prod/kustomization.yaml → images.newTag

Both writes push straight to loom’s main. No PR, since loom stays unprotected on purpose (Argo’s the thing that has to trust it, and Argo only watches Git). Both Argo Applications already have syncPolicy.automated, so neither overlay needs a manifest change to pick this up. Argo notices on its own poll timer, or immediately with Access & consoles → Skip the poll wait. That write authenticates as a small GitHub App (installed once on loom, Contents: Read and write) rather than a personal token or SSH deploy key. One App, reused as-is across every app repo’s CI, minting a short-lived installation token per run instead of a long-lived credential to rotate.

Enforce the merge policy by restricting the PR merge button itself: app repo → Settings → General → Pull Requests → uncheck Allow merge commits and Allow squash merging, leave only Allow rebase merging. When main hasn’t diverged, GitHub’s rebase-merge performs a genuine fast-forward and keeps the SHA identical. No script required. This has to be a repo setting rather than branch protection: branch protection and rulesets don’t enforce on a private repo on a personal GitHub account: both require a GitHub Team organization. Two real gaps follow from that: nothing stops a raw git push origin staging:main bypass from the command line, and it’s unverified whether “Rebase and merge” silently stamps a new SHA when main has diverged but the rebase hits no textual conflict. See Still open.

If a hotfix bypasses staging

Only possible if something merges into main other than a fast-forwarded staging, e.g. a hotfix branch cut from main directly. The hotfix merge itself usually succeeds (it’s its own fast-forward). The failure shows up on the next staging → main merge attempt, because main’s tip is no longer an ancestor of staging’s. Nothing in CI catches this. It’s not gated by any workflow, only by whichever way you bring staging into main yourself:

• A raw git push origin staging:main from the command line (or an IDE’s push dialog) is rejected outright by git itself, no branch protection required:

  ! [rejected]        staging -> main (non-fast-forward)
  error: failed to push some refs to 'main'

  You’d see this directly in your terminal (or IDE). Never in an Actions log, since nothing automated runs this push; promote.yml only reacts after a push to main already succeeded.
• Through a PR + “Rebase and merge” (the normal path), expect a real content conflict most of the time rather than a clean rejection. The hotfix and whatever’s new on staging usually touch overlapping lines. GitHub refuses to merge and asks you to resolve it, same as any conflicting PR.

Resolve it by bringing the hotfix into staging’s history, not by forcing main:

git checkout staging
git merge main   # pulls the hotfix commit into staging. Expect a real conflict here, not a clean merge
# resolve the conflict markers in the affected file(s), then:
git add <file>
git commit
git push

Retry the staging → main promotion (PR + rebase-merge, or a raw push if you’re just testing) once that’s pushed. Don’t rebase staging onto main instead. It rewrites staging’s commit SHAs, which orphans any image already built and tagged against the old ones.

This is why no hotfix bypass is a hard rule for this design, not a preference: every commit reaching main has to have gone through staging first, or the invariant that makes promotion free breaks.

Promoting more than one staging commit

Fine, and the normal case. Fast-forward doesn’t care how many commits sit between main’s old tip and staging’s current one. Promoting always ships exactly one image regardless: whatever’s tagged on staging’s tip at merge time, representing the cumulative diff of however many commits got it there.

Adopting per app

For each app in workloads/:

1. On the app repo (not loom): Settings → General → Pull Requests → only Allow rebase merging enabled. Convention-only, not a technical block. See above for why.
2. Add LOOM_APP_ID and LOOM_APP_PRIVATE_KEY as repo secrets (Settings → Secrets and variables → Actions), same values as every other app repo. One GitHub App, installed once on loom, reused everywhere.
3. Split the app’s existing push: main-only build workflow (see Build & CI) into the two triggers in the table above. Watch for the :latest-tag trap: if the workflow tags latest conditioned on is_default_branch, moving the build trigger to staging freezes that tag forever, since staging is never the default branch. Drop the tag from the workflow rather than ship a silently-stale one.
4. loom-side: nothing. Both overlays already auto-sync.
5. workflow_dispatch Promote (see Deploy an app change) becomes redundant for that app once this ships. Drop it or keep it as a manual override, per app.

Still open

• Whether GitHub’s “Rebase and merge” silently stamps a new SHA when main has diverged from staging but the rebase hits no textual conflict, rather than refusing to merge. A hotfix reaching main and a subsequent staging commit will almost always touch overlapping lines, forcing a real conflict and manual resolution before the button is ever exercised. So the clean-rebase case is a real but low-probability gap, not a proven-safe one.