Boring on purpose, reproducible by design.
Everything here was executed for real against a live Cloudflare account — including the mutual-binding bootstrap that bites every fresh deploy, and one production-shaped bug that only staging could reveal. The checked-in script is the record; this page is the narrative.
The staging deploy, walkable
Click a step or use the buttons. The order is load-bearing — the api and ingest workers bind each other, so a fresh account needs the bootstrap branch.
Local development
One command preflights, migrates, builds, starts, cross-wires, and health-gates all eight workers — then one Ctrl-C tears every process group down.
terminal# install + full verification
pnpm install
pnpm typecheck && pnpm test && pnpm build
# the whole platform, locally
pnpm dev
# same stack + idempotent demo org/user/vault/agent key
pnpm dev:seed
# force fresh Astro builds before the Workers start
pnpm dev -- --build
| port | supervised group | owns |
|---|---|---|
| 5170 | [stack] | router + api + mcp + ingest — test through the router |
| 5173 | [app] | Hono SSR console, primary in its own Wrangler process |
| 5174 | [blog] | Astro assets Worker, serving the last build |
| 5175 | [www] | Astro assets Worker, serving the last build |
| 5176 | [docs] | Astro/Starlight assets Worker, serving the last build |
scripts/dev.mjs first creates missing .dev.vars from their examples, sharing
one generated INGEST_INTERNAL_TOKEN between api and ingest. It reclaims only listeners
proven to belong to this repository — including a stale parent Wrangler that would otherwise respawn
workerd children — then applies the integration D1 migrations. Missing dist/ directories
trigger the appropriate Astro build; --build forces all three.
Wrangler's dev registry cross-wires the five process groups. The ready gate waits for
/healthz, /app/auth/sign-in, /, /blog/, and
/docs, including hashed static assets; the seeded form prints the demo email, password,
vault id, and hk_dev_… key. One Ctrl-C signals every detached process group.
Wrangler 4.110.0 serves assets from non-primary Workers in a multi-config process as 500. Every asset-bearing vertical therefore runs primary in its own process and joins the router through the dev registry. The old www/docs asset caveat no longer exists.
Local dev now runs fully offline: scripts/dev.mjs points api and ingest at
wrangler.dev.jsonc configs that carry no ai/vectorize bindings, so
wrangler opens no remote tunnel. Semantic search stays real via SEMANTIC_MODE=qmd: a
supervised sidecar (scripts/semantic-bridge.mjs, port 5178) mirrors vault pages to
.tmp/qmd-mirror/<vaultId> and runs the local qmd engine
(init → collection add → update → embed, then typed lex+vec queries, no rerank, 8s cap,
CPU-forced) — any bridge failure degrades to lexical with the loud engine badge.
BILLING_MODE=fake makes checkout self-activate; DEV_SEED=1 enables the seed
endpoint. The readiness script fails the build if dev flags ever appear in a deployable config.
Configuration & secrets
Secrets survive deploys; vars are per-invocation; the deploy script is the single record of which values staging runs.
| name | kind | where | meaning |
|---|---|---|---|
| BETTER_AUTH_SECRET | secret | api + app · same value | session signing — mismatched values = mysterious 401s |
| BETTER_AUTH_URL | var | api + app | router origin + /app/auth — cookies stay host-only on the router host; production: https://usehelios.co/app/auth |
| BETTER_AUTH_TRUSTED_ORIGINS | var | app + api | extra CSRF-trusted origins — localhost pair in dev; deploy-production.sh empties it (the configured origin is auto-trusted) |
| SSR_INTERNAL_TOKEN | secret | api + app · same value | first-party SSR exemption from the org rate limiter — dev value in .dev.vars; production via wrangler secret (never a config var) |
| SEMANTIC_MODE | var | api + ingest | real | fake | qmd | off — qmd is the offline local-dev bridge; readiness forbids fake/qmd in prod |
| BILLING_MODE | var | api + app | real (Polar) | fake — fake self-activates checkouts |
| POLAR_* · product ids | secret | api | required only when BILLING_MODE=real |
| INGEST_INTERNAL_TOKEN | secret | api + ingest · same value | authorizes only the forced agent:helios-ingest workflow principal |
| TURNSTILE_SITE_KEY / _SECRET_KEY | var / secret | app | human check on auth forms — widget renders only with the site key; siteverify skipped when the secret is absent (dev) |
| GOOGLE_ / GITHUB_CLIENT_ID+SECRET | secret | app | OAuth sign-in — provider buttons render only for configured pairs |
| REQUIRE_EMAIL_VERIFICATION | var | app | gate sign-in on verified email — false in dev, true in prod |
| EMAIL (send_email) + EMAIL_FROM | binding / var | app | verification, reset, welcome, and reminder mail — no-ops with a warning off-platform |
Workflows + cron ship with the kernel: sign-up triggers the welcome/onboarding-reminder
workflows (day 3/7/14, deterministic instance ids); a daily 0 0 * * * cron runs the
subscription-expiration workflow as the missed-webhook safety net. Named production envs must
re-declare every workflow, DO, and cron binding — they do not inherit.
Custom-domain routes for usehelios.co (apex + www→apex 301) are checked into the router
config, and scripts/deploy-production.sh is the launch path: it preflights the
per-worker Wrangler secrets (aborting with the missing list), builds the static verticals
with SITE_URL=https://usehelios.co, applies remote D1 migrations, and deploys in
the staging order with production vars — BILLING_MODE=real,
POLAR_SERVER=production, REQUIRE_EMAIL_VERIFICATION=true, emptied
trusted origins. Post-deploy: onboard Email Routing for usehelios.co, point the production
Polar webhook at the router origin, set the Turnstile site key. pnpm readiness
plus green dist checks (which pin the canonical domain) gate the launch.
Care & feeding
Each vault maintains itself; the operator watches queues and tails.
What runs on its own
- Nightly, per vault (DO alarm): full
validate(), export-tar snapshot to R2backups/, idempotency pruning, usage counters to the OrgDO. - On writes: a 60-second debounced alarm batches dirty chunks to the ingest queue; the consumer embeds and marks. Failures redeliver (×3) — then surface in freshness.
- On hosted ingest: the Workflow extracts text, HTML, or PDF, compiles content channels through Workers AI, and writes code-structured pages plus 2–3 inbox seedlings. Invalid model output gets one corrective retry, then fails visibly for workflow retry.
- Recovery layers: nightly tars → DO point-in-time (30 days) → user-facing export.
Runbook
| symptom | first move |
|---|---|
| Search shows “lexical-fallback” persistently | wrangler tail helios-ingest — the consumer may be failing silently on a model call; the badge is the only surface symptom by design |
| Fresh sign-ups can't create vaults | check ensurePersonalOrg path (see incident below); assert member row exists in D1 |
| 402s users dispute | billing page meters == enforcement constants; check subscriptions row + OrgDO counters |
| Vault deletion questioned | tombstone in OrgDO + org_audit row; residue test defines “gone” |
| Anything weird at the edge | reproduce on the vertical's own workers.dev URL to split router vs app |
Deploy order when contracts change: packages → api (DO classes live there) → mcp/ingest → app → router last. Never rename a DO class binding without a migration tag; VaultDO migrations must stay additive-safe because vaults wake at their own pace across versions.
Case study: the bug only staging could catch
A truly fresh production sign-up could never create a vault — and every local gate was green. Kept here because the lesson is the operations story.
Symptom: first live smoke on staging — fresh email, sign-up 200, create vault →
403 Organization membership is required.
Why local missed it: the app's sign-up form handler bootstrapped an org inline, so the golden-path test (which drives the form) passed. The staging smoke hit Better Auth's raw endpoint — the path any scripted or API-first customer takes — and nothing on that path created an org. The local gate had quietly narrowed to one of two doors.
Fix (shape matters): a lazy, race-safe ensurePersonalOrg at the single
principal→org choke point, with deterministic ids derived from the user id — concurrent first
requests upsert identical rows and converge without a transaction; interrupted sign-ups self-heal on
the next request. The duplicate app-side bootstrap was deleted: one convention, one place.
apps/api/src/routes/api.ts// ids are DETERMINISTIC functions of the user id — racers collide into
// the same rows; ON CONFLICT DO NOTHING makes the race a no-op
const orgId = `org_${userId}`;
A clean-database test now drives the real Better Auth sign-up (no seed helpers), asserts the membership exists, creates a vault with the fresh session, and counts exactly one org after a second resolution. Both sign-up doors were re-proven locally and on staging before redeploy.