docs / 03 · operations
chapter 03 · operate & deploy

Boring on purpose, reproducible by design.

Everything here was executed for real against a live Cloudflare account — including the mutual-binding bootstrap that bites every fresh deploy, and one production-shaped bug that only staging could reveal. The checked-in script is the record; this page is the narrative.

interactive · the real sequence, step by step

The staging deploy, walkable

Click a step or use the buttons. The order is load-bearing — the api and ingest workers bind each other, so a fresh account needs the bootstrap branch.

    
            
    01

    Local development

    One command preflights, migrates, builds, starts, cross-wires, and health-gates all eight workers — then one Ctrl-C tears every process group down.

    terminal# install + full verification
    pnpm install
    pnpm typecheck && pnpm test && pnpm build
    
    # the whole platform, locally
    pnpm dev
    
    # same stack + idempotent demo org/user/vault/agent key
    pnpm dev:seed
    
    # force fresh Astro builds before the Workers start
    pnpm dev -- --build
    portsupervised groupowns
    5170[stack]router + api + mcp + ingest — test through the router
    5173[app]Hono SSR console, primary in its own Wrangler process
    5174[blog]Astro assets Worker, serving the last build
    5175[www]Astro assets Worker, serving the last build
    5176[docs]Astro/Starlight assets Worker, serving the last build

    scripts/dev.mjs first creates missing .dev.vars from their examples, sharing one generated INGEST_INTERNAL_TOKEN between api and ingest. It reclaims only listeners proven to belong to this repository — including a stale parent Wrangler that would otherwise respawn workerd children — then applies the integration D1 migrations. Missing dist/ directories trigger the appropriate Astro build; --build forces all three.

    Wrangler's dev registry cross-wires the five process groups. The ready gate waits for /healthz, /app/auth/sign-in, /, /blog/, and /docs, including hashed static assets; the seeded form prints the demo email, password, vault id, and hk_dev_… key. One Ctrl-C signals every detached process group.

    why five processes

    Wrangler 4.110.0 serves assets from non-primary Workers in a multi-config process as 500. Every asset-bearing vertical therefore runs primary in its own process and joins the router through the dev registry. The old www/docs asset caveat no longer exists.

    Local dev now runs fully offline: scripts/dev.mjs points api and ingest at wrangler.dev.jsonc configs that carry no ai/vectorize bindings, so wrangler opens no remote tunnel. Semantic search stays real via SEMANTIC_MODE=qmd: a supervised sidecar (scripts/semantic-bridge.mjs, port 5178) mirrors vault pages to .tmp/qmd-mirror/<vaultId> and runs the local qmd engine (init → collection add → update → embed, then typed lex+vec queries, no rerank, 8s cap, CPU-forced) — any bridge failure degrades to lexical with the loud engine badge. BILLING_MODE=fake makes checkout self-activate; DEV_SEED=1 enables the seed endpoint. The readiness script fails the build if dev flags ever appear in a deployable config.

    02

    Configuration & secrets

    Secrets survive deploys; vars are per-invocation; the deploy script is the single record of which values staging runs.

    namekindwheremeaning
    BETTER_AUTH_SECRETsecretapi + app · same valuesession signing — mismatched values = mysterious 401s
    BETTER_AUTH_URLvarapi + approuter origin + /app/auth — cookies stay host-only on the router host; production: https://usehelios.co/app/auth
    BETTER_AUTH_TRUSTED_ORIGINSvarapp + apiextra CSRF-trusted origins — localhost pair in dev; deploy-production.sh empties it (the configured origin is auto-trusted)
    SSR_INTERNAL_TOKENsecretapi + app · same valuefirst-party SSR exemption from the org rate limiter — dev value in .dev.vars; production via wrangler secret (never a config var)
    SEMANTIC_MODEvarapi + ingestreal | fake | qmd | off — qmd is the offline local-dev bridge; readiness forbids fake/qmd in prod
    BILLING_MODEvarapi + appreal (Polar) | fake — fake self-activates checkouts
    POLAR_* · product idssecretapirequired only when BILLING_MODE=real
    INGEST_INTERNAL_TOKENsecretapi + ingest · same valueauthorizes only the forced agent:helios-ingest workflow principal
    TURNSTILE_SITE_KEY / _SECRET_KEYvar / secretapphuman check on auth forms — widget renders only with the site key; siteverify skipped when the secret is absent (dev)
    GOOGLE_ / GITHUB_CLIENT_ID+SECRETsecretappOAuth sign-in — provider buttons render only for configured pairs
    REQUIRE_EMAIL_VERIFICATIONvarappgate sign-in on verified email — false in dev, true in prod
    EMAIL (send_email) + EMAIL_FROMbinding / varappverification, reset, welcome, and reminder mail — no-ops with a warning off-platform
    scheduled machinery

    Workflows + cron ship with the kernel: sign-up triggers the welcome/onboarding-reminder workflows (day 3/7/14, deterministic instance ids); a daily 0 0 * * * cron runs the subscription-expiration workflow as the missed-webhook safety net. Named production envs must re-declare every workflow, DO, and cron binding — they do not inherit.

    production launch checklist

    Custom-domain routes for usehelios.co (apex + www→apex 301) are checked into the router config, and scripts/deploy-production.sh is the launch path: it preflights the per-worker Wrangler secrets (aborting with the missing list), builds the static verticals with SITE_URL=https://usehelios.co, applies remote D1 migrations, and deploys in the staging order with production vars — BILLING_MODE=real, POLAR_SERVER=production, REQUIRE_EMAIL_VERIFICATION=true, emptied trusted origins. Post-deploy: onboard Email Routing for usehelios.co, point the production Polar webhook at the router origin, set the Turnstile site key. pnpm readiness plus green dist checks (which pin the canonical domain) gate the launch.

    03

    Care & feeding

    Each vault maintains itself; the operator watches queues and tails.

    What runs on its own

    • Nightly, per vault (DO alarm): full validate(), export-tar snapshot to R2 backups/, idempotency pruning, usage counters to the OrgDO.
    • On writes: a 60-second debounced alarm batches dirty chunks to the ingest queue; the consumer embeds and marks. Failures redeliver (×3) — then surface in freshness.
    • On hosted ingest: the Workflow extracts text, HTML, or PDF, compiles content channels through Workers AI, and writes code-structured pages plus 2–3 inbox seedlings. Invalid model output gets one corrective retry, then fails visibly for workflow retry.
    • Recovery layers: nightly tars → DO point-in-time (30 days) → user-facing export.

    Runbook

    symptomfirst move
    Search shows “lexical-fallback” persistentlywrangler tail helios-ingest — the consumer may be failing silently on a model call; the badge is the only surface symptom by design
    Fresh sign-ups can't create vaultscheck ensurePersonalOrg path (see incident below); assert member row exists in D1
    402s users disputebilling page meters == enforcement constants; check subscriptions row + OrgDO counters
    Vault deletion questionedtombstone in OrgDO + org_audit row; residue test defines “gone”
    Anything weird at the edgereproduce on the vertical's own workers.dev URL to split router vs app

    Deploy order when contracts change: packages → api (DO classes live there) → mcp/ingest → app → router last. Never rename a DO class binding without a migration tag; VaultDO migrations must stay additive-safe because vaults wake at their own pace across versions.

    04

    Case study: the bug only staging could catch

    A truly fresh production sign-up could never create a vault — and every local gate was green. Kept here because the lesson is the operations story.

    Symptom: first live smoke on staging — fresh email, sign-up 200, create vault → 403 Organization membership is required.

    Why local missed it: the app's sign-up form handler bootstrapped an org inline, so the golden-path test (which drives the form) passed. The staging smoke hit Better Auth's raw endpoint — the path any scripted or API-first customer takes — and nothing on that path created an org. The local gate had quietly narrowed to one of two doors.

    Fix (shape matters): a lazy, race-safe ensurePersonalOrg at the single principal→org choke point, with deterministic ids derived from the user id — concurrent first requests upsert identical rows and converge without a transaction; interrupted sign-ups self-heal on the next request. The duplicate app-side bootstrap was deleted: one convention, one place.

    apps/api/src/routes/api.ts// ids are DETERMINISTIC functions of the user id — racers collide into
    // the same rows; ON CONFLICT DO NOTHING makes the race a no-op
    const orgId = `org_${userId}`;
    the regression that guards it

    A clean-database test now drives the real Better Auth sign-up (no seed helpers), asserts the membership exists, creates a vault with the fresh session, and counts exactly one org after a second resolution. Both sign-up doors were re-proven locally and on staging before redeploy.