Run it in production

Last reviewed against the codebase: 2026-07-25.

This page is the operator’s runbook for the hosted environment. It covers why the hosted stack is built the way it is, the exact commands to bring it up, deploy a change, verify it, and put it back to rest - plus the real failure modes that bite a new maintainer and how to recover from each.

The infrastructure is Terraform (IaC) and follows a hibernate / wake model so it costs roughly nothing when idle. Only operators with AWS SSO access to the dedicated Fire Path account run any of this.

Note

For the tier-by-tier resource inventory, security groups, and network topology, see Overview & architecture and the generated Configuration reference. This page does not duplicate them - it tells you how to drive the stack.

Why hibernate / wake at all

This is a solo-run, life-safety product that must be enterprise/Government-grade when live but must not burn money while it is dormant between validation sessions. A permanently-on stack (RDS Multi-AZ + NAT + ALB + ECS) costs on the order of hundreds of AUD/month; a dormant one costs a few dollars. So the design splits every resource into two classes:

• Baseline (always-on, ~AUD $3/mo) - VPC, subnets, security groups, the S3 gateway endpoint, the Route 53 hosted zone, ACM certs, the KMS CMK, the long-lived secrets (JWT signing key, NASA FIRMS key), IAM roles + GitHub OIDC, and the Terraform state bucket. These are cheap, slow to recreate, or hold identity/keys, so they are never torn down.
• Ephemeral (expensive, torn down at rest) - NAT Gateway + Elastic IP, RDS Multi-AZ Postgres, ElastiCache Redis, the ALB, and the ECS services. These are created on wake and destroyed on hibernate.

The whole model is driven by three Terraform variables - enable_data_tier, enable_compute, and dev_mode - flipped by two scripts in infra/scripts/.

Data is not persisted across hibernate

dev_mode=true means RDS is destroyed without a final snapshot on hibernate. Every wake starts with an empty database, which is why the self-bootstrap step (below) exists. This is deliberate for the validation-cycle posture - do not treat the hosted DB as a system of record between wakes.

The three stages

Stage	What’s running	Idle cost	Brought up by
COLD (rest)	Baseline only	~AUD $3/mo	hibernate.sh
WARM (data)	+ RDS + Redis + NAT	~AUD $235/mo	wakeup.sh data
HOT (full)	+ ALB + ECS api + wind-worker	~AUD $290–320/mo	wakeup.sh full

terraform output stage prints the current stage at any time (for example COLD (~AUD $3/mo idle)), which is the quickest way to answer “is anything up?”.

Warn before waking

A WARM or HOT wake incurs real, hourly AWS cost the moment it starts provisioning. Both scripts print the exact resources they will create with their per-item cost and require an interactive y to proceed. Pause, confirm the cost posture, wake for the specific task, and hibernate the moment you are done.

Waking the stack

Both scripts hard-pin the correct AWS profile (firepath-admin) and clear any inherited region, then run a preflight identity check before doing any work. If the SSO token is missing or points at the wrong account, they fail fast with a copy-pasteable fix instead of spending a minute building a Docker image first.

# From the repo root, after: aws sso login --profile firepath-admin
bash infra/scripts/wakeup.sh data   # WARM: RDS + Redis only
bash infra/scripts/wakeup.sh full   # HOT:  + ALB + ECS api + wind-worker

Expected preflight output on success:

── Preflight: AWS identity check ─────────────────────────────────────
✅ Authenticated to firepath account <id> as firepath-admin.

What wakeup.sh full does, step by step

A single full command brings up the entire stack hands-off. The script narrates six numbered steps:

1. Build two images - the api-server (linux/arm64, cheaper Graviton ECS task) and the wind-worker (linux/amd64, built FROM the local WindNinja base image). The worker build fails loudly if the WindNinja base image is absent locally rather than deploying a broken worker (see VENDOR.md to rebuild/pull it).
2. Push both to ECR, reconciling the api-server repo in both AWS and Terraform state first (see the ECR gotcha below).
3. terraform apply - ALB + ECS api service + the wind-worker tier, with fire clustering enabled (enable_fire_clustering=true) for the cycle.
4. Force-deploy the ECS api service so it pulls the freshly-pushed :latest.
5. Verify DB self-bootstrap - poll /api/readyz (via the ALB directly, before DNS is updated) until it returns 200, confirming the container came up and migrated + seeded its own empty database.
6. Point DNS - every wake produces a new ALB hostname, so the script re-points api.firepath.software at it (automatically if a Cloudflare API token + zone id are present, otherwise printing the exact record to set).

WARM (data) mode skips steps 1, 2, 4, 5, and 6 - it only applies the data-tier Terraform and returns.

Why the app bootstraps its own database

Because dev_mode recreates RDS empty every wake, something must migrate + seed it. The chosen design is self-bootstrap on boot: the api task migrates and seeds its own database on startup when DB_BOOTSTRAP_ON_BOOT=true (the Terraform default, validated on the 2026-07-13 wake). There is no manual seed step and no SSM tunnel in the normal flow - step 5 just waits for /readyz to go green. The seed admin password persists in Secrets Manager (firepath-ai/prod/seed-admin), so a fresh wake is fully hands-off and still yields a real admin login.

bootstrap-prod-db.sh still exists as an emergency manual fallback only - it opens an SSM port-forward to RDS through a running task, pushes the Drizzle schema, and seeds. Run it by hand only if step 5 reported /readyz never became ready and the task logs show the self-bootstrap failed:

bash infra/scripts/bootstrap-prod-db.sh
# prereqs: session-manager-plugin, jq, pnpm
# set SEED_ADMIN_PASSWORD for a real admin login, else the admin is device-only

Deploying a change and confirming it is live

Re-running wakeup.sh full against an already-HOT stack rebuilds, pushes, and force-deploys the new image - that is the deploy path. ECS tasks take ~2–5 min to become healthy. Confirm the deploy with the two health signals:

curl https://api.firepath.software/api/healthz   # process is up
curl https://api.firepath.software/api/readyz    # DB reachable + self-bootstrap done

/healthz proves the container answers; /readyz returning healthy means the DB self-bootstrap completed and the service can serve real requests. If /readyz is 503, the task is still provisioning or migrating - wait and re-poll. See Verify it works for the full verification checklist (tests, typecheck, health probes).

For the routine “how do I change X” tasks - config/feature flags, an env var, a schema change, a new route - see Update common things.

Returning to rest

bash infra/scripts/hibernate.sh

hibernate.sh prints exactly what it will DESTROY (NAT + EIP, RDS, Redis, the data/redis secrets, ECS + ALB) and what it PRESERVES (the baseline listed above), requires a y, then destroys the ephemeral tier by applying enable_data_tier=false enable_compute=false. On success:

✅ Hibernated. Idle cost: ~AUD $3/mo.

Verify with terraform output stage - it should print the COLD line.

After hibernate, the api.firepath.software CNAME still points at the now-gone ALB, so API requests return DNS errors. This is expected: the SPA detects it and shows a “Service is paused” banner. No action is required unless you want to tidy the stale DNS record.

Gotchas and how to recover

These are the real failure modes that have bitten this stack, each already defended in the scripts - but worth knowing so you recognise them.

• Wrong AWS SSO directory leaking through. A shell whose AWS_PROFILE defaults to an unrelated profile at a different SSO start URL makes every aws call silently hit the wrong account (confusing “token does not exist” errors). Both scripts force AWS_PROFILE=firepath-admin and unset AWS_DEFAULT_REGION, and the preflight aborts with an actionable message if the resolved account is not the Fire Path one. If the browser opens the wrong SSO directory on aws sso login, open a fresh terminal.

• ECR “RepositoryAlreadyExists” aborting every wake. hibernate.sh force- deletes the api-server ECR repo, so after a clean hibernate the repo is absent from both AWS and Terraform state. The historic bug: a wake recreated the repo in AWS for the push but never imported it into state, so the next terraform apply tried to CREATE an existing repo and failed 400. The script now reconciles both independently and idempotently - ensures the repo exists in AWS, then attempts a terraform import and treats “already managed” as success

  • so this cannot recur.

• Postgres extensions must exist before push. drizzle-kit cannot CREATE EXTENSION itself, so postgis + citext must be enabled first or the push dies with “type citext does not exist”. The self-bootstrap and the manual fallback both run lib/db/migrations/0000_init_postgis.sql before pushing the schema.

• RDS master password breaks the seed URL. RDS-managed master passwords contain URL-unsafe characters. Node’s strict WHATWG new URL() (used by node-pg/drizzle in the seed) throws ERR_INVALID_URL on a raw one, while psql/libpq is lenient - which is why the schema push can succeed but the seed dies. The bootstrap script percent-encodes the user and password for the node-pg URL.

• Orphaned Elastic IP after hibernate. terraform destroy releases the NAT’s Elastic IP, but if the NAT’s ENI is already gone the release errors InvalidNetworkInterfaceID.NotFound, the apply exits non-zero, and the EIP is left allocated (~AUD $3.6/mo) with state out of sync. hibernate.sh sweeps any unassociated project-tagged EIP, releases it, and re-runs the destroy to reconcile - so a partial destroy self-heals.

• New ALB hostname every wake. Because the ALB is ephemeral, each wake mints a new DNS name and the api.firepath.software CNAME must be re-pointed or clients hit a dead host (“hotspots gone after a wake”). Set CLOUDFLARE_API_TOKEN (Zone:DNS:Edit on firepath.software) and CLOUDFLARE_ZONE_ID once - for example in a gitignored env file you source before waking - and step 6 updates DNS automatically (TTL 60s, DNS-only). Without them the script prints the exact record to set by hand.

• NASA FIRMS key. On a normal hibernate/wake cycle the JWT and FIRMS secrets persist, so nothing is needed. Only re-populate the FIRMS key if you deliberately destroyed it; the post-wake checklist prints the exact aws secretsmanager put-secret-value command.

Static sites - the docs + marketing (Cloudflare Pages)

The developer docs and the marketing site are static Astro builds deployed to Cloudflare Pages (projects firepath-docs and firepath-marketing). These are entirely separate from the AWS stack above - they have no hibernate/wake cost and can be deployed while the API stack is COLD.

Deploys go through a wrapper (scripts/deploy-cf-pages.mjs) so authentication is deterministic and no one is ever hand-held through an auth fix:

# Developer docs - from Fire-Path-AI/
pnpm --filter @workspace/docs-site run deploy:pages

# Marketing - runs build + redaction, then deploys
cd Fire-Path-AI/artifacts/marketing-site && pnpm run deploy

deploy:pages chains build first, and build itself runs the full pipeline: gen (regenerate the drift-gated reference) → check:reference → astro build → check:redaction (the redaction gate on the built dist/) → check:links. So a docs deploy cannot ship stale reference, a redaction leak, or a broken link.

Authentication model (and the failure it prevents)

The wrapper resolves auth the same way locally and in CI:

• FIREPATH_CF_DEPLOY_TOKEN set → used as the API token (CI / token path). The token needs “Cloudflare Pages: Edit”, and you must also set CLOUDFLARE_ACCOUNT_ID - a Pages-scoped token cannot enumerate accounts, so wrangler can’t auto-detect which account to deploy to (it fails with “Failed to automatically retrieve account IDs”). Store the token as a secret, never in the repo; the account id is not secret. (Alternative: give the token Account Settings: Read so it self-detects - but the account-id env is simpler.)
• Not set → the wrapper strips any ambient CLOUDFLARE_API_TOKEN for that one process and falls back to the local wrangler login OAuth session (OAuth can list accounts, so no account id is needed).

Two footguns the wrapper handles

1. A stale/expired CLOUDFLARE_API_TOKEN exported in a shell silently shadows a valid wrangler login → the upload fails with Cloudflare auth error 10000. The wrapper strips the ambient token on the OAuth path so this can’t happen.
2. wrangler whoami needs account-list permission a Pages-scoped token lacks, so the wrapper only preflights whoami on the OAuth path - it never false-fails a working deploy token. You should never have to manually unset a token to deploy.

Automated deploys (CI)

.github/workflows/deploy-pages.yml deploys the docs automatically on every push to main that touches the docs site, and the marketing site via a manual dispatch. Both authenticate from the repo secret CLOUDFLARE_PAGES_TOKEN and the repo Variable CLOUDFLARE_ACCOUNT_ID; if the token is absent the jobs skip with a warning rather than failing. One-time setup: add the secret CLOUDFLARE_PAGES_TOKEN (Pages: Edit) and the Variable CLOUDFLARE_ACCOUNT_ID (Settings → Secrets and variables → Actions) - after that, docs publishing is hands-off.

Releasing the marketing site

The developer docs are Cloudflare-Access-gated and noindex, so they deploy freely. The marketing site is public and search-indexed. It went live at www.firepath.software on 2026-08-03; the former human-in-the-loop publish gate (MARKETING_PUBLISH_OK=1) has been removed now that the truthfulness review is done, so deploy publishes directly:

# Local
cd Fire-Path-AI/artifacts/marketing-site
pnpm run deploy   # ci (astro build + redaction gate) → Cloudflare Pages deploy

For CI, run the Deploy Cloudflare Pages workflow with target marketing (manual dispatch). The copy rules still apply on every change - conservative claims only (no “proven / validated / production-ready”, no uptime/accuracy figures) - enforced by review, not a gate.

This publishes publicly

www.firepath.software is public and SEO-indexed. Keep the copy conservative and truthful; indexing + caching are hard to fully un-ring.