Operations

Last reviewed against the codebase: 2026-07-25.

How the AWS API backend is operated day to day. The three Cloudflare surfaces (marketing, docs, web-app shell) run 24/7 and are covered in Infrastructure - this page is about the one surface that hibernates: the API + database.

Everything here is scripted. There is no click-ops: a wake, a deploy, and a hibernate are each a single command whose behaviour is described below, grounded in the actual scripts under infra/scripts/ and the GitHub Actions workflow under .github/workflows/.

Who this page is for

A new maintainer who needs to bring the backend up, ship a code change, respond to an alarm, and put it back to sleep - ideally from this page alone, even after a long gap. Where a step has bitten us before, the gotcha and its recovery are called out inline.

The cost model - persist the data tier, hibernate compute

Only the COMPUTE tier (ECS + ALB + NAT) hibernates by default; the DATA tier (RDS + Redis) stays up so real accounts and data survive a hibernate/wake. persist_data defaults true (Terraform), and hibernate.sh / wakeup.sh default PERSIST_DATA=true. A small persistent tier plus DNS, storage, and secrets is what survives between sessions.

• COLD (default resting state, ~AUD $35–40/mo) - compute destroyed (ECS + ALB + NAT); the persistent data tier (RDS + Redis) and the always-on baseline (VPC, DNS zone, TLS certs, secrets, Terraform state) remain up.
• HOT (~AUD $290–320/mo while up, + task-hours) - the whole stack: data tier + ALB + ECS api-server + wind-worker + fire clustering. End-to-end validation on real infrastructure.

The persistent tier is sized for cost-safe testing, not final HA: RDS db.t4g.micro, single-AZ, 20 GB gp3, and Redis single-node cache.t4g.micro. When persisting, RDS gets deletion protection and takes a final snapshot on any destroy. Prod-grade HA is a later step (bump to db.t4g.medium + multi-AZ + a Redis replica, ~AUD $180/mo). See DB-PERSISTENCE-PLAN.md for the full transition and cost detail.

Why keep the data tier up rather than tear it fully down? The old fully-cold posture (~AUD $3/mo) destroyed RDS + Redis every hibernate, so every wake started from an empty, freshly-bootstrapped database. Keeping a small single-AZ data tier warm trades a modest idle cost for real durability - accounts and data now survive a wake instead of being reproduced from seed each time.

Persistence is the default - the live prod tier follows on the next transition

These are the code/infra defaults. The prod transition that actually stands up the durable tier is run by the owner and may not be applied to the live stack yet, so persistence takes effect on the next wake / after the transition (see DB-PERSISTENCE-PLAN.md) - it is not a live guarantee that today’s prod data is already durable.

Full teardown is the explicit opt-out

A deliberate full teardown destroys RDS and all data (and takes a final snapshot first). It is opt-in only: PERSIST_DATA=false bash infra/scripts/hibernate.sh (or --ephemeral). The default hibernate leaves the data tier up.

The full cost/mode reference lives in COLD-WARM-HOT-MODE.md in the repo root.

Waking the stack

Two wake modes, both via infra/scripts/wakeup.sh:

Command	Brings up	Use for
bash infra/scripts/wakeup.sh data	RDS + Redis only (data tier + NAT)	schema/data work without paying for compute
bash infra/scripts/wakeup.sh full	the whole stack - data tier + ALB + ECS api-server + wind-worker + fire clustering	end-to-end validation on real infrastructure

Both modes are feature-flag driven (enable_data_tier, enable_compute, enable_wind_worker, enable_fire_clustering, persist_data), so a mode is just a set of Terraform variables - nothing is hand-edited. The flags themselves are documented in Infrastructure.

The script is cost-gated: it prints exactly what it will create and its monthly cost, then waits for a y/N confirmation before touching AWS. Reading that block before you type y is the single best habit - it is the last checkpoint before spend.

What a full wake actually does, in order

The script narrates six steps. Understanding them makes failures diagnosable:

1. Build the api-server image (linux/arm64, from Fire-Path-AI/Dockerfile) and the wind-worker image (linux/amd64, from Fire-Path-AI/Dockerfile.worker, built FROM the local WindNinja base). Building both in the same run means enabling the worker tier can never point at a missing image.
2. Push both images to ECR (see the ECR-state gotcha below).
3. terraform apply with the HOT variable set - brings up the ALB, both ECS services, and turns fire clustering on.
4. Force a new ECS deployment so the api-server pulls the freshly-pushed :latest.
5. Verify the DB self-bootstrapped by polling /api/readyz on the ALB until it returns 200.
6. Point api.firepath.software at the new ALB (auto if Cloudflare creds are set, else printed instructions).

A data wake runs only the Terraform apply - no image build, no ECS, no DNS step.

Preflight: the AWS profile is hard-pinned - and that matters

Fire Path AI has a dedicated AWS account and SSO directory. The wake and hibernate scripts force AWS_PROFILE=firepath-admin and clear AWS_DEFAULT_REGION internally, precisely because a developer’s shell often has a leftover AWS_PROFILE pointing at an unrelated SSO directory. If that leaked through, aws ecr get-login-password and friends would silently authenticate against the wrong directory and fail with confusing “token does not exist” errors ~50 seconds into a Docker build. The script fails fast instead, checking aws sts get-caller-identity up front and asserting the account id matches. If you see the preflight error, run aws sso login --profile firepath-admin; if the browser opens the wrong SSO directory, your parent shell is leaking AWS_PROFILE - open a fresh terminal.

Gotcha: the ECR-in-state reconcile

hibernate.sh deletes the api-server ECR repository (force-delete). After a clean hibernate the repo is absent from both AWS and Terraform state. On the next full wake the push needs the repo to exist in AWS, and the subsequent terraform apply needs it to be in state - otherwise the apply tries to create a repo that already exists and aborts the whole wake with RepositoryAlreadyExistsException. The script reconciles these two facts explicitly and idempotently: create-if-absent in AWS, then terraform import (tolerating “already managed” as success). This was a real recurring bug - an earlier version created the repo for the push but never imported it, so every wake collided at apply. If a wake ever aborts at Step 2/3 on a RepositoryAlreadyExists error, the import branch is the place to look.

Timing expectations

• data - ~12 min wall on first stand-up (single-AZ RDS provisioning is the slow part); much faster once the persistent data tier is already up and only compute is being reconciled.
• full - ~15 min wall + ~5–8 min for the two Docker builds/pushes. Step 5 then polls /readyz for up to ~10 min while the api migrates (additively) against the persistent RDS instance and the idempotent seed runs. A /readyz returning 503 during this window is expected - it means “still provisioning/migrating”, not “broken”.

Self-bootstrap on boot - no manual DB step

The single most important operational property: the api task migrates and seeds its own database on boot. When DB_BOOTSTRAP_ON_BOOT is set (the Terraform default), a freshly-woken task runs its migrations and seed against the RDS instance itself. Migrations are additive and the seed is idempotent (insert-if-absent) - there is no destructive re-seed in prod, so with the data tier persisting, real accounts and data carry across a wake rather than being recreated. There is no SSM tunnel and no manual seed command in the normal flow - a full wake goes from a hibernated stack to a schema-complete, running backend unattended. Step 5 of the wake simply confirms it happened by polling /readyz until it is 200.

Why self-bootstrap instead of the old tunnel flow? The predecessor bootstrapped the DB from the operator’s laptop over an SSM port-forward through a running task (see the emergency fallback below). That worked but coupled every wake to a local toolchain (session-manager-plugin, pnpm, correct DATABASE_URL encoding) and to the operator being present. Self-bootstrap moves the whole thing into the container that already has the code and the credentials, making a wake genuinely hands-off. It was validated on the 2026-07-13 wake.

The seed admin’s password persists in Secrets Manager (firepath-ai/prod/seed-admin), so a fresh wake produces a real admin login without any local .env or prompt. Prod bootstrap seeds without demo data (seed({demoData:false})), so real users never see placeholder fires, assets, or places - this was a fix after prod once seeded a fake fire into the shared default org. Real “what’s around me” places come from the OSM/Overpass ingest (PLACES_INGEST_ENABLED, off by default) - an honest empty state until it is enabled and validated on a wake; no hand-invented coordinates.

Emergency fallback only - bootstrap-prod-db.sh

infra/scripts/bootstrap-prod-db.sh still exists but is no longer in the wake flow. Run it by hand only if Step 5 reported /readyz never became ready and the task logs show the self-bootstrap failed. It opens an SSM port-forward to RDS through a running task, reads the RDS master credential from Secrets Manager (kept in-shell, never printed), enables the postgis + citext extensions, pushes the schema, and seeds. Two gotchas are baked into it because they bit us: (1) extensions must be created before drizzle-kit push or the push fails with “type citext does not exist” - drizzle can’t CREATE EXTENSION itself; (2) RDS-managed master passwords contain URL-unsafe characters that Node’s strict WHATWG URL parser rejects (ERR_INVALID_URL) even though psql/libpq accept them, so the script percent-encodes the user and password. Prereqs: session-manager-plugin, jq, pnpm, and an authed firepath-admin SSO session.

Auto-DNS - the ALB hostname changes every wake

Because compute hibernates, a wake destroys and recreates the ALB, so the load balancer gets a brand-new AWS hostname every time (only the data tier persists - the ALB does not). Step 6 of a full wake re-points api.firepath.software at the new ALB so the public API endpoint follows the ephemeral load balancer automatically.

• If CLOUDFLARE_API_TOKEN (scoped Zone:DNS:Edit on firepath.software) and CLOUDFLARE_ZONE_ID are exported, the script updates the api CNAME via the Cloudflare API (TTL 60s, DNS-only / proxy off). Cached clients self-heal within ~1 minute. Set these two vars once in a gitignored env file and source it before waking.
• If they’re not set, the script prints the exact record to set by hand (Type CNAME, Name api, Target = the new ALB hostname, TTL 60, Proxy off).

The three Cloudflare surfaces are never recreated, so their DNS never churns - only api. needs attention on a wake. See Infrastructure.

503 immediately after a wake = stale api. DNS

If the API returns 503 right after a full wake, a stale api. CNAME pointing at the previous (now-destroyed) ALB is the first suspect. Re-run Step 6, or check terraform output -raw alb_dns_name and confirm the Cloudflare api record matches it.

Readiness checks

The api-server exposes two probes (defined in Fire-Path-AI/artifacts/api-server/src/routes/health.ts). The distinction is load-bearing - see the incident section:

• GET /api/healthz - cheap liveness. Returns 200 {status:"ok"} whenever the process is up, regardless of dependencies. It’s the ALB target-group health check and a fast “is the container running?” signal, and it is deliberately never written to the audit log.
• GET /api/readyz - readiness. Verifies dependencies (database reachable, and ingest freshness) before the task is considered ready to serve. Healthy is 200 {status:"ready", db:"ok"}; db:"down" or a 503 means a dependency is unhealthy. A full wake verifies the task came up healthy via /readyz rather than assuming success.

/healthz staying 200 while /readyz is 503 is expected and correct - liveness is not readiness. Do not “fix” a 503 by restarting a container that is passing liveness; find the failing dependency.

You can watch both continuously with the record-keeping monitor:

# Rehearse the whole loop COLD against a local api-server (no spend, no AWS session needed):
API_BASE=http://localhost:3000/api ITERATIONS=3 infra/scripts/wake-watch.sh

# During a real HOT wake, prod defaults + live log/alarm checks, infinite loop:
infra/scripts/wake-watch.sh

wake-watch.sh probes /healthz + /readyz every INTERVAL seconds, greps the api-server log for life-safety markers (AUDIT LOG INSERT FAILED, undelivered, pool error, ingest staleness, …), counts CloudWatch alarms in ALARM, and writes a timestamped evidence file under wake-evidence/. It degrades cleanly: endpoint probes always run, and the AWS log/alarm checks only run when an authed firepath-admin session is present - otherwise they report “skipped (local)”. The design rule is “no output = not checked”, so every check leaves a durable record rather than relying on eyeballing.

Deploying code to a HOT stack

The first image on a wake comes from wakeup.sh full itself. Subsequent code deploys during the same HOT session go through the manual GitHub Actions workflow, .github/workflows/deploy.yml:

• Manual trigger only (workflow_dispatch), because the stack is COLD by default - a deploy is a deliberate action taken during a HOT wake, never an auto-push. It requires typing deploy into a confirm input, and a guard step fails the run if you don’t.
• Credential-free - it assumes an IAM role via GitHub OIDC (trust pinned to the repo’s main ref), so no long-lived AWS keys live in CI.
• It builds + pushes an image tagged with the 12-char commit SHA (and :latest), renders a new ECS task definition from the currently-deployed one, deploys with wait-for-service-stability: true, then runs a post-deploy smoke test curling /api/healthz (must contain "status":"ok") and /api/readyz on the live ALB.

Because every image is tagged with its commit SHA, rollback is just re-deploying a known-good tag - there is no separate rollback tooling to learn.

The smoke test tolerates ingest staleness, on purpose

Right after a cold DB, /readyz may report ingest sources as stale (nothing has been ingested yet). The smoke test requires the endpoint to answer but does not require a fully-fresh body, so a green deploy does not by itself prove ingest is caught up - check /admin/telemetry/overview for that.

Observability

The Terraform observability module (compute-gated, so it has zero footprint while hibernated) provisions, on every HOT wake:

• An SNS alert topic that emails a configured address on any alarm.
• CloudWatch alarms across the stack, including metric-filter alarms per life-safety log string - AUDIT LOG INSERT FAILED, EMERGENCY ALERT UNDELIVERED, READINESS: ingest sources are stale, UNCAUGHT EXCEPTION / UNHANDLED PROMISE REJECTION - plus ALB 5xx and p99 latency, ALB unhealthy hosts, ECS CPU / memory / no-running-tasks, and (when the data tier is up) RDS CPU / free storage / connections.
• A code-defined ops dashboard tying those signals together - the single place to correlate ALB / ECS / RDS signals.

RDS alarms are gated on the static data-tier flag (known at plan time), so a fresh wake’s plan stays deterministic.

Alarms don’t exist while COLD - watch the strings by hand

Because the module is compute-gated, the alarms are only in place during a HOT wake. If you’re operating a wake before the module has applied, or investigating between wakes, the same log strings can be watched directly in CloudWatch Logs Insights (that’s exactly what wake-watch.sh automates).

The alarm-to-response mapping is the incident runbook, summarised next.

Incident response by alarm class

Each alarm maps to a likely cause and a fix. The authoritative version (with diagnosis commands) is OPS-RUNBOOK.md in the repo root; the essentials:

• API down / /readyz 503 (db:"down"). RDS unreachable - security group, RDS not yet available, connection storm, or credentials rotated without a redeploy. Confirm RDS is available and its SG allows the ECS SG, verify the connection secret, force a new ECS deployment. Remember /healthz staying 200 here is correct.
• Ingest stale (READINESS: ingest sources are stale). Most often an expired/over-quota NASA FIRMS key: rotate NASA_FIRMS_API_KEY in Secrets Manager and redeploy; the mirror endpoint is tried automatically. DEA is a best-effort secondary feed (FIRMS stays authoritative); an OpenMeteo failure keeps the last-good layer rather than half-wiping it. Every external call has a 15s timeout + retry, so a single blip self-heals - sustained staleness means an upstream outage or a credential problem.
• Alerts not delivered (EMERGENCY ALERT UNDELIVERED). Push provider outage, all device tokens stale, or web VAPID unset. Emergency Warnings never age out - they retry until delivered or the alert clears; stale tokens are auto-revoked; dedup is race-safe. If web push is silent, set the VAPID_* env vars. Email fallback is on by default (THREAT_EMAIL_FALLBACK_ENABLED=true in the prod api-server task def): it self-gates on the Resend key, so with no RESEND_API_KEY on the stack it is a safe no-op, and with the key it sends email as a backup channel when a push delivery fails. SMS fallback stays off (needs Twilio).
• Audit trail dropping (AUDIT LOG INSERT FAILED). A compliance/forensics gap - treat as high priority. Check the audit_log table exists and its schema matches; the log line reports how many events were dropped.
• Process crash loop (UNCAUGHT EXCEPTION). By design the process logs the stack and exits(1) so ECS replaces the task cleanly - continuing in an undefined state is worse for a life-safety system. A repeating crash is a deterministic bug: roll back to the last-known-good image tag.
• Auth failure spike (500s on protected routes). A 500 (not 401) means a server-side problem - a missing JWT_SECRET or a DB outage during the token-revocation check. A spike of 401s instead is normal (expired tokens) and not an incident.

The alarm signals and their healthy values are enumerated in the runbook’s “Signals to watch” table.

Hibernating

infra/scripts/hibernate.sh tears compute back to COLD while leaving the data tier up (the PERSIST_DATA=true default):

bash infra/scripts/hibernate.sh   # prints what it will destroy, then asks y/N

• Destroys (the compute cost) - NAT gateway + Elastic IP, the ECS services + ALB.
• Preserves - the persistent data tier (RDS + Redis + the database-url / redis-url secrets), so real accounts and data survive; plus the baseline VPC + subnets + security groups + S3 gateway endpoint, the DNS hosted zone, ACM TLS certs, the KMS key + JWT/FIRMS secrets, IAM roles + OIDC, and Terraform state. When persisting, RDS keeps deletion protection and any destroy takes a final snapshot first.

A deliberate full teardown (PERSIST_DATA=false bash infra/scripts/hibernate.sh, or --ephemeral) additionally destroys RDS, ElastiCache Redis, and the data-tier secrets - all data is lost, taking a final snapshot first. That is the explicit opt-out, not the default.

The script clears known teardown blockers idempotently before the destroy - it disables ALB deletion-protection (the live flag can drift from Terraform state) and empties the ECR repo so it can be destroyed - then runs the destroy as a terraform apply with the tiers flagged off.

The Elastic IP orphan - why the script sweeps and re-runs

terraform destroy releases the NAT’s Elastic IP, but if the NAT’s network interface is already gone the release errors with InvalidNetworkInterfaceID.NotFound; the apply then exits non-zero and leaves the EIP allocated (~AUD $3.6/mo of silent cost) with Terraform state out of sync. This bit us on more than one hibernate. The script therefore captures the destroy’s exit code, sweeps any unassociated project-tagged EIP and releases it, and re-runs the destroy to reconcile state. A hibernate lands cleanly without manual cleanup - but if you ever see a lingering unassociated EIP, this is why, and terraform output stage printing COLD is the confirmation the reconcile worked.

After a hibernate, the api.firepath.software CNAME points at a now-dead ALB hostname - requests return DNS errors and the web-app SPA shows its “Service is paused” banner. That’s expected; cleaning up the stale record is optional.

Known infra follow-up (non-blocking)

The Terraform S3 backend still locks state via a DynamoDB table, and Terraform now deprecates that in favour of native S3 state locking (use_lockfile = true). It’s a warning, not an error - wakes still succeed. The migration (bump required_version to >= 1.11, swap the lock line, terraform init -reconfigure, then decommission the DynamoDB table) should be done COLD, off the critical path of a HOT wake, and every operator/CI running Terraform must switch at the same time (mixed checkouts don’t share a lock). Full steps are in OPS-RUNBOOK.md.

Where the detail lives

• Infrastructure - hosting topology, the four surfaces, Terraform modules + feature flags, the hibernate/wake cost model at the architecture level.
• Configuration reference - every environment variable (generated from the code), including DB_BOOTSTRAP_ON_BOOT, JWT_SECRET, NASA_FIRMS_API_KEY, and the VAPID_* keys referenced above.
• Overview & architecture - the application tiers, the ingest → predict → alert pipeline, and the fire-science model behind the wind-worker.