Troubleshooting

Last reviewed against the codebase: 2026-07-25.

The failures most likely to trip up a new maintainer, why they happen, and how to recover. Each entry names the signal you’ll actually see, the underlying cause, and a runnable fix. Where a problem is inherent to the hibernate/wake cost model (rather than a bug), that is called out so you don’t chase a phantom.

Run everything from the repository root unless noted; the pnpm workspace lives in Fire-Path-AI/, while docker-compose.yml and infra/ sit at the repo root.

App login returns 503 / 530

Signal: the web app loads its shell but every API call fails; the browser console shows 503 (service unavailable) or Cloudflare 530 (origin unreachable).

Why: the front end is a static SPA served by Cloudflare; it is almost always up. A 5xx here means the API origin is unreachable - the two front ends are decoupled on purpose so the marketing/app shell never depends on a hot backend.

Diagnose in order - each step isolates a layer:

1. Is the API process up? In local dev it listens on :3000.

   curl -s http://localhost:3000/api/healthz     # liveness - process is up
   curl -s http://localhost:3000/api/readyz      # readiness - DB reachable too

   A 200 from /healthz but a non-200 from /readyz means the process is alive but the database is not reachable - go to step 2. See Verify for what each probe asserts.

2. Is Postgres up?

   docker compose up -d postgres
   docker compose ps postgres          # State should be "running (healthy)"

3. Is the tunnel up (hosted origin only)? In local-against-api.firepath.software development, the origin is Joel’s machine exposed via cloudflared. A 530 with the local api-server clearly running means the tunnel daemon is down - restart it, then retry.

Against the deployed environment, a 5xx almost always means the stack is hibernated or mid-wake - this is expected, not a fault. See Run it in production for the wake procedure.

The one rule for “up but not working”

Check /readyz before anything else. /healthz only proves the process answers; /readyz proves it can actually serve (DB reachable). The 503/530 vs. readiness split is the fastest way to tell “no process” from “process, no database.”

Tests fail with DATABASE_URL / JWT_SECRET unset

Signal: vitest fails with an error tied to a missing environment variable.

Why: by design the suite passes with all production secrets unset - CI is never given real credentials, so tests inject dummies or mock the boundary. If a test fails on a missing variable, the usual causes are:

• a stale real value leaking in from your shell (a previously exported DATABASE_URL pointing at something that no longer exists), or
• a missing dev default - you never created .env.

Fix:

cd Fire-Path-AI
cp .env.example .env          # dev defaults point at the docker-compose Postgres

Ensure JWT_SECRET is set to any non-empty dev string - auth fails hard without it. To reproduce CI’s “no secrets” contract locally, unset the variable and run the suite; it must still pass:

env -u DATABASE_URL -u JWT_SECRET pnpm --filter @workspace/api-server run test

If it fails only when the real value is unset, a test is relying on a live credential - that belongs to a HOT wake, not the unit suite.

Test logins don’t appear (local) or you fear they leak (prod)

Signal: the one-tap local test sign-in buttons are missing, or you’re worried they could be reachable on a deployed service.

Why / reassurance: ENABLE_TEST_LOGINS=true gates the convenience logins, and the endpoint hard-404s whenever NODE_ENV === production. That means the flag being accidentally left on cannot expose test logins on a deployed service - the NODE_ENV guard fires first.

Fix (local): set ENABLE_TEST_LOGINS=true in your .env and restart the api-server. See Run it locally for the full env setup.

Terrain wind shows “unavailable”

Signal: predictions render, but terrain-refined wind reports “unavailable” and the model falls back to the coarse synoptic wind field.

Why: terrain refinement (WindNinja) runs on a separate wind-worker ECS tier built from an amd64 image. It is not part of the default API deploy.

Not deployed by default

The wind-worker is brought up only by wakeup.sh full. On a wakeup.sh data wake - or in local dev without the worker running - terrain wind is genuinely absent and the fallback is correct behaviour, not a bug. The wind-worker has also historically not been deployed even on full wakes; treat terrain wind as an opt-in capability until you confirm the worker tier is live.

Fix: run the full production wake (see Run it in production) and confirm the worker tier and its Redis queue are up. Capability status is tracked on the architecture overview.

pnpm install is slow or refuses a package

Signal: install stalls, or a specific just-published package version won’t resolve.

Why: the workspace enforces a minimum release age on npm packages as a supply-chain defence - a brand-new release is deliberately held back until it has aged. This is a security control, not a failure.

Fix: none needed - wait, or pin to an already-aged version. Do not disable the setting to “unblock” an install; that removes the protection for everyone.

Docs-site: drift gate fails (check:drift)

Signal: pnpm run ci (or check:drift) fails in the docs-site with a committed-vs-generated mismatch.

Why: the generated reference sections (/reference/*) are produced from the codebase - the Drizzle schema, config.ts, the API spec, and a curated repo tree. When source changes, the committed Markdown goes stale and the drift gate catches it before publish.

Fix - regenerate, verify, then commit:

pnpm --filter @workspace/docs-site run gen          # runs gen:datamodel/config/api/repotree
pnpm --filter @workspace/docs-site run check:drift  # should now pass

Commit the regenerated Markdown. Full guidance is in Update common things.

Docs-site: redaction or link gate fails

Signal: the docs build fails at check:redaction or check:links.

Why: check:redaction runs over the built dist/ and blocks publish if the output contains secrets or personal data (absolute paths, account IDs, resource IDs/ARNs, private emails, sample place names). check:links blocks broken internal links. Both are hard publish gates in the build/ci scripts.

Fix: read the gate output - it names the offending file and the matched value. Remove the sensitive value (or fix the link target), then rebuild:

pnpm --filter @workspace/docs-site run build   # gen → check:reference → build → redaction → links

Because the gate runs on the compiled output, a value can slip past source review and still be caught here - that is the intended backstop.

Wake fails: ECR repo “not in Terraform state”

Signal: a production wake aborts complaining the container registry isn’t managed by Terraform, or re-creates on every wake.

Why: the image push needs the ECR repo to exist before Terraform runs, but the repo must also live in Terraform state. Earlier wakes created the repo in AWS for the push but never imported it into state, so every subsequent wake tripped over it. wakeup.sh now imports the repo unconditionally and treats “already managed” as success.

Fix: re-run wakeup.sh - the import step is idempotent. If it still aborts, read /tmp/firepath-ecr-import.err; a genuine failure (not an “already managed” no-op) is printed verbatim. See Run it in production.

Wake fails: type "citext" does not exist

Signal: the DB bootstrap fails on schema push with a missing-type error for citext (or postgis).

Why: drizzle-kit push cannot itself CREATE EXTENSION. The extensions must exist before the schema push, or the push references a type that isn’t there yet.

Fix: bootstrap-prod-db.sh enables extensions first (idempotently) via the init SQL, then runs push-force. If you’re bootstrapping by hand, enable postgis + citext before pushing. The production bootstrap sequence lives in Run it in production.

Sign-in doesn’t work after a fresh production wake

Signal: the stack is up and healthy, but you can’t sign in through the web UI.

Why: the prod admin needs a real email/password credential to sign in through the UI. The bootstrap resolves the seed-admin password in order: (1) an explicit env override, (2) a persistent Secrets Manager value (the recommended source - set once, survives every wake), (3) empty → a passwordless device-only admin that the web UI cannot log in as.

Fix: ensure the seed-admin password exists in Secrets Manager (or pass it as an env override to the bootstrap). A silently-empty password produces a device-only admin and a UI that rejects every login - check that first.

Wake succeeds but api.firepath.software still 5xx

Signal: the wake reports healthy (the ALB answers /readyz), yet the public hostname is still unreachable.

Why: every wake produces a new ALB hostname, so DNS must be repointed each time. The wake auto-updates the Cloudflare record when CLOUDFLARE_API_TOKEN and CLOUDFLARE_ZONE_ID are exported (TTL 60s, DNS-only); without them it prints manual instructions instead and DNS stays stale.

Fix: either export those two variables before waking so DNS updates automatically, or follow the printed manual repoint. Then confirm origin health directly against the ALB before blaming DNS:

curl -s https://api.firepath.software/api/readyz    # after DNS propagates (~60s TTL)

An AWS command hits the wrong account / SSO directory

Signal: wakeup.sh / hibernate.sh open a browser to an SSO start page that isn’t the Fire Path one, or AWS calls target an unexpected account.

Why: a developer shell can default AWS_PROFILE to an unrelated profile at a different SSO directory, and that leaks into the scripts. Both scripts hard-pin AWS_PROFILE=firepath-admin and clear AWS_DEFAULT_REGION to defend against this, but an already-broken SSO session in the environment can still confuse you.

Fix: if the browser opens the wrong directory, your AWS_PROFILE is leaking through - open a fresh terminal and re-authenticate:

aws sso login --profile firepath-admin

Never set AWS_PROFILE globally in your shell profile for this repo; let the scripts pin it.

Hibernate leaves an orphaned Elastic IP (cost leak)

Signal: after hibernate.sh the monthly cost doesn’t drop to the ~AUD $3 baseline; a NAT Gateway or its Elastic IP is still billing.

Why: hibernate destroys the expensive ephemeral tier (NAT Gateway + EIP, RDS Multi-AZ, ElastiCache, ECS/ALB - roughly AUD $230/mo) and preserves the cheap always-on baseline (VPC, Route 53 zone, ACM certs, KMS + secrets, IAM). Live infra can drift from Terraform state, so a destroy mid-flight can leave a resource - an orphaned EIP still bills even when detached.

Fix: re-run hibernate.sh - its pre-destroy steps clear known blockers idempotently. Then confirm nothing expensive survived (no running RDS/ECS, no detached-but-allocated EIP). Note that data is intentionally lost on hibernate (no final snapshot) - you re-seed on the next wake; that is by design, not data loss to recover from. The full cost model is in Run it in production.

Reading the health signals

• GET /healthz - liveness: the process is up and answering.
• GET /readyz - readiness: the database is reachable and the service can serve.

When something is “up but not working,” check /readyz first - it is the single signal that distinguishes a live process from a serving one, and it short-cuts almost every diagnosis on this page. See Verify.