Verify

Last reviewed against the codebase: 2026-07-25.

How to confirm a change is sound before it ships - the same signals CI enforces, run by hand. This is a life-safety application, so “it compiles” is not the bar: a change is verified when the types build, the suites are green with production secrets absent, the running process reports ready (not just alive), and - if you touched anything the reference is generated from - the docs gates stay green.

Run everything below from Fire-Path-AI/ unless a step says otherwise. The order here mirrors the PR workflow, so working top-to-bottom reproduces CI locally.

1. Type-check the workspace

pnpm run typecheck

This is a two-stage check, and the order matters - it is why there is a dedicated script rather than a bare tsc:

1. typecheck:libs - tsc --build --force compiles every workspace library (lib/*) first, emitting their declaration files.
2. typecheck - then runs each artifact’s own typecheck (tsc --noEmit) across artifacts/** and scripts.

Why libs first: the apps import the libs by their built .d.ts outputs. If you type-check an app before its libs have been built, tsc resolves stale or missing declarations and reports phantom errors (or, worse, misses real ones). CI runs the same root script (see the typecheck job in .github/workflows/pr.yml), so a green pnpm run typecheck locally means a green typecheck job.

Gotcha: stale lib declarations

If you change a type in a lib/* package and an app still reports the old shape, your .d.ts outputs are stale. Re-run pnpm run typecheck (the --force on the libs stage rebuilds them) - do not try to fix it by editing the app.

The client app additionally guards its design system. pnpm --filter @workspace/firepath-ai run theme:check runs typecheck + a theme audit + a WCAG contrast audit; the web CI job runs this, so a token/colour change that passes locally but fails contrast will be caught there.

2. Run the tests

Both the API server and the client use Vitest. They are separate suites with very different shapes.

API server

pnpm --filter @workspace/api-server run test          # unit + route integration
pnpm --filter @workspace/api-server run test:coverage # same, with the coverage ratchet
pnpm --filter @workspace/api-server run test:watch    # re-run on change (local dev)

test:coverage enforces ratchet thresholds defined in artifacts/api-server/vitest.config.ts (currently statements 59 / branches 50 / functions 62 / lines 61). These floors sit just under the measured baseline and are a one-way ratchet: raise them as new tests land, never lower them. A change that deletes or fails to cover code enough to drop below a floor fails the suite even if every assertion passes - that is intentional, and it is what makes “we have a coverage gate” a real claim.

Why coverage isn’t ~85% yet

The whole-repo number is held down by modules that genuinely cannot be unit-covered in a COLD (no-AWS) checkout - the WindNinja simulation shims, the scheduler run-loops, and the live ingest adapters (DEA, Open-Meteo, FFDI). Those are validated at a HOT wake, not in unit tests. The testable core targets ≥85% statements/lines and ≥90% on route integration; the config comments track the climb.

Client

pnpm --filter @workspace/firepath-ai run test

The client suite is pure-logic only - it runs Vitest over the shared lib/** math (spread geometry, offline-tile math, formatting) that must be correct on a phone during an evacuation. Component/UI behaviour is deliberately not unit-tested here; it is covered by the workspace typecheck plus the on-device EAS build. CI runs this as the client-tests job.

Secrets must be UNSET

Both suites must pass with production secrets absent. They inject dummy values or mock the external call, so a missing real key is never the reason for a red suite - and a suite that needs a real secret is a bug. The api-test CI job proves this: it sets only throwaway values (JWT_SECRET: ci-test-only-..., a local DATABASE_URL) and never receives production credentials.

Reproduce a CI-only failure

If the suite passes locally but the api-test job fails, you probably have a real secret exported in your shell that CI doesn’t. Reproduce it by unsetting the key (unset RESEND_API_KEY, etc.) and re-running - the test should still pass. If it doesn’t, the test is depending on a real secret and must be fixed.

The api-test job also mirrors the local DB setup: it boots Postgres, applies the PostGIS init migration, pushes the Drizzle schema (push-force), seeds fixtures, then runs test:coverage. See Start/stop locally for the same sequence by hand.

3. Service health - alive vs ready

With the API running (default port 3000, all routes mounted under /api), two endpoints distinguish “the process is up” from “the process can actually serve”:

curl http://localhost:3000/api/healthz
curl http://localhost:3000/api/readyz

• GET /healthz - liveness. Cheap and dependency-free; always returns {"status":"ok"} if the process is running. The orchestrator (ALB / Cloudflare) uses this to decide whether to kill and replace a task. It must stay dependency-free - otherwise a transient DB blip would make the orchestrator destroy an otherwise-healthy process.

• GET /readyz - readiness. Actually verifies the process can serve. It runs SELECT 1 against Postgres and returns HTTP 503 with {"status":"not_ready", "db":"down"} if the DB is unreachable; on success it returns 200 with {"status":"ready","db":"ok","ingest":[...]}, where ingest reports the age (in minutes) of each source’s last cycle and flags any older than 60 min as stale.

Why both exist (real defect history): /healthz used to be the only signal, and it returned a static "ok" even while Postgres was down and every real route was 500-ing - a green light that lied. /readyz was added (audit finding M1) so readiness reflects the truth. Stale ingest is surfaced, not fatal: a stale FIRMS feed sets stale: true (visible on the dashboard and to a CloudWatch alarm) but does not flip readiness to 503 - taking the whole API down because one upstream feed lagged would be the wrong failure mode.

/readyz returns 503 on a fresh wake - this is expected

On a fresh WARM/HOT wake the process self-bootstraps its database in the background, so /healthz comes up immediately but /readyz will report not_ready (503) until the DB is reachable. Poll /readyz until it flips to 200; don’t treat the early 503 as a failure. See Start/stop production.

4. Documentation gates

Any change that touches code the reference is generated from must keep the docs site green. Run the full gate chain exactly as CI does (the docs-site job):

pnpm --filter @workspace/docs-site run ci

This runs, in order:

1. gen - regenerate all four reference sections from source.
2. check:drift - fails if the committed reference is stale (i.e. gen produced output that differs from what’s checked in).
3. check:reference - every page the nav/links point at actually exists.
4. astro build - the site compiles.
5. check:redaction - the built dist/ contains no secrets or PII (runs the shared denylist; the marketing site runs the same list in its own job).
6. check:links - no broken internal links in the output.

The most common local failure: drift

check:drift is the step that bites maintainers. If you changed the Drizzle schema, an API route, an env var, or the workspace layout without regenerating, drift fails. The fix is not to edit the generated Markdown by hand - re-run gen, then commit the regenerated files:

pnpm --filter @workspace/docs-site run gen

See Update common things for which generator maps to which reference section, and the repository guide for the output itself.

5. Load smoke test (optional)

A ramped k6 smoke test proves routing + the auth middleware + the DB read path hold under concurrency - so the HOT wake only has to confirm the behaviour at prod scale, not discover a problem in it. With the local stack up and the server on :3000:

docker compose up -d postgres redis
k6 run Fire-Path-AI/artifacts/api-server/load/api-smoke.k6.js

The script (artifacts/api-server/load/api-smoke.k6.js) targets http://localhost:3000/api by default. Point it at another environment by overriding API_BASE:

API_BASE=https://api.firepath.software/api k6 run \
  Fire-Path-AI/artifacts/api-server/load/api-smoke.k6.js

This is a smoke test, not a capacity benchmark - it validates that the tier stays correct under load, not how many requests it can serve. Real capacity numbers come from a HOT wake.

What “verified” means

A change is ready to ship when, from a clean checkout with production secrets unset:

• pnpm run typecheck is green (libs build, apps + scripts type-check);
• the API and client Vitest suites pass, and coverage does not regress below the ratchet;
• a running server answers /healthz and /readyz with 200;
• if you touched a generated source, pnpm --filter @workspace/docs-site run ci is green (no drift, no broken links, no redaction hits).

These are exactly the jobs in .github/workflows/pr.yml, so a local pass predicts a green PR. When something still only fails in CI, it is almost always a real secret leaking through your shell (see §2) - not a difference in the code.