Testing & quality

Last reviewed against the codebase: 2026-07-25.

How quality is enforced across the codebase, and - just as importantly - why it is enforced this particular way. This is a life-safety product with an enterprise/Government target, so the guiding principle throughout is verify, don’t trust: prefer failure modes over happy paths, assume races by default, and treat load behaviour as a correctness property rather than a performance nicety.

Every layer below runs in CI on each pull request (and on pushes to main); together they are the gate a change must pass before it can merge. The pipeline is defined in a single workflow (.github/workflows/pr.yml) and is designed so each job is reproducible by hand - there is no CI-only magic. If a job passes in CI it should pass on your machine with the same commands, and vice versa.

The layers of testing

Layer	Tool	Job in CI	What it covers
API integration	Vitest	api-server vitest	Routes exercised against a real local Postgres (PostGIS), Express in-process - no mocked DB layer
Client unit	Vitest	Client unit tests	Pure client logic only (lib/**), in a plain Node env (e.g. offline map-pack geometry/size/eviction math)
Load smoke	k6	(manual / pre-release)	Ramped concurrent load against the local api-server (routing + auth + DB read path)
Docs gates	Node scripts	docs-site + marketing-site	Drift, reference-existence, redaction, and link checks on these docs
Typecheck	tsc	Typecheck	Whole-workspace parse + type-resolve
Supply chain	pnpm audit + gitleaks	Dependency + secret scan	Known CVEs and accidentally-committed secrets
SPA theme + a11y	theme/WCAG audits	SPA theme + WCAG	Colour-token rules and contrast/dichromacy ladders

The three surfaces that gate behaviour are the API integration suite, the client unit suite, and typecheck. The rest guard the way the code is shaped (types, tokens, dependencies) and what leaves the building (the published-docs redaction gate). k6 is the one deliberate exception - it is not in the per-PR pipeline, and the reasoning for that is spelled out under Load (k6) below.

API integration (Vitest)

The api-server suite (artifacts/api-server, ~400 cases across ~70 files) is 100% integration - there is no mocked database. It runs against a real PostGIS Postgres, because the geometry columns on fires, assets, forecast_points and the predictions tables genuinely require PostGIS; a mocked or plain-Postgres layer would let bugs in the spatial queries pass review. Express is started in-process against that database, so a test exercises the real router, the real auth/RBAC middleware, and the real SQL - the whole request path a client hits.

Why it is deterministic (and how the determinism is built)

A shared-database integration suite is only trustworthy if it behaves identically on a freshly-seeded CI database and on a developer’s weeks-old local one. Two mechanisms in vitest.config.ts make that true, and it is worth understanding both because they are the first things to suspect if the suite ever goes flaky:

1. Pinned file order. Vitest’s default sequencer orders files by their cached duration from the previous run - which changes every run, so the order (and therefore the shared-DB state each file sees when it runs a whole-database scan) shifts run-to-run. That was the root cause of a historical ~0.6% run-to-run coverage jitter. A custom AlphabeticalSequencer sorts files by path so the order is identical every time. Cheap, fully reversible, and it made coverage reproducible (proven by five identical runs).
2. One file at a time. Integration tests hit one shared Postgres and spin up Express in-process, so they must run sequentially or they race on shared rows (global delivery/evaluator scans, seed data). This is fileParallelism: false. Note the sharp edge: Vitest 4 removed poolOptions.forks.singleFork (it was silently ignored, so files ran in parallel and the suite went non-deterministic). fileParallelism: false is the supported replacement - if you see mysterious cross-test contamination after a Vitest upgrade, check that this flag still exists and is honoured.

The global sweep (test-owned data cleanup)

A global setup/teardown (test/setup.ts) sweeps all test-owned rows in one FK-safe pass, before the first file and after the last. Test data is identifiable purely by naming convention, which is what makes a blanket sweep safe:

• organizations.slug LIKE vitest-%
• users.auth_provider_user_id LIKE vitest-%
• fires.external_id LIKE vitest-%

Two design choices here matter for a maintainer:

• Why a global sweep and not per-file afterAll deletes. The request-audit middleware writes its http.GET row fire-and-forget on res.on('finish'). A per-file afterAll that drops the test’s org/user can win the race against that async insert, which then fails its FK and logs AUDIT LOG INSERT FAILED - noise in an otherwise-green run. A global sweep runs when no request is in flight, so the race can’t happen.
• The demo seed is left alone. The sweep deliberately does not mutate seed rows (e.g. bumping observed_at), because that would change the evaluator’s workload suite-wide and perturb the delivery/threat tests. Tests that need fresh, in-window data create and own it themselves (see fires.test.ts).

If the suite hangs or errors on connect

The tests assume DATABASE_URL, JWT_SECRET, and PORT are set. Load them via dotenv or set -a; source .env; set +a before running. A suite that hangs at startup is almost always a missing or wrong DATABASE_URL, or Postgres not being up - bring the local stack up first (see Run locally).

What a good change adds

Every change is expected to add the relevant happy-path, error, boundary, auth/RBAC, and concurrency cases - not deferred “tests later”. The suite already models this: alongside the route happy-paths there are dedicated IDOR tests (idor.test.ts, predictions-idor.test.ts), RBAC/revalidation tests, an auth device-bypass regression test, and true concurrency tests such as evaluator-two-writer-race.test.ts and cluster-alert-dedup.test.ts. When you touch a route, match that shape.

Client unit (Vitest)

The client’s Vitest config (artifacts/firepath-ai/vitest.config.ts) is scoped to lib/**/*.test.ts pure modules, in a plain Node environment, so React Native / Expo code is never loaded. This is a deliberate boundary, not a gap: loading the native/Expo runtime under Vitest is slow and brittle, and component behaviour is already covered by typecheck plus the on-device build. What these tests guard is the platform-agnostic logic that would be expensive to discover a bug in on a device - today that is the offline map-pack geometry/size/eviction math (lib/offline/mapPacks.shared.test.ts).

There is also a small pure-logic suite for the shared spread model (lib/spread-model), run the same way.

Load (k6)

A ramped smoke test (artifacts/api-server/load/api-smoke.k6.js) proves the app tier holds up under concurrent load cold, so a production wake only has to confirm behaviour at scale rather than discover a problem. It registers once for a JWT, then hammers liveness (/healthz), readiness (/readyz, which exercises the DB ping + ingest-freshness check), and an authenticated org-scoped read (/assets). It asserts <2% failed requests, p95 < 1s, and <2% check errors. It is a health check under load, not a capacity benchmark.

Run it against the local stack (from the repo root, with Postgres + Redis up and the api-server on :3000):

docker compose up -d postgres redis
# start the api-server with DATABASE_URL / JWT_SECRET / REDIS_URL / PORT=3000
k6 run Fire-Path-AI/artifacts/api-server/load/api-smoke.k6.js

Point it at production instead by overriding the base URL:

API_BASE=https://api.firepath.software/api k6 run Fire-Path-AI/artifacts/api-server/load/api-smoke.k6.js

The ramp is small on purpose (0 → 25 virtual users over ~50s). It exists to catch a broken app tier - connection-pool exhaustion, a missing index turning a read into a table scan, an auth path that falls over concurrently - not to establish a throughput number.

k6 is not in the per-PR pipeline - this is intentional

Running k6 on every PR is too slow and, more importantly, meaningless against an ephemeral CI service container. It is run manually before a release cut (or nightly against a warm backend). The same reasoning parks three other checks out of the MVP pipeline until there’s real user impact to gate against: smoke tests against an ephemeral preview (needs a per-PR preview env - adds idle cost), blue/green ECS deploy + auto-rollback (needs always-on prod traffic to mean anything), and EAS preview builds on PR (burns build minutes). Each lights up the first time it would actually protect a user.

Docs-site gates

The published docs are treated as a first-class deliverable, which means they get their own CI gates so they can neither drift from the code nor leak anything. The docs ci script (artifacts/docs-site, run by the docs-site job) chains five gates in order:

1. gen - regenerate the code-derived reference (reference/repository, reference/api, reference/data-model, reference/configuration) directly from the source.
2. check:drift - fail if the committed generated reference differs from a fresh gen (i.e. the code changed but the reference wasn’t regenerated + committed). Untracked generated files are also flagged, so a new generator’s output must be committed.
3. check:reference - every generated section exists, is non-empty, and every sidebar link resolves.
4. astro build - build the static site.
5. check:redaction - no PII, secret value, or infrastructure identifier in the built output.
6. check:links - no broken internal links in the built HTML.

The redaction gate (scripts/redaction-gate.mjs) is the load-bearing one for a public publish. It scans the built dist/ HTML/JSON/XML (not the framework/vendor bundles, which are third-party and produce false positives) against a denylist of ~10 rules: developer machine paths, 12-digit AWS account IDs, AWS resource IDs (vpc-/subnet-/sg-/…), ARNs, infra endpoints (*.rds/cache/elb.amazonaws.com), employer identity, AI-tooling references, personal/work emails, personal names/handles, and the seeded sample address. Any hit exits non-zero and fails the build. The exact same denylist is run over the marketing site (the marketing-site job), so both published surfaces are scanned by identical rules - the reason the marketing scaffold can sit in the repo before it’s cleared for a public, indexed release.

The drift gate will fail an unrelated PR if the reference is stale

If you change a route, a DB column, or a config var and don’t re-run pnpm --filter @workspace/docs-site run gen and commit the regenerated reference/* files, the drift gate fails - by design. Fix it by regenerating and committing, not by editing the generated files by hand (they’re overwritten on the next gen). See the generated reference for what these files contain.

Typecheck, supply chain, and theme

These jobs guard the shape of the codebase and run with no DB, no services, and no secrets:

• Typecheck runs pnpm typecheck from the repo root, which chains tsc --build across every workspace lib and then tsc --noEmit per artifact. Pure parse + type-resolve.
• Dependency + secret scan runs pnpm audit --audit-level=high (surfacing known CVEs) and gitleaks over the diff (catching committed secrets). pnpm audit is currently non-blocking (|| true) so a newly-disclosed upstream CVE doesn’t wedge unrelated PRs; the plan is to promote it to blocking once the baseline is clean.
• SPA theme + WCAG runs the SPA’s npm run theme:check - tsc --noEmit, then a theme-audit.mjs pass (seven token rules: HEX / RAW / RGBA / NAMED / ALPHA / SC2-NAME / SC2-IMPORT), then wcag-audit.mjs (composition contrast + dichromacy ladders). Zero failures required; this is the same gate run locally before every device build, and it exists because colour-token/contrast regressions are an accessibility (and, for evacuation UI, life-safety) defect, not cosmetics.

A theme-check gotcha for a new maintainer

tsc --noEmit on the SPA fails with TS6305 unless the composite libs (api-client-react / db / api-zod) have their dist/ built first. theme:check doesn’t chain through the workspace-root script, so CI runs pnpm run typecheck:libs immediately before it. If you run theme:check by hand and hit TS6305, run pnpm run typecheck:libs first.

Coverage expectations

The api-server suite enforces a coverage ratchet in CI (run via pnpm run test:coverage, whose thresholds live in vitest.config.ts). The thresholds sit just under the current measured baseline, so coverage can only rise, never regress. This is a ratchet, not a wish: a change that drops coverage below the committed floor fails the PR.

• The floors enforced in CI today are statements 59% · branches 50% · functions 62% · lines 61% (vitest.config.ts). These are what actually block a PR - not the target below.
• The resale target is ≥85% statements/lines, with ≥90% on route integration. This is the destination the ratchet climbs toward; it is not enforced yet - do not present it as the current gate.
• The current whole-repo floor is held below that target by genuinely wake-only or external-binary modules - live ingest (DEA / Open-Meteo / FFDI feeds), the scheduler run-loops, the terrain-wind WindNinja worker, the process entrypoints (index.ts, worker-main.ts), and the on-boot DB self-bootstrap (db/bootstrap.ts). These are excluded from coverage and validated on a production wake rather than in unit tests, because they can’t be meaningfully unit-covered cold without a live RDS or the WindNinja binary.
• The rule: raise the floors as cold-testable coverage lands; never lower them.

The coverage number being reproducible is itself a deliberate property - see Why it is deterministic above; without the pinned file order, a ratchet on a jittery number would flap.

CI never gets real secrets

A hard rule with a security rationale: tests must pass with production secrets UNSET. CI injects non-secret stand-ins - e.g. JWT_SECRET: ci-test-only-not-a-real-secret-but-long-enough and a throwaway DATABASE_URL matching docker-compose - so a leaked CI log or a compromised runner never exposes a real credential, and token replay isn’t a concern because every PR gets a fresh database.

The practical consequence for you: anything that needs a real secret cannot be a CI test. If a behaviour only works against real FIRMS, a real email provider, or real RDS, it belongs in a production wake validation, not the suite. To reproduce a “works in CI but I set the real key locally” difference, unset the key locally and re-run - that’s exactly the environment CI sees.

How to run them

The commands are intentionally the same by hand as in CI. Bring up the local Postgres/Redis first (see Run locally), then:

# Typecheck (no DB/services needed)
cd Fire-Path-AI && pnpm typecheck

# SPA theme + WCAG (build libs first - avoids TS6305)
cd Fire-Path-AI && pnpm run typecheck:libs
cd Fire-Path-AI/artifacts/firepath-ai && npm run theme:check

# api-server integration suite (needs Postgres + Redis up)
cd Fire-Path-AI
set -a && source .env && set +a
pnpm --filter @workspace/db run push
pnpm --filter @workspace/api-server run seed
pnpm --filter @workspace/api-server run test            # or: test:coverage for the ratchet

# Client pure-logic suite (no DB/services)
pnpm --filter @workspace/firepath-ai run test

# Docs gates
pnpm --filter @workspace/docs-site run ci

For the exact health-check and end-to-end verification steps see Verify; for bringing up the local Postgres/Redis the integration and load suites need see Run locally. The full CI pipeline itself is defined in .github/workflows/pr.yml.