QA & testing

Tests run on Vitest, with a separate Playwright layer for the browser. The strategy has three tiers:

1. A fast default suite that runs against an in-memory Database fake — no real Postgres, no network, no provider keys. This is what npm test runs and what the green gate (and CI) depends on.
2. A slower integration suite (*.integration.test.ts in packages/adapters) that exercises the real adapters against a live DATABASE_URL / provider keys and skips itself when those are absent — so the gate stays green offline. It can provision its own throwaway Neon branch per run (see Integration tests).
3. An opt-in end-to-end suite (Playwright + Clerk, under e2e/) that drives the whole stack — SPA, API Worker, database, real Clerk auth — in a browser. Not part of npm test; see End-to-end tests.

Pure domain logic (packages/core, packages/contracts) is tested directly with no I/O at all.

Test taxonomy

Layer	Where	What it proves	Needs
Service / route	api/test/**.test.ts (~120 files)	Domain services + Hono routes, wired to the fake DB	nothing — fast & deterministic
Pure logic	packages/core/**.test.ts (~10), packages/contracts/**.test.ts (1)	Algorithms, validation, Zod contracts	nothing
Integration	packages/adapters/test/*.integration.test.ts (~19 files)	The real Neon/Gemini/Resend adapters: RLS, jsonb round-trips, FTS, real queries	DATABASE_URL and/or provider keys — else skipped (or NEON_API_KEY + NEON_PROJECT_ID to self-provision a branch)
Source guards	packages/adapters/test/*.guard.test.ts	Static invariants (see the jsonb guard below)	nothing
End-to-end	e2e/tests/*.spec.ts (Playwright)	The whole stack in a browser: SPA + API Worker + DB + real Clerk auth	a live stack, a Clerk dev instance, a test DB — opt-in, not in npm test

A representative offline run (npm test, no DB / keys, as of 2026-06-27 — counts drift over time):

Test Files  ~128 passed | ~22 skipped
Tests        ~679 passed |  ~68 skipped

The skipped figure is not a failure — the skipped files are exactly the *.integration.test.ts suites, gated off when no DATABASE_URL / provider key is present (see Integration tests). Point a DATABASE_URL (or NEON_API_KEY + NEON_PROJECT_ID) at a throwaway branch and those ~19 suites run instead of skip — a recent real-Neon run was 82 integration tests green, and is where the GDPR-erasure DELETE-grant bug (migration 0088) and several stale-fixture bugs were caught that the fake-DB gate could not see. The counts move as the codebase grows; treat them as a snapshot, not a target.

The fake Database

api/test/helpers/fake-database.ts exports makeFakeDatabase(): Database — an in-memory store that mirrors the real adapter's contract, including:

• Tenant scoping. forTenant(ctx) is the only entry point, exactly as in production; every in-memory row carries a tenant field and every query filters on it, so the fake exercises the same isolation discipline the real RLS enforces.
• Keyset pagination. pageBy reproduces the adapter's cursor paging (sorted by id, limit + 1 to detect more, nextCursor).
• Business logic the adapter runs in SQL. The fake re-implements the parts service code depends on: the reportTxnMatch predicate (date window, survey set, channel/agent/subscriber/dimension segmentation with AND/OR, completed-only analytics, alerted/notes toggles), the anonymity threshold suppression (ANONYMITY_THRESHOLD, default 5), the badge auto-award engine, the lead-rule engine, and the suppression-rule matcher.

Because the fake mirrors behaviour rather than re-deriving it, a service test passes here for the same reasons it passes against Neon. Where the fake intentionally simplifies (e.g. it reports all-time instead of honouring the days window for some aggregates), the divergence is commented in-line.

This is the ports & adapters payoff: services depend on the Database port, so a test swaps in the fake adapter with no production code change. The same seam fakes the other ports — tests inject a fake ai ({ completeJson: async () => … }), and email / SMS / voice / object-store ports are similarly trivial to stub. There is no vi.mock() of provider SDKs anywhere; the hexagonal boundary makes it unnecessary.

How to add a service test

A domain service takes its dependencies by constructor injection, so a test wires the fake DB (and a fake ai where needed) and calls the service directly:

import { describe, it, expect } from 'vitest';
import { makeFakeDatabase } from './helpers/fake-database';
import { VerbatimService } from '../src/domains/verbatim';

const ctx = { clientAccountId: 'acct_1', userId: 'u1', roles: [], permissions: [] } as any;

it('scores un-scored verbatims once', async () => {
  const db = makeFakeDatabase();
  const ai = { completeJson: async () => ({ results: ['positive'] }) } as any;
  const svc = new VerbatimService(db, ai);
  // …seed via db.forTenant(ctx).transaction(...), then assert on svc.analyze(ctx, surveyId)…
});

The same pattern covers route tests (build a minimal Hono app, set ports + tenant on the context, mount the router) and the ack/retry contract — runBatch takes an injected process, so the queue contract is testable with no DB at all (see Queues).

Keep the fake in sync. When you add a repository method to packages/adapters/src/repositories/<domain>.ts (assembled by src/repositories.ts) (or a new port method), add the matching stub to fake-database.ts. A missing stub surfaces as a TypeError the first time a test exercises the path — not a compile error — so add the stub in the same change as the repository method.

Integration tests (real Neon + providers)

The packages/adapters/test/*.integration.test.ts suites verify what the fake can't: that the real SQL, RLS policies, and provider calls behave. They self-gate with:

const DB_URL = process.env.DATABASE_URL;
const suite = DB_URL ? describe : describe.skip;   // skipped when unset

(provider suites gate on GEMINI_API_KEY / RESEND_API_KEY the same way). They cover the real Neon adapter (neon.integration.test.ts — tenant isolation, round-trips), admin, roles, survey config, dashboards, reporting, reports (Excel/PDF), search (Neon FTS), segments, short URLs, email templates, feature flags, response detail, verbatim + verbatim-coding, alert config, and channel sessions.

To run them, point at a throwaway Neon branch (never a real one — the suites create and delete rows) and run the gate normally:

export DATABASE_URL="postgres://…@…neon.tech/voc?sslmode=require"
npm run db:migrate    --workspace db    # the branch needs the schema first
npm test                                # the integration suites now run instead of skipping

The integration harness strips Neon's -pooler host for a direct connection and cleans up its test rows in afterAll. In CI these env vars are intentionally unset (cost + secrets), so the suites skip cleanly and the gate stays green.

Self-provisioning a throwaway branch

You don't have to hand-craft a branch and paste its URL. Neon branches are copy-on-write off a parent, so one is ready instantly. Set a Neon API key + project and the gate provisions and destroys its own branch around the run:

export NEON_API_KEY=napi_…          # a Neon API key
export NEON_PROJECT_ID=…            # the project to branch
# optional: export NEON_PARENT_BRANCH=…  # a migrated base to branch from
npm test                            # integration suites now run against a fresh, disposable branch

The Vitest global setup (packages/adapters/test/setup/neon-branch.global.mjs, wired in vitest.config.ts) creates the branch, points DATABASE_URL at it, runs an idempotent migrate, and deletes it on teardown. Without the two env vars it's a no-op — plain npm test stays unit-only. The same logic is exposed as a CLI for CI / scripts:

node db/scripts/neon-branch.mjs create [name]   # → "<branchId>\t<connectionUri>"
node db/scripts/neon-branch.mjs delete <branchId>

Proving tenant isolation

Tenant isolation has its own dedicated check, separate from the suite:

npm run db:verify-rls --workspace db

It asserts every tenant-owned table has RLS enabled + forced with a tenant_isolation policy, and that an unset tenant context is default-deny (a cross-tenant read returns zero rows, never an error). It is a release gate — see Multi-tenancy & RLS and step 2 of the deployment runbook (DEPLOYMENT.md).

End-to-end tests (Playwright + Clerk)

The integration suite proves the adapters; the E2E suite (under e2e/, run with npm run e2e) proves the product — it drives a real browser through the whole stack: the SPA, the API Worker, a database, and real Clerk auth. It is opt-in and deliberately not part of npm test, because it needs a live stack, a Clerk development instance, and a throwaway database.

Turnkey local config

playwright.config.ts loads a gitignored e2e/.env.local (copy e2e/.env.example) and sources the Clerk dev keys from the app's own files (CLERK_PUBLISHABLE_KEY ← frontend/.env, CLERK_SECRET_KEY ← api/.dev.vars), so in practice the only thing you supply is the test user. e2e/global.setup.ts calls clerkSetup() once for a Testing Token (a dev instance otherwise bot-blocks scripted sign-in).

Sign-in: use a Clerk test email

e2e/auth.setup.ts signs the user in programmatically and saves storage state under e2e/.auth/, which the authed specs reuse. Use a Clerk test email — any +clerk_test subaddress (e.g. e2e+clerk_test@yourco.com): it signs in by email code (the fixed test OTP, no real inbox), which is deterministic and sidesteps Client Trust.

Client Trust is an attack protection that returns needs_client_trust on a programmatic password sign-in from a new device — it can't be completed head-lessly, and @clerk/testing's password helper silently no-ops on it (looks like a hang). auth.setup.ts falls back to password but fails loudly with the fix (use a test email, or disable Client Trust on the dev instance).

The test user must belong to exactly one Organization (the workspace gate treats the active org as the tenant, auto-provisioned on first sign-in), and needs the in-app workspace admin (client_admin) role for the data-mutating specs — set it in Admin → Users or via an identity.member_standard_role row (role_key = 'client_admin'). RBAC specs additionally use role-scoped +clerk_test users (agent / analyst / manager); they self-skip when those aren't configured.

Fast fixtures via the API

Driving the builder UI for every fixture is slow and brittle, so e2e/helpers/api.ts seeds through the Worker using the signed-in session token: seedSurvey({ questions, theme, pageRules, publish }), submitResponse / seedNpsResponses (public capture), and apiGet/apiPost for assertions. Specs then focus their browser interactions on the surface under test. Channel flows that aren't browser-drivable (SMS/IVR conversations, inbound webhooks) are exercised at the /public API layer; email/suppression outcomes are asserted via the API/DB, not a real inbox.

Self-provisioning the database

The API Worker runs as a separate wrangler dev process that reads DATABASE_URL from api/.dev.vars (not process.env). So when NEON_API_KEY + NEON_PROJECT_ID are set, Playwright's globalSetup (e2e/neon-db.ts) — which runs before the webServer boots — creates a Neon branch, backs up and rewrites api/.dev.vars to point at it, and migrates; globalTeardown restores api/.dev.vars and deletes the branch. Without those vars it's a no-op and the run uses whatever DATABASE_URL is already in api/.dev.vars (point it at a throwaway branch — the specs mutate data).

The rewrite only takes effect on a freshly-started worker — don't reuse a stale wrangler dev. The backup is api/.dev.vars.e2e-backup; if a run is hard-killed, restore from it before re-running.

Local gotchas

• wrangler runtime vs compatibility_date. If api/wrangler.toml's compatibility_date is newer than the pinned wrangler's bundled workerd runtime, npm run dev -w @voc/api won't boot ("requires compatibility date …, but the newest supported by this server binary is …"). Run the API with npx wrangler@latest dev until the devDependency is bumped (set CLOUDFLARE_HYPERDRIVE_LOCAL_CONNECTION_STRING_HYPERDRIVE = DATABASE_URL for the Hyperdrive binding).
• Use 127.0.0.1 (not localhost) for E2E_BASE_URL / E2E_API_URL to dodge IPv4/IPv6 skew on Windows.
• Add RATE_LIMIT_DISABLED=true to api/.dev.vars: the whole suite shares one client IP, so its API-seed traffic would otherwise trip the per-IP rate limits (/v1/* 300/min, /public/surveys/* 30/min). The flag is read from the Worker env and is never set in production.

Run

npm run e2e:install        # once: install the Playwright Chromium browser
# fill e2e/.env.local: E2E_USER_EMAIL=e2e+clerk_test@…  (Clerk keys auto-sourced from frontend/.env + api/.dev.vars)
npm run e2e                # boots the API Worker + SPA, runs the specs
npm run e2e:report         # open the last HTML report

Set E2E_NO_WEBSERVER=1 to drive your own already-running stack. npm run e2e:typecheck type-checks the specs without running them. The full env-var table + the multi-role recipe live in e2e/README.md.

Test catalog

Shared journey steps live in e2e/helpers/journeys.ts; API fixtures in e2e/helpers/api.ts. Specs target role/label/data-testid (the app exposes data-keith-nav on sidebar links + data-testid="widget-type-…" on the dashboard palette). Status: ✓ implemented & green · ▢ planned.

Area	Coverage
Pipeline & auth	✓ stack + Clerk pipeline (authed dashboard / signed-out → /login / /health); ✓ +clerk_test code sign-in; ✓ multi-role sign-in (admin/agent/analyst/manager/no-org); ✓ no-org → workspace gate · ▢ sign-out, org switch
Survey authoring	✓ create→add NPS+text→publish; ✓ duplicate → (copy) draft · ▢ all question types in builder; reorder; edit; option routing; theme; template; AI generation (ai flag)
Public runtime	✓ flagship loop (build→publish→anon submit→console); ✓ single/multi/text submit + capture; ✓ required + email validation; ✓ branching reveal; ✓ rating; ✓ single-as-dropdown; ✓ multi-page Next/Back; ✓ deep-link ?a1= prefill · ▢ partial save; option-action early completion; agent/cd/rq attribution; review redirect; recording (fake media); closed/unknown survey
Responses console	✓ console loads with controls; ✓ open a response detail (?open=) shows its answer; ✓ add a note; ✓ amend an answer + edit history; ✓ CSV export · ▢ filters (survey/channel/date/state, AND/OR); in-flight drawer; funnel
Dashboards & reporting	✓ create→configure+add widget→save+reload-persist; ✓ survey-scoped NPS widget renders its score; ✓ golden-data NPS summary · ▢ external-API widget; edit/duplicate/remove; restore/delete/Overview-protected; reports + scheduled exports; executive (flag) + KPI
Alerts & leads	✓ create an alert rule; ✓ detractor response → alert (raised at capture); ✓ promoter response → lead · ▢ each rule kind; queue resolve/escalate; transform → lead detail; edit/remove
Distribution	✓ a suppressed address is not invited · ▢ invite logged + link; SMS E.164; frequency cap; send-window hold; unsubscribe; short-url /r/:slug → /s/:id
Verbatim & coaching	✓ a text answer surfaces in the verbatim feed; ✓ wordcloud reflects a distinctive word · ▢ sentiment; code a verbatim; review queue; coaching badges/wall/team
Compliance (GDPR)	✓ GDPR response erasure removes the answer; ✓ per-answer PII redaction masks in place · ▢ retention sweep; erase-subject; bulk redact; deletions log/export
Admin & account	✓ account page renders; ✓ save a workspace setting; ✓ create a segment; ✓ create an email template · ▢ invite member + role + survey scope; roles; agents; danger-zone delete; file-drop; webhooks; audit; system features/ops/cron
RBAC	✓ analyst can view responses but is denied user-admin; ✓ manager denied user-admin; ✓ agent denied the survey builder · ▢ agent own-responses scope; survey-scope lockdown; cross-tenant isolation
Cross-cutting	✓ every nav destination mounts with no runtime error; ✓ a11y (axe) smoke — no critical WCAG 2 A/AA violations on the public survey + the app shell · ▢ responsive/dark-mode; console-error-free

Several fixes on the E2E branch came from this suite: running as a normal client_admin (not a system admin) exposed a featureGate leak onto /v1/me and an over-broad dev proxy; the axe smoke caught real critical a11y violations (an unlabelled date-range select; unlabelled public-survey inputs); and the dashboard specs surfaced two builder races — a slow "New dashboard" silently discarding in-progress widget adds, and an empty dashboard's first add never marking it dirty (Save stuck disabled). E2E earns its keep by exercising the app as a non-privileged user.

Static guards

Some invariants are cheaper to assert by scanning source than by running code, so they live as *.guard.test.ts and run in the normal gate (no DB):

• jsonb-encoding.guard.test.ts — our postgres.js clients use fetch_types: false (required for Hyperdrive / PgBouncer). Under that setting, binding an already-stringified value straight to a ::jsonb cast double-encodes it — Postgres stores a jsonb string instead of the object/array, and reads coerce it to empty. The guard fails if the broken …}::jsonb pattern reappears in repositories.ts, neon.ts, or api/src/scheduled.ts (the correct form is …}::text::jsonb, or pass a raw object). This caught a real, silent corruption bug; keep the guard green.

The lint layer enforces a related invariant: voc/no-raw-sql-outside-adapters (a custom rule in eslint-rules/, set to error in eslint.config.js) keeps the raw sql\`tag confined topackages/adapters, so route/service code can never issue tenant-scoped SQL outside the withTenant` boundary. CI labels this step "the withTenant raw-SQL guardrail".

The gate

A change isn't done until the gate is green. It is:

npm run typecheck            # tsc --noEmit across all workspaces
npm run lint                 # eslint (incl. voc/no-raw-sql-outside-adapters)
npm run build --workspace frontend   # the SPA must build
npm test                     # vitest run across api/ + packages/ (+ frontend/) via root vitest.config.ts

npm test is vitest run driven by the root vitest.config.ts, which collects the project's own suites (api/**, packages/**, frontend/**) and excludes node_modules, build output, .docusaurus, and the installed Claude/agent skill templates under .claude//.agents/ (whose own bundled tests would otherwise break the gate).

Capture the real exit code — don't pipe to tail/grep, which masks it.

CI

.github/workflows/ci.yml runs the same gate on every push to new/main and on pull requests:

npm ci  →  typecheck  →  lint  →  npm test  →  npm audit --audit-level=high  →  build -w @voc/frontend

Because CI sets no DATABASE_URL or provider keys, the integration suites skip there. The plumbing to run them in CI now exists — a Vitest global setup self-provisions a disposable Neon branch from NEON_API_KEY + NEON_PROJECT_ID (see above) — so wiring the integration / RLS suite into CI is now a matter of adding those secrets, not new code. API contract tests against the published OpenAPI, the E2E Playwright run, and per-PR preview deploys remain noted as TODO in the workflow.

Coverage

There is no coverage tooling configured — no @vitest/coverage-v8/c8, no thresholds, no CI coverage gate. The deliberate trade-off is to invest in behavioural depth (the fake DB mirrors real SQL semantics; integration suites verify the real adapters) rather than chase a line-coverage number, which on a thin-route/fat-service codebase tends to over-reward trivial getters.

To take a one-off measurement:

npm i -D @vitest/coverage-v8
npx vitest run --coverage          # add --coverage.reporter=text-summary for a console summary

If a coverage gate is ever introduced, scope thresholds to packages/core and api/src/domains (the logic-bearing code), not the whole tree — adapters and routes are thin and better served by the integration + contract suites.

What is not covered (yet)

Know the gaps so you don't assume protection that isn't there:

• No frontend unit/component tests. frontend/ has zero Vitest/RTL tests today; the SPA is verified by tsc + the production build + the E2E suite + manual/visual QA. (vitest.config.ts already includes frontend/**/*.test.tsx, so adding React Testing Library tests needs no config change.)
• The E2E suite runs by hand, not in CI. The Playwright + Clerk harness (above) is green against a live stack but stays opt-in — it needs that stack + a Clerk dev instance + a DB, so it isn't part of npm test or the CI workflow yet. Depth coverage (the ▢ rows in the catalog) is an in-progress expansion. (The unrelated legacy Cypress suite in legacy/ is the old platform and is not run.)
• The RLS / integration suite is a CI TODO. db:verify-rls and the *.integration.test.ts suites run by hand / at deploy; the self-provisioning Neon-branch plumbing exists but isn't wired into CI yet.
• Provider integration suites only run with keys. Gemini/Resend behaviour is exercised locally when keys are present, not in CI.

Conventions

• Tests must run without network or DB by default. If a new test needs Postgres or a provider key, it belongs in an *.integration.test.ts file with the describe.skip gate — never in the default suite.
• File naming: *.service.test.ts / *.route.test.ts (unit), *.integration.test.ts (real adapter, gated), *.guard.test.ts (source scan), *.spec.ts under e2e/ (Playwright, opt-in).
• Mirror, don't mock. Inject the fake DB / fake ai; don't vi.mock provider SDKs.
• Anonymity & isolation are tested at both layers — the fake mirrors the anonymity-threshold suppression and tenant scoping; the integration suite confirms the real RLS enforces them. See Contributing for the non-negotiables these tests defend.