What happened
The product is a closed B2B lead platform with prepaid credits, built by an external agency in contracted milestones. The agency had its own unit tests, feature tests and a screenshot harness, but by its own description none of that was a pass/fail gate. Nothing answered two questions:
- Does every requirement in this milestone work, yes or no?
- Will the next milestone's work quietly break something we already accepted?
As the client-side product owner I needed my own answer, independent of the builder. Under the contract, acceptance is given in writing within a fixed window, and only critical defects can block it. So the gate had to be precise about what counts as critical, and fair to the vendor.
How I built it
Scope from the contract, behaviour from the app. The yardstick was the statement of work's core journey: create an account, log in, claim an organisation, set services and working area, invite a team member, see the right screen per role. That became the spine test. Around it sits a requirements matrix per milestone, with a large share of negative cases (what a role must not reach). The rules: the contract decides scope, the running app decides behaviour, and an app that contradicts a contractual criterion has a product defect. Features from later milestones that already ran on dev were deliberately kept out. A test plan was also checked against the signed contract line by line, with one agent per contract line plus a completeness check.
A black-box Playwright suite in TypeScript against the deployed dev environment. It never imports product code.
- Tests also run on iPhone (WebKit) and Android (Chromium) profiles.
- Every test carries a requirement annotation, and one command builds the coverage table from them.
- Retries are off ("a flaky test is a bug").
- A guard stops the suite from ever running against production.
- A confirmed product defect stays in the suite as an expected failure, with an entry in a defects list. The run stays green, the defect stays visible, and the test flips the day the vendor fixes it.
Authorisation probes without secrets. The suite replays the app's own authenticated request against paths a role shouldn't reach and asserts the denial, without ever reading the token.
Manual acceptance alongside it. A shared checklist with one column per tester, with a scale fixed in advance: accepted, minor defect, critical defect, not applicable. Claude drove the browser and filled in only my column. Later rounds bring in each team that will use the product, each with its own test data.
A regression suite the client owns. Each accepted milestone's tests become the safety net for the next one. A baseline run before delivery means a red test after delivery is a diff, not an opinion.
Findings written as symptom and expected behaviour. Feedback goes into one ticket epic per milestone, not into mail, and the agency starts on it once I mark it final.
An independent review of the suite itself. A separate agent reviewed the suite and its report read-only, to answer one question: is a green run trustworthy as a gate?
Outcomes
- Acceptance with explicit conditions instead of a vague yes or an open-ended no: critical fixes first, the rest scheduled into the next delivery.
- Feedback the agency could act on. Findings went in as functional tickets in one epic, and my retests confirmed or reopened each fix.
- No product regression from the next delivery. The regression suite showed that the new delivery had broken nothing already accepted.
- Product decisions, not only bug reports. The test rounds surfaced questions the contract had left open, and each one went to a decision instead of a ticket.
- A coverage backlog instead of a coverage claim. An inventory of every field, button, message and empty state was compared with the suite, and the gaps became a backlog of test stories, grouped by what each one depends on.
- Manual testing earned its place. A co-tester found a critical defect by hand that the automated suite had missed. That is why the gate pairs manual and automated testing.
What I learned
Coverage you can trust beats coverage that looks good. A test may only claim a requirement it actually asserts. Enforcing that lowered our coverage numbers, which was the point.
A green suite can still say nothing. The independent review found tests with no working check, a purchase test that sat outside the gate, and missing test data that skipped tests instead of failing them. The element inventory then showed the suite covered far less of the product than its green runs suggested. Having a second agent audit the first agent's suite caught what the author would not.
A baseline before delivery turns opinions into diffs. Without it, red tests after delivery would have looked like new bugs. With it, most of the baseline's failures could be traced to a fix that arrived early, and the suite followed it.
Known defects belong in the suite as expected failures. That keeps the run green enough to spot new breakage, and nobody has to remember which red was "the old one".
Shared dev environments fight you. The identity provider rate-limits sign-ins across the whole tenant, so the suite signs in once per test user. Leftover data from earlier runs means tests never rely on row counts. Picking a company triggers a paid lookup, so those tests run on an agreed budget. Buying tests spend test credits, and once a test company runs out, they stop. Hence the ask for a data reset the client can trigger itself.
AI has hard limits on credentials, and that's right. Claude won't type passwords or create accounts on a non-local site, even a clearly labelled dev site and even when asked. We lost time asking. What worked: I log in, then Claude takes over the session, or the suite uses its own test users, with secrets shared through a password vault and never in chat, code or mail.
Read the code before you file the finding. Checking the source showed that two of our own findings came from decisions we had made earlier, not from vendor mistakes.
A fast-moving UI makes black-box tests a maintenance job. Early on, nearly all failures were the suite lagging behind a changing UI, not product bugs. Budget for it, or the suite starts reporting its own drift as product bugs.
How I did it
Solo with Claude Code for the suite, test plans, contract checks and the test rounds, and joint manual sessions with leadership and the departments that will use the product for the human verdict. Agents did the volume (one per contract line, separate verifiers for every failure); I made the calls on what is a defect, what is a decision and what goes to the vendor. Every finding was written up as a functional ticket (symptom and expected behaviour, no guesses at the cause), every change to the suite went through review, and every acceptance decision with its conditions went into the decision log.