Home / Work / The Client's Own Test Gate for an Agency-Built Product

The Client's Own Test Gate for an Agency-Built Product

Internal Tooling ยท 2026

An agency builds our B2B platform in milestones. Its tests were no pass/fail gate, so as the client I built my own: a contract-based test matrix, manual and automated rounds, and a regression suite we own.

  • Manual + automatedGate
  • Client-ownedRegression suite
  • Playwright
  • TypeScript
  • Acceptance Testing
  • Product Ownership
  • Vendor Management
  • QA
  • Claude Code
The Client's Own Test Gate for an Agency-Built Product

What happened

The product is a closed B2B lead platform with prepaid credits, built by an external agency in contracted milestones. The agency had its own unit tests, feature tests and a screenshot harness, but by its own description none of that was a pass/fail gate. Nothing answered two questions:

As the client-side product owner I needed my own answer, independent of the builder. Under the contract, acceptance is given in writing within a fixed window, and only critical defects can block it. So the gate had to be precise about what counts as critical, and fair to the vendor.

How I built it

Scope from the contract, behaviour from the app. The yardstick was the statement of work's core journey: create an account, log in, claim an organisation, set services and working area, invite a team member, see the right screen per role. That became the spine test. Around it sits a requirements matrix per milestone, with a large share of negative cases (what a role must not reach). The rules: the contract decides scope, the running app decides behaviour, and an app that contradicts a contractual criterion has a product defect. Features from later milestones that already ran on dev were deliberately kept out. A test plan was also checked against the signed contract line by line, with one agent per contract line plus a completeness check.

A black-box Playwright suite in TypeScript against the deployed dev environment. It never imports product code.

Authorisation probes without secrets. The suite replays the app's own authenticated request against paths a role shouldn't reach and asserts the denial, without ever reading the token.

Manual acceptance alongside it. A shared checklist with one column per tester, with a scale fixed in advance: accepted, minor defect, critical defect, not applicable. Claude drove the browser and filled in only my column. Later rounds bring in each team that will use the product, each with its own test data.

A regression suite the client owns. Each accepted milestone's tests become the safety net for the next one. A baseline run before delivery means a red test after delivery is a diff, not an opinion.

Findings written as symptom and expected behaviour. Feedback goes into one ticket epic per milestone, not into mail, and the agency starts on it once I mark it final.

An independent review of the suite itself. A separate agent reviewed the suite and its report read-only, to answer one question: is a green run trustworthy as a gate?

Outcomes

What I learned

Coverage you can trust beats coverage that looks good. A test may only claim a requirement it actually asserts. Enforcing that lowered our coverage numbers, which was the point.

A green suite can still say nothing. The independent review found tests with no working check, a purchase test that sat outside the gate, and missing test data that skipped tests instead of failing them. The element inventory then showed the suite covered far less of the product than its green runs suggested. Having a second agent audit the first agent's suite caught what the author would not.

A baseline before delivery turns opinions into diffs. Without it, red tests after delivery would have looked like new bugs. With it, most of the baseline's failures could be traced to a fix that arrived early, and the suite followed it.

Known defects belong in the suite as expected failures. That keeps the run green enough to spot new breakage, and nobody has to remember which red was "the old one".

Shared dev environments fight you. The identity provider rate-limits sign-ins across the whole tenant, so the suite signs in once per test user. Leftover data from earlier runs means tests never rely on row counts. Picking a company triggers a paid lookup, so those tests run on an agreed budget. Buying tests spend test credits, and once a test company runs out, they stop. Hence the ask for a data reset the client can trigger itself.

AI has hard limits on credentials, and that's right. Claude won't type passwords or create accounts on a non-local site, even a clearly labelled dev site and even when asked. We lost time asking. What worked: I log in, then Claude takes over the session, or the suite uses its own test users, with secrets shared through a password vault and never in chat, code or mail.

Read the code before you file the finding. Checking the source showed that two of our own findings came from decisions we had made earlier, not from vendor mistakes.

A fast-moving UI makes black-box tests a maintenance job. Early on, nearly all failures were the suite lagging behind a changing UI, not product bugs. Budget for it, or the suite starts reporting its own drift as product bugs.

How I did it

Solo with Claude Code for the suite, test plans, contract checks and the test rounds, and joint manual sessions with leadership and the departments that will use the product for the human verdict. Agents did the volume (one per contract line, separate verifiers for every failure); I made the calls on what is a defect, what is a decision and what goes to the vendor. Every finding was written up as a functional ticket (symptom and expected behaviour, no guesses at the cause), every change to the suite went through review, and every acceptance decision with its conditions went into the decision log.