The problem
A website rebuild was being delivered by an external vendor, tracked in Jira with code in GitLab. Keeping the CEO informed meant somebody manually assembling status every week: slow, subjective, and — the real issue — easy to spin. A human writing their own project's status report is not a neutral narrator.
I built a system that reads Jira, GitLab, Slack, Confluence and Sentry, computes a metrics snapshot, writes the prose over that snapshot, and emails the report to me and the CEO. Since August 4th that runs fully automatically, with no human reading the prose before it lands in the CEO's inbox.
Which is a genuinely uncomfortable thing to ship. Most of the design is about earning it.
Compute and narrate, strictly separated
The single most important decision: the LLM may not compute anything.
Plain Python calculates every number — 274 pytest tests, socket-blocked so nothing can accidentally reach a live API during a test run. The model receives a fixed brief and writes prose over it. It may not compute, round, or estimate.
Then an audit layer extracts every number from the generated prose and rejects anything not present in the brief, with a single named-problem retry.
This caught a real fabrication before it shipped: an "8-week median" with no source anywhere in the brief. Plausible, well-written, entirely invented. That single catch is the whole argument for the layer.
No API key exists anywhere
The weekly run is a scheduled cloud coding routine — the assistant is the narrator under the subscription, so there's no API key in the system at all. Slack I/O goes through the routine's connector. The only mail credential lives in a dispatch-only GitHub workflow with no schedule trigger, and tests assert both of those properties rather than trusting them.
Honest metrics by design
- Forecasts come from measured throughput percentiles, never vendor estimate hours
- Scope is tracked in three separate buckets, because Jira status genuinely cannot distinguish approved from unapproved work
- The headline number cross-checks Jira-closed against GitLab-merged — one source can be gamed, agreement between two is harder
- RAG status is rules-based, never a judgement call
Outcomes
- Live weekly reports to the CEO, fully automated, with a promised-vs-delivered record per week usable for vendor invoice reconciliation
- The prose audit caught a fabricated statistic before it shipped
- Surfaced that 12 PRDs/SRSs had been waiting 40 working days on our own side for sign-off — while the previous, human-written reporting claimed only 3 tickets needed decisions
- Being extended into a generic transactional-mail layer with a designed HTML email template
That third bullet is the point of the whole project. The bottleneck was on our side, and the manual report had never once said so.
What we learned
The live project beats the plan. The real Jira board contradicted my assumptions repeatedly — team-managed project quirks, nobody logging time, 34 issues closed as Won't Do, a 40-bug bulk-close skewing throughput. Every rule had to bend to observed data.
A prompt alone doesn't hold for autonomous runs. The first routine run committed its snapshot to a working branch instead of main — correct, green, and completely invisible to the next run. The fix was prevent / detect / recover layers, not better prompting. Autonomy fails quietly, not loudly.
Green tests are not done. A post-build audit found three defects no test caught. The worst: the three scope buckets overlapped, so the forecast blended approved and awaiting-decision work — defeating the exact design decision meant to prevent that.
Number-auditing LLM prose works, but is weakest on single digits. That's an acknowledged, documented limit rather than a hidden one.
Cost-consciousness as a first-class requirement: a cheap model for narration, trimmed briefs, and tables rendered in Python instead of spent as tokens.
How it was built
Solo, in 5 stage-gated stages: the maths validated against fixtures first, live APIs only later. At each gate I reviewed real numbers, not code. Then a defect audit after "done", and the production run handed to a scheduled cloud routine whose prompt (ROUTINE.md) lives in the repo like code.
History is git. Every weekly snapshot and the exact prose that was sent are committed, never regenerated — so "what did we tell the CEO in week 3" has one answer.