An engineering audit of what your agents actually shipped
Stefan-Iulian Tesoi · · 6 min read
Compare three lists for the same period: the items marked done, the commits merged, and the behaviour a user can now observe. Anything appearing in one list and not the others is the finding. Unreported work turns up about as often as overreported work, which is the part most teams do not expect.
An engineering audit of delivery is not a performance review and not a metrics exercise. It is a reconciliation, of the kind an accountant would recognise, between three records that are all supposed to describe the same month.
What does an engineering audit of delivery answer?
One question: what did the team ship, stated so that someone who was not there could check it. Not how much, not how fast, and not whether anyone worked hard.
The question got harder when agents started writing the code, for an unglamorous reason. A developer who finishes something tells someone, and the telling is what kept the tracker roughly honest. A coding agent finishes something and updates whatever it was told to update, which may be one system, or none. The work is real either way; the record is now optional in a way it never quite was.
A tracker was never the source of truth. It was a summary that a person kept synchronised by talking. Remove the talking and what remains is a summary of whatever the last integration happened to write.
Which three lists do you compare?
Three, because any two of them can agree and still be wrong together.
- Items marked done, for the period, with their closing dates. This is the claim.
- Commits merged, from the Git history of every repository the team touched, merges excluded. This is what actually changed.
- Observable behaviour, from the deployed product: the feature exists, the endpoint answers, the bug no longer reproduces. This is the only list a customer would recognise.
The third is the one teams skip, and skipping it turns the exercise into delivery reporting about throughput. Merged is not deployed, and deployed is not working. A release sitting behind a feature flag nobody enabled is present in the second list, absent from the third, and indistinguishable from success in the first.
What does each kind of gap mean?
Each combination has a usual cause, and the causes need different responses.
| Present in | Absent from | Usually means |
|---|---|---|
| Backlog | Git, product | The item was closed on a claim nobody checked |
| Git | Backlog, product | Work happened off the record, or on a branch that never shipped |
| Git, product | Backlog | Real delivery with no item, invisible to any roadmap |
| Backlog, Git | Product | Merged but not released, or released behind a flag |
The third row surprises people, because every instinct says the risk runs the other way. Overreporting is the failure everyone watches for. Underreporting is the one that quietly makes a roadmap useless, since a milestone's progress is derived from items, and work with no item contributes nothing to it however much of the product it built.
Row one is the divergence covered in why done and the repository disagree. Rows two and three are a different problem, and the rest of this is about them.
What about work that shipped without an item?
It is the most common finding, and this blog is the example closest to hand.
The site has 27 published posts. The epic they are supposed to hang from has four items, all filed in the last three days, by an agent following a skill that tells it to file one. Twenty-three posts were written, given hero images, validated, built and deployed with nothing in the backlog to show for them. None of that work is missing from Git. It is missing from the record anyone would consult to answer what shipped, which is the only record a roadmap can read.
The honest reading is not that someone was careless. Filing the item is a separate action from doing the work, it is unenforced, and the post goes live whether or not it happens. Any step in that shape decays. The fix is to make the record a by-product of the work rather than a chore after it, which is what a skill step or a commit convention buys, and what a backlog an agent can read sets out at the item level.
Laimonade records when a commit rather than a person moved an item, keeping the commit, its author and a match confidence against the item, so a status that looks wrong can be traced to what caused it instead of argued about. Two of those four carry no such record, which says a person moved them: a small thing, and worth being able to know for certain. What the system does with that is in how Laimonade works, and the view a lead wants from it is in Laimonade for engineering leaders.
How do you make the audit cheap enough to repeat?
Keep it to an hour a month and accept a rough answer. An audit that takes a day gets scheduled quarterly, then skipped, then abandoned.
An hour is enough when the three lists are pulled mechanically:
- Export closed items for the period from the tracker, title and date only.
- Pull commit subjects from each repository for the same window. This takes seconds and is the list nobody disputes.
- Diff the two by eye. At a normal team's volume that is 40 to 80 items against 200 to 400 commits, and the mismatches stand out without tooling.
- Spot-check the product for the five largest items only. Checking everything is what makes this expensive, and the large items are where a flag-shaped gap hides.
- Write down the causes, not the count. "Three items closed on unverified claims" is a number. "Closed by whoever ran the deploy, without opening the item" is a thing to change.
The standard each closed item should have met is the subject of definition of done when an agent wrote the code. An audit that keeps finding the same gap is reporting that the definition is not being applied, rather than that the team is careless, and a month of causes is usually enough to see which single step accounts for most of them.
Frequently asked questions
How long does a monthly delivery audit take?
About an hour once the exports are scripted, for a team of five to ten. The first one takes two or three hours, because nobody has pulled the three lists side by side before and the gaps found are accumulated drift rather than one month's worth. Later runs compare against a known state and get faster.
What do you do about work with no backlog item?
File it after the fact, dated when it shipped, then fix the step that let it happen. Back-filling alone is a treadmill and the same gap reappears next month. The durable change is making the item a by-product of doing the work — a commit convention, a skill step, a pre-merge check — so nobody has to remember a separate action.
Can this replace a retrospective?
No, and they answer different questions. A retrospective asks how the work felt and what to try next. An audit asks whether the record of what shipped is true. It is more useful as an input to the retrospective than as a substitute, because it replaces impressions of a slow month with a list of what actually merged.
Is this just engineering metrics under another name?
No. Engineering metrics measure rates — deployment frequency, lead time, change failure rate — and assume the underlying records are accurate. An audit tests that assumption. Running metrics against a tracker where a quarter of the work has no item produces confident numbers about a fraction of the delivery, which is worse than no numbers.