Spikes halved our bug escape rate

When Anthropic dropped Fable 5, I was excited to give it a try, and like a lot of people, I was very impressed with what it could do. One of the things I loved was that it would write spikes when planning out code it was going to implement. That habit alone seemed to be a big part of why it got better results.

I started experimenting with spikes on non-Fable models (especially since Fable became unavailable almost immediately after launch). Both Opus 4 and Sonnet 4 would write spikes when asked, and it improved their results too.

For sugarmaple.ai, we use a heavily opinionated, Spec Kit-driven process, so on July 12th we added spikes to it. The change simply mandated that during the planning phase, the agent should write spikes to fill knowledge gaps around libraries and APIs. The spikes are committed as part of the plan, so the implementing agent has them as a reference.

Across the roughly 90 features that have gone through spec and planning so far, I’ve been tracking how often something got past that stage and turned up later as a defect or a spec gap nobody caught in time.

What the July change did

Since that change, by my read of the numbers, the escape rate has roughly halved. Here’s what that means, and where it came from.

First, what I counted. I went back through every feature from 001 through 101, 90 feature directories in all, and pulled out the cases where the spec or plan itself was wrong: not a correct plan implemented badly, but a plan that asserted something false or left something out. There were 47. An escape is one of those defects that got past the start of implementation without anyone noticing, and surfaced later in real use, in the hands of the next feature’s author, or in an audit. A defect caught mid-implementation, before merge, isn’t an escape: it cost us some rework, but it never reached anyone.

I split the features at 036-explicit-actor-jwt, the first plan where a spike shows up, and charged each defect to the feature whose plan was actually wrong, not the one that happened to find it. Before spikes, 31 features produced 16 escaped defects, an escape rate of 52% per feature. After, 59 features produced 16 escapes, or 27%. Defects found in real use fell from 35% of features to 15%.

Put another way, we shipped nearly twice as many features in the second window, 59 against 31, and exactly the same number of defects escaped: 16 and 16. Nearly doubling throughput without doubling what reaches users is the whole point of a better planning phase, and spikes delivered it.

The defects didn’t vanish; they got caught earlier. The share of features with a defect caught at plan time, in clarify, plan review, research or a spike, grew from 3% to 8%. The share caught mid-implementation grew from 3% to 14%. That second number is spikes paying off twice. Because the spikes are committed with the plan, the implementing agent has working code against the real library or API to compare with, and when reality doesn’t match the plan it notices during the build instead of after the merge.

In 064-kafka-client-migration, the plan assumed a replacement Kafka client was installable. A spike found there was no install path on Node 26, and the migration was deferred before anyone wrote a line of code against it. Before spikes, that assumption would have gone straight into tasks, and we’d have found out halfway through the build, or in production. A spike is a small piece of throwaway code, and it turns an assumption into a fact before anything gets built on top of it.