The end of end-to-end testing at Sugarmaple

Sugarmaple has always used SDD from the very first commit. And that first commit used principles in its constitution that I’ve been using from very early on in my agentic software development. And every spec that had more than one user story was creating mandated end-to-end tests. For the first 15, 20, maybe even 30 features, this worked really well. However, it stopped working well once the end-to-end test inventory got above 300 and both flakiness and total execution time increased.

As my frustration with end-to-end tests started to increase, I realized that contract tests and end-to-end tests are testing the same thing in different ways. If you’ve got a sequence of transactions that form your end-to-end test, then each of those transactions has an input and output contract that could be tested independently of the combined test.

The irony is that while the SDD constitution mandated TDD and BDD testing methodologies from the start, contract-driven development was one of the more foundational architectural principles of the system. Every service was defined in terms of its OpenAPI contract and its Kafka schemas. Contract testing should have been the third leg of the testing system from the start.

What PR #176 did last week

Last week’s PR (#176) took the browser suite from 353 scenarios to 69. The in-cluster infra suite went from 22 to 12.

First we did an audit of tests that were already redundant. This allowed us to get a jump start by just deleting tests that weren’t adding value.

We had already defined a subset of the tests, about a sixth of them, as the smoke test set. We did remove some of these, but this was the starting point: every other test was eligible for deletion.

But we didn’t just delete them. For every remaining test we evaluated whether it could be decomposed cleanly into contract tests. If it could, we wrote the contract tests and deleted the end-to-end test in the same commit with the mapping in the commit message. The secondary goal was to get the tests passing in under 20 minutes (including 10 minutes to set up the cluster in CI).

In the week before this change we implemented only three or four SDD runs, but the following week the number jumped to 15. Both the speed and the reliability were a productivity unlock, without a loss of release confidence.

Going forward, a new scenario gets admitted only if it proves something observable solely through the real public surface, and only if no existing journey can absorb it as one more assertion. We plan to hold the suite at under 70 rather than let it grow back.