Oct 25, 2026

Software factories start with verification

Goal

You want a software factory or even merely hyper-fast agentic development, without devolving into a low quality, buggy product[1]. In the old days[2], I would spin off half a dozen agents, bottling lightning to produce oodles of functionality - and then end up spending much of my day testing their work.

To remove that bottleneck, we need agents to be able to check their own homework - verify their code works, from start to finish.

Compass

Traditionally, you build out lengthy, ponderous verification incrementally. Start at unit tests and work your way up towards end-to-end testing, going through the valley of integration tests and the marshes of Lean proofs[3], detouring through scenario enumeration, with a pit stop in spec driven development. All this methodical work sounds like eating your vegetables; surely it must be the right approach? Ultimately, this approach doesn’t remove the bottleneck of needing to manually test for a long, long time.

Instead, true north is: your agent should be able use your application the same way you do - if you can dogfood / smoke test anything before launch, your agent should be able to do the same. Be it your your website, or mobile apps, or your IaaS platform - get it all into your agent’s hands, ideally leveraging out-of-the-box tools such as Chrome MCP, mobile emulators, and Maestro.

Summit

Thereafter, your agent can (token intensively - aka expensively!) verify its work! It will be incomplete, until you (ask it to) discuss user stories & scenarios, both successful & failing - as well as of course races and complex scenarios.

Every feature you now launch should have the smoothest developer/smoke/user-acceptance testing known to humanity[4].

Speed

From here, you want to focus on making verification faster: detect defects earlier during the development process, and accelerating the feedback loop.

From fastest to slowest:

1. Unit tests
2. Integration tests
3. System tests: bake the agent’s manual mcp piloting into system tests, covering core business scenarios[5]
4. Deterministic simulation testing: necessary in domains with a high burden of correctness such as payments [6], databases, distributed system control planes; or your product core (verified kernels)
5. Formal methods: see footnote [3]. Apply to domains with a high burden of correctness
6. (True North) End-to-end testing

Map

In conclusion, the best way to boost agentic velocity is to focus on verification, and orient directly on the destination: end-to-end verification of your actual product. From there, keep accelerating and improving the correctness of your systems by building out the middle pieces, preferably justified by the next project that needs them.

References

• Pragmatic Engineer: Inside OpenAI’s agentic software factory
• Pragmatic Engineer: How building software is changing at Anthropic
• Toward Recursive Self-Improvement: How GLM Built Its Own Inference Infrastructure
• SpaceX AI’s Lauren Tan’s Software Factory talk
• Github: Migrating the GitHub Copilot runtime to Rust
• Antithesis: Deterministic Simulation Testing
• Anthropic & Theorem: Formal Verification
• Hillel Wayne: Why Don’t People Use Formal Methods?

[1] “slop” is the industry term

[2] the start of this year

[3] not my area of expertise, but from what I understand the challenge is guaranteeing the proof-assistant program actually corresponds to your program [3.1], as well as difficulties with modeling networking (CAP theorem, finite time guarantees, etc) [3.2]

[3.1] see both Why Don’t People Use Formal Methods? as well as recently Anthropic & Theorem’s brilliant approach with using the generating ASM as the basis, albeit with plenty of caveats - on a “single-threaded static binary with a fixed memory budget on a RISC-V CPU using Linux OS”

[3.2] see Verdi’s proof of RAFT

[4] intelligences beyond human e.g. LLMs, whales, dolphins are not sufficiently understood by this author

[5] note there will be practical tradeoffs e.g. CI time - even with a healthy parallelism factor this can result in system tests being the long pole for your CI suite.

[6] I work in payments, and spend much of my time considering race conditions & potential inconsistencies - and can now cheaply build out DSTs to consistently reproduce races.

Even better, I can ask my agent to discover races potentially tediously (with subagents etc each piloting their own mcps & tools, mimicking benign human users & adversarial botnets or fraudsters), and then record those network responses from third party services such as PSPs to feed into a DST framework.

Then, can use the whole harness of tests, including the DSTs via dependency injection, as a verifier for fixes - be it tactical bugfix or architectural rework.

A bug once upon a time only handed to your most seasoned engineers carefully analyzing for days can be fixed by just linking a slack thread or race detection log to your agent.