Proof
Evidence, not reviews.
Real tasks on real code.
18 Oct 2026: 14 tasks taken from open-source projects (Hugo, Gson, jsoup, Pydantic, SQLAlchemy, Scrapy, pytest, ESLint, Hono and the GitHub CLI). Each was checked afterwards against tests the AI never saw. 290 minutes in all, $16.60 of AI time.
- 3
- delivered, all correct
- 0
- wrong changes delivered
- 5
- correct, but held back by our checks
- 6
- not solved or didn't finish
| Task | Result | Minutes |
|---|---|---|
| Hugo: relative partials (feature, Go) | Delivered, correct | 73 |
| Gson: unpaired surrogates (fix, Java) | Delivered, correct | 4 |
| SQLAlchemy: in-place mutation (fix, Python) | Delivered, correct | 13 |
| ESLint: new-cap globals (feature, JavaScript) | Correct, held back | 73 |
| Pydantic: create_model namespace (feature, Python) | Correct, held back | 25 |
| jsoup: cyclic append (fix, Java) | Correct, held back | 32 |
| Pydantic: NameEmail newline (fix, Python) | Correct, held back | 11 |
| Team board API rescue (fix, JavaScript) | Correct, held back | 5 |
| Scrapy: generated requests (feature, Python) | Not solved | 48 |
| GitHub CLI: clone over SSH (fix, Go) | Not solved | 43 |
| pytest: walrus rewrite (fix, Python) | Not solved | 45 |
| Hono: strict slash (fix, TypeScript) | Not solved | 69 |
| Bookshelf API rescue (fix, Python) | Not solved | 9 |
| Gson: character offset (feature, Java) | Didn't finish | 3 |
What it means: when our checks let a change through, it was right every time in this run. They are still too strict, so five correct changes were thrown away. That is the next thing we're fixing.
What we put the system itself through.
Measured on 3 October 2026, on our own machines.
When parts of it break
We broke our own system on purpose, one part at a time, while it worked, and measured how it came back.
6 s
of failed database writes when the main database server was deleted; a replica took over
16 s
for every AI worker to be replaced after all of them were killed at once
~4 s
for tool calls to work again after a minute-long outage of the gateway every agent uses
How: A probe made a tool call and a database write every second, through each fault and after it.
When everything is lost
Backups only count if they restore. We restored the whole database from them into a new server and compared.
211 s
from asking for the restore to a database answering
0
rows lost: every table matched, 79,100 tool calls and 34,606 agent steps included
How: A nightly backup plus a continuous log of every change since. Under load, at most the last 5 minutes could be at risk.
When it breaks in production
A sample shop runs beside us. When its alerts fire, our operations agents investigate and repair it, within strict limits.
3 of 3
injected incidents found and repaired end to end
How: Each repair goes through the same policy gateway as every agent action, is checked afterwards, and is undone if the service doesn't recover.
When someone attacks it
We scanned every library we ship for known vulnerabilities and tested the walls around the sandbox where AI-written code runs.
170 → 2
known vulnerabilities in our own dependencies, none critical or high left
0
internal services or internet hosts reachable from the sandbox, beyond the package cache and its own storage
How: Dependencies checked against the OSV database; the sandbox's network walls tested from inside it.
What this doesn't prove
These are our own tests on our own systems, not an outside audit, and the tasks are smaller than most real apps. No customer has used Warrantee yet. Backups still live on the same machines as the system they protect until we move to the cloud. Every new run replaces the numbers above, with its date.