sandbox-conformance

Which sandbox actually
holds the line?

Every sandbox for an AI coding agent makes claims — it can't read your keys, can't reach the network, can't plant a git hook. This lab turns those claims into contracts anyone can run, and measures, guarantee by guarantee, which sandbox holds it and which doesn't.

how a result means something

Attack, control, effect

A contract has three parts, so a result is never just "it returned an error". A sandbox passes only when it stops the attack and the ordinary work still runs.

attack
something the agent must not be able to do. It has to work without a sandbox first, or the result is vacuous and proves nothing.
control
legitimate work that must keep working. A sandbox that blocks the attack by breaking the control fails the contract.
effect
what is checked on disk or on the wire afterwards: a protected file is unchanged, a planted marker is absent.
attack grep -q hunter2 ~/.netrc control echo ok > work/f.txt effect ~/.netrc unchanged · secret absent
the measurement

Guarantee by guarantee

Five sandboxes, measured on one Linux host (kernel 6.8, Landlock ABI 4) against the full contract set. Filter by category, search a contract, or show only where the sandboxes disagree.

* no policy field asks for this guarantee: the result reflects the sandbox's defaults. ‡ the contract counts this as an attack, but on review it is not a breach. § depends on a newer kernel than this host's. Hover a marked cell for the review note; the notes explain a result, they never change it (all notes).

reading the five

What each one is, and isn't

Counts are not a ranking. A capability a sandbox doesn't express is a gap, not a vulnerability; a difference of defaults is not a bug.

the rules of the lab

How to trust a number

Unsupported ≠ weak

When a sandbox can't express a contract's policy, the cell reads unsupported — a capability gap, never a vulnerability. nono and landrun are allow-only, so they show many.

Defaults ≠ bugs

Many escapes are on guarantees no policy field asked for. They reflect a sandbox's defaults, not a configured rule that broke.

Evidence, not claims

Every result ships with its raw output and host stamp, and the lab tests itself against reference sandboxes whose behaviour is known. A finding under coordinated disclosure is not shown at all until the vendor authorizes it.

Maintained by the author of TartarOS. TartarOS is where I try each escape the lab finds, to see whether it can be mitigated, so it is built against these contracts: read its column with that in mind. That is also why the contracts come from the other projects' public test suites. TartarOS is measured here since 2026-10-22, with the same corpus, host and runner as the others, and its result file records the conflict of interest. Contracts and adapters are welcome: see CONTRIBUTING.md. Reports sent to vendors' security programs (2026-10-27): sandbox-runtime 4, one still open; codex 2, one still open; nono and landrun none. An open report's details stay unpublished until it is fixed. Every contract credits its upstream source.