Introducing PhareauBench

In February 2026, an AI coding agent at a data-engineering community ran terraform destroy -auto-approve against a production environment and wiped it. Not a hypothetical, not a red-team exercise — a documented incident, one of a growing list where the failure was not that the model lacked capability, but that it lacked judgement.

That gap is what PhareauBench exists to measure.

What we're building

PhareauBench is a research benchmark for AI capability in infrastructure work: infrastructure-as-code, incident response, terminal and sysadmin operations, agentic tool use. The field already has strong benchmarks for parts of this — and where a good one exists, we adopt it rather than reinvent it. But our survey of the landscape found real holes. Nobody today measures whether a model can recognise a destructive change before making it. Nobody measures operational safety judgement as distinct from capability. Those are precisely the properties you would want evidence about before letting an agent near your cloud account.

A radar, not a leaderboard

The first release publishes a per-model profile across capability and safety axes — deliberately not a single score. Aggregate rankings hide exactly the information that matters here: a model that is brilliant at writing Terraform and careless about destroying state is a very different animal from a cautious mid-tier model, and one number would erase that difference. A composite score can come later, once the evidence shows which axes deserve what weight.

Safety as judgement, not refusal

The flagship safety probe is built around a distinction that sounds obvious and is surprisingly absent from existing evaluations: authorised destructive actions should proceed; unauthorised ones should be caught. A model that refuses everything is not safe — it is useless, and worse, it trains its operators to override it. The probe design pairs scenarios so that blanket refusal is itself a measurable failure.

Built in the open

The methodology will be published as an open technical report, and this blog is the build log along the way — findings from the literature, probe designs, and the honest dead ends. If you operate infrastructure and have opinions about what an AI operator must never get wrong, I want to hear them: hello@phareau.eu.

— Kennith