Coding agents skip API Platform unless AGENTS.md says what to build

I maintain API Platform, so I wanted to know what a coding agent does with it. We gave agents an API to build on API Platform, starting with no project instructions, and most of them wrote Symfony controllers around the framework instead of using it. The API worked and passed the tests, but API Platform ended up as a routing table. I’ll show why they do that, and what made them use the framework as designed. Every run in this post used Claude Haiku.

Controllers around the framework

The first contracts asked for plain JSON: a bookshop and one of our products, with a fixed list envelope like {"items": [...], "total": 5} and a fixed error body. A test suite calls the API over HTTP and never reads the code.

The agents delivered, but the code is not what we would write. A typical run declares an #[ApiResource] class whose operations point to hand-written controllers, with read, deserialize, validate, write and serialize all set to false. The agent did not find these options in our documentation. It grepped vendor/ until it understood which listener does what, and then switched them all off.

I can’t really blame it. Nothing in a fresh project tells an agent how API Platform wants to be used. The behavior lives in attributes, in config/packages/api_platform.yaml and in listeners. We have bin/console debug:api-resource, but agents reach for debug:router, the command they know from Symfony. On top of that our defaults go against a plain JSON contract: the default output is JSON-LD with its own error format. For an agent that reads code, turning the serializer off is the shortest path. This isn’t specific to us: models prefer the libraries they saw most, and agents do worse on convention-heavy frameworks like Django than on a minimal one like Flask.

Can we make the framework win?

That raised the question we actually cared about: how do we improve an agent’s understanding of the framework? To try it, we first needed a task where using API Platform as designed is clearly the right answer, otherwise there is nothing to measure. So we wrote a contract that asks for what API Platform is for. Same bookshop rules, served as JSON:API 1.1 with relationships, filters and pagination, plus an OpenAPI document on /openapi.json. JSON:API is a standard that API Platform implements natively: install api-platform/json-api, enable the jsonapi format and your resources speak it.

A run is scored on 281 checks, HTTP assertions on every endpoint plus a validator for the OpenAPI document and the JSON:API schema. Before giving it to agents, we built the API ourselves the way the documentation intends, to make sure the contract can be done with API Platform. It passes every check.

Without instructions, most agents still wrote controllers.

Improving the framework knowledge

We compared 4 conditions, 3 runs each:

• no additional context, skills and plugins disabled
• the official AGENTS.md from api-platform/skillset, 2.4k characters that point to the documentation
• the official skillset, that same file plus a Claude Code plugin of 15 skills
• a targeted AGENTS.md that we wrote

We ran every condition on a patched API Platform version. Our first tries on 5.0.4 led to six fixes in API Platform, see the bugs agents found for us below. An agent used the framework as designed when its code relies on resources with state providers and processors and the native jsonapi format, which we checked in the code each agent left behind.

The official AGENTS.md sends the agent to the documentation, and in two runs out of three the agent still wrote controllers. The skillset does better, but the agent never invoked a single skill. Vercel saw the same on Next.js, their skill went unused in 56% of their evals, and a benchmark paper found that 39 of 49 public skills brought no gain. Whatever it changed came through the same AGENTS.md.

The targeted file starts with the decision:

Then it gives a recipe per extension point (provider, processor, Doctrine extension, filters), the JSON:API configuration and the pitfalls we hit while writing the reference. Note that I wrote it knowing what the contract asks for, which favours it. Still, the difference with the official file is clear: pointing to the documentation tells the agent where to learn, it doesn’t tell it what to build.

The negation in that first line is worth a note. Models follow negated instructions poorly, and as some research paper shows, it even gets worse as they scale. Anthropic’s own guide says to tell Claude what to do instead of what not to do. Our line does both: it says what to build, names what not to do and gives the reason. I suspect that “not with Symfony controllers”, right at the top, does a lot of the work, but 3 runs don’t prove it, maybe we’ll evaluate this ourselves at some point.

The bugs agents found for us

On 5.0.4, the runs that used API Platform as designed scored 270 to 278 out of 281, while the bypass runs scored the maximum. Most of the lost points were our bugs. We verified each case against the code and opened six issues, all fixed on main:

• #8747 pagination links duplicate the page parameter and next does not advance
• #8748 validation errors have no status or title, and source.pointer is not a JSON Pointer
• #8749 _id is not a valid member name
• #8750 PATCH with a mismatching data.id or data.type must answer 409
• #8751 the OpenAPI 3.0 downgrade produces an invalid document
• #8756 a JSON:API schema without readable properties gives attributes.properties: []

Four of them shipped in 5.0.5, the other two (_id and the PATCH 409) are on main for 5.1. The _id fix is opt-in, set strict_member_names: true.

With all six fixes, the framework loses no point anymore on this contract. Designed runs score 277 to 281, and what remains are the agents’ own mistakes. Two notes. These patched numbers are a re-score of the same archived apps, we did not run the agents again after the last fix. And the contract asked for OpenAPI 3.0 or 3.1 while API Platform emits 3.2 by default, so agents had to downgrade, which is where #8751 bit them. Part of that is a bias of our contract.

It still costs more

Using the framework as designed cost $1.03 to $1.61 per run (the notional cost the CLI reports), against $0.30 to $0.69 for the runs that bypassed it. That is 2 to 3 times more. A research paper measured something close on bug-fix tasks, context files added over 20% cost without raising success. Another one measured the opposite, 16.6% fewer output tokens with an AGENTS.md, so the cost question is far from settled. Code size did not move, 1,493 to 2,213 lines whatever the agent read.

The targeted file doesn’t fix that. The tokens move from discovery to building, but the total stays the same. This is our next problem: a smaller recipe, and making debug:api-resource the place an agent finds what an operation can do instead of vendor/.

What you can do today

If you use API Platform with a coding agent, first upgrade to API Platform 5.0.5, it ships four of the six JSON:API fixes. Then give it an AGENTS.md that states the decision before anything else. We proposed one in api-platform/skillset#4, a generalized version of the file we measured. Copy it in your project and adapt the format section to your API.

Everything we ran is on GitHub: the contracts, the harness, the scorer, and the code and transcript of every run, in coopTilleuls/agentic-api-bench.For the broader analysis across frameworks, see Agents read your terminal, not your docs on Tilia.What does your AGENTS.md say, does it point to the documentation or tell the agent what to build?