More tasks does not mean more coverage. That's the lesson from round five of the dappa planner grand prix, where the cheapest model in the field produced the longest task list and the worst test results. If you're picking an AI planner by price alone, this will change your thinking.

The setup

Six frontier models were each handed the same brief: plan a Supabase database build. Same schema, same constraints, same output format. Each plan then fed into a live dappa autonomous coding loop, and every run hit a real pgTAP gate - no manual checking, no eyeballing the output. Pass or fail, the gate decides.

The metric that matters isn't how many tasks the planner produced. It's how many pgTAP tests passed at the end of the run, and what the whole thing cost.

What GLM-5.2 did

GLM-5.2 came in at 2.6 cents all-in. It produced 8 tasks - the most of any model in the field. On paper that looks thorough. In practice, only 43 tests passed the pgTAP gate. That's the worst result of the round.

The problem isn't that the plan was short. The problem is that task count and task depth are different things. GLM-5.2 sliced the work into more pieces, but each piece demanded less from the coder. Fewer assertions per task means fewer tests get written, and fewer tests means the gate has less to verify. You end up with a longer to-do list and a thinner codebase.

This is the trap: a plan that looks detailed because it has many steps can still be shallow if none of the steps say “and prove it works like this.”

Luna still wins

Luna (the model that won rounds one through four) ran 5 tasks, cost 92 cents all-in, and passed 101 tests. That's the best test count in the field and the best tests-per-dollar ratio by a wide margin.

For comparison, running Fable end-to-end - a heavier, more expensive path - produced 63 passing tests at $7.41. Luna beat that by 8x on cost and still came out ahead on coverage. The gap isn't close.

Five tasks, each one demanding real assertions, beats eight tasks that dont ask the coder to prove anything.

How to read a plan before you run it

The number to look at isn't task count. It's what each task requires the coder to demonstrate. A well-written task will name the function, the expected return, and the failure condition. Something like:

Task: create_reservation(animal_id, customer_id, deposit_paid)
- Returns reservation row with status = 'pending'
- Raises exception if animal_id not in stock
- pgTAP: ok( create_reservation(...), 'reservation created' )
         throws_ok( $$ SELECT create_reservation(-1, 1, true) $$, 'animal not found' )

A shallow task from a cheap planner looks like this instead:

Task: add reservation logic to the database

Same position in the list. Completely different demand on the coder. The second version will produce whatever the coder feels like writing, which is usually the happy path and nothing else.

When you're reviewing a plan, count the assertions, not the tasks. If a task has no named failure condition, it will probably produce no test for that condition.

The set -u crash

Running this round also exposed a bug in dappa itself. The loop was using set -u in its shell wrapper, which treats any unset variable as an error and exits. One of the cheaper models returned a plan with a field dappa wasn't expecting, the variable lookup failed, and the whole run crashed before the gate could fire.

The fix was straightforward - default the variable before the check - but it's worth noting because it only surfaced under pressure from an unusual plan shape. Thats the value of running diverse models against the same build: they stress-test the tooling as well as the plan.

The practical rule

Spend on the plan. The planner is the cheapest part of the loop by a large margin even when you pick the most expensive one. The coder runs many more tokens than the planner, and the gate runs on your own infrastructure. Saving 89 cents on the planner to get 58 fewer passing tests is not a trade worth making.

  • Judge a plan by what each task demands, not how many tasks there are.
  • Count named assertions per task before you commit to a run.
  • A gate that fires on every run is the only honest measure - gut-feel code review is not a substitute.
  • Unusual plan shapes will find bugs in your tooling. Run diverse models and fix what breaks.

Luna at 92 cents all-in, 101 passing tests, five focused tasks. That's the benchmark. Everything else in round five is measuring itself against it.

Want to see how we build these loops?

The planner grand prix runs on the same autonomous coding infrastructure we use for client builds. If you want to understand how it fits together - or whether a build like this makes sense for your own workflow - see the work we've done so far and get in touch from there.