AI Agents

Every Model Has a Battlefield: Typed-Decision Models vs LLMs in Agent Loops

10 hours ago 9 min read

In a sequential agent loop, a typed-decision model beat every LLM we tested. Across 10 Wikipedia races with eight models, TypeSafe's Jev won all ten, using 7.3 seconds of model time and $0.002 in total. The fastest LLM, Gemini 3.1 Flash-Lite, needed 17.2 seconds. Premium models (Claude Opus 5, GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.1 Pro) needed 85 to 154 seconds, even though they often found shorter routes.

Outside loops, the picture flips. On one-off text tasks, Jev tied or lost most rounds against a small LLM. The point of this article is that model choice depends on the shape of the task, and a loop is the shape where speed per decision decides everything.

Race run on 23 September 2026; single-step and text-task tests on 22 September 2026. Prices are those listed by each provider at the time.

Which AI model is fastest in an agent loop?

In our Wiki Race, eight models raced from the same Wikipedia page to a target page, one link click per decision. Every model reached the target in all 10 races. What separated them was model time and cost.

Total model time over 10 races: Jev 7.3 s (10 wins, 0.3 s per decision, $0.002, 3.1 clicks); Gemini 3.1 Flash-Lite 17.2 s (0.7 s, $0.011, 3.4); Claude Haiku 4.5 22.1 s (0.9 s, $0.048, 3.4); GPT-5.4 mini 33.3 s (1.1 s, $0.029, 3.8); Claude Opus 5 84.5 s (4.3 s, $0.335, 2.7); GPT-5.6 Terra 132.6 s (6.5 s, $0.106, 3.0); Claude Sonnet 5 145.7 s (2.2 s, $0.237, 3.4); Gemini 3.1 Pro 154.2 s (6.4 s, $0.259, 2.8) — Total model time across ten races, with wins, median time per decision, total cost and average clicks. Every model reached the target in all ten races.

The models fall into two tiers. The fast tier (Jev, Flash-Lite, Haiku, GPT-5.4 mini) spent 7 to 33 seconds across all ten races. The premium tier spent 85 to 154. Within the fast tier, Flash-Lite was the steady runner-up, second in 7 races; Haiku was close on speed at about four times the cost.

Wiki Race mid-run from Coffee to Moon: Jev has finished in six clicks and 1.7 seconds while four premium models are still on their first decision — Coffee to Moon, 7.2 seconds in. Jev has finished: six clicks in 1.7 seconds of model time. Gemini 3.1 Pro, Terra, Sonnet and Opus haven't made their first click yet.

What is a typed-decision model?

A typed-decision model picks, scores or classifies against options you define, instead of generating text. You send a question and a typed answer space (a list of options, a yes/no, a score range) and it returns a judgment over that space: which option, and how likely. Jev, TypeSafe's "System One" model, is one. At the time of testing it cost $0.042 per million input tokens, with output free.

A language model does the same job by writing its answer token by token, and every token takes time. That difference produces three practical consequences:

Output is cheap or free. You pay for the options you send in, not for words coming out.
Latency is short and flat. There's no answer to stream, so a call takes about the same time whether you send 10 options or 250.
It can't explain itself. No reasoning, no text. If the step needs prose or code, it's the wrong tool.

Our explainer on how chatbots like ChatGPT produce answers covers the generation side of that picture.

Why do agent loops change which model wins?

In a loop, every step waits for the previous one. A batch of independent tasks hides a slow model, because you can send all requests at once. A loop can't: step 12 starts only after step 11 has answered. Per-decision latency multiplies by the number of steps, and so does cost.

Most agents are loops: pick a tool, read the result, pick the next tool. Each step is a choice among many options with a short answer, which is exactly the shape a typed-decision model is priced for.

At the average pace in our races, an agent making 40 sequential decisions per task would spend about 14 seconds of model time with Jev, 3.3 minutes with Claude Opus 5 and 4.4 minutes with GPT-5.6 Terra. Per decision, that's about $0.0001 for Jev, $0.0005 for Flash-Lite, $0.0053 for Terra and $0.0197 for Opus. The 40-step figures are an extrapolation from measured averages, not a separate measurement.

How the Wiki Race benchmark works

Each race starts every model on the same Wikipedia page and asks it, one click at a time, which link leads closest to the target. The rules are identical for everyone:

Each model sees the first 250 links on the current page, in page order, with visited pages removed.
If the target is linked on the current page, code clicks it automatically for every model.
Only model time counts. Wikipedia page loads are excluded.
Every model gets one uncounted warm-up call first. Cold connections added up to about 700ms to a first call in our tests, enough to decide a short race.
Each provider answers in its best-tested format: Jev scores every link, Claude and Gemini return the link's number, OpenAI models return the link's name.
A race stops at 25 clicks.

We ran 10 races in one session, from start pages with no obvious link to their targets.

Does a smarter model win the race?

No. The premium models chose better routes and still lost on time. Claude Opus 5 found the shortest route, or tied for it, in 8 of 10 races. Jev did in 6. From Coffee, Jev wandered through Caffeine, Alkaloid and an asteroid named 4197 Morpheus to reach the Moon in six clicks, and still finished first.

The fairest comparison is a race where several models took the identical route. Same pages, same clicks, so the only difference is the model:

Identical routes. Bitcoin to Ancient Egypt via Cryptography and Egypt: Jev 0.8 s, Opus 11.8 s, Terra 12.0 s. Ice hockey to Leonardo da Vinci via Europe: Jev 0.3 s, Opus 5.8 s, Gemini Pro 6.0 s, Sonnet 11.9 s, Terra 17.7 s. Guitar to Black hole via Electromagnetism and Astrophysics: Flash-Lite 1.3 s, Sonnet 8.0 s, Opus 10.3 s, Gemini Pro 11.3 s, Terra 17.2 s. Honey bee to Jupiter via Bees in mythology and Zeus: GPT-5.4 mini 2.0 s, Opus 10.9 s, Terra 14.7 s — Races where several models clicked exactly the same pages. Same route, same clicks: the only difference is model time.

In a loop, thinking well is not the same as doing well.

Extra reasoning can even hurt. In a single-step test from the Beer page toward Mount Everest, GPT-6 Astra spent about 8 seconds per answer and picked the Andes every time.

Honey bee to Jupiter, finished: eight lanes with their routes, click times and costs; Jev first in 0.7 seconds, Sonnet last at 39.4 seconds — Honey bee to Jupiter. GPT-5.4 mini, Opus and Terra took the same route through Zeus in 2.0, 10.9 and 14.7 seconds. Sonnet spent 26.5 seconds on its first click.

Race-by-race results

Race-by-race model time in seconds, columns Jev, Flash-Lite, Haiku, mini, Opus, Terra, Sonnet, Pro. Beer to Mount Everest: 0.4, 0.6, 0.8, 4.7, 1.9, 3.9, 2.2, 3.1. Pizza to World War I: 0.8, 1.4, 2.1, 2.0, 6.3, 22.7, 10.1, 19.6. Bitcoin to Ancient Egypt: 0.8, 1.4, 1.9, 3.5, 11.8, 12.0, 12.9, 24.6. Tennis to Photosynthesis: 0.6, 1.6, 4.2, 3.4, 6.6, 10.3, 6.7, 10.7. Ice hockey to Leonardo da Vinci: 0.3, 1.6, 2.6, 3.1, 5.8, 17.7, 11.9, 6.0. Coffee to Moon: 1.7, 2.3, 1.9, 4.8, 20.4, 21.8, 21.6, 42.3. Chess to Volcano: 1.0, 3.4, 1.7, 2.8, 7.1, 10.0, 6.3, 10.0. Guitar to Black hole: 0.7, 1.3, 1.9, 4.2, 10.3, 17.2, 8.0, 11.3. Honey bee to Jupiter: 0.7, 3.0, 3.1, 2.0, 10.9, 14.7, 39.4, 22.0. Danube to Albert Einstein: 0.3, 0.6, 1.9, 2.8, 3.4, 2.3, 26.6, 4.6 — Model time and clicks per race for all eight models. Darker cells are faster; the dot marks the fastest model in each race.

The full results, including every model's route, are available as raw race data (JSON).

Scoreboard after ten races with eight racers: Jev 10 first places, 0.7 seconds per race, $0.0022 total; Gemini 3.1 Pro last at 15.4 seconds per race — Session totals after all ten races. Jev averaged 0.7 seconds per race; Gemini 3.1 Pro averaged 15.4.

When should you not use a typed-decision model?

Before the race, we tested Jev on one-off text tasks against Gemini 3.1 Flash-Lite and GPT-5.6 Terra. It didn't win those, and the losses show where it doesn't fit:

Fine-grained judgment: marking filler sentence by sentence, Jev agreed with Terra 69% of the time; the LLMs agreed with Terra 96 to 98%.
Single calls without a loop: scoring a whole post, Jev was about 3x faster (0.35s against about 1s) at the same cost per call, with no quality edge. On one call, that difference rarely matters.
Large batches: a single request is capped at 64k tokens, and batching made results worse. Sent in parallel, the batch worked, and Flash-Lite was still cheaper.
Anything that needs text: summaries, explanations, code, drafts.

Where it did win outside the race was classification with many options: on 30 deliberately ambiguous articles, it tagged 29 correctly against 26 for Flash-Lite and 24 for Terra, two to five times faster.

The practical pattern is a mixed agent: a fast decision model in the inner loop, and a reasoning model called only for the rare steps where being right matters more than being quick.

How to benchmark models for your own agent loop

Use two steps: benchmark one loop step across many candidates, then race two or three finalists on the full task.

Step 1: benchmark a single loop step

Freeze one real step of your loop into a prompt file and run it across candidates. We froze the first Beer → Mount Everest step (the question plus the 250 links) and ran it across ten OpenAI models with llm-preflight, an open-source tool that sends the same prompt to several models and reports validated output, latency and cost side by side. The relevant part of the config:

{
  "warmups": 1,
  "save_responses": true,
  "prompts": [
    {
      "name": "wikirace-step",
      "prompt_file": "wikirace-step.txt",
      "system_prompt": "You are a fast game-playing agent. Reply with raw JSON only.",
      "validation": {
        "json_schema": {
          "type": "object",
          "required": ["link"],
          "properties": { "link": { "type": "string" } }
        }
      }
    }
  ],
  "environments": {
    "openai-speed": {
      "suite_repetitions": 5,
      "models": [
        { "provider": "openai", "model": "gpt-6-astra" },
        { "provider": "openai", "model": "gpt-5.6-terra" },
        { "provider": "openai", "model": "gpt-5.4-mini" },
        { "provider": "openai", "model": "gpt-4.1-nano" }
      ]
    }
  }
}

Trimmed: the full run used ten models. Preview the cost before paying for it:

pip install llm-preflight
llm-preflight benchmark.json --env openai-speed --profiles wikirace-step --dry-run
llm-preflight benchmark.json --env openai-speed --profiles wikirace-step

Single-step benchmark, p95 latency and most frequent pick: gpt-5.4-mini 1.21 s, Beer in Tibet; gpt-4.1-nano 1.42 s, Acerum (off target); gpt-5.6-terra 8.20 s, Beer in Tibet; gpt-6-astra 12.42 s, Andes (off target); gpt-5.6-sol 21.06 s, Andes (off target) — One frozen race step across five of the ten OpenAI models. All answers were valid JSON; only two picked a link that actually leads toward Mount Everest.

Non-reasoning models answered in 1.2 to 1.7 seconds at p95; reasoning models took 8 to 21. Every model returned valid JSON, so on validity, speed and cost the tool recommended the cheapest, gpt-4.1-nano, whose favourite link was "Acerum", near the top of the alphabetical list. Validity and quality are different tests: keep the saved responses and read the picks.

Step 2: race the finalists on the full loop

Take a fast model, a premium model and the model you're evaluating, and run your whole task several times:

Clock the model, not the world. Leave page loads, tool execution and file reads out of the total.
Warm up before the clock starts, or short runs measure connection setup.
Share the shortcuts. Anything code decides is decided the same way for every model.
Report every run, including the ones your favourite lost.

FAQ

How much faster is a typed-decision model per decision?

In our race, Jev's median decision took 0.3 seconds, against 0.7 for Gemini 3.1 Flash-Lite, the fastest LLM, and 2.2 to 6.5 seconds for premium models. On a single one-off call that gap rarely matters; across a loop of dozens of steps it decides whether the agent feels interactive.

When is a reasoning model worth it in an agent?

For the rare step where a wrong choice is expensive or irreversible. Claude Opus 5 found the shortest route in 8 of 10 races but used 84.5 seconds of model time against Jev's 7.3, so it doesn't pay for the routine steps of a loop. Use a fast model for the inner loop and call the reasoning model only for the high-stakes decisions.

How much does one agent decision cost?

In our races: about $0.0001 per decision for Jev, $0.0005 for Gemini 3.1 Flash-Lite, $0.0053 for GPT-5.6 Terra and $0.0197 for Claude Opus 5, at prices listed in September 2026.

What is the best LLM option if I need text output in the loop?

Gemini 3.1 Flash-Lite: second in 7 of 10 races, 17.2 seconds of model time and $0.011 in total, and unlike a typed-decision model it can also write. Claude Haiku 4.5 was close on speed (22.1 seconds) at about four times the cost.

How reliable are these results?

They come from 10 races with eight models in one session on 23 September 2026. An earlier 10-race test with three of the models showed the same order. Single slow responses swing totals, and vendor pricing can change, so rerun the numbers on your own task before committing to an architecture.