τ²-Bench Airline tests whether a model can do an airline support agent's job: follow a policy manual, talk to a simulated customer, and call the right tools to search flights, change bookings, and issue refunds. The model doesn't need any domain knowledge; it's scored entirely on how well it executes tool calls across multi-step task trajectories. We run it continuously against the same provider endpoints that serve OpenRouter traffic, so a score reflects both the model and the provider running it. We use this benchmark because it has a high floor, so we can assess provider variance and not model capability. Our routing algorithm for tool call requests uses these same signals to send traffic to the best performing endpoints.
No verified runs yet. Check back soon.
It's a tool-calling benchmark that is hard to game. Grading depends on live tool-call trajectories rather than memorized answers, so it resists training-data leakage better than Q&A-style evals. It exercises every tool-calling failure mode (wrong arguments, skipped policy checks, giving up, hallucinated confirmations) at a relatively low cost per run. The relative scores also carry more signal than the absolute ones. The same model can score differently across providers, and those deltas are what Exacto routing uses to pick higher-accuracy endpoints.
Each task is a simulated airline support conversation with a scripted user, a toolbox (flight search, booking changes, refunds, loyalty policies), and a gold reference solution. A task passes only if the final database state and the messages to the user match the reference; partial credit is not awarded.
There is still headroom. Top models fail roughly one in five tasks, and the airline domain is the hardest τ²-Bench split. Accuracy differences here separate models that follow multi-step policies from ones that merely chat well.
The floor is high, though. Many tasks reward inaction. A refusal task with an empty gold action list passes for any agent that changes nothing. Even weak models score well above zero, so the meaningful spread sits at the top of the range.
The benchmark is public, so tasks may appear in training corpora. Contamination inflates scores less here than in Q&A-style evals, though, since a leaked task still has to be executed correctly, step by step, against a live database.
Scoring fidelity has limits. The checker verifies two things: the final database hash and exact substring matches in the agent's messages. Each task's natural-language assertions ("agent should refuse the cancellation") are metadata, and no judge model reads the transcript. So a savings calculation fails if the agent says "$23,552.50" when the checker greps for "23553".
The user simulator matters too. We pin it to gemini-2.5-flash so agent scores stay comparable, but the sim is itself an LLM with failure modes of its own. It can stop the conversation before the agent finishes, leak its hidden task instructions, or keep a stuck agent looping until the 200-step ceiling kills the run. Swapping the sim model shifts absolute scores, which is why cross-paper τ²-Bench numbers rarely line up exactly.
Every task ships a gold solution: a list of tool calls, strings the agent must say, and natural-language assertions. After the conversation ends, the checker replays the gold tool calls against a fresh database and compares hashes with the agent's final database. It then greps the agent's messages for each required string. The reward is the product of those two checks:
reward = db_match × communicate_met // each ∈ {0, 1}
db_match = hash(agent DB) == hash(gold DB)
communicate_met = every required string appears in an agent message
any run that hits MAX_STEPS instead of a clean stop scores 0 outrightThe rollouts below are from real runs, with gemini-2.5-flash as the user simulator throughout.
Task 17 agent: openai/gpt-5.1
For reservation FQ8APE: add 3 checked bags, swap the passenger to Omar Rossi, and upgrade basic economy to economy, paying with a gift card.
The agent looked up the user, found the right reservation among several, confirmed the changes and payment method, then made all three writes: passenger swap, cabin upgrade, and bags. The final database hashes match the gold state and the run ends on USER_STOP, so reward is 1. This is what the eval is designed to measure: multi-step tool use under policy constraints, done correctly.
Task 0 agent: openai/gpt-4o-mini
Cancel reservation EHGLP3. The booking is more than 24 hours old, basic economy, no travel insurance, so policy says no cancellation.
The agent pulled the reservation, saw a basic economy fare booked more than 24 hours ago with no insurance, and cancelled it anyway. It even promised a $208 refund. The checker compares the final database against the gold database (unmodified), the hashes differ, reward is 0. The same model refused this exact cancellation in a different epoch; sampling variance flips the outcome.
Task 18 agent: openai/gpt-4o-mini
Downgrade all five business-class reservations to economy, then report the total amount saved. The correct total is $23,553.
The agent executed all five downgrades correctly, and the database check passed. Then it computed the savings from a partial list of fares and told the user $14,965 instead of $23,553. The communicate check greps every agent message for the literal string "23553", finds nothing, and zeroes the whole task. One wrong arithmetic answer erased five correct database writes.
Task 11 agent: meta-llama/llama-3.1-8b-instruct
Remove passenger Sophia from reservation GV1N64. Policy forbids changing the passenger count; the correct move is to downgrade both passengers to basic economy and refund $5,244.
The agent’s very first move is a write with fully invented arguments: a reservation ID the user never gave and a passenger "John Doe" born 1990-01-01 who exists nowhere in the data. After finding the real reservation it sends update_reservation_passengers with a one-passenger array against a two-passenger booking, gets "number of passengers does not match", and retries the identical call four more times, at one point hallucinating "Sophia Smith, 1992-01-01" as the second passenger. It never consults the policy (removing a passenger is forbidden; the correct move is a cabin downgrade), quotes a fabricated $200 refund, and both checks fail. The arguments are well-formed JSON that the tool schema accepts; they are just wrong about the world, which is why small models can look fine on schema-level tool-call metrics and still fail tasks like this.
Task 14 agent: openai/gpt-4o-mini
Rebook the cheapest business round trip and split payment across gift cards, one certificate, and a credit card. The correct split puts $44 on the card.
The agent built a booking where the payment amounts did not sum to the ticket price. The tool rejected it with the same error every time, and the agent retried the identical call dozens of times until the run hit its 200-step ceiling. Any run that terminates on MAX_STEPS scores 0 before the database is even compared. This transcript is 202 messages long; the excerpt below is the loop.
Task 0 agent: openai/gpt-4o-mini
Same task as the policy-break failure: the user wants to cancel EHGLP3, and policy says no.
This run earned a legitimate pass: the agent checked the reservation, cited the 24-hour rule, and refused. But look at what the checker actually verified: an untouched database and an empty communicate list. An agent that stonewalled every request, or transferred to a human immediately, would score identically. Refusal tasks measure "did nothing break", so they inflate scores for overly cautious models.
Scores aggregate all successful runs, weighted by task count, with a minimum of 45 graded tasks per model-provider pair. A model's headline score uses its default routing (not pinned to a provider) when one exists; otherwise it falls back to the median provider. The standard deviation is measured across runs for that representative result. Cost, time, and token figures are per-task averages from the same runs. Best value is the cheapest Pareto-optimal model within 5 points of the top score.
These are the same measurements that power Exacto routing. See the docs for how routing works, or browse all models to try one.
These scores are available through OpenRouter's public benchmarks API, so you can retrieve the same model-level results programmatically.
GET https://openrouter.ai/api/v1/benchmarks?source=openrouter Authorization: Bearer <API key>
Use task_type=agentic to filter to tau_bench_verified_airline. Each item represents one model and includes accuracy, accuracy_stddev, avg_cost_per_task, total_tasks, and last_run_timestamp. See the benchmarks API docs.