Full Dataset

Tau3-Bench is the hard, single-domain extension of the Tau-Bench family. It targets a realistic retail-banking agent (“Rho-Bank”) and adds the defining τ³ mechanic: dynamically discoverable tools. Instead of being handed every capability up front, the agent is given a small base tool set and must search a knowledge base, unlock the right hidden tool at runtime, and only then call it — the way real enterprise agents gate sensitive operations. Tau3-Bench contributes 750 samples to the 3,000-sample Tau-Bench family (the other 2,250 are the three-domain Tau2-Bench). Every sample is fully instrumented: a structured intent, a complete dialog, an executable Python evaluator, and a reference payload (full environment snapshot) for deterministic grading.

​

What Tau3-Bench is

Each sample is a multi-turn banking conversation in which the agent must complete one or more high-stakes operations — filing a card dispute, activating a debit card, resetting a PIN, investigating an interest discrepancy — while satisfying a strict, knowledge-base-grounded policy. Three things make Tau3-Bench substantially harder than Tau2-Bench:

• Discoverable tools. The operation the user wants is performed by a tool that is not in the base list. The agent must call KB_search to learn the tool exists, unlock_discoverable_agent_tool to make it available, and call_discoverable_agent_tool to run it — with the correct arguments.
• Mandatory identity verification. Before any account-specific action, the agent must look up the user, match identity fields against what the customer provides, and log_verification with a timestamp from get_current_time.
• Policy grounded in a knowledge base. Rules like provisional-credit eligibility, the correct dispute category, or which activation flow matches a card’s issue_reason live in the KB, not the prompt. The agent must retrieve and apply them, and obtain explicit confirmation before high-impact writes.

​

Task categories

Tau3-Bench is a single domain (banking), but spans a wide range of banking sub-tasks, often combined within one conversation:

​

At a glance

​

What’s inside

​

Discoverable tools — the τ³ mechanic

The base tool set deliberately does not include the operations that change account state. Instead it includes a gateway: So filing a dispute looks like: KB_search (find the policy + the tool name file_credit_card_transaction_dispute_4829) → unlock_discoverable_agent_tool → call_discoverable_agent_tool. The agent must discover the right tool (e.g. the new-account activation tool, not the replacement one) and supply correct arguments it has gathered from earlier lookups. This is why discovery and KB tools dominate the trajectories below.

​

Trajectory length

Banking dialogs are long and tool-dense. The table reports the datapoints as mean / median / p90. With only ~5 user turns but ~14 tool calls per dialog, the agent does most of the work autonomously between user messages — searching the KB, unlocking tools, and chaining lookups — which is exactly where models go wrong.

​

Tool usage

Banking trajectories are dominated by discovery and verification, not by the final write:

• KB_search is the single most-used tool — the agent constantly grounds its policy decisions and finds tool names in the knowledge base.
• unlock_discoverable_agent_tool and call_discoverable_agent_tool are next — every state change goes through the discover-then-call path.
• get_current_time + log_verification appear in nearly every trajectory, anchoring the mandatory identity-verification step.
• Identity lookups (get_user_information_by_name / _by_id / _by_email) and account reads (get_credit_card_accounts_by_user, get_credit_card_transactions_by_user) gather the arguments the discovered tools need.

​

How challenging is the data

Banking is the hardest slice in the Tau-Bench family. A passing trajectory has to clear every gate at once: verify identity correctly, discover and unlock the right hidden tool, apply KB-grounded policy (the correct dispute category, the matching activation flow, provisional-credit eligibility), gather exact arguments from prior lookups, and obtain explicit confirmation before high-impact writes. Evaluated on the banking corpus, frontier models pass under 30% of tasks: The dominant failure modes are exactly the τ³ additions: failing to discover/unlock the correct tool, choosing the wrong policy branch from the KB, skipping or mis-logging identity verification, and writing before confirmation. This is what makes the slice a strong training and evaluation signal.

​

Evaluation method

Each task’s auto-generated Python evaluator scores a trajectory on three independent axes, then combines them: overall_pass requires all three checks to pass. Because grading is executable and snapshot-based, results are deterministic and reproducible — and usable directly as an RL reward signal.

​

Training utility

Supervised fine-tuning (SFT) a smaller open-weight model on Tau3-Bench banking trajectories yields strict-pass gains on the knowledge-grounded Banking domain, where success depends on retrieving and applying the right product, procedure, or policy from the large Rho-Bank knowledge base (~700 docs). Evaluation runs 97 tasks with num_trials=1, max_steps=200, and a gpt-5.5 user simulator (temperature = 0), graded by combined database-state and per-action checks. Training data. ~628 SFT samples (latest iteration) in multi-turn messages format, produced after expanding knowledge-base and policy coverage. The system prompt embeds the full Rho-Bank customer-service <policy> document, and each assistant turn carries an explicit reasoning field that works through policy compliance (e.g. requiring identity verification before acting on an account). Dialogues are genuinely multi-turn and long-horizon — typically ~11–19 assistant turns and ~9–21 tool calls — using banking tools such as KB_search, identity-verification, and account-lookup calls, which teaches the retrieve-then-act pattern the domain rewards. Base model: Qwen3.6-27B. Metric. Strict pass@1 (reward = 1.0: database state and all action checks pass), temperature = 1.0. Results. Strict pass@1 improves from 10/97 (baseline) → 11/97 (an early 189-task iteration) → 15/97 (latest) — +5 tasks, ≈+50% relative. How it compares. On the public τ-Bench banking leaderboard (banking_knowledge, Pass^1), our ~27B model lands right among models many times its size: A ~27B model matches Gemini 3 Pro and xAI’s Grok-fast tier on this knowledge-grounded domain — and beats a 397B open model (Qwen3.5-397B, 9.8) by more than five points. The frontier still leads (GPT-5.5 37.4, GPT-5.4 30.7, Claude Opus 4.7 25.3), which is exactly the headroom further training targets. What drove it. The gains are on the knowledge-grounded service tasks — those where the agent must first locate the right product/procedure/policy in the knowledge base and then act on it. Two representative wins: (1) product recommendation — matching a customer’s profile (spend pattern, income, no-annual-fee preference) to the right Rho-Bank credit card by looking up product details in the knowledge base; and (2) procedure execution — walking a traveling customer through setting up a travel notification so their card isn’t blocked abroad, following the documented steps. Expanding the knowledge-base and policy coverage in the training data teaches the model to retrieve the correct policy and issue accurate, policy-compliant tool calls, so it satisfies both the database-state and the per-action checks more often.

​

Access & licensing

The full Tau3-Bench corpus — all banking environments, dialogs, intents, evaluators, and reference payloads — is available for commercial licensing, including model training. For licensing, contact support@eigenai.com. A free 10-task sample is available now under the CC BY-NC-ND 4.0 license — see Demo Samples.