Want to try it first? A free 10-task sample is available on the Demo Samples page.
What APEX Agent is
APEX Agent is a corpus of long-horizon, tool-using tasks in finance and law. Each task is a single user prompt describing a deliverable — a number, a memo, a filled-in spreadsheet, a legal analysis — paired with a virtual workspace mounted with the relevant files (PDFs, spreadsheets, documents, and correspondence). The agent works the task using tools for filesystem, PDF reading, spreadsheets, document editing, code execution, and correspondence (email, chat, calendar), then returns a final answer that is graded against a hand-written rubric. Tasks fall into three streams:- IB (Investment Banking) — questions about company filings, 10-Ks, equity research notes, and deal memos. PDF-heavy.
- MC (Management Consulting) — questions about financial models in Excel: navigating multi-tab workbooks, applying formulas, and producing analysis. Spreadsheet-heavy.
- Law — long-document analysis: locating relevant clauses, citing them correctly, and chaining textual facts into an argument. Text-retrieval-heavy.
At a glance
What’s inside
Difficulty profile
To characterize the dataset’s difficulty distribution, IB+MC tasks are difficulty-classified by running a strong open-weight baseline agent. Each task is assigned to one of three tiers — where “solve” means a perfect score, every rubric criterion passed:
Across the IB+MC corpus the difficulty gradient is broad: a large solvable core plus a substantial hard tier (~48%) that defeats both the baseline and current frontier models. Easy- and medium-tier samples ship with successful trajectories (SFT-ready, also RL-trainable); hard-tier samples ship without trajectories but remain RL-trainable via the environment and reward verifiers.
How challenging is the data
As a reference point, frontier closed-source models were evaluated on sampled subsets of the IB+MC corpus. Balanced pilot — 100 tasks (50 IB + 50 MC):
Hardest-tier subset — 200 tasks:
On the hardest tasks, frontier models land around 30%. These tasks demand multi-step domain reasoning, file-format-compliant outputs, and multi-row trajectory handling that current models do not reliably deliver out of the box — which makes the corpus a strong training and evaluation signal.
Trajectory length
APEX Agent tasks are genuinely long-horizon. The table below summarizes, for IB+MC baseline rollouts, the assistant turns (steps) and tool calls per rollout — shown as mean / median / p90, broken out by domain and difficulty tier:- IB easy tasks have the longest trajectories (mean 22 steps, p90 45) — reading PDFs and 10-Ks involves many page reads.
- MC trajectories are shorter and more uniform (mean 17–19 steps) — spreadsheet navigation is more direct.
- Tool calls exceed step counts throughout, since a single assistant turn can issue several tool calls in parallel.
Tool usage
The IB and MC streams exercise different tools, reflecting their different source material:- IB (PDF-heavy) — rollouts are dominated by
pdfs_read_pdf_pages,filesystem_search_files, andpdfs_search_pdf, withcode_execution_code_execused for numerical work. Code execution rises in prominence on harder tasks, which demand more quantitative analysis. - MC (spreadsheet-heavy) — rollouts are dominated by
excel_read_tab,excel_list_tabs_in_spreadsheet, andfilesystem_search_files, navigating multi-tab financial models. The document toolword_read_document_contentalso appears on memo-writing tasks.
Training utility
Supervised fine-tuning (SFT) a smaller open-weight model on successful APEX Agent trajectories yields substantial lifts on held-out tasks from the public Mercor APEX-Agents benchmark — a third-party eval set distinct from the training corpus. The IB+MC and Law slices were trained separately because they exercise different agent capabilities — IB+MC emphasizes data, formula, and numerical computation, while Law emphasizes information retrieval and text-based reasoning over long documents. Training data. The dataset provides ~1,000 samples per domain (task + trajectory + reward verifier + environment) for SFT or RL; the hard tier additionally supports RL via its environments and verifiers. The proof-of-concept SFT run below used a 500-per-domain subset drawn from the easy and medium difficulty tiers — half the available SFT-compatible samples per slice. Base model: Qwen3.6-27B. Evaluation sampling:pass@1, max-steps=30, temperature=1.0.
Metrics. Strict pass = fraction of rollouts where every rubric criterion was passed (reward = 1.0). Mean reward = average fraction of rubric criteria passed per rollout (captures partial credit).
IB+MC results (320 held-out tasks: 160 IB + 160 MC):
Strict pass improves +3.7 pp overall (≈54% relative). MC sees the largest lift — strict pass doubles (4.4% → 8.8%) and mean reward rises +77% relative, reflecting that MC tasks (Excel-tab navigation, multi-row trajectories) benefit most from the cohort-trained protocol.
Law results (160 held-out tasks):
Strict pass nearly doubles (6.9% → 12.5%, +5.6 pp, ≈82% relative), and mean reward rises +0.107. The baseline already produced partial credit on many Law tasks, and SFT concentrates on pushing partial-credit rollouts into fully-correct ones.
Caveats. Baseline rollouts were graded by
claude-haiku-4.5; SFT rollouts by the stricter claude-sonnet-4-5 (≈2–3 pp lower strict pass on shared spot-checks), so the SFT lifts above are a lower bound. All results are pass@1; the headline deltas of 3.7–5.6 pp are robust to sampling noise.