> ## Documentation Index
> Fetch the complete documentation index at: https://docs.eigenai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Full Dataset

> The full Enterprise Bench corpus — long-horizon, tool-using agent tasks set inside realistic simulated companies, spanning 27 enterprise environments and two task families, synthesized by EigenData-CLI.

**Enterprise Bench** is a corpus of **long-horizon, tool-using agent tasks set inside realistic simulated companies**, synthesized by EigenData-CLI. Each task drops an agent into a self-contained **enterprise environment** — one company backed by up to \~40 connected SaaS systems (CRM, ticketing, docs, chat, code, observability, billing, HR) that all share **one consistent world state** — and asks it either to *operate* the business (multi-system write and coordination tasks) or to *answer questions* about it (read-only investigative QA). The full corpus spans **27 enterprise environments** and **10,800 samples**. Each sample bundles a task, the environment, a reward-verification script, and — where available — a successful agent trajectory, usable for either supervised fine-tuning or reinforcement learning.

The difficulty, trajectory, and tool-usage sections below characterize the **bottom-up read-only QA family**, the slice evaluated end-to-end.

<Note>
  Want to try it first? A free, individually-audited 20-task sample (10 operations + 10 investigation) is available on the [Demo Samples](/products/eigendata-cli/datasets/enterprise-bench/demo) page.
</Note>

## What Enterprise Bench is

A simulated company in Enterprise Bench is not a single API — it is a **federation of enterprise systems over a shared database**. Salesforce, Slack, Linear/Jira, Notion, GitHub, PagerDuty, Stripe, Grafana, SonarQube, Optimizely, LaunchDarkly, Greenhouse, Lattice, and more are all mounted as MCP tool servers backed by the same coherent world: the Linear ticket Slack is discussing is the same ticket the PagerDuty incident references, the Salesforce account the email mentions is the one billing tracks in Stripe. An agent works the task with native tool calls — searching, reading, and (on write tasks) mutating records across systems — exactly as a human operator would move between browser tabs.

Tasks fall into two complementary families, both set in these enterprise environments:

* **Top-down — operations & coordination (WRITE).** A persona-driven, multi-turn request to *get something done* across several systems: triage a CVE across Linear and Slack, onboard a customer across Salesforce/Stripe/Calendar, run a pre-launch readiness check and start an experiment. The agent discovers context across systems, then executes coordinated writes. Graded on the **final state of the world** — a DB-diff against a golden delta plus an LLM rubric of goal / process criteria.
* **Bottom-up — investigation & QA (READ-ONLY).** An answer-mined question over the same environments — *"has this feature flag ever actually been enabled?"*, *"are these two engineers duplicating work?"*, *"how many of these incidents are unresolved and are they all linked to one ticket?"* The agent investigates with read-only tools and returns a grounded, cited answer. Graded on **answer recall against mined ground truth** (entity / fact / source recall, minus a distractor penalty).

## At a glance

| Property                    | Value                                                                                                                                                                                                                                                              |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Enterprise environments** | **27** industry verticals — each a self-contained simulated company                                                                                                                                                                                                |
| **Connected systems**       | up to **\~40 enterprise SaaS systems** per environment (Salesforce, Slack, Linear/Jira, Notion, GitHub, PagerDuty, Stripe, Grafana, SonarQube, Optimizely, LaunchDarkly, Greenhouse, Lattice, Snyk, Zendesk, Google Workspace, …), all over one shared world state |
| **Samples**                 | **10,800** — task + environment + reward verifier, usable for SFT or RL; SFT-ready samples additionally ship a successful agent trajectory                                                                                                                         |
| **Task families**           | **2** — top-down (multi-system writes), bottom-up (read-only QA)                                                                                                                                                                                                   |
| **Task / answer formats**   | write-and-coordinate; QA in `factual` / `count` / `comparison` / `list` / `explanation`                                                                                                                                                                            |
| **Grading**                 | **top-down:** DB-diff against a golden delta **+** LLM rubric (goal / process / policy criteria). **bottom-up:** recall metrics (entity / fact / source recall) − distractor penalty                                                                               |
| **Layout**                  | tau-bench four-folder (`intent` / `datapoints` / `evaluators` / `reference_payloads`), one `NNNNNN` id per sample                                                                                                                                                  |

## What's inside

| Component            | Description                                                                                                                                                                                                               |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Tasks**            | A persona, a goal, and constraints (top-down) or an investigative question with answer type, difficulty, and required reasoning (bottom-up). Most carry the evidence chain / gold sources the answer or write depends on. |
| **Environments**     | A self-contained simulated company per environment — up to \~40 SaaS systems over one shared relational world, so cross-system references resolve consistently.                                                           |
| **Reward verifiers** | An executable scorer per task. Top-down diffs the final DB state against a golden delta and runs LLM rubrics; bottom-up scores the cited answer with recall metrics. Both return a reward in `[0, 1]`.                    |
| **Trajectories**     | Agent rollouts with chain-of-thought reasoning and native tool calls — SFT-ready samples ship a successful (passing) rollout.                                                                                             |
| **Tool schemas**     | The action space of the systems a sample operates in — every tool of every MCP the trajectory touched, in OpenAI function format (not just the tools a given rollout happened to call).                                   |

## Task categories

Enterprise Bench is organized by **enterprise environment**: each of the **27 verticals** below is a distinct simulated company, and serves as a **case study** for the kind of cross-system enterprise work an agent must handle. Each environment is realized as a concrete named company (the `ai_ml_startup` world ships as *Lattice AI*, `ecommerce_scale` as *Cartable*, `mobility_fleet` as *RideGrid*, …), populated with its own people, teams, tickets, documents, incidents, accounts, experiments, and feature flags — so a task at Lattice AI is about INT8 quantization rollbacks and GPU quota, while one in `freight_logistics` is about denied-party screening and shipment latency.

| Sector                        | Environments                                                                                                               |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| **Software & infrastructure** | `ai_ml_startup`, `b2b_saas_startup`, `dev_tools_company`, `oss_data_infra`, `nexus_cloud_platform`, `cybersecurity_vendor` |
| **Payments & fintech**        | `payments_api`, `enterprise_fintech`, `insurance_carrier`                                                                  |
| **Commerce & marketplaces**   | `ecommerce_scale`, `omnichannel_retail`, `proptech_marketplace`, `travel_booking`                                          |
| **Media & consumer**          | `streaming_media`, `gaming_studio`, `creative_agency`, `edtech_platform`                                                   |
| **Regulated & public sector** | `govtech_platform`, `healthcare_saas`, `legaltech_saas`, `nonprofit_ngo`, `biotech_rnd`                                    |
| **Logistics & physical ops**  | `freight_logistics`, `mobility_fleet`, `climate_iot_hardware`, `regional_telecom`                                          |
| **Professional services**     | `consulting_firm`                                                                                                          |

Both task families instantiate across these worlds, giving broad capability coverage without re-using surface content.

### Expanding the categories

The 27 environments are case studies, not a closed set. Each is a world + tool servers + task generators + reward verifiers, so the corpus grows along three axes:

* **More verticals** — manufacturing/ERP ops, banking back-office, clinical/EHR workflows, retail POS and supply chain, telco network operations, and other industries, each with its own systems and domain content.
* **More task families** — beyond top-down writes and bottom-up QA: scheduled / long-running operations, approval and escalation chains, policy-and-compliance enforcement, cross-system reconciliation, and incident-response runbooks.
* **Deeper system coverage** — pulling more of each environment's \~40 connected systems into tasks (HR/Workday, IaC/ArgoCD, secrets/Bitwarden, observability/Datadog/Sentry, and the wider doc and CRM surface).

Each addition ships the same way — a coherent world plus an executable reward verifier per task — so it is immediately RL-trainable and (with a passing rollout) SFT-ready.

## Difficulty profile

Bottom-up tasks carry an explicit difficulty label and reasoning tags; the distribution skews hard by design (counts from the evaluated read-only slice):

| Tier          | Share | What scales                                                                                                            |
| ------------- | ----: | ---------------------------------------------------------------------------------------------------------------------- |
| **Medium**    |   11% | A single cross-system hop; the answer is reachable once the right two systems are joined.                              |
| **Hard**      |   65% | Multi-hop joins, near-duplicate records to disambiguate, contradiction between a config's nominal and effective state. |
| **Very hard** |   24% | Deeper chains, aggregation over scattered records, and answer-mined facts that only a thorough sweep surfaces.         |

Answer types are dominated by **factual (60%)** and **explanation (36%)**, with a sharp tail of **count, comparison, and list** questions (≈4% combined) that prove the most punishing — they require complete, exhaustive enumeration rather than locating a single fact. Top-down tasks scale on a different axis: number of systems touched, number of coordinated writes, and the order constraints between them.

## How challenging is the data

As a reference point, a frontier-scale **open-weight model — Qwen3.5-397B** (`qwen3-5-397b`) — was evaluated on the **bottom-up read-only QA family**: **1,932 answer-mined questions across all 27 environments**, scored both by each task's recall verifier and by an independent LLM rubric.

**Headline (LLM-judge rubric, reward ∈ \[0, 1]):**

| Metric                                                |                    Value |
| ----------------------------------------------------- | -----------------------: |
| Mean rubric reward                                    | **0.558** (median 0.605) |
| Mean recall reward (entity/fact/source − distractors) |               **≈ 0.40** |
| **Strict pass** (rubric = 1.0)                        |  **13.6%** (262 / 1,932) |
| Reward ≥ 0.9                                          |                    24.4% |
| Reward = 0.0 (total miss)                             |                    12.0% |

The global mean sits near **0.56**, but the signal is in the variance.

**By answer type** — locating one fact is tractable; exhaustive enumeration is not:

| Answer type   | Tasks | Mean reward |
| ------------- | ----: | ----------: |
| `explanation` |   702 |       0.607 |
| `factual`     | 1,150 |       0.550 |
| `count`       |    30 |       0.433 |
| `comparison`  |    29 |   **0.165** |
| `list`        |    21 |   **0.095** |

**By environment** — capability is sharply uneven across verticals (selected; patched rubric mean):

| Hardest                |  Mean |   | Easiest                          |  Mean |
| ---------------------- | ----: | - | -------------------------------- | ----: |
| `nonprofit_ngo`        | 0.434 |   | `omnichannel_retail`<sup>†</sup> | 0.884 |
| `regional_telecom`     | 0.445 |   | `mobility_fleet`                 | 0.684 |
| `creative_agency`      | 0.491 |   | `govtech_platform`               | 0.667 |
| `enterprise_fintech`   | 0.501 |   | `b2b_saas_startup`               | 0.652 |
| `nexus_cloud_platform` | 0.501 |   | `healthcare_saas`                | 0.629 |

<sup>†</sup> `omnichannel_retail` is a small slice (n=13); treat its mean as indicative, not robust.

**Why it loses points.** A second-pass audit of the non-perfect answers (given the gold and the docked criteria) found the misses are overwhelmingly **real agent errors, not grader noise** — three failure modes dominate:

* **Wrong-similar-entity (≈39%)** — the model grabs a near-duplicate of the right record (the adjacent ticket, the other engineer, PR #2 instead of PR #4). The environments deliberately plant look-alikes that demand ID / assignee / status cross-verification.
* **Retrieval-gives-up (≈28%)** — when one tool returns thin data the model concludes *"no history available"* instead of trying the audit log, transition history, or another system where the fact actually lives.
* **Fabrication (≈17%)** — on sparse data the model invents plausible-but-unsupported evidence (a Slack message, a metric, an ID) rather than stating absence.

A further ≈5% are format failures (empty / unparseable final answers). These tasks demand exact cross-system grounding, complete enumeration, and the discipline to **state absence rather than invent** — which a frontier-scale open model does not reliably deliver out of the box, and which is precisely what makes the corpus a strong training and evaluation signal.

## Trajectory length

The bottom-up tasks are read-only but genuinely investigative. The table below summarizes, for the evaluated read-only slice, the **assistant turns** (steps) and **tool calls** per rollout — shown as mean / median / p90, broken out by difficulty tier:

| Tier          | Steps (turns) | Tool calls     |
| ------------- | ------------- | -------------- |
| **Medium**    | 9.1 / 10 / 12 | 17.5 / 16 / 30 |
| **Hard**      | 8.7 / 10 / 12 | 16.7 / 17 / 28 |
| **Very hard** | 9.0 / 9 / 12  | 18.6 / 18 / 30 |
| **All**       | 8.8 / 10 / 12 | 17.2 / 17 / 28 |

* **Tool calls (\~17) far outnumber assistant turns (\~9)** — the agent issues several searches and reads in parallel per turn.
* **Trajectory length is roughly flat across tiers** — the difficulty comes from *which* records to find and reconcile, not from longer rollouts.
* **The top-down write family runs longer still** — in the demo set, 21–49 turns and 10–26 tool calls across 4–5 systems each, since it interleaves discovery with coordinated writes and multi-turn confirmation.

## Tool usage

The read-only investigation family is **search-dominated**: the agent sweeps chat, ticketing, docs, and code, then reads the specific records it needs. Across the 1,932-task slice, tool calls concentrate in:

* **Chat (Slack)** — ≈29% of all calls; `slack__search`, `slack__read_channel`, `slack__list_channels`.
* **Ticketing (Jira / Linear / Asana / Zendesk)** — `jira__search_tickets`, `jira__get_ticket`, `linear__search_tickets`, `zendesk__search_tickets`.
* **Docs (Confluence / Notion)** — `confluence__search_docs`, `notion__search_docs`.
* **Code & incidents (GitHub / PagerDuty)** — `github__list_pull_requests`, `github__search_code`, plus incident lookups.

The top-down write family additionally exercises the **mutating** tools — `linear__link_tickets`, `salesforce__add_note` / `log_activity`, Stripe customer creation, `notion__add_space_member`, calendar writes — and is graded by the resulting database delta, not by the answer text.

## Access & licensing

The full **Enterprise Bench** corpus — all environments, tasks, reward verifiers, and trajectories — is available for **commercial licensing, including model training**. For licensing, contact [support@eigenai.com](mailto:support@eigenai.com).

A free 20-task sample is available now under the [CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/) license — see [Demo Samples](/products/eigendata-cli/datasets/enterprise-bench/demo).