An answer key for a data set that never changes under your tests.

01 / What it is

Questions, requests, and the answers.

Every item names a question somebody would actually ask an agent, the tool calls that answer it, and the value the answer has to be. Nothing in it is invented: each one was read off a pinned host and each one cites the test in our repository that would fail if it moved. An item nobody can cite does not go in the pack, however true it is today.

{
  "id": "gh-pull-number",
  "service": "github",
  "question": "Which pull request number in olympus-labs/parthenon carried the fix for the dropped-events incident?",
  "tool_calls": [
    {
      "method": "GET",
      "host": "gh-2026-03-g12.snap.sandboxapis.dev",
      "path": "/repos/olympus-labs/parthenon/pulls/53"
    }
  ],
  "expected": {
    "path": "number",
    "equals": 53
  },
  "difficulty": "read",
  "guarded_by": "packages/conformance/src/parity/pinned-cross-category.test.ts#closed_by_pull resolves to the closing PR on the GITHUB host, via the tracker's branchName"
}

The 53 items carry one of three difficulties. 40 are read: one field off one response. 7 are join, where two responses have to agree and the second call is only reachable once the first has been read. 6 are reason, where the answer is computed from what came back rather than copied out of it.

02 / Why pinned hosts

The same answer every run.

A measurement taken against a moving target measures the target. Point an eval at a real GitHub organization and a merged pull request, a new reviewer or a renamed branch changes the score without anything about the agent changing, and the run you did this morning is not the run CI does tonight. So every host in this pack is a pinned snapshot: one compiled data set, addressed by a content hash, served under its own hostname.

03 / Running it

Three commands.

curl -O https://sandboxapis.dev/evals/olympus-labs-g12.json
curl -O https://sandboxapis.dev/evals/run.py
pip install requests && python3 run.py

It prints a line per item and a score. There is a Node 22 harness beside it that needs no dependencies at all (node run.mjs), and both cache by request, so a full run costs 12 requests rather than one per item. Anonymous access is enough to finish it; a free key raises the limit and rides your own budget instead. Keys and rate limits has the numbers.

53/53  all correct  (olympus-labs-g12-2026-09, 12 requests)

04 / Your agent

One function to replace.

Out of the box both harnesses read the answer straight out of the response, which scores the data set and proves the pack is right. There is exactly one function to change, it already receives the question, the fetched responses and an agent flag, and the model call you need is written in a comment inside it. Put yours there, run with --agent, and the same expectations now score your agent instead.

def answer(item, responses, agent):
    """THE AGENT HOOK. With no flag this reads the answer out of the response, which
    scores the DATA SET. Put your model call under "if agent:" and the same
    expectations score the MODEL instead, which is what makes this an eval:
        if agent:
            reply = anthropic.Anthropic().messages.create(model="claude-sonnet-4-5",
                max_tokens=256, messages=[{"role": "user", "content": item["question"] + json.dumps(responses)}])
            return json.loads(reply.content[0].text)["answer"]
    """
    return at(responses[-1], item["expected"]["path"])

That is the whole difference between a smoke test and an eval: the fetch and the assertion stay where they are, and the middle beat becomes a model call. The agent tutorials teach the same shape one service at a time.

05 / The items

Every question in the pack.

Grouped by the service that answers it. The guarded_by field in the JSON names the test behind each row.

06 / Downloads

Four files.

Version olympus-labs-g12-2026-09. The version string changes whenever the generation or the item set does, so a customer pins it the way they pin a host.

Next

Ask for more items.

This pack reads one incident told by 5 services. If the corner of the data set your agent works in is missing, or you want harder questions, say so and we will add them: tell us what you need. Items are added, never renumbered, so a score stays comparable across versions.

Agent tutorials · SandboxAPIs for agents · MCP setup · Versioning and pinning

All services