How do I evaluate an agent that reads Jira?

Give it a Jira whose answers you already know. Point the agent at a pinned, read-only Jira host, ask it questions from an answer key, and score what it says. The host returns the same bytes every run, so a change in the score is a change in the agent.

01 / The host

Use the pinned Jira host.

An eval needs a host whose data never moves, so use jira-v3-g12.snap.sandboxapis.dev, the pin the eval pack is verified on: the same paths and JSON as Jira Cloud, holding a simulated company's projects, issues, sprints and history.

The live host, jira.sandboxapis.dev, re-rolls daily and can carry a newer generation of the data set than jira-v3-g12.snap.sandboxapis.dev; every answer on this page is for jira-v3-g12.snap.sandboxapis.dev.

02 / Connect the agent

Through MCP, or through the SDK it already uses.

With the MCP server, one environment variable points its Jira tools at the pin. The agent then calls orient first, which returns every base URL, the pinned hosts and its remaining budget:

terminal

claude mcp add sandboxapis --env SANDBOXAPIS_BASE_URL_JIRA=https://jira-v3-g12.snap.sandboxapis.dev -- npx -y @sandboxapis/mcp

Or keep the agent's own Jira client and change its server URL:

python

# pip install jira==3.8.0
from jira import JIRA

jira = JIRA(server="https://jira-v3-g12.snap.sandboxapis.dev", basic_auth=("agent", "any-token"))
print(jira.issue("FATE-51").fields.summary)

03 / The answer key

Questions whose answers are known in advance.

The eval pack holds 53 items across 5 services, 11 of them on Jira. Each has a question in plain words, the request that answers it, and the expected value at a path in the response. Every expected value cites a test in this repository that fails if the answer changes. Here is one:

jira-summary · read

question:  What summary does the Jira host give FATE-51?
request:   GET https://jira-v3-g12.snap.sandboxapis.dev/rest/api/3/issue/FATE-51?fields=summary,status,issuetype,assignee,created,resolutiondate,customfield_10016,project
expected:  fields.summary = "Postmortem follow-up: events dropped during a rebalance"

The harness scores the data set alone, with no model, by checking the last response; with --agent it scores your agent's answer to the same question instead:

terminal

curl -O https://sandboxapis.dev/evals/olympus-labs-g12.json
curl -O https://sandboxapis.dev/evals/run.py
pip install requests && python3 run.py

The same items are on Hugging Face as sandboxapis/olympus-labs-evals, for tooling that loads datasets from there.

04 / Why it holds

Every run gets the same answer.

Every host in the pack is a g12 pin serving one artifact, sha c27f762cc1d2, and the build fails if today's compiler stops reproducing those bytes. Run the eval tonight, next week or after a model upgrade, and the questions have the same answers. Next: the Jira agent tutorial, or the four-hop walk to see the same incident on Slack, GitHub and GitLab.

What this is not

Read this before you rely on it.

SandboxAPIs is read-only: a write returns the provider's own error unless your key has opted in to scratch writes, and those never reach anybody else. Coverage is bounded, and the manifest that says exactly where it stops is public. The data is a simulated company, not yours and not a copy of anybody's. And it reproduces what each vendor documents and publishes, not the vendor's undocumented behaviour, so keep one test against the real service for that.