A new benchmark called ITBench-AA, developed in partnership between Artificial Analysis and IBM Research, is putting AI agents to the test on actual Site Reliability Engineering (SRE) tasks — and the results are sobering. Every frontier model scored below 50%, revealing a significant gap between the market hype around autonomous agents and their real-world operational readiness.

The Numbers Don’t Lie

Claude Opus 4.7 took first place with 46.7%, followed by GPT-5.5 at 45.8%, and Qwen3.7 Max at 42.5%. These are the best scores any model achieved on the ITBench-AA SRE benchmark — and none of them crossed the halfway mark.

“The best result on ITBench-AA SRE went to Claude Opus 4.7 at 46.7%,” the benchmark report notes. Every model tested fell short of 50%, including those marketed as frontier-class systems.

Why ITBench-AA Is So Demanding

The benchmark uses a rigorous metric: average precision at full recall. If an agent produces even a single false negative in a task response, that task scores zero.

“It may sound harsh, but this level of severity is actually quite consistent with how the real world works,” the analysis points out. When an SRE team investigates a production incident, missing the actual root cause — while getting everything else right — means the incident remains unresolved.

Each task presents the agent with incident snapshots containing alerts, events, traces, metrics, logs, and topology data. The agent must return a structured JSON output identifying the exact entities behind the root cause. This requires multi-step causal reasoning, filtering out misleading signals, and operating under ambiguity — skills that remain a challenge for even the most advanced models.

Why the Best Models Still Fall Below 50%

Three factors explain the underwhelming results:

Multi-step reasoning with high error cost. In real incidents, simultaneous symptoms and cascading effects confuse the models. A pod crash might trigger alerts across multiple services, making it hard to separate cause from correlation.

More steps ≠ better diagnosis. GPT-5.5 scored 45.8% with an average of 31 reasoning turns per task. Gemini 3.1 Pro Preview, by contrast, used an average of 83 turns and scored only 30%. The models tend to over-analyze, getting lost in collateral symptoms, upstream dependencies, and failure injection mechanisms rather than homing in on the root cause.

Structural ambiguity. Enterprise environments mix application layer issues, network problems, cluster events, rollouts, and resource constraints simultaneously. Models lack the surgical precision required to pinpoint the actual cause in these noisy environments.

Context from the Original ITBench (IBM)

The original ITBench, published on arXiv (2502.05352), already signaled the same pattern across multiple IT domains:

  • SRE tasks: 13.8%
  • CISO (security) tasks: 25.2%
  • FinOps tasks: 0%

ITBench-AA extends these findings with more granular and production-realistic scenarios, confirming that the gap is neither narrow nor easy to close.

What This Means for Enterprise AI Strategy

The benchmark carries direct implications for any company investing in AI agents for IT operations:

Don’t treat agents as autonomous operators yet. The data shows that enterprises are “too early” to trust complete IT ops automation without strong human oversight. The gap between what the hype sells and what the models deliver is still wide.

Where the real value lives today: Operational copilot functions — pre-triage, evidence organization, hypothesis suggestion, and assisted automation. The value is real, but the autonomy level does not match the hype.

Strategy impacts are concrete:

  • ROI expectations need recalibration
  • Governance models must account for agent limitations
  • Access policies and security perimeters need to reflect reduced autonomy

Economic risk is real. An agent that errs doesn’t reduce risk — it redistributes it to a layer that’s harder to audit. The consequences include availability incidents, alert fatigue, and unauthorized production changes.

The Silver Lining

“Finally, a benchmark that measures something that matters,” is how the AI community received ITBench-AA. Despite the low scores, the benchmark itself is good news because it forces the market to replace the fantasy of total autonomy with operational reality.

ITBench-AA measures genuine readiness: shell access, real incident snapshots, observability artifacts, rigorous scoring, and a sharp focus on root cause identification. It provides the kind of honest signal that enterprises need to make informed investment decisions.

The Smart Move Right Now

The most intelligent path forward isn’t to buy into the promise of full automation. It’s to invest in assisted automation, better observability, audit trails, and use cases where the agent accelerates the human operator without becoming a single point of failure.

Enterprise AI agents will get there — but ITBench-AA makes it clear that the timeline is longer, and the path harder, than most vendors would like you to believe.