AgentLens: code-agents evaluation shifts from outcome to process

Researchers presented AgentLens — a benchmark for evaluating code agents in July 2026. Unlike standard tests that reduce results to simple pass/fail, AgentLens analyzes the entire trajectory of an agent's work: how it follows instructions, uses available tools, tests its code, recovers from errors, and communicates with the user.

Why Old Approaches Are Insufficient

A typical benchmark for code agents works in an extremely simple way: given a task—solved or not. The result is reduced to a single binary value: 1 or 0, pass or fail. This is convenient for quick model ranking, but completely doesn't reflect real user experience.

People working with code agents in production see much more. They interact with the agent throughout its entire work process, follow its reasoning, see the errors it makes, observe how it recovers and changes strategy. They ask clarifying questions, request code rewrites, point out bugs.

Standard benchmarks capture none of this. They don't explain why the agent chose that particular approach, what intermediate steps seemed logical to it, where it got stuck. The final result may be technically correct, but the path to it may be lengthy, inefficient, full of failed attempts.

How AgentLens Works

AgentLens combines several evaluation approaches into a single system:

• Formal verification — objective checks where possible (code syntax, logic, unit test results)
• LLM-written trajectory-reviews — specially trained models analyze the agent's entire path, evaluating each step and decision
• Side-by-side comparisons — system shows visual comparison between agent versions, highlighting behavior differences

The result of each run is not simply a score from 1 to 10, but detailed text explanation in natural language. The system describes: why the agent received that particular score, which decisions were logical and correct, where it erred, how it recovered from errors, was it efficient in tool usage.

Where AgentLens is Used

The benchmark proved useful not only for ranking different models. Developers use it for three main purposes:

• Behavior diagnostics — deep understanding of where the agent systematically makes mistakes, which task classes are harder for it, which tools it underutilizes
• Own agent version comparison — tracking whether quality improved between releases or regression occurred in some aspect
• Automatic monitoring — overnight testing pipelines catch issues before they hit production systems

What This Means

AgentLens symbolizes a broader trend in the AI community: evaluation of code agents transitions from simple metrics to comprehensive analysis of the entire work process. Instead of binary yes/no, the system now gives full picture: which decisions the agent made, why, what worked, what didn't.

This helps developers not only improve final results, but also better understand how their AI systems reason, what types of tasks come easy to them, and which cause problems.

The benchmark was released as open source on GitHub, allowing researchers and developers worldwide to use it for their projects.