How to evaluate AI agent libraries: the example of transformers and traces over answers
An AI agent can return the correct answer but get there via the most expensive route: reading unnecessary code, guessing APIs, failing on errors and rebuilding the solution from scratch. For library developers, this is no longer a minor detail but a separate layer of tool quality. Using the transformers library as an example, we examine how to set up an evaluation bench that analyzes agent traces, tokens, time and errors.
AI-processed from Habr AI; edited by Hamidun News
Developers of libraries for AI agents face a problem: blind checking of the final agent response does not show how efficiently the agent arrived at that response. An agent can return a correct string and at the same time take the most expensive route to it — read unnecessary code, guess API signatures, fail on errors, and reassemble the solution from scratch.
Why One Correct Answer Is Not Enough
For a library developer, an incorrect path by an agent is already not a cosmetic problem, but a signal about a new layer of quality of the tool itself. If an agent regularly spends extra steps guessing the API or reading unnecessary code, it indicates that the documentation, function signatures, or library structure are not clear enough for agent use — even if the end result is formally correct.
- The example analysis is built on the transformers library
- Not only the final answer is evaluated, but also the path the agent took to reach it
- Key metrics: call traces, token consumption, execution time, errors
- Behavioral markers are separately recorded — for example, repeated API guessing
What Is Included in the Evaluation Stand
The evaluation stand built on the transformers example collects complete traces of agent work: what tool calls it made, how many tokens it spent, how long the solution took, and what errors occurred along the way. Behavioral markers are separately accounted for — repeating patterns that indicate the agent is unsure about the API and acts by trial and error instead of immediately applying the required tool.
What This Means
As libraries and SDKs are increasingly used not by humans directly, but by AI agents, the quality of the tool begins to be measured not only by the correctness of documentation for humans, but also by how predictably and cheaply the agent is capable of working with it. Evaluation by traces and behavioral markers becomes a new standard for developers who want their library to be convenient to use specifically by agents.
Need AI working inside your business — not just in your newsfeed?
I build production AI for companies — custom CRM, internal tools, autonomous agents, workflow automation. Owned by you, shaped to your process, no per-seat tax. Built by Zhemal Khamidun, CPO of AlpinaGPT (AI platform, 6,000+ users).
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.