Sentient (@SentientAGI): Why Do Multi-Tool Analyst Agents Fail?

Why Do Multi-Tool Analyst Agents Fail? To study limitations of complex reasoning agents, we built CryptoAnalystBench (198 open-ended analyst-aligned questions across 11 categories) with a state-of-the-art evaluation pipeline for response scoring and error categorization. We’ve successfully uncovered the most common errors plaguing such agents, and proposed engineering strategies to help developers solve for them. Evaluating Agent Capability Using CryptoAnalystBench As the world builds AI systems for law, finance, medicine, and other high-stakes use cases, we need to understand how these systems actually fail, and thus how to improve them quickly. Current RAG, search, or tool-calling evaluations reduce agent performance to single isolated numbers. This paper is the first to consider how to evaluate agents’ capabilities on these things in combination–both in answer quality and error implications — using CryptoAnalystBench. Learn more about CryptoAnalystBench. Introducing a New 7-error Taxonomy for Modern Agentic Systems To perform proper error analysis on modern agentic systems, we built (i) a claim verification pipeline and (ii) an LLM-as-a-judge with our new 7-Error Taxonomy. To create the error taxonomy, we built a multi-tool agent harness and expert-annotated its responses to questions from CryptoAnalystBench with failure modes and 1-10 scores on four criteria: temporal relevance, data consistency, depth, and query relevance. We then created a detailed rubric for an LLM judge to automatically identify these error categories. With this method, we enabled AI to turn agent execution traces into failure signatures, exposing what broke and how to fix it. To study any model-specific error tendencies, we also produced responses with 5 models: GPT-OSS-120B, Qwen3-235B-A22B, Kimi K2.5, GPT-5.2, and GLM 4.7. Explore the exact LLM judge rubrics on GitHub. Key Finding #1: LLM judges are better at binary error identification than scaled scoring. The LLM judge, while presenting poor correlation in score assignment with human judges, showed top-tier alignment on error identification, at 93.45% accuracy. This implies that LLM judges can be used reliably for error analysis, but not necessarily human-aligned subjective judging. Key Finding #2: Hallucination isn’t the dominant failure mode anymore. Across models, fabricated claims are <6% (often ~1–2%). Similarly, all models keep citation preciseness >85%. Top performers are GLM-4.7 (97.1%), Kimi (95.7%), and GPT-5.2 (95.5%). The remaining gap is mostly uncited factual assertions (but not hallucinations). Key Finding #3: Higher-order failures are nuanced and model-specific. Most common failures are from temporal staleness, missing risk assessments, and issues with answer directness / framing. For individual models, we found that: Qwen3-235B-A22B has the highest rate of temporal staleness (17.2%) and source reconciliation failures (9.6%), yet no shallow synthesis errors, implying that the model’s training procedure likely emphasized aggressive data reasoning and integration. GPT-5.2 has the lowest error rates across all dimensions besides near-matching Kimi K2.5 on data consistency (1% vs. 2%) and lack of risk assessment (9.6% vs 10.1%) Both GPT-OSS-120B and GPT-5.2 excel in answer relevance and framing over their Chinese counterparts, with GLM-4.7 being the worst (16.2%). Mitigation Strategies for Building More Reliable Agents Using this error analysis, we generalize the following mitigation strategies at the level of prompt engineering: Dictate that the LLM prefers structured APIs over unstructured search for numerical data Auto-inject timestamps or date windows from APIs and search into LLM context Implement CoT-like prompting for task decomposition to force depth The result is hopefully more relevant, deep, and context-aware LLM responses. Takeaways for AI Developers Building Advanced AI Agents If you’re building multi-tool reasoning agents, we recommend creating a similar error identification rubric to track down the exact reasons your agent fails in production scenarios. Our CryptoAnalystBench, rubrics, and evaluation pipelines are thus meant to be an example for any AI developer trying to build similarly advanced AI agents. If we want high-stakes agents we can trust, such failure analysis and (later) auto-improvement of AI systems is necessary to tighten the feedback loop between users and engineers. Huge shoutout to the authors and contributors: Anushri Eswaran, Oleg Golev (@oleg_golev), Darshan Tank, Sidhant Rahl, and Himanshu Tyagi (@hstyagi). Resources: Read the Paper: https://arxiv.org/abs/2602.11304 Check out the GitHub repo: https://github.com/sentient-agi/CryptoAnalystBench Read the CryptoAnalystBench blog: https://www.sentient.xyz/blog/cryptoanalystbench