Sonic SVM (@SonicSVM): Toward the SMART Benchmark for Trading Agents

Toward the SMART Benchmark for Trading Agents Agents are taking on an increasing share of crypto trading, from research to execution. As this penetration rises, the key question is no longer whether an agent can complete a trading workflow, but whether it can trade smartly. SMART Benchmark(v1) measures trading decision quality for builders and users by scoring auditable, dimension-level decision actions, enabling more comparable evaluations under shared task constraints while reducing outcome noise. Get your trading agent analyzed by SMART benchmark on Manic: https://benchmark.manic.trade/ Why We Need a Benchmark for Trading Decision Quality Today’s benchmarks largely cover two categories: Both matter, but neither directly measures crypto trading decision quality. Trading is fundamentally result-oriented, yet outcomes (e.g., PnL) are a noisy evaluation target: they entangle market regime, timing, randomness, and context/memory state with the agent’s actual decision quality. This motivates a benchmark designed around decision quality. To do that, we first need clear definitions of what a trading agent is, and what trading smartness looks like in practice. A trading agent should be defined accordingly. It is not an LLM wrapper that can place orders. It is a decision system that: knowing what information to look at, extracting a thesis from messy, incomplete signals, converting the thesis into a concrete trade, stating position logic and invalidation conditions, revising judgment when new evidence arrives, balancing speed with rigor (not collapsing into paralysis or reckless confidence). That leads to a key requirement: benchmark the harness layer directly. Since performance is heavily shaped by the underlying model, the benchmark should isolate what the agent system adds on top of the base LLM. Once those layers are explicit and measurable, progress becomes directional rather than anecdotal. Introducing the SMART Benchmark Manic’s benchmark is designed to reward stable decision frameworks, not “one correct answer.” The key design choice is to make the agent’s outputs auditable and scorable, while keeping market randomness from dominating the score. To do that, Manic has defined 5 dimensions that cover the agent’s decision harness from input grounding to risk handling. Why these 5 dimensions (and what we exclude): We decompose a single end-to-end trading decision into (1) getting the right live context, (2) gathering and validating supporting evidence, (3) forming a thesis from that evidence, (4) turning it into an executable trade, and (5) applying disciplined risk rules. These map directly to the five dimensions above. We intentionally do not benchmark portfolio construction or long-horizon strategy optimization in v1, because the focus is the per-decision chain where harness quality is most auditable and comparable. Base model vs harness: We operationalize scores on tool-grounded actions and structured outputs (see scoring for details), which serve as a proxy for harness behavior. This reduces—but does not eliminate—base-model effects; leaderboards should be interpreted with base-model context in mind. Anti-gaming (v1): Any benchmark can be overfit. v1 mitigates this by grounding tasks in live/variable inputs (e.g., real-time quotes and refreshed task instances) and scoring for consistency between evidence and actions rather than “one right answer.” How scoring works: action > outcome The benchmark scoring is designed to emphasize: Action > outcome: scores primarily reflect whether the agent produced an executable, internally consistent decision process (even if the market later moved against it). Structured > stylistic: we score what the agent does and outputs (fields, choices, updates), not fluency, tone, or “sounds like an analyst.” The hybrid scoring mechanism of the SMART Benchmark : Rule-based checks where ground truth or behavior is objectively verifiable (e.g., schema compliance, tool/API evidence). LLM rubric only where judgment is inherently qualitative (e.g., tension identification, update discipline). What to expect next Crypto, as an environment, is open and adversarial, information is messy and fast-moving, and decisions are directly tied to real value transfer, which makes it the best proving ground for trading agents. The SMART Benchmark is our attempt to evaluate trading decision quality in a way that’s reproducible and actionable, by scoring auditable decision actions (what the agent looked at, what it concluded, what it would do, and what would change the decision) instead of noisy outcomes like short-term PnL. We hope this helps builders and teams surface failure modes earlier, compare agents under shared constraints, and iterate toward trading skills that remain disciplined when the market gets uncertain. This is a v1 release, and we will continue iterating the benchmark as agents (and the market) evolve.