CertiK (@CertiK): From Internal to Public: how CertiK AI Auditor catches vulnerabilities with low noise.

From Internal to Public: how CertiK AI Auditor catches vulnerabilities with low noise. 1. Why We're Opening CertiK AI Auditor Now Judging by results alone, CertiK AI Auditor is a mature AI security product. We evaluated it against 35 real security incidents from 2026 (the test data was not used in model training or knowledge base construction), all with root causes in verified Solidity code. The results: 28 out of 35 exact hits on the first run (80%). An additional 3 cases were partial hits on the first run and reached exact hits on the second pass, bringing the cumulative exact hit rate to 31/35 (88.6%). The remaining cases included 2 partial hits across both runs and 2 misses. But these results are not the reason we decided to open the tool for external testing. CertiK AI Auditor was not originally built for external release. It was an internal tool, built to serve our auditors. Its original goal was specific: help the audit team identify the issues worth prioritizing - earlier and faster. Whether an AI tool "looks smart" is irrelevant. What matters is whether auditors choose to keep using it, and whether development teams actually save time. 2. From a Failed First Version to an Evolving Framework CertiK AI Auditor did not start in its current form. After the first version shipped, our internal auditors gave candid feedback: the system could generate findings, but it wasn't consistently reducing their workload. The issue wasn't that it couldn't find vulnerabilities at all. It was the noise - duplicate results, risk assessments lacking sufficient context, and false positives that required manual effort to dismiss. In a real audit workflow, a tool that creates more triage work than it saves hasn't actually solved the problem. This feedback changed our product direction. We stopped treating detection coverage as the sole metric and began requiring that detection rate and false positive rate both meet a meaningful bar. The former determines whether the tool can find real issues; the latter determines whether anyone will actually keep using it. Over the following year, we ran extensive experiments: agent architectures, general-purpose workflows, task-specific workflows, different models, and different orchestration strategies. The goal was never to prove one AI paradigm "superior" - it was to determine which approaches consistently delivered better results against real cases and benchmarks. Along the way, we found that the effective boundary of each paradigm shifts as models improve. Agents were too unpredictable when LLMs were weaker - structured workflows outperformed them. But as models grew more capable, agents showed higher ceilings, while workflows required increasing manual adjustment to keep pace with model improvements. Each model upgrade effectively meant rebuilding the pipeline. We didn't need to commit to one paradigm. We needed a framework flexible enough to evolve continuously as models advanced. That framework took shape in mid-2025 and its core architecture remains in use today. What survived wasn't a single approach, but the set of design choices that proved effective through repeated validation. 3. Multi-Scanner: High Detection Through Validated Complementary Coverage The first key conclusion from our experiments: no single scanner reliably covers all vulnerability types. Different scanning approaches vary significantly in sensitivity to different issue classes. The same model under different orchestration can exhibit different strengths and blind spots. Rather than betting on one "universal scanner," the more effective strategy is combining multiple detection methods whose coverage has been validated as complementary. Our benchmarks illustrate this directly. For example, we selected 7 confirmed vulnerabilities and ran 4 scanners against each one, 10 times per scanner, recording the pass rate (detection rate) for each scanner–vulnerability pair. Several patterns stand out: No single scanner performs best across all vulnerabilities. Some scanners achieve near-100% pass rates on certain issue types while scoring 0% on others. "Easy" and "hard" vulnerabilities are starkly separated. Some vulnerabilities are caught at 100% by all scanners; others are detected sporadically by only one or two. Ensemble coverage exceeds any individual scanner. Taking the best pass rate across all scanners for each vulnerability, every issue reaches ≥80% coverage - a profile no single scanner achieves alone. The point is not to stack as many scanners as possible. What matters is which combinations actually close each other's blind spots in real projects and benchmarks, without significantly increasing noise. We deploy a set of primary scanners for baseline coverage, supplemented by specialized scanners that activate under specific conditions. For example, when the system encounters a Move or Solana project, it engages scanners equipped with domain-specific security knowledge curated by our auditors with relevant expertise. 4. Multi-Stage Validator: Low False Positive Rate as a Core Goal Higher detection does not automatically translate into a better user experience. As scan signals increase, another problem immediately surfaces: noise. For developers and auditors, finding issues is not the finish line. If the output is cluttered with duplicates, overstated claims, or findings lacking context, it still consumes significant time in manual triage. Our goal is to deliver results closer to actionable issues, not a pile of raw alerts. To achieve this, CertiK AI Auditor applies multi-round deduplication and multi-stage validation after scanning. The deduplication stage performs multiple rounds of function-level and file-level merging, consolidating duplicate findings from different scanners into single issues. The multi-stage validator then examines each finding through successive stages, each evaluating from a different angle. Early stages focus on structural and syntactic correctness, while later stages assess semantic validity, contextual relevance, and exploitability. Evidence from all stages is aggregated into a final verdict. Verdicts are not a simple valid/invalid binary. The system supports four states: valid, invalid, partially valid, and discussion. Real security findings often contain mixed claims. Some warrant refinement rather than outright dismissal. The impact is visible in our internal case study. Starting from 367 raw findings, multi-round deduplication reduced the set to 84 distinct issues - a 77% reduction. Multi-Stage Validation then classified these as: 25 valid, 47 false positives, and 12 out of scope. This matters because we're not just trying to maximize the number of flagged concerns. We care whether the final output is something a development team would actually review and act on. Validated findings pass through an Enhancer stage that assigns severity levels, refines descriptions, and generates remediation recommendations - bringing the output closer to actionable security intelligence. 5. Auditors in the Loop: Why This System Is Not a Black Box There is another important reason CertiK AI Auditor has been able to make steady progress toward these goals: it was not developed in isolation from real audit practice. This system was built to serve auditors, and auditors have been shaping it continuously. How real vulnerabilities are defined, which results are worth reviewing, which cases should enter the benchmark, which emerging attack patterns should be encoded into the knowledge base - these decisions come from accumulated audit experience, not from the model itself. Specifically, auditor involvement spans several critical layers of the system: Knowledge Base: Auditors with diverse domain expertise curate structured security knowledge that serves as an external knowledge source during model inference. When a new attack pattern emerges, it can be injected into the system immediately, benefiting every scanner. Benchmark Design: Auditors design test cases that fall outside model training data, probing the boundaries of model capability. We classify benchmarks into two categories - capability evals (frontier problems where scanners perform weakly, driving capability improvement) and regression evals (problems scanners already handle reliably, ensuring no degradation). This creates a ratchet effect: the capability baseline can only move upward. Feedback Loop: Continuous expert review of system outputs directly drives knowledge base updates, benchmark expansion, and scanner tuning. This also means we do not treat missed cases as exceptions to be hidden. Misses are often the starting point for benchmark expansion and system iteration. In one internal test, a reentrancy vulnerability was detected by three scanners but incorrectly classified as invalid by the validator. That case was added to the benchmark dataset to improve validator calibration. What we care about is not just "did we catch it this time," but "why was it missed, which stage failed, and how do we absorb this back into the system." This makes CertiK AI Auditor closer to an engineering system that can be continuously refined - rather than a black-box model that occasionally produces impressive results. 6. Public Testing We do not claim that CertiK AI Auditor can replace human auditing. Web3 security still involves substantial challenges that depend on business logic, system-level interactions, and expert judgment. Our auditors, after using this system for over six months, continue to demonstrate vulnerability intuition and a holistic perspective that the system does not match. The system also has clear capability boundaries. That said, we believe this tool is ready to serve as an additional security layer in the development process, particularly well-suited for: Pre-deployment self-review Protocol upgrade and diff review Pre-audit triage and screening Post-audit verification The purpose of this release is not to declare the problem solved. It is to take a tool that has been refined internally over an extended period and has shown initial validation against real incidents and real audit cases, and subject it to broader testing across the Web3 development ecosystem. If what you care about is not just "can AI find vulnerabilities," but "can AI genuinely help development teams surface the security issues worth addressing, earlier," we hope CertiK AI Auditor is worth trying. If you’re interested in participating in the beta, you can reach out to one of the team members below. @xy9301 @hudsonjameson @de3p4c1to Learn more about CertiK AI Auditor here. Appendix Stats for 2026 incident detections: