How We Built an Autonomous AI Security Triager: From LLM to 20,000+ Real Reports

In nine months, the number of reports researchers send to HackenProof every month more than doubled, from 2,564 in December 2025 to a peak of 7,785 in May 2026. It wasn't spam. Frontier AI models made every bug hunter sharper, and triage became the bottleneck.

AI Made Every Bug Hunter Sharper Overnight

The shift came with Claude Opus 4.6. Frontier models got good enough at reading code and reasoning about attack paths that every researcher got sharper at once, including everyone submitting to HackenProof.

Our report volume shows it. Monthly submissions climbed from 2,564 in December 2025 to 4,937 in March and peaked at 7,785 in May 2026. Since then, they have held between 5,700 and 6,700 a month, still more than double the December baseline.

Our triage team was sized for years of steady bug bounty growth. Within a few months, it had to absorb more than twice the load.

It Wasn't Spam

The obvious suspect was low-effort AI spam. The data said otherwise: most of the new reports were valid findings. The problem was repetition. In one case, a single real vulnerability arrived as 40+ separate reports, each describing it a different way.

That changes the economics of triage. Spam can be filtered in seconds. A valid duplicate still has to be read, matched to the original and answered, and someone has to recognize the same bug written 40 different ways.

Three Fixes We Tried First

Before building an AI triager, we tried three cheaper fixes. Only one worked, and even that one was outpaced by the models.

Paid submissions went live in June 2026, which is why volume dropped after May's peak. It never fell back to the old baseline, because the reports coming in were real.

The Real Bottleneck Was Judgment

Spam was never the bottleneck. Judgment was. Every valid report, duplicate or not, still needs a person to read it, check scope, decide severity, and write a response. That is the core of vulnerability triage, and it doesn't scale by hiring.

So we asked a different question: what if triage itself could reason?

How The Triager Reasons

The triager reasons over three sources before it suggests anything:

1. The report itself: what's claimed, what's attached, what's missing.
2. The codebase in scope: whether the vulnerable path is real, whether it's reachable, whether it's already fixed.
3. Past triage decisions: whether a report of this exact shape has been judged before.

From those, it proposes an action, and a human approves it. A stage can act alone only after it has earned that right through decisions people approved. We call the principle "propose, don't assume."

First Try: OpenClaw

The first version ran on OpenClaw with a deliberately simple setup: one program, one cron job, one prompt file, repeated for every program on the platform. It proved the idea worked and broke in three ways:

• Expensive. The more reports it triaged, the bigger its context grew and the more tokens it spent.
• Not scalable. Every new program added more for it to remember.
• No shared knowledge. What it learned on one program stayed with that program.

Rebuilding on LangChain: The Triage Graph

The second version moved to LangChain, which gave us control over execution and over how much effort each step spends.

Each triage run is a graph. It selects programs from a scan set, scans their open reports and skips the ones already decided, then fans out one triage branch per report in parallel. The branches are collected into verdicts and a review queue. If there is nothing new to triage, the run ends early.

Inside each branch, a report moves through a cascade of stages, cheapest first. Each stage may end a report only on terms it can be trusted with, and the output is always a proposal: a state, labels, and a comment for a human to review.

Two shortcuts close reports early: confident junk from screen and duplicate families from dup_check. Both still pass through challenge before anything is proposed. A report with nothing to judge halts straight to finalize. Every outcome lands in a review queue for a human, and confirmed outcomes flow back into recall as precedent for the next report.

Two Reports Through The Pipeline

Two example traces show how the cascade ends a report early without cutting corners.

Five Rules The Graph Enforces

The graph is built around five rules that no stage can override:

• Cheapest first. Rule-based steps and cheap models run before expensive ones, so a deep model only sees what survives.
• Evidence, not verdicts. Stages pass on the evidence behind a decision, not just the decision.
• A valid finding reaches a human. No automated path closes a valid report without a person seeing it.
• No comment, no closure. A report can't be closed without a drafted response to the researcher.
• Learn only from approved reports. Only outcomes a human approved become precedent.

Results: 70% of Reports Triaged With No Human In The Loop

Today, 70% of submissions on HackenProof are triaged fully autonomously, with zero humans in the loop. Our goal is 85% by the end of 2026.

That is a step beyond where we stood in March, when we launched the MCP server and wrote that AI does not replace the security engineer's judgment. For valid findings, it still doesn't: every valid report reaches a person. What changed is that a stage can now act alone on the decisions it has earned the right to make, after people approved enough of them.

The rest still reach people, but they arrive with context loaded, duplicates matched, and a decision already drafted. Triagers spend their time on the judgment calls that actually need them, and researchers get answers faster, even as AI keeps making them better at finding bugs.