Documentation

What it is

ef-vulns-bench is a benchmark for evaluating LLMs and agents on detecting security vulnerabilities in Ethereum execution-layer and consensus-layer client code. It is built on the Ethereum Foundation's public disclosure archive: 118 structured findings drawn from real-world security reviews across 24 upstream client repositories — Geth, Erigon, Reth, Nethermind, Besu, Lighthouse, Prysm, Teku, Nimbus, Lodestar, Grandine, Vyper, and others.

Every finding carries verified introduced / last-available / fix commit hashes, file-and-line components, and a long-form summary written by EF protocol-security reviewers. Labels are trustworthy because they came from the people who triaged the original vulnerability.

Why a benchmark on real findings matters

Most code-vulnerability benchmarks lean on synthetic bugs or CWE-style toy examples — useful, but easy to overfit and not representative of what an auditor actually sees. ef-vulns-bench is built from real, published, verified vulnerabilities in production Ethereum clients, so it grades the kind of reasoning a real reviewer needs: understanding the report, finding the right code path, and reasoning about commit-relative state.

The dataset also addresses one analytical problem most code-bench datasets ignore: cross-finding presence at a commit. If finding F1 says "bug introduced at C1" and F2 says "bug at C2", commit C2 often still contains F1. Naive labelling would mark only F2 as the ground truth at C2; this dataset computes the full set of findings present at every evaluation commit using git ancestry analysis.

The four tasks in detail

Tasks are organized along two axes. Scope: T1 and T3 reason over the whole codebase at a commit; T2 and T4 reason over a single commit's diff. Shape: T1 and T2 are open-ended (list every applicable finding); T3 and T4 are closed (yes/no for one specific report). Only T3 currently has a public leaderboard.

T3 presence categories

T3 negative cases split into adversarial buckets, each testing a different reasoning failure. The leaderboard reports accuracy per category in addition to overall.

Metrics

Accuracy

Fraction of completed predictions that match the ground-truth label. Null/failed predictions are reported as a coverage gap, not a wrong answer.

Precision (per label)

Of predictions of this label, how many were correct.

Recall (per label)

Of cases with this label, how many were caught.

F1

Harmonic mean of precision and recall.

Stratified accuracy

Accuracy computed separately within each adversarial bucket — exposes failure modes that overall accuracy hides.

How models get on the board

Each model is run by the triager-agent harness over the T3 test set. Its prediction file is committed under results/t3/<harness>/<model>/. This site re-scores those files weekly against the parquet ground truth from ef-vulns-bench, then publishes the normalized leaderboard.

To add a model: open a PR to triager-agent with a test_preds.jsonl in the conventional directory and a short README.md describing the run.

Citation

Coming soon — see ef-vulns-bench for the canonical citation.