A public benchmark for LLM evaluation of internal control over financial reporting.
Existing financial-domain benchmarks test whether models can find errors in financial statements. None of them test whether a model can evaluate a control — judge whether it is designed effectively, classify the severity of a deficiency, or cite the right standard.
AuditAgent Bench measures that. It draws on SEC filings, Single Audit findings, and hand-authored control scenarios, across four tasks.
The framing is ICFR rather than SOX. SOX 404 is one jurisdiction’s mechanism for a concept that also exists under NI 52-109 in Canada and Uniform Guidance for federal award recipients. COSO underlies all of them, so the taxonomy is anchored on COSO components rather than any single regime’s severity language.
Given a disclosed control weakness, assign severity (deficiency, significant deficiency, or material weakness) and the relevant COSO component.
Given a process narrative and a control description, judge whether the control is designed effectively. If not, identify the gap: segregation of duties, precision, evidence, timeliness, or review.
Given a mix of clean and deficient Item 9A disclosures, flag them correctly. The false-positive rate on clean items is the headline metric — prior work found error-detection benchmarks where most models flagged nearly every clean input.
Given an identified deficiency, cite the correct COSO principle or PCAOB standard.
Dataset construction in progress. No results have been published yet.
All source text is public: SEC EDGAR filings, plus synthetic scenarios authored for Task B. Every ground-truth label in a released version is human-reviewed. The dataset is released under CC-BY 4.0; the eval harness under MIT.