Methodology

Dataset construction, labeling, evaluation, and known limits.

Data sources

Tasks A, C, and D draw on ICFR disclosures retrieved from SEC EDGAR: Item 9A of a 10-K, Item 15 of a 20-F, and the equivalent section of a 40-F. Foreign private issuers filing 20-F and 40-F are registrants complying with SOX rather than with their home regime, so they broaden the company population without changing the regulatory one. Every item records its own jurisdiction and regime.

Task A also draws on Single Audit findings from the Federal Audit Clearinghouse. These are internal control over federal award compliance rather than ICFR in the Section 404 sense — a distinct sub-population, tagged as such and reported separately. They are included because they carry something SEC filings cannot: findings explicitly classified as material weakness or significant deficiency.

Task B items are hand-authored control-design scenarios, including minimal pairs — two scenarios identical except for one deliberate change. LLM-drafted variants are reviewed before inclusion, and every synthetic item records which model drafted it.

Why severity is only scored on part of the corpus

SEC rules compel public disclosure of material weaknesses. Significant deficiencies are reported to the audit committee, and lesser deficiencies are not reported at all. A severity label taken from a periodic filing is therefore almost always "material weakness" — not because that reflects the distribution of real deficiencies, but because it is the only tier the disclosure regime surfaces.

Severity is scored only on items whose source could have expressed a different tier: Single Audit findings and synthetic scenarios. Task A's primary label is the COSO component, which is well-populated across every source and is jurisdiction-neutral.

Why clean items are rated, not just counted

The two labels in Task C are not equally strong. A deficient label comes from management stating a material weakness. A clean label comes from nobody having said anything — absence of disclosure, not absence of deficiency.

This matters because the false-positive rate on clean items is the headline metric. If a model flags a filing that management called effective but which was later restated, that is not a false positive; the model may be right and the label wrong. Two controls apply:

Filings later contradicted by an Item 4.02 8-K ("Non-Reliance on Previously Issued Financial Statements"), or by a subsequent-period material weakness covering the same period, are excluded from the clean pool entirely.
Remaining clean items are rated attested where a registered public accounting firm issued an ICFR opinion behind the assertion, or management_only where the assertion is management's alone. The headline false-positive rate is reported on the attested subset; the management-only rate is reported alongside and labelled as the weaker measurement it is.

Labeling

Every item carries a label_source field recording how its ground truth was established:

Value	Meaning
`disclosure`	Taken from the filer's own characterization in Item 9A
`human`	Assigned by a human reviewer
`llm_draft_human_verified`	Drafted by a model, reviewed and accepted by a human

No item enters a released version without human review. Published results report the breakdown by label_source.

Contamination policy

EDGAR filings are in the training data of every model evaluated here. This is disclosed rather than assumed away, and three mitigations are applied:

Filings from recent fiscal years are prioritized, past the training cutoff of most evaluated models.
A paraphrased subset of disclosure items is maintained. Both the verbatim and paraphrased scores are reported; the gap between them separates memorization from reasoning.
Task B items are synthetic and therefore contamination-free by construction. Task B is the reasoning control group.

Synthetic authorship carries its own conflict: a model that drafts scenarios should not be evaluated on them without that being visible. Every synthetic item records the model that drafted it, and results are reported split by drafting model.

Evaluation

One standardized prompt template per task, versioned alongside the dataset. Models are asked for structured JSON output; unparseable responses are recorded as parse failures and reported, not salvaged by pattern-matching the answer out of prose. Every model runs on the full item set — no sampling. Every raw response is retained.

Model versions are pinned to full identifiers and recorded with each result. A change to a prompt template produces a new results version; scores are never compared across prompt versions.

Limitations

English-language filings only. Version 1 is predominantly SOX-404 reporting, including foreign registrants who file under it.
Version 1 excludes IT general control scenarios.
No fine-tuning, tool-use, or multi-agent evaluation.
Task A labels derived from disclosures inherit the filer's own severity characterization, which is a judgment made under disclosure incentives. This is why severity is scored only on gradient-bearing sources.
Clean labels remain an absence of disclosure rather than an independent verification. Restatement exclusion and attestation rating narrow the problem; they do not eliminate it. A company with an undisclosed and never-restated weakness still sits in the clean pool.
Single Audit findings concern internal control over federal award compliance, which overlaps with but is not identical to ICFR under Section 404.
Synthetic scenarios are cleaner and better organized than real disclosure prose. A model may perform differently on them for stylistic reasons alone.

Licensing

Dataset: CC-BY 4.0. Code and eval harness: MIT. COSO and PCAOB frameworks are referenced for taxonomy and cited by principle or standard number; their text is not reproduced.