Reading a model card like an auditor

TL;DR

• The 2018 model card proposal asked for nine sections, including training data, disaggregated results and out-of-scope uses, in one to two pages.
• Of five current cards compared here, only Meta’s Llama 4 card reports training compute and emissions; none reports results by demographic group; the DeepSeek and Qwen cards have no training data or safety section at all.
• The EU AI Act’s Annex XI now requires documentation of training data provenance, compute and energy from providers of general-purpose AI models, with Commission enforcement powers from 2 August 2026.
• The 2025 Stanford Foundation Model Transparency Index found the industry average fell from 58 to 40 out of 100, with training data, compute and post-deployment impact the least disclosed areas.

A model card is the document that ships with a model and says what it is, what it was trained on, how it was evaluated and where it should not be used. Nearly every release now has one, but the word covers everything from a 135-page system card to a README with a parameter count and a benchmark table. This article explains what the original template asked for, compares what five current cards from Meta, Google, Anthropic, OpenAI and DeepSeek disclose and omit, sets out the legal requirements now attached to this documentation, and gives a table of audit questions with what a good answer looks like.

The 2018 template

The term comes from Model Cards for Model Reporting, posted to arXiv in October 2018 by Margaret Mitchell and eight co-authors and presented at the FAT* conference in January 2019. The paper proposed “short (one to two page) records we call model cards” with nine sections: Model Details (developer, date, version, licence, contact); Intended Use (primary uses, primary users, out-of-scope uses); Factors, which “could include demographic or phenotypic groups, environmental conditions, technical attributes, or others”; Metrics; Evaluation Data; Training Data; Quantitative Analyses, split into unitary results (per factor) and intersectional results (at the intersection of factors); Ethical Considerations; and Caveats and Recommendations.

Two features matter for a reader today. First, disaggregation, meaning results broken down by group or condition rather than one aggregate score, was the central idea. The abstract describes cards that “provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups.” Second, the paper anticipated that training data would often be withheld. The section “May not be possible to provide in practice,” it says, and when full detail is not possible, “minimal allowable information should be provided here, such as details of the distribution over various factors in the training datasets.” The fallback was a summary, not silence.

The companion document for data is Datasheets for Datasets by Timnit Gebru and colleagues, first posted in March 2018 and published in Communications of the ACM in 2021. By analogy with the electronics datasheet, it proposes that every dataset document “its motivation, composition, collection process, recommended uses, and so on.” For large language models, cards and datasheets have rarely shipped together.

The Hugging Face template

The most widely used operational version is Hugging Face’s Model Card Guidebook (Ozoani, Gerchick and Mitchell, 2022) and its annotated template, which adds an Environmental Impact section asking for hardware type, hours used, cloud provider, compute region and carbon emitted. Its note on Evaluation is candid: it “is ideally disaggregated”, “However, this section is most commonly used to report aggregate evaluation performance on different task benchmarks.”

Five current cards compared

The table compares five documents fetched for this article: two cards for open-weight models (Llama 4, Gemma 3), two system cards for closed models (Claude Sonnet 4.6, GPT-5) and one Hugging Face card for an open-weight model from a Chinese lab (DeepSeek-V3.1). A system card is the term OpenAI and Anthropic use for a document covering the deployed system, its safeguards and its risk-framework decisions, not only the model.

What each card does well

The Llama 4 model card is the only one that reports compute in a checkable form: “Model pre-training utilized a cumulative of 7.38M GPU hours of computation on H100-80GB (TDP of 700W) type hardware,” with “1,999 tons CO2eq” of estimated location-based emissions. It describes training data as “a mix of publicly available, licensed data and information from Meta’s products and services,” with a cutoff of August 2024, and its LiveCodeBench entry carries a problem date range (10/01/2024 to 02/01/2025), which matters because that benchmark’s problem set changes over time.

The Gemma 3 card gives token counts per size (the 27B model “trained with 14 trillion tokens”), describes filtering including CSAM filtering “at multiple stages,” and states shot counts. It also contains a scope statement of the kind the 2018 Caveats section was designed to hold: “A limitation of our evaluations was they included only English language prompts.”

The Claude Sonnet 4.6 system card, dated 17 February 2026, runs to 135 pages across capabilities, safeguards, an alignment assessment, agentic safety and evaluations under Anthropic’s Responsible Scaling Policy. Its data paragraph says the model “was trained on a proprietary mix of publicly available information from the internet up to May 2025, non-public data from third parties, data provided by data-labeling services and paid contractors, data from Claude users who have opted in to have their data used for training, and data generated internally at Anthropic.” It reports per-language results, states that because “Claude Sonnet 4.6 is not a frontier model” it was not tested by government partners before release, and carries a dated changelog: on 6 March 2026 a BrowseComp score was revised from 74.72% to 74.01% after “an improved cheating detection pipeline” flagged nine unintended solutions.

The GPT-5 system card, dated 13 August 2025 and 60 pages, is almost entirely a safety document. It names four external evaluators, reports “more than 5,000 hours of work from over 400 external testers and experts,” and explains a precautionary decision: OpenAI chose “to treat gpt-5-thinking as High capability in the Biological and Chemical domain” while stating “we do not have definitive evidence that this model could meaningfully help a novice to create severe biological harm.” A later arXiv version added monitorability evaluations.

What each card omits

Neither closed-model card discloses parameter counts, architecture or compute. The GPT-5 data section is one paragraph: the models “were trained on diverse datasets, including information that is publicly available on the internet, information that we partner with third parties to access, and information that our users or human trainers and researchers provide or generate.” No quantities, no cutoff, no source list. The Sonnet 4.6 card gives a cutoff and a crawler policy but also no quantities. The Gemma 3 card names TPU generations but no chip counts, hours, energy or emissions, and reports no dangerous-capability evaluations. The Llama 4 card names no external red-teaming organisation and gives no per-language results despite listing 12 supported languages.

The DeepSeek-V3.1 card reports 671B total and 37B activated parameters, context-extension token counts, a benchmark table with dated windows and an MIT licence, but no training data description, safety evaluation, compute figure or disaggregated results. The Qwen3-235B-A22B card follows the same pattern: 235B total and 22B activated parameters, Apache 2.0, and nothing on data, compute or safety. For these models the licence is often the most consequential document; see our map of open-weight licences.

None of the five reports results by demographic group. The closest any comes to the 2018 paper’s intersectional analysis is the per-language tables and the BBQ bias benchmark in the Anthropic and OpenAI cards.

The audit questions

The checklist below is written for a reader assessing a card, not a developer writing one.

Regulatory pull

Documentation is no longer only a norm. Under Article 53 of the EU AI Act, providers of general-purpose AI models must “draw up and keep up-to-date the technical documentation of the model, including its training and testing process” per Annex XI, give downstream integrators documentation per Annex XII, “put in place a policy to comply with Union law on copyright and related rights,” and “draw up and make publicly available a sufficiently detailed summary about the content used for training.” Article 53(2) exempts models “released under a free and open-source licence” with public parameters from the first two duties, but not from the other two, and not if the model is designated as posing systemic risk.

Annex XI is, in effect, a statutory model card. It requires intended tasks, acceptable use policies, release date, architecture and parameter count, modality and licence, plus “information on the data used for training, testing and validation, where applicable, including the type and provenance of data,” “the computational resources used to train the model (e.g. number of floating point operations),” and “known or estimated energy consumption of the model.” Systemic-risk models must add evaluation results and “measures put in place for the purpose of conducting internal and/or external adversarial testing (e.g. red teaming).” Of the five cards above, only Llama 4 comes close to the compute item, and none gives a FLOP figure.

The Commission’s General-Purpose AI Code of Practice was published on 10 July 2025 with three chapters: Transparency, which includes a Model Documentation Form; Copyright; and Safety and Security, for systemic-risk models only. The training-content summary template followed on 24 July 2025. According to the Commission’s guidelines page, the obligations applied from 2 August 2025, “the Commission’s enforcement powers enter into application” from 2 August 2026, and models placed on the market before 2 August 2025 must comply by 2 August 2027. The Commission’s Q&A gives the systemic-risk presumption as “a threshold of 10^25 floating-point operations (FLOP) used for training the model.” A card that omits compute omits the number that decides which tier of the law applies.

The transparency index

The 2025 Foundation Model Transparency Index from Stanford’s Center for Research on Foundation Models, published in December 2025, scores developers out of 100. Its abstract reports that “the average score out of 100 fell from 58 in 2024 to 40 in 2025,” that “Companies are most opaque about their training data and training compute as well as the post-deployment usage and impact of their flagship models,” and that IBM scored 95 against “the lowest scorers, xAI and Midjourney, at just 14.” Alibaba, DeepSeek and xAI were scored for the first time. The authors note that the five Frontier Model Forum members they score “end up in the middle of the Index” and suggest they “avoid reputational harms from low scores but lack incentives to be transparency leaders.” Full per-company scores are on the results page.

The paper that coined the term expected training data to be withheld and asked for a distributional summary in its place. Seven years later, the most common answer is still a one-sentence list of source categories.

Limits of the card

A card is a self-report, and even commissioned external evaluations are chosen by the developer. Independent reproduction of a score depends on the setup being disclosed, which is why versions, shots and formats matter more than the numbers. For how evaluation harnesses differ, see our guide to comparing evaluation frameworks; for why a reasoning model’s score depends on its thinking budget, see our explainer on test-time compute.

Closed-model cards describe a system that can change without a new card; the Sonnet 4.6 changelog is the exception. Open-weight cards describe base weights, not the fine-tuned derivative a buyer deploys. And no card examined here reports the intersectional results the 2018 paper placed at the centre of the idea. The template exists, the law now names most of its fields, and the index measures the gap. Reading a card well means reading it against all three.