Research quality check
Automated quality checks
The pipeline can collect, analyse and publish without manual approval of every record. Unsupported or disputed labels are withheld. Evidence checks and model agreement do not establish factual accuracy; every record remains linked to its source.
Historical comparison · 17 October 2026 · codebook 1.0
Blind model cross-check
Two Gemini model runs independently coded the same seeded random sample of 200 articles. The second run used publisher-extracted full text for all 200 records and did not see the first run’s labels.
| Field | Matching results | Agreement |
|---|---|---|
| Candidate identity set | 175 of 200 | 88% |
| Coverage role | 159 of 200 | 80% |
| Story frame | 128 of 200 | 64% |
| Issue-tag set | 65 of 200 | 32% |
| Lead voice | 112 of 200 | 56% |
These are model-to-model agreement rates, not human validation, accuracy, or evidence that either label is correct. The weak issue-tag and lead-voice agreement means those comparisons should be treated cautiously. These results apply to that 200-article sample and its codebook 1.0 labels. They are not a quality score for the current archive, later records or other codebook versions. Each current record shows its own codebook and available check status.
Optional human review
This optional tool lets a person compare identity, tone, voice, coverage role and issue tags with the original sources. It is not required for automatic updates. No human agreement rate has been completed or published.
How the optional review sample works
The page selects 200 articles with existing coded labels from the current archive using a stable hash of each record ID. This keeps the sample repeatable across visits to the same archive. Reviewers see the published labels only after saving their independent labels for that article.
Agreement is calculated only after the sample is complete. It will compare reviewer and model labels for identity matches, tone, voice, coverage role and issue tags; an uncoded model field is excluded from that field’s denominator. This pilot is not an estimate of political bias, and it does not replace a larger multi-region validation study.
Current state: no independent agreement rate is published until a reviewer completes and exports the sample.
Read the automated quality standards ↗ · Return to Media Watch ↗