# Benchmark protocol

## Goal

Select a transcription, word-timing, and diarization pipeline that produces source-linked AI Engineer knowledge-graph evidence from YouTube and master recordings. The benchmark must beat the published Podhood transcript on correctness while preserving its useful navigation and structure.

## Silver benchmark release: `100-talk-v1`

The first multi-provider release freezes 100 retained conference recordings
from the completed 1,042-recording production corpus. The exact IDs, original
audio checksums, durations, sampling diagnostics, and unavailable labels live
in `cohorts/100-talk-v1/cohort.json`.

- Every system receives the exact retained audio bytes identified by SHA-256.
- ElevenLabs Scribe v2 is the machine-generated **silver reference**, never
  human ground truth.
- Source-derived dimensions (duration, bytes, checksum, and published title)
  support representativeness claims. Scribe speaker clusters, log probability,
  keyterm density, word rate, and edge silence are sampling diagnostics only.
- Eight title-derived multi-presenter candidates are reserved as stress
  fixtures, but must not be called panels until a human verifies the format.
- Accent, true speaker count, overlap, room/audio condition, and publication
  date remain unavailable until an authoritative metadata join or human review.
- Managed acquisition is manual, resumable, concurrency-limited, and protected
  by an explicit dollar cap. Required CI only scores frozen artifacts offline.

The silver release reports whole-transcript substitutions, insertions, and
deletions against Scribe; technical-entity disagreement; source-token timestamp
coverage before drift; speaker-count and optimal cluster-mapped attribution
disagreement; latency, throughput, retries, and actual or clearly estimated
cost. Every public label must say **silver disagreement**, never unqualified
WER, accuracy, DER, or JER.

## Human-gold test set

The later human-gold release uses at least 24 recordings totaling roughly 18
hours, selected from the frozen 100 where practical.

- Eight standalone talks: podium and lapel microphones, audience questions, varied accents, dense slides, and technical names.
- Eight interviews or panels: two to six speakers, short backchannels, interruption, cross-talk, and similar voices.
- Four long-livestream excerpts: music, applause, silence, remote callers, room noise, and edit boundaries.
- Four multilingual or code-switched sessions.

Where both the YouTube encode and a production master exist, retain both as separate cases sharing one editorial identity. Freeze the decoded audio properties and SHA-256 before running systems.

## Human gold

Two annotators independently produce:

1. Verbatim transcript and a separately normalized scoring transcript.
2. Technical entity spans with canonical forms and alias policy.
3. Speaker-turn RTTM with overlapping speech retained.
4. Speaker identities verified against the event roster and the recording.
5. Word boundaries on a stratified subset containing names, numbers, code, overlap, and silence.
6. UEM regions defining what audio is scored.

An adjudicator resolves disagreements without seeing system names. Store annotator IDs, tool version, revision, adjudication notes, and checksums.

## System execution

- Run every system against the same decoded audio bytes.
- Pin model aliases, immutable revisions where available, libraries, configuration, hardware, and warm/cold state.
- Preserve native word and segment timing before any forced alignment or cleanup.
- Run ASR and diarization as independent lanes. Do not let a strong transcript conceal weak speaker attribution.
- Run an unconstrained speaker-count pass and an oracle-count diagnostic; score them separately.
- Preserve overlap-aware diarization even if the product also needs an exclusive single-speaker view.
- Store raw provider results outside Git in content-addressed storage; check in normalized redacted receipts and hashes.

## Evaluation

- Report raw and normalized WER.
- Report technical named-entity error rate and entity precision/recall by people, companies, products, acronyms, versions, numbers, and code identifiers.
- Report collar-free DER with `skip_overlap=false`, its miss/false-alarm/confusion components, JER, speaker-count error, and named-identity error.
- Report a 250 ms-collar DER only as a secondary diagnostic.
- Score timestamp coverage against source tokens before measuring drift. Missing, merged, or untimed units receive an explicit penalty.
- Measure start, end, and midpoint absolute error against human word-boundary gold.
- Run browser seek tests: a deep link must land inside the cited utterance or at most 500 ms before it.
- Score overlap-only DER and attribution separately.
- Have blinded reviewers score summary and chapter claims against cited transcript ranges.
- Measure latency, RTF, throughput, peak memory, cost per recorded hour, correction minutes per recorded hour, and operational complexity.

## Reporting

Publish overall, per-format, and worst-decile results. A global average cannot hide a failed panel, multilingual session, or high-value technical entity. Preserve `observed`, `inferred`, and `unavailable` claims separately.