# Source data needed for a useful AI Engineer retrieval benchmark

Status: data-provider backlog, informed by human review of question batches 1 and 2. This is not a benchmark result or a claim that the missing facts are unavailable forever.

## What the current corpus supports

The pinned 2025 and 2026 program snapshots are already useful for:

- finding a talk from a natural-language description;
- returning speaker names from a talk;
- finding other talks connected through explicit speaker records;
- resolving co-presenters and numbered workshop series;
- answering basic company or role questions when those fields are present.

Human review kept 27 of 40 batch-two questions. The strongest accepted questions used talk descriptions or meaningful speaker-to-talk relationships. Room/day lookup, snapshot-record mechanics, and administrative schedule completeness were judged low-value.

Until the following source layers exist, describe this suite as **basic program retrieval**, not comprehensive first-party conference knowledge retrieval.

## P0 — provide before the next benchmark expansion

### 1. Recording and timestamped transcript coverage

Provide a durable mapping for every published talk:

```text
talkId
videoId
sourceUrl
startSeconds
durationSeconds
sourceRange
language
captionSource          # human, platform captions, ASR provider
rawTranscriptPath
cleanTranscriptPath
transcriptChecksum
reviewStatus           # raw, machine-cleaned, human-reviewed
```

Keep raw timestamped output as well as cleaned reader-facing Markdown. Timestamp links must use absolute source-video seconds. Workshop and livestream ranges must preserve the authoritative event boundaries.

This unlocks exact quotations, “where in the talk?” questions, Q&A retrieval, demo narration, and links that open at the supporting moment. A planned demo in an abstract must remain distinct from an outcome observed in video.

### 2. First-party recording and resource links

Attach typed resources to stable talk IDs:

```text
resourceId
talkId
type                   # recording, slides, repository, demo, paper, dataset, notebook
url
title
source
publishedAt
verifiedAt
availabilityStatus     # announced, published, removed, unknown
```

This is needed for questions such as “where is the recording?”, “where is the promised harness?”, and “can I get the slides or code?” An abstract stating future intent is not evidence that an artifact was published.

### 3. Canonical cross-event people

Provide a reviewed identity layer instead of leaving every event-scoped speaker record isolated:

```text
personId
displayName
aliases[]
eventSpeakerIds[]
organizationAffiliations[]
identityEvidence[]
reviewStatus
```

Exact normalized-name matches can seed candidates, but the durable artifact should record the accepted merge and any exceptions. Benchmark gold should consume the curated `personId`, not recreate identity decisions from names during retrieval.

This unlocks reliable cross-year questions such as “what did this person present in 2025 and 2026?” while retaining an auditable correction path for collisions.

### 4. Historical organization and role normalization

For each speaker appearance, preserve the affiliation at the time of the event:

```text
personId
eventSpeakerId
organizationId
organizationName
roleTitle
validAtEvent
source
```

Do not overwrite historical affiliations with a speaker’s current company. Organization IDs and aliases should support questions by company without treating spelling variants as separate entities.

## P1 — high-value enrichment

### 5. Talk-content entities and claims

Derive structured enrichment from reviewed transcripts and abstracts:

- products, models, frameworks, datasets, companies, and research papers mentioned;
- explicit claims and recommendations with timestamped evidence;
- demonstrations and whether an outcome was actually observed;
- audience Q&A segments;
- curated topics and subtopics;
- key points labeled as generated/derived, never transcript quotations.

Each derived item needs a source talk ID, timestamp range where applicable, generator/version, checksum, and review status.

### 6. Playlist and editorial taxonomy

Map public playlist entries to canonical talks by stable video and talk IDs. Preserve:

- playlist ID and title;
- video ID;
- canonical talk ID;
- editorial topic labels;
- ordering and snapshot date;
- unresolved or multi-talk video mappings.

Playlist membership is useful editorial evidence, but it should not replace event-program identity or be counted twice when one video appears in several playlists.

### 7. Series, repeats, and replacement semantics

Explicitly distinguish:

- numbered workshop parts;
- repeated performances of the same session;
- duplicate records for one occurrence;
- genuinely separate sessions sharing a title;
- moved, cancelled, or superseded entries.

Use a stable `seriesId` or `canonicalSessionId` plus a typed relationship. Do not infer “moved” or “duplicate” from matching titles alone.

## P2 — only if product needs justify it

### 8. Aggregate audience and outcome evidence

Possible additions include aggregate attendance, ratings, or published follow-up outcomes. Keep attendee identity and movement out of the public retrieval corpus unless there is a separate privacy-reviewed requirement.

## Do not optimize the benchmark around

- room/day lookup merely because schedule data is easy to obtain;
- internal record-duplication mechanics that attendees do not care about;
- birth dates or unrelated personal facts;
- claims that a planned demo succeeded without recording evidence;
- exhaustive schedule listing as a proxy for useful knowledge retrieval.

## Promotion condition

Do not call the suite a durable first-party conference knowledge benchmark until, at minimum:

1. canonical person IDs cover the reviewed cross-event speakers;
2. published recordings are mapped to canonical talk IDs;
3. timestamped transcripts cover multiple events and topic families;
4. talk resources have typed, verified links;
5. the benchmark reports coverage for each source layer rather than treating missing data as a retrieval failure.