swyxbench Working paper · AI systems / information retrieval Source ↗

Working paper 001 · preliminary dataset

Static relationship projection versus runtime graph expansion for first-party conference retrieval

swyxbench research archive · AI Engineer World’s Fair program snapshots, 2025–2026

1,612 documents2,560 relationships40 frozen questionsLocal service cost: $0

Abstract

We compare deterministic lexical retrieval, per-field BM25F, source-backed alias matching, corpus-fitted latent semantic indexing, static relationship projection, bounded runtime graph traversal, and optional neural retrieval on an incomplete AI Engineer program corpus. On the frozen 40-question regression set, BM25F with source-derived relationship labels reaches 0.9643 Recall@10 and 0.8347 nDCG@10, while runtime graph expansion reaches 0.8504 and 0.7974 respectively; both attain 1.0 multi-hop Success@10. These observations support retaining explicit relationships without adding a graph database for this particular fixture. They do not establish performance on a complete conference knowledge corpus.

1

Research question

Does preserving repository-owned speaker–talk relationships improve retrieval enough to justify a dedicated graph backend? The practical comparison is not “graph versus no graph”: it is whether runtime traversal adds value beyond a deterministic lexical index enriched with the same source-derived relationships.

Canonical facts remain versioned repository JSON with stable IDs and explicit source provenance. Graph databases, hosted vector stores, and production index services are optional deployment choices, not sources of truth.

2

Dataset and evidence boundary

The portable fixture contains 1,612 sanitized documents and 2,560 relationships generated from pinned AI Engineer World’s Fair 2025 and 2026 program snapshots. Each source-derived document retains its upstream path, checksum, JSON pointer, and stable source entity ID.

AuthoritativeProgram entities, speaker references, stable IDs, and source-owned relations.
DerivedSanitized documents, expansion labels, rankings, figures, and measured receipts.
UnavailableComplete reviewed transcripts, attendee behavior, recording links, released artifacts, and resolved cross-event person IDs.

The frozen questions include exact speaker/talk lookup, topical discovery, event/year filters, multi-hop relations, three caption excerpts, and unanswerable prompts. This distribution overweights graph-related talks and does not represent the full conference portfolio.

3

Experimental design

Every arm receives the same source-derived corpus, query text, structured filters, and relevance judgments. Retrieval and support decisions are produced before evaluator-only labels are joined. The required benchmark runs offline without paid services.

  1. Lexical baselines: SQLite FTS5 BM25 and independently normalized field-aware BM25F.
  2. Entity assistance: source-backed alias/title trigrams fused with BM25F rankings.
  3. Relationship comparisons: static source-derived speaker–talk projection versus bounded runtime traversal.
  4. Semantic controls: corpus-fitted 64-dimensional LSI; pinned BGE-small and MiniLM remain optional exploratory arms.
  5. Safety: unsupported-answer rate, provenance-valid cited evidence, and answerable false abstention are measured separately from ranking quality.
4

Results

Grouped bar chart comparing Recall at 10, nDCG at 10, MRR, and multi-hop success for BM25, aliases, static relations, runtime graph expansion, and LSI.
Figure 1. Local retrieval arms evaluated on the same 40 frozen questions. Values are fixed-set regression measurements; the incomplete source corpus and graph-skewed query mix limit external validity.
Table 1. Exact measured scores for the required offline retrieval systems.
SystemRecall@10nDCG@10MRRMulti-hop@10Decision
Loading checked receipt…

Static relationship projection improves Recall@10 by 0.114 relative to runtime graph traversal while matching its multi-hop success. Runtime traversal has the strongest MRR, indicating better first-hit ordering for some questions, but does not justify a graph backend for the present corpus.

Architecture and evidence gates

5

Exploratory neural comparison

Pinned BGE-small exact retrieval and MiniLM cross-encoder reranking provide optional research controls. They are excluded from required offline CI and their newly surfaced relevance pool has not yet been fully adjudicated.

Grouped bar chart comparing optional BGE-small dense retrieval and MiniLM reranking systems across the frozen retrieval metrics.
Figure 2. Optional neural systems on the same incomplete fixed set. Because the candidate pool is provisional, these results are exploratory and should not be interpreted as a model leaderboard.

Inspect the optional neural result receipt →

6

Limitations and threats to validity

  • The corpus is not the complete AI Engineer knowledge dataset; it largely contains public schedule metadata.
  • Only one video contributes transcript excerpts, so transcript retrieval and quotation quality are not broadly tested.
  • The original 40-question set is biased toward graph and retrieval talks.
  • Cross-event speaker identities, recording links, released artifacts, and rich transcript-derived entities are not yet available.
  • Support and traversal policies were debugged against this fixed set; the reported scores are regression evidence, not untouched generalization.
  • Latency observations depend on one local machine and are not production performance guarantees.

Read the prioritized source-data requirements → Review the more diverse candidate questions →

7

Reproducibility and artifacts

git clone https://forge.smol.ai/swyx/bench.git
cd bench
corepack enable
pnpm install --frozen-lockfile
pnpm bench check aie-retrieval

The checked receipt contains the corpus checksum, source receipts, environment, frozen query/judgment checksums, ranking metrics, source-derived vector configuration, and evaluation gates.

Measured result JSONSource receiptsSuite manifestSource code ↗