Y Combinator · Video summary

Going In Deep On Data | YC Paper Club

The YC Paper Club's "Data Edition" explores the critical role of data in AI development, highlighting how expert data is the bottleneck to AI progress and the shift from model-centric to data-centric approaches.

Y Combinator · August 20, 2026 · 53-min episode · 4-min read

Key takeaways

⚡Diffusion language models offer significant speed advantages, producing over 1,000 tokens per second, which is crucial for real-time AI applications like voice agents.00:00
💰The initial perception of data as a commodity has been overturned, with data businesses generating over $100 billion in market cap, demonstrating its critical value in AI.00:56
🔄In production AI, the focus shifts dramatically from model architecture to data, with 97% of effort dedicated to data curation and analysis to address real-world complexities.03:00
🛠️Datasets and RL environments are complex products, not simple files, requiring significant craftsmanship and continuous updates to remain relevant in dynamic environments.04:15
🛑Expert data is the primary bottleneck to AI progress, as humans are essential for defining 'good' data and verifying performance at the frontier of AI capabilities.05:47
↔️Cross-lingual transfer scores are not symmetric, meaning the benefit of training with language A for language B does not necessarily equal the benefit of training with B for A, highlighting the complexity of multilingual models.48:54

Chapters

0:08Introduction to YC Data Club
0:20Why Data Matters: The Shift from Commodity to Critical Asset
1:40Data vs. Model: The Production AI Perspective
5:09The Challenge of Test Distribution in AI
9:03Expert Data as the Bottleneck to AI Progress
10:46Apps to Smartphones, Data is to LLMs: The Future of Data Businesses
15:53Scaling Expert Supervision: Snorkel AI's Approach
18:39Data 1.0: The Limitations of Manual Labeling
23:30Data Programming: Encoding Expert Supervision in Software
28:41Data 2.0: Expanding Frontier and the Data Challenge
30:35Scaling SWE 'Taste' with Senior SWE-bench
35:03Scaling Expert Rewards: The Reliability/Flexibility Trade-off
39:09Building a Calibrated 'Validation Agent' for Dynamic Rewards
42:16Live Leaderboard: Senior SWE-bench Results
43:55The Team Behind Senior SWE-Bench
44:14Conclusion: The Need for More Data Research
46:55Open Benchmarks Grants
47:24Inception: Diffusion Language Models for Production
48:30Diffusion vs. Autoregressive LLMs: Benefits and Drawbacks
49:58Why Diffusion LLMs are Especially Good for Voice
51:41Mercury 2 for Voice: Benchmarks
53:29The Need for Good Data to Build and Evaluate Good Models
55:24TauForge: An Agentic Pipeline to Generate RL Environments
56:41High-Level Overview of TauForge
1:03:21Improving Upcoming Model Performance with TauForge
1:05:29Mercury 2: Parallel Generation Changes the Pareto Frontier
1:07:04ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining
1:08:55Motivation: Scaling Laws Beyond English
1:11:44Optimizing for Thai (th): The Problem Setup
1:15:45Measuring Language Synergies
1:18:25Cross-Lingual Transfer Matrix
1:20:54Language Transfer Scores are Not Symmetric
1:22:09Language Script > Language Family
1:22:22Next Step: Estimating Loss and Multilingual Mixture
1:25:56Modeling the Curse of Multilinguality

The digest

For years, the prevailing wisdom in AI development, particularly in machine learning, was that the model itself was the primary driver of performance. Researchers and engineers poured countless hours into refining architectures, tweaking algorithms, and exploring new neural network designs. However, a significant shift has occurred, revealing that expert data, not just the model, is the true bottleneck to AI progress.

The Evolution of Data's Value

In the early days of AI, roughly between 2016 and 2023, data was often considered a commodity. The sentiment was, "I've downloaded ImageNet; how much more data could I possibly need?" This perspective led many, including venture capitalists, to believe that data businesses had a terminal value of zero. Yet, in the span of a decade, the data industry has generated over $100 billion in market capitalization, proving this initial assessment profoundly wrong. Companies like Scale AI, Surge AI, and Mercor have demonstrated the immense value in curating and managing high-quality data.