Confluent Developer · Video summary
Hacking Kafka Streams with Sophie Blee‑Goldman | Ep. 15 | Confluent Developer Podcast
Sophie Blee-Goldman shares how her team hacked Kafka Streams to overcome partitioning limits by implementing an async processor wrapper to enable parallel processing within partitions.
Watch on YouTube ↗Key takeaways
- 🎯The maximum parallelism in Kafka Streams is fundamentally limited by the number of partitions in upstream topics, not by the number of available cores or threads, creating a bottleneck for high-growth applications.08:52
- 🚀Evaluate your Kafka Streams application's throughput limitations and identify if partitioning is a bottleneck for your growth.09:51
- 💡Modifying Kafka Streams' core threading model to allow multiple threads to process records from a single partition in parallel is challenging without forking the project, due to its reliance on ordered processing and offset commits.15:27
- 🚀A workaround for Kafka Streams' parallelism limits can be achieved by creating a wrapper around the user's processor that injects custom logic before the user processor is invoked, effectively hijacking the stream thread.20:01
- 🔍By introducing an 'async processor' wrapper, each stream thread can hand off records to a separate thread pool for actual processing, allowing more threads to work on a partition than the number of partitions allows.22:43
- 💡Consider implementing an async processing wrapper or similar pattern if you need to scale beyond Kafka's default partition parallelism limits.22:43
Chapters
- 0:00🎙️ Introduction to Sophie Blee-Goldman
- 1:23🧑💻 Sophie's Background and Current Role
- 3:00🏢 Early Career: Internship at Google
- 6:11🔗 Challenges in Open Source Collaboration
- 8:13📈 The Core Problem: Scaling Kafka Streams
- 10:15⏳ Approaching the Throughput Limit
- 11:10⚙️ Constraints: No Forking Kafka Streams
- 15:14💡 The Hack: Injecting a Solution
- 18:27🧵 Hijacking Stream Threads
- 22:43🔄 Introducing an Asynchronous Thread Pool
- 23:27🚀 Enabling Higher Parallelism
- 24:53🎉 Solution Success and Naming
- 25:11🎬 Conclusion and Further Resources
- 25:57✅ Maintaining Ordering Guarantees (Per Key)
- 30:06🔗 Handling Commits and Flushes
- 32:37🔌 Custom Kafka Clients
The digest
High Level Overview
This discussion with Sophie Blee-Goldman, a Kafka community expert, unpacks a significant scalability challenge within Kafka Streams: the inherent limit of parallelism set by the number of Kafka partitions. Facing the constraints imposed by Kafka's architecture and a tight one-month deadline for a high-growth AI startup client, Sophie and her team engineered an innovative solution that injects asynchrony and parallelism into Kafka Streams without forking the open source codebase. This episode is essential for software engineers and architects working with Kafka who encounter scalability bottlenecks, providing insights into working within open source constraints while creatively hacking Kafka’s threading model.
Keep reading in The Digest
The full 5-minute article, 10 more sections, and every new episode from Confluent Developer and the creators you follow.
- Structured Deep Dive
- Background and Context: Kafka Streams Scalability Limitations [08:30–13:05]
- The Fundamental Challenge: Kafka Streams Threading Model [15:40–17:00]
- Constraints of Open Source Contribution and Time Pressure [13:15–15:10]
- The Core Innovation: Injecting Parallelism via Processor API Wrapping [18:29–23:40]
- Maintaining Correctness: Key-Based Ordering Guarantees [26:40–30:00]
- Handling Offset Commits and Kafka Client Integration [30:20–33:40]
- Key Framework: Async Processor Architecture for Kafka Streams
- Practical Implications & Takeaways
- Action-Oriented Summary
More from Confluent Developer
Inside OpenAI’s Streaming Backbone with Aravind Suresh | Ep. 24 | Confluent Developer Podcast
OpenAI's streaming backbone uses abstractions and automation to manage 10x scale and deliver reliable infrastructure for AI products, prioritizing user experience over complex primitives.
From Git Blame to Principal Engineer with Sage Pierce | Ep. 22 | Confluent Developer Podcast
Principal Engineer Sage Pierce discusses scaling through others and the challenges of event-driven views in microservice architectures, introducing the IMI pattern.
Inside $3M GPU Racks: Powering Modern AI with Bryan Oliver | Ep. 16 | Confluent Developer Podcast
Discover how GPU scheduling and variability-aware techniques are transforming AI workloads in cutting-edge data centers with Bryan Oliver.
Turning Chaos into Push-Button Provisioning with Dhiraj Suri | Ep. 14 | Confluent Developer Podcast
Transforming disparate systems into a seamless provisioning process is crucial in enterprise software development, as highlighted by Diraj Suri's experiences.