AI Engineer · Video summary
Building Conversational Agents — Thor Schaeff and Philipp Schmid, Google DeepMind
In this workshop, Thor Schaeff and Philipp Schmid from Google DeepMind guide participants through building conversational agents using the Gemini API, emphasizing hands-on experience with API key creation, tool integration, and the development of coding agents capable of executing commands and generating outputs.
Watch on YouTube ↗Key takeaways
- Creating an API key is essential for using the Gemini API, and it can be done easily through the AI.dev platform without needing a credit card.
- The interactions API, launched in December, offers a unified approach for developers to work with models and agents, making it simpler and more aligned with industry standards.
- New features like server-side state management and asynchronous execution improve efficiency, especially for long-running tasks, by reducing the need to keep connections open.
- Implicit caching is enhanced with server-side state, leading to 2-3 times better cache rates, which can significantly reduce costs for API usage.
- The workshop emphasizes hands-on experience with coding agents capable of executing commands and generating outputs, demonstrating practical applications of the API.
- Create your API key on AI.dev to participate fully in the workshop and explore the capabilities of the Gemini API.
Chapters
- 0:14🎯 Introduction and Multilingual Setup
- 3:10🎯 Building and Testing the API Key
- 25:04🎯 Creating a Basic Agent Class
- 27:50🎯 Running the Agent Example
- 31:28🎯 Adding Tools: File Read/Write
- 33:20🎯 Handling Function Calls and Self-Recursion
- 34:44🎯 Running the Enhanced Agent
- 41:41🎯 Building a Multi-Tool Agent
- 1:01:40🎯 Voice Interaction Setup and Grounding
- 1:11:40🎯 Using Gemini Live API for Audio and Video
- 1:25:21🎯 Handling Latency and Session Management
- 1:27:00🎯 Business Applications and Use Cases
- 1:33:22🎯 Speaker Identification and Personalization
- 1:42:51🎯 Evaluating Live Voice Agents in Production
- 1:43:20🎯 API Availability and Data Sovereignty
- 1:45:50🎯 Handling Hallucinations and Ensuring Reliability
- 1:47:04🎯 Final Q&A and Feedback
The digest
High-Level Overview
This detailed presentation and hands-on workshop by Thor Schaeff and Philipp Schmid from Google DeepMind introduces the new Gemini API suite for building advanced conversational agents. Covering APIs for chat, multi-turn interactions, and multimodal real-time conversations, the session focuses on practical developer tools, agent architecture, and live coding demos. It is designed for AI engineers and developers interested in leveraging Google DeepMind’s Gemini ecosystem to build conversational AI that seamlessly integrates stateful dialogue management, tool use, and real-time audio/video inputs.
Keep reading in The Digest
The full 6-minute article, 14 more sections, and every new episode from AI Engineer and the creators you follow.
- Structured Deep Dive
- Introduction to Gemini API and Workshop Setup (00:14–06:00)
- The Interactions API: A Unified Interface for Models and Agents (09:00–14:10)
- Agent Architecture Essentials (14:30–15:10)
- Coding a Simple Multi-Turn Agent (25:00–36:00)
- Practical Details on Using Agent Skills and Tooling (18:00–25:00)
- Gemini Life API and Real-Time Multimodal Interaction (49:00–01:18:00)
- Context Management, Caching, and Privacy (43:00–44:30, 47:00–49:00)
- Key Challenges and Best Practices (45:00–47:00, 01:23:00–01:30:00)
- Key Frameworks and Models
- Interactions API Framework
- Agent Components Model
- Practical Implications & Takeaways
- Action-Oriented Summary
More from AI Engineer
Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs
In this hands-on workshop, Angelos Perivolaropoulos from ElevenLabs guides participants through the process of training an LLM from scratch using basic libraries like Torch, emphasizing the importance of tokenization and model architecture in building effective language models.
Skill Issue: How We Used AI to Make Agents Actually Good at Supabase — Pedro Rodrigues, Supabase
In this workshop, AI Engineer Pedro Rodrigues discusses how to enhance agent performance at Supabase using skills—structured workflows that improve agent interaction and efficiency. He shares insights on creating, testing, and implementing these skills, highlighting their significance in the development of agent-friendly applications.
Ralph Loops: Build Dumb AI Loops That Ship — Chris Parsons, Cherrypick
In this workshop, AI Engineer Chris Parsons introduces Ralph Loops, a method for building simple AI workflows that enhance productivity by iteratively refining tasks, using real coding examples like a Pomodoro timer. Participants learn to implement these loops practically, emphasizing the importance of context and feedback in AI-assisted coding.
Mergeable by default: Building the context engine to save time and tokens — Peter Werry, Unblocked
In this video, Peter Werry from Unblocked discusses the importance of a context engine for AI agents, emphasizing how it optimizes task execution by providing relevant context and reducing cognitive load for users. He debunks common myths surrounding context engines and illustrates their effectiveness in enhancing productivity and understanding within organizations.
Context Is the New Code — Patrick Debois, Tessl
In this talk, Patrick Debois presents the concept that context is the new code, emphasizing the importance of creating, testing, and distributing context in the software development lifecycle, particularly when using AI coding agents. He advocates for a systematic approach to context development that involves generating reusable prompts, evaluating their effectiveness, and leveraging feedback to improve overall coding practices.
Mastering AI Pricing: Flexible & Agile Monetization — Mayank Pant, Stripe
In this video, Mayank Pant from Stripe discusses the challenges and strategies of AI pricing, emphasizing the need for hybrid pricing models that adapt rapidly to changing features and customer needs, allowing companies to maintain competitive margins while fostering customer trust and engagement.