Latent Space · Video summary
OpenAI’s New Agent Stack: Computer Use, Decisions API, UltraFast, Dots—Ari Weinstein & Nikunj Handa
OpenAI's Ari Weinstein and Nikunj Handa discuss the latest advancements in computer use agents, including the new Decisions API, UltraFast inference, and the personal assistant product, Dots.
Watch on YouTube ↗Key takeaways
- ⚡Computer use agents are now faster than the average human at accomplishing tasks, with the next goal being superhuman performance.00:00
- 💰The new GPT-6.1 Soul model offers significant cost and speed advantages for computer use, being a fifth of the cost of Astra and a seventh for computer use specifically.01:52
- ☁️Dots, the new personal assistant, provides each user with a dedicated Linux virtual computer in the cloud, enabling it to run full desktop applications and web browsers.04:03
- 🐛Computer use agents can now reliably start tasks and are good at debugging, retrying, and introspecting problems, a major improvement over past models.08:06
- 📸App Shots capture not just screenshots, but all metadata, code, and accessibility representations, giving the language model full context.12:49
- ⏳A significant bottleneck for computer use speed is often the loading time of the websites or applications being automated.15:00
Chapters
- 0:00The Next Frontier: Superhuman Computer Use
- 0:14Tips for Developers Using the Agents API
- 0:16OpenAI Dev Day Announcements Recap
- 0:30API Side Changes: Async Function Calling & Mid-Turn Steering
- 0:31Computer Use Advancements and Milestones
- 0:36Future of Decisions API and Agent Building
- 0:36Dots: Personal Computers in the Cloud
- 0:40Measuring and Improving Computer Use Performance
- 0:42The Broader Vision for Computer Use Agents
- 0:43Computer Use in Dev Workflows and Testing
- 0:43Aha Moments in Computer Use Development
- 0:46The Decisions API: Inspiration and Implementation
- 0:49Decisions API vs. Computer Use
- 0:49UltraFast Inference and Websockets
- 0:58Automating Personal Tasks with Computer Use
The digest
OpenAI is pushing the boundaries of AI agents, aiming for a future where computer use is not just faster than the average human, but literally superhuman. This vision, discussed by Ari Weinstein and Nikunj Handa, involves a suite of new tools and models designed to make AI agents more efficient, cost-effective, and capable of real-time interaction.
The Evolution of Computer Use Agents
Ari Weinstein, who leads the product and engineering team for computer use agents, highlights the significant progress made in recent months. He notes that computer use is now faster at accomplishing tasks than the average human in most cases. The next frontier is to achieve superhuman performance, where agents can use software as fast or faster than expert computer users. This will enable the creation of products that offer much more real-time experiences and significantly reduce the time users spend on intricate computer operations.
Weinstein reflects on his career, which has consistently focused on automation, from his work on Shortcuts at Apple to his current role at OpenAI. He emphasizes that the models have become extraordinarily capable in the last year, particularly in their ability to debug, retry, and introspect when tasks encounter problems. Modern computer use agents often write and execute JavaScript code, performing multiple actions at once, and leverage multimodal interfaces like screenshots, accessibility features, and Playwright to adapt to various tasks.
Keep reading in The Digest
The full 2-minute article, 2 more sections, and every new episode from Latent Space and the creators you follow.
- Key Announcements from OpenAI Dev Day
- The Decisions API and UltraFast Inference