Talk
Human IS the Loop - Embedding user first, voice enabled agents into mobile with Flutter and Gemini
The current model of detached, autonomous agents does not produce compelling and engaging user experiences and adding a "human in the loop" only throws the detachment into sharper relief. - The agent does the thinking, the user ticks a box - The agent decides what to attend to, the user provides guidance when needed - The user prompts, the agent creates and acts To facilitate truly engaging user experiences we need to build different types of agents. Agents that focus on the user first and foremost. These types of agents should: - Understand what problem the user is trying to solve, and what they want to achieve (context, attention, intent) - Switch context when the users attention changes but remember where they were up to (context switching and memory) - Be experts in a particular domain and understand concepts and workflows that are effective (knowledge, orchestration, workflows) - Natively understand the same functional and visual surfaces available to the user and navigate them on behalf of the user (tools/functions/state mutation, ui, routing & navigation) Using an example implementation with Flutter, signals, go_router and Gemini, attendees will learn how to:- - Create an on-device agentic harness which integrates memory, voice input, tool calls and LLM interaction - Seamlessly combine an agentic loop with bespoke orchestrated workflows - Wire tool calls into state management and app routing - Track and respond to changes in user attention and intent and manage state for backgrounded activities - Handle interruptions like timers and asynchronous events
