Two Robots, One Physical-AI Loop: What Breaks Between Perception, Reasoning and Action

See how two physical robots, Puppy and Brian, close the perception-reasoning-action loop. Discover real failures and a practical architecture for reliable embodied AI.

Gemini Live API ROS Python Docker JSONL

Overview

I built Puppy and Brian, two real robots used as a live Physical AI lab. The demo shows how we close the perception → reasoning → action loop in the physical world.

Instead of asking an LLM to read every raw sensor stream or directly control motors, the robots use a layered architecture: a fast deterministic runtime handles sensing, safety, reflexes, and hardware control; Gemini Live handles real-time conversation and social interaction; and a slower reasoning layer proposes plans that must be validated before reaching the robot.

I will show the live robots, interaction logs, architecture, and the real failures we found while operating them: unstable face recognition, sensor blindness caused by head movement, corrupted long-term memory, unsafe social gestures, and confusing test logs.

Tech stack

Gemini Live API

A high-speed WebSocket interface for building real-time, bidirectional voice and video interactions using Gemini 2.0 Flash.

The Multimodal Live API enables sub-second latency for voice-to-voice applications by streaming audio and video directly to Gemini 2.0 Flash. It supports over 40 languages and provides 10 distinct built-in voices (such as Lyra and Orion) for natural, interruptible conversations. Developers use a single WebSocket connection to handle complex tasks like real-time vision processing and tool use (function calling) without the lag of traditional text-to-speech pipelines. This is the production-ready engine for building AI agents that see, hear, and respond instantly.

https://ai.google.dev/gemini-api/docs/multimodal-live

View projects
ROS

ROS (Robot Operating System) is the open-source software framework providing libraries and tools for building complex, modular robotic applications.

ROS is not a traditional operating system, but an open-source meta-OS (or framework) that streamlines robot software development. It provides essential OS-like services: hardware abstraction, low-level device control, and robust inter-process communication (IPC) via concepts like Nodes and Topics. The core mission is enabling code reuse across different robotic platforms, offering over 2,000 specialized packages for functions like navigation and perception. Developers leverage its C++ and Python APIs to create distributed systems, with the newer ROS 2 architecture (e.g., Jazzy Jalisco) specifically addressing real-time and embedded system requirements.

https://www.ros.org/

View projects
Python

Python: The high-level, general-purpose language built for readability, powering everything from web backends to advanced machine learning models.

Python is the high-level, general-purpose language prioritizing clear, readable syntax (via significant indentation), ensuring rapid development for any team . Its ecosystem is massive: use it for robust web development with frameworks like Django and Flask, or leverage its power in data science with libraries such as Pandas and NumPy . The Python Package Index (PyPI) provides thousands of community-contributed modules, offering immediate solutions for tasks from network programming to GUI creation . The language is actively maintained by the Python Software Foundation (PSF), with the stable release currently at Python 3.14.0 (as of November 2025) .

https://python.org

View projects
Docker

Docker is the open-source platform that packages applications and dependencies into standardized, portable containers for consistent execution across any environment.

Docker is the industry-standard containerization platform, enabling developers to build, ship, and run applications efficiently. It uses the Docker Engine (the core runtime) to create lightweight, isolated environments called containers: these units bundle an application’s code, libraries, and configuration. This self-contained approach guarantees consistency, eliminating the 'it works on my machine' problem across development, testing, and production environments (local workstations, cloud, or on-premises). Docker debuted in 2013 and now serves over 20 million developers monthly, simplifying complex workflows like CI/CD and microservices architecture by leveraging tools like Docker Hub for image sharing and Docker Compose for multi-container applications.

https://www.docker.com

View projects
JSONL

JSON Lines (JSONL) is a text format for storing structured data where each individual record occupies a single line.

JSONL streamlines large-scale data processing by treating every line as a valid JSON object. Unlike standard JSON arrays, this format allows for incremental parsing and streaming: you can append new records without rewriting the file or loading 50GB into memory. It is the industry standard for high-throughput logging, ETL pipelines, and training datasets for LLMs like GPT-4. By using the .jsonl extension and UTF-8 encoding, developers ensure compatibility with Unix tools like grep and awk while maintaining the rich structure of nested data.

https://jsonlines.org/

View projects

Two Robots, One Physical-AI Loop: What Breaks Between Perception, Reasoning and Action

Related talks