StreamingTelligence/Gelato

Open protocol and SDKs for connecting AI controllers to real-world devices with authenticated commands, media streaming, and device-controlled execution. Presented by Streamingtelligence Lab

Summary

Gelato is an open protocol and SDK stack for connecting AI controllers with real-world devices such as phones, smart glasses, and cameras. It defines authenticated pairing, command delivery, media streaming, device observations, and optional model integrations, while keeping execution authority on the device.

What it is useful for

Gelato is useful when an application needs to turn networked hardware into an interface for an AI controller. For example, the included Android and iPhone applications support GUI control, camera or screen observation, and voice-assistant interactions. The Rokid glasses implementation handles camera frames, microphone input, and audio replies, while the Seeed XIAO ESP32S3 Sense implementation provides camera observation from a headless board.

The architecture separates three roles:

• The device agent advertises its capabilities, publishes observations, receives commands, and decides whether to execute them.
• The Controller manages pairing, identity, authenticated sessions, routing, state, policy, commands, cancellation, and media streams.
• An optional model runtime converts device input into content or proposes an action. Proposed actions are still routed through the device session rather than executed directly by the model.

This makes Gelato relevant to building remote device-control systems, multimodal assistants, and new device integrations without tying the protocol to one model provider. The repository contains the Gelato v1 specification, JSON schemas, device and server SDKs, model-interface components, and example MVPs.

The supported-device entries are explicitly MVP implementations, not a guarantee that every capability is production-ready. Model integrations also have documented boundaries: some accept only text or images, some do not propose actions, and the one-shot HTTP adapter is not a persistent session. Physical-device validation remains a manual release check.

How engineers can use it

The documented local Controller setup requires Node.js 22 or newer:

npm ci
npm run build
npm start

Open http://127.0.0.1:8090, run npm run token in another terminal, and enter the generated token in the Controller console. From there, create a one-use pairing link and follow the setup guide for the target device.

For remote devices, the README specifies a reachable HTTPS Controller with WebSocket upgrades; the loopback HTTP setup is intended only for local development. Relevant guides are under examples/device/ and examples/server/control-room/README.md.

Engineers adding unsupported hardware can start with docs/protocol.md, the schemas in device-sdk/schemas/, and docs/implementing-a-device.md. Optional model integrations are documented in model_interface/README.md.

Before submitting changes, run the repository’s documented validation sequence:

npm run ci

This checks public protocols, Controller behavior, security boundaries, SDKs, package exports, and local integration paths without requiring service credentials, external APIs, or physical devices.