Jarvis
A local AI room assistant inspired by Iron Man, built to make my environment more interactive.
What Jarvis is
Jarvis is a local AI room assistant. It runs language on-device, dispatches work to focused subagents, reaches out through tools and connectors, controls the physical room, and renders itself as a glowing orb projected onto the wall. The goal isn’t a smart speaker, it’s an environment that listens, sees, and responds like it’s part of the room.
Why I'm building it
I wanted a system I understood from the metal up, not a smart speaker that hides its reasoning behind a brand.
Building Jarvis forces me to learn agents, local models, computer vision, hardware control, and interface design all at once, in one coherent project.
And honestly: making a room respond to you like it's alive is the most fun engineering I've ever done.
Current capabilities
What actually works today. Experimental pieces are marked as such, and nothing here is dressed up as finished.
Language runs on-device, no cloud round-trips.
Focused agents handle distinct jobs and report back.
Calls out to APIs and services through defined tools.
Sets lighting scenes and reacts to events.
Renders as a glowing orb on the wall, no visible screen.
A vision pipeline that watches and observes.
MediaPipe hand landmarks for gesture experiments.
Early experiments naming what the camera sees.
How the signal flows
From a spoken request to a glowing response on the wall, inspect each stage.
Voice (“Hey Jarvis” wake word + push-to-talk), and increasingly gesture and presence.
Hover or tap a node to inspect
Computer vision experiments
- Hand tracking via MediaPipe landmarks, detecting grabs and gestures.
- A vision pipeline that observes the room and flags fresh observations.
- Early object-identification experiments, naming what the camera sees.
- The honest bottleneck: running vision and a local model at once is heavy on a laptop.
New local compute
More on-device compute means bigger local models, faster inference, and richer multimodal experiments.
- Selected for the AMD AI Developer Program — AMD sent over a Sapphire Radeon™ AI PRO R9700.
- A generational jump from a GTX 1080: far more headroom for local LLMs and deep-learning work.
- Directly targets the Jarvis bottleneck — running vision and a local model together without a laptop choking.
- Next up: benchmarking the card, testing local models on AMD, and sharing what the hardware makes possible.
Where it's going
- Gesture control of the room, reach out and adjust the environment.
- Tighter, faster local pipelines so vision + AI run together smoothly.
- Voice interaction that feels conversational, not command-line.
- More of the room mapped into one addressable environment.
Lessons learned
- Latency is the product. A correct response that arrives late feels broken.
- Read animation state once per frame, never couple a render loop to React.
- On a projector, you design with light you add, not pixels you draw.
- Tool definitions and context discipline beat raw model size for reliable agents.