VOS and VibeOS

Why I built it

I wanted an assistant I could talk to rather than type at, closer to Jarvis in the Iron Man films than to a chat box. I built it with Roman Slack, and then I used it quite a bit myself. That turned out to be the useful part. Every time I used it I ran into something broken and had to fix it, so I was probably improving it more than using it. I'd do that on every project from now on: use the thing early, because that is where the real problems show up.

What it is

VOS is the assistant. We built it between September 2025 and February 2026. Under the hood it is not one AI but a small team of them. One agent, the primary agent, is the only one that talks to you. When you ask for something it can't do itself, it hands the job to a specialist: one for weather, one for searching the web, one for your calendar, one for notes, one for maths, one for browsing websites. Think of a front desk with a few colleagues behind it. You ask the desk, the desk walks your question over to the right colleague, and brings the answer back to you.

What it can do

You can type to it, talk to it, or phone it. It keeps a memory, so something you said last week can come up again without you repeating it. The window-style app at the top is Roman's design. I wrote the talking part, the phone screen and parts of the notes and settings screens. The first thing that worked from end to end was asking for the weather in New York and getting the answer back through three separate programs. After that it grew one specialist at a time.

What went wrong

Plenty. Agents sometimes got stuck halfway through a job or handled the same message twice, and I spent a lot of time on that. On a phone call you can't interrupt it: while it is speaking it ignores you, so it doesn't hear its own voice and answer itself. I never got past that trade-off. And the plan was much bigger than what exists. It included a security agent that watches the others, games, and a team of agents that writes code. None of that got built, and the launcher still has a few icons, like the shop, that open nothing.

VibeOS

VibeOS is the smaller sibling. Roman started it in late January 2026 as a minimal agent, written in Rust (a programming language). There is one agent, Vi, that lives inside a virtual computer and works the way you would at a keyboard: it types commands into a terminal. A second character, Joi, is the voice. You talk to Joi, and if what you asked for is a job she passes it to Vi, then tells you the result in her own words. The glowing orb screen is Roman's work. I wrote the part that decides what Vi does next, the lock on the door of the virtual computer and a small web dashboard.

One bug I'm glad I fixed: the AI would sometimes describe the result of a command before the command had run, which means it made the answer up. Now it pauses mid-sentence, the real command runs, and it carries on from the real output.

Roman and I also worked on Rookery, a feed for research papers. I wrote the part that decides which papers to show you. The site is no longer online.

What's next

Both are prototypes, and I'm not working on either of them any more. The last work on them was in February 2026. What I spend my time on now is a newer system of agents set up like a company. That one is not public.

What I wanted

An assistant I could talk to rather than a chat box: one agent owns the conversation, hands work to specialists, remembers what I said last week, and can take a phone call. I built VOS with Roman Slack from September 2025 to February 2026. The plan was a lot bigger than what exists. This is about the part that got built.

I also used it a fair amount myself, and that shaped most of the work. Using it kept turning up problems, which I then had to fix, so I probably spent more time improving it than using it. That is the lesson I take into every project since: use it early and often, because the bugs you find that way are the ones that matter.

I wrote the SDK every agent is built on, the API gateway, the primary agent and most of the specialists, the call system, the Twilio gateway, and an API for editing system prompts with versioning and rollback. Roman wrote the browser agent, backend auth, the memory visualisation, the first small web frontend and, late on, the call fast mode. Roman finished the notes agent, and the voice gateway is shared work. The Flutter client is Roman's repo: he designed it and wrote the window system, and I added voice mode, the notes UI, the phone app and the prompt settings.

September 2025October 2025

How VOS works

Every agent is the same SDK class with a different tool list. A message from the client goes through the API gateway into RabbitMQ as a typed notification. The primary agent reads it and, if needed, sends an agent_message to a specialist's queue. The specialist runs its tools and reports back. In text chat only the primary agent talks to the user. In a phone call the call can be transferred to a specialist, which then speaks through the call tools.

What is RabbitMQ?

A message broker. Every agent has its own queue, which works like a mailbox. A sender drops a message in and moves on, and the agent picks it up when it is free. A message only leaves the queue once the agent confirms it handled it, so a crash in the middle doesn't lose it.

The loop inside each agent is a three-state machine kept in the database: IDLE, THINKING, EXECUTING_TOOLS. A turn loads history, calls Gemini in JSON mode, and requires at least one tool call. An empty list is rejected. Tools publish their results back to the agent's own queue as new notifications, so a multi-step task is several turns, not one long call.

Memory has two modules, both set to run every turn by default (the interval is an environment variable). A retriever runs before the model call and a creator after the turn. The creator is itself an LLM call, and it answers CREATE_NOW, WAIT or IGNORE for what just happened. The retriever searches Weaviate and feeds the results into the turn. Memories are either private to one agent or shared across agents. Notes use the same store for semantic search.

What is semantic search, and what is Weaviate?

Weaviate is a vector database. It stores each piece of text as a long list of numbers that captures roughly what it means, so a search can match on meaning instead of exact words. Asking about "my trip next month" can find a note that only says "flight to Lisbon on the 14th". That is the whole trick behind both the agents' memories and the notes search.

Voice and calls

Voice has two paths. Voice mode streams audio to voice_gateway, which transcribes with AssemblyAI (I moved over from Deepgram in November) and speaks with ElevenLabs or Cartesia, chosen per session. Call mode goes through a call bridge in the same gateway. It waits 1.2 s after the last transcript before sending text to the agent, and it drops transcripts while the agent's own speech is playing, to stop echo. Calls can be transferred between agents, each with its own voice. Phone calls arrive through twilio_gateway, which converts Twilio's 8 kHz mulaw audio to PCM and back. A fast mode, Roman's, uses a smaller model and allows only the speak and hang-up tools.

Why wait 1.2 seconds before answering?

Speech-to-text hands back a transcript in pieces, one each time you pause for breath. If every piece went straight to the agent it would answer half-sentences. So each new piece restarts a 1.2 s timer, and only when the timer runs out does the combined text go to the agent. It costs a little delay on every turn and it stops the agent jumping in early.

VibeOS

A Rust agent, vos_agent, runs inside a headless Ubuntu VM under QEMU and executes bash through a PTY. A gateway exposes it over WebSocket with an API key, rate limiting and TLS. Memory is SQLite with full-text search. Roman started it in late January 2026 as a minimal agent, with the simplest solutions and minimal code as the rule, and built most of the voice pipeline, the later plugin system, the memory rework and the WhatsApp and Twilio work. I wrote the agent loop, the TLS and rate-limited gateway, the first version of agent memory (remember, recall, forget), a multi-agent delegate tool and a web dashboard. The voice side is a separate agent, Joi (Python, Groq, Hume TTS). She listens, decides whether a request is a task for Vi, the Rust agent, and speaks the reply. In voice mode Vi never speaks to you directly; in the chat app it does.

What is a PTY?

A pseudo-terminal: a fake terminal window that a program can type into and read from. It lets the agent run bash the way a person at a keyboard would, with a real shell, working directory and interactive programs, instead of launching one command at a time with no shell around it.

My part of the loop went through three versions. First a loop capped at five iterations. Within days I replaced that with a 200-iteration safety valve that ends only when the model calls an explicit stop tool, rejects turns with no tool call, and turns every error into context for the next turn. The third version, in February, runs tools inline. The model writes one continuous stream, and when it writes a tool tag such as <bash time="15"> the stream pauses, the real command runs, and its output is spliced in before the model carries on. I made the change because the model had been inventing tool results before they ran. I also replaced regex detection of "is this a task" with a Groq model in Joi.

Results

I never benchmarked either system, so I have no latency, accuracy or cost numbers. The first thing that worked end to end in VOS was weather: ask the primary agent, it delegates to the weather agent, the answer comes back. After that the specialists, memory, voice and calls came one at a time. VOS has no automated tests. VibeOS has close to 190 small unit tests in Rust plus an integration test for the gateway. The limits I set in the code are these.

Limits I set in the code. Not measured behaviour.

Setting	Value
VOS agent: LLM call timeout	90 s
VOS agent: requeue on transient error	3 times
VOS agent: stuck state reset	300 s
VOS call: transcript debounce	1.2 s
VibeOS: tool loop, iterations	200
VibeOS: bash output kept, chars	20,000

What didn't work

Agents got stuck or double-handled messages, and most of the reliability work went there: a fix for a race on message history in October, a requeue system in September and a reset for stuck agents in December. The memory creator saved the same thing over and over until I gave it a duplicate check and made it ignore when unsure. I changed models several times, from Gemini 2.5 Flash to a 3 Pro preview and on to Flash variants, and I never compared them properly, so I can't tell you which was best.

Call mode drops speech while the agent is talking. That stops echo and it also means you can't interrupt it. Roman's Twilio work in VibeOS was untested when he added it and was later switched off.

Unfinished

I'm no longer working on VOS or VibeOS. I moved on to a newer system of agents organised like a company, and that is what I work on now. It is not public. Both projects stay as they were in February 2026.

The original plan had a Watchdog security agent, event subscriptions, a games and apps suite and a coding team of agents. None of that exists. There is a registry service for apps, but nothing that generates them. Basic document tools exist (create, read, list, delete), but the agent-to-agent document flow I planned isn't finished. The app's launcher has icons for a shop, reminders and tasks that open nothing. In VibeOS, Joi doesn't have a personality yet, so in February Roman added a direct Vi mode to test voice without her.

Also with Roman: Rookery

Roman Slack and I also made Rookery, a social-media-style feed for research papers. I built the backend and the AI that decides which papers to show you; Roman built the interface and handled publicity. A waitlist was online, but the site never opened to the public. It builds paper embeddings from the citation graph with a GraphSAGE network, and the adaptive feed mixes mostly relevant papers with fresh ones and a share of deliberately adjacent topics. I evaluated it with simulated users. The site is no longer online.