Talking to the Assistant I Already Know

Designing voice and video for a personal AI assistant, with continuity, conversational control, and privacy at the center.

The problem: my assistant only worked when I could type

I wanted to talk to my assistant when my hands were busy or I was away from a keyboard, and to show it things instead of describing them.

But adding a microphone and a camera raised a more basic question: would I still be talking to the same assistant?

I wanted my existing agent’s reasoning, with access to our conversation history and memories. A convincing voice would be worth little if it answered without that context. And even a capable agent would be hard to use if it cut me off, took too long to answer, or was unclear about what it could hear and see.

The design challenge: make my existing assistant available through speech and vision, without losing continuity or my control over the conversation.

My role and scope

This is the voice and video layer of Lakehouse, my personal AI assistant: a web and iOS experience running on an always-on Apple silicon computer. I was its primary user, speaking Mandarin and English and sometimes switching mid-sentence.

I set the experience priorities, evaluated voice options, and made the product tradeoffs. An AI coding assistant did most of the implementation, debugging, and benchmarking. I started from an open-source project that already provided wake-word calling, speech detection, and streamed playback, and built on top of it.

The evidence comes from one person’s everyday use, side-by-side listening, and small audio tests. What it does and doesn’t establish is summarized at the end.

1. “I want to speak without managing the interface.”

Handsfree interaction starts before the first answer: how do I get the assistant’s attention, and how does it know I’ve finished?

I considered continuous transcription, push-to-talk, and a wake word that opens a conversation. I chose a wake word for voice and kept video’s existing push-to-talk. That gave voice a handsfree way in without sending everything I said through the conversation pipeline.

The next problem came from ordinary speech. I pause to think, and switching languages adds another pause. The assistant often took those pauses as the end of my turn, so I lengthened the pause it allowed mid-conversation, while keeping a shorter window for catching the wake word.

The tradeoff: more room to finish a thought means more waiting before every answer. I chose finishing the sentence over the fastest possible reply. A fixed pause is still a compromise; it can’t tell thinking from finishing.

Two gaps remained. To keep the assistant from hearing its own voice, the microphone closed while it spoke, which also meant I couldn’t interrupt it. And the wake word swallowed my first sentence, so I had to repeat what I’d just asked. Handsfree should mean not managing the assistant at any point in the exchange; both gaps broke that.

2. “I want my agent’s brain, with a voice.”

The central requirement was continuity. Speech recognition captures what I say, the existing agent decides what to say back, and the voice speaks it.

Speech handles input and output. The agent owns the answer. In calls, memory comes from the ongoing session; nothing is retrieved per turn.

Keeping those responsibilities separate meant I could evaluate speech and vision tools on their own, without handing the assistant’s reasoning to a new model.

I did test an integrated audio-and-vision model alongside the separate components. In the version I tested, transcription wasn’t good enough to adopt, and its emotion judgments were unreliable: it recognized some simple sounds, but when I asked an open-ended question about a recording of silence, it described a vocal condition anyway. I removed it.

The tradeoff: separate components need more coordination, but let me tune recognition, perception, reasoning, and voice quality individually. The integrated model’s broader abilities didn’t outweigh its unreliability at the tasks I needed.

Memory is part of continuity, and here the experience and the mechanism turned out to differ. Calls felt as if the assistant remembered me, so I assumed voice turns retrieved memory. They don’t. Voice and video calls run in a fast mode that skips memory retrieval entirely, to get the first answer out sooner; nothing is looked up when I speak. That behavior came with the open-source project I started from.

What carries continuity instead is what the conversation already holds. The agent keeps a running session for up to ten idle hours, so a call inherits whatever memories were retrieved during earlier text turns in that session. It also always has its stable persona and the last 20 messages. The result is uneven: a call that follows an afternoon of texting starts with far more context than one started cold after a long gap.

The tradeoff: skipping retrieval keeps the first answer fast, but makes a call’s memory depend on what came before it. The next step I’d try is to run retrieval during calls with a hard time limit of about a second, and continue without retrieved memory if it runs over.

3. “Give me time to speak, then answer without a long silence.”

Responsiveness meant two different things: patience while I was speaking, and speed once I’d finished.

Letting me finish and answering promptly are separate design decisions. The sequence has no measured time scale.

I compared local and cloud speech recognition, then several local recognizers. Small tests suggested local recognition could meet my needs without routinely sending raw audio to a third party. I later switched recognizers after testing bilingual speech, silence, and background noise.

But the recognizer was only one source of delay. The system also waited for a sizeable chunk of the reply before sending anything to be spoken, so even after the agent had started answering, I heard nothing. I lowered the amount of text needed to start speaking and let it break earlier, at a comma.

The tradeoff: shorter segments start playback sooner but give the voice less context for natural phrasing, so speed and expressiveness had to be judged together. Estimates from recorded replies supported the change; I didn’t measure the full live delay from question to answer.

For the voice itself, I stayed with ElevenLabs after considering alternatives, and chose separate Mandarin and English voices after hearing how a voice built for one language sounded in the other.

I also corrected my own cost assumption. My first comparison treated every response as spoken, but I used voice only occasionally. A lower unit price alone wasn’t a reason to switch when voice quality, data handling, and the effort of switching also mattered.

4. “Let me show you something without making the camera a permanent observer.”

Video could spare me from describing a scene, but it also widened what the assistant could collect and claim to know.

So video was a session I explicitly opened and closed. Within it, a local vision model periodically turned a sampled frame into a short scene description. Sampled frames weren’t saved to disk, and each description expired after a short interval.

The interface is proposed; the routes are documented. Local vision describes one path, not the privacy of the whole session.

That addressed two separate concerns. Processing frames locally limited their exposure, and expiring descriptions stopped an old observation from standing in for the present. If vision became unavailable, the conversation carried on without current visual context.

The tradeoff: periodic descriptions are less detailed and less immediate than continuous video understanding, but they let less visual information into the conversation.

The privacy boundary isn’t complete. Video’s separate push-to-talk path uploads clips to my own server, and whether a clip goes further depends on the model: with Claude, only its transcript reaches the model; with Gemini or other API models, the clip itself is sent. Local speech recognition also keeps a cloud fallback, which I plan to remove so recognition stays entirely local. And by design, what I say reaches the conversation’s model provider, and the assistant’s reply goes to ElevenLabs to be spoken. So I can’t describe the whole experience as on-device.

The project turned data routing into a product question: what leaves the phone, what reaches my server, what reaches another provider, and what is kept? Every path needs an answer, including the ones taken when something fails.

5. “If something goes wrong, don’t make me guess.”

Several of the most consequential failures showed no error at all. They looked like an assistant that ignored me, stopped listening, or was working from an outdated view.

It heard me. Then my words vanished. The bracketed phrase is a placeholder, not a real transcript.

These failures changed how I evaluate. The unit that matters is a whole conversational turn: activation, recognition, response, playback, and the return to listening. A success message from each component isn’t evidence that the turn worked.

What changed, and what still needs proof

The work established wake-word voice, revised pause handling, local-first recognition, earlier playback, language-specific voices, and session-scoped visual descriptions. It also uncovered failures that kept the assistant from responding reliably.

The strongest result is a clearer set of boundaries: the agent owns the answer; speech and vision handle input and output; visual context expires; and any claim of local processing has to account for every route.

Open gaps: interruption, the lost first sentence, memory retrieval during calls, call ownership across multiple clients, a retention rule for stored clips, and removing the cloud recognition fallback. The tests were small and informed implementation choices; they don’t establish general accuracy, production readiness, or end-to-end performance.

What I’d carry into a production voice agent

In a production voice agent, with customer support as the clearest case, I’d keep continuity and conversational control as priorities, under stricter release requirements. This is proposed work:

• Interruption: let callers stop an answer, cancel queued speech, and continue from what they actually heard.
• Turn-taking: tune pause behavior separately for short answers, account details, and longer explanations, across languages and accents.
• Continuity and handoff: decide which prior information is appropriate to retrieve, and pass relevant context to a human representative so the caller doesn’t have to repeat it.
• Privacy and cost: define allowed processing and retention for every route, then measure cost per completed interaction, including retries and escalations.
• Responsiveness: measure the full path to first audio, and recovery from failures, rather than individual component timings.

This project taught me to judge voice by whether a person can finish, correct, and continue a thought. A natural-sounding answer matters. So does knowing that the assistant heard the right words, used the right context, and is ready to listen again.