Lakehouse: One Assistant, Many Harnesses

Designing continuity, coordination, and control in a personal multi-agent system.

Role: System designer and product owner. I framed the problems, defined the architecture and interaction rules, and directed AI coding agents to implement them.

Timeline: 2026–ongoing. A working personal system, evaluated through my own daily use.

The problem: context stops at the edge of the tool

I use AI across chat apps, terminals, and my phone. A conversation might begin in ChatGPT, continue in a coding tool, and return later in a different environment. The work continues, but the context often does not. I have to explain earlier decisions, repeat preferences, and reconstruct what remains unfinished.

Platform memory helps within some of these environments, but it gives me no consistent way to manage that history across them. Adding a second assistant introduces another problem: I want the assistants to collaborate without treating everything I have told either one as shared information.

Lakehouse grew out of those daily frictions. It is a personal system with two resident assistants, Claude and Codex, private conversations, a shared group room, and memory that can be accessed from multiple tools. I use “harness” to mean the environment in which an assistant runs, such as a chat app or coding terminal.

I organized the design around three questions:

• Continuity: How can meaningful context survive changes in session, model, and harness?
• Coordination: How can multiple assistants share enough to work together while retaining separate histories?
• Control: How can I understand what has been remembered, where it lives, and who can use it?

These questions shaped both the architecture and the assistant’s behavior. Daily use then exposed a further challenge: even when an assistant has useful context, bringing it into the conversation can make the experience worse.

Architecture at a glance

Each assistant has its own memory store. Different harnesses connect to the appropriate store; the Lakehouse runtime provides private and group conversations and other capabilities.

1. Continuity: deciding what should survive

The first problem was repeated reconstruction. A decision made in one environment could be absent in the next, even when I considered the work part of the same ongoing conversation.

I moved durable memory outside the harness. Lakehouse uses Sostenuto, an open-source memory layer I built and published. Each assistant’s store can be reached from the environments where that assistant works. Stable identity guidance is also maintained outside an individual session.

This made continuity something I could design independently of a particular chat interface. But a shared destination for memory was only the beginning. I still had to decide what to preserve, how it would get there, and when it should return.

Preserve the work that matters

A complete transcript and a useful memory serve different purposes. From working sessions, I wanted the system to retain decisions, open threads, and working preferences—the things that help us resume—without promoting every exchange into permanent context.

The memory layer distinguishes dated events, durable memories, and session summaries. Unresolved threads can surface again, while related observations reinforce an existing memory instead of automatically creating another entry. Updates retain version history.

That changed the design question from “How much can the assistant remember?” to “What would help it continue appropriately?” An unfinished decision deserves a different place in recall from a passing remark.

Connect through the mechanisms each tool supports

The environments do not offer equivalent access. Some support hooks that supply context or capture work automatically. Others depend on the assistant choosing to use a memory tool. Imported history can fill in the past, but it does not provide continuous recall on new turns.

I kept those differences explicit instead of presenting every connection as seamless.

The connections serve the same goal through different mechanisms. Local Codex distillation captures work; ChatGPT history import provides backfill. Neither is equivalent to automatic recall before every reply.

Connecting more harnesses also created overlapping capture paths. A Lakehouse turn running through Codex could already be recorded by Lakehouse and then be captured again by the coding-session distiller. I excluded those turns from the second path so the same interaction would not be treated as separate evidence.

Keep the limits visible

The implemented result is a memory layer that multiple harnesses can reach, with selective capture and explicit routes for retrieval. It gives the system a foundation for resuming work across environments.

It does not establish that changing the model preserves the same experience. Models still differ in voice, interpretation, and tool use. Retrieval also has a cost: the current real-time voice path skips memory retrieval to reduce latency. Continuity remains uneven across surfaces, and the design has to account for those differences.

2. Coordination: sharing context without merging histories

Once two assistants could participate in the same system, continuity became a question of boundaries as well as persistence.

Consider a discussion in the group room. When I later return to a private conversation, I want the assistant to know what happened in that discussion. I do not want its private history with me to become available to everyone in the group.

I chose a separate memory store for each assistant. They can share an experience without sharing their entire history. Neither assistant reads the other’s private store through the normal memory path.

Make the handoff explicit

In the group room, assistants respond in sequence. The next assistant receives the preceding response and tool results, giving it a basis for building on the discussion.

When I return to private chat, relevant recent group messages enter that assistant’s context. Messages retain their speaker attribution, including which statements were the assistant’s own. Private memory is excluded from group context.

This creates a deliberate asymmetry: a shared discussion can inform a later private conversation, while a private conversation does not automatically become shared knowledge.

The principle also informed my Beyond Personal Agents essay: sharing a room is not the same as sharing access.

The implemented handoff lets an assistant carry group progress into private work while preserving separate memory stores. The remaining design question is how much shared history to bring forward. Too little leaves gaps; too much makes the current conversation compete with accumulated background.

3. Control: making “I’ll remember” mean something specific

An assistant saying “I’ll remember that” sounds reassuring. It leaves several questions unanswered: Did it actually save anything? Was the information stored in the platform’s memory or in my own system? Which assistant can retrieve it later?

As I used memory across more environments, these distinctions became part of the experience. A verbal promise alone could not tell me what had happened.

Give memories a traceable location and history

Lakehouse’s durable memory is held in stores I manage. Entries identify the surface they came from, and content updates retain their history. Separate stores define the ordinary access boundary between assistants.

The local service can be reached from supported external tools without exposing an inbound public listener. The database remains on my machine, although retrieved information is sent to the model handling the conversation. Owning the store gives me control over persistence; it does not mean every use of its contents stays local.

These choices make storage, retrieval, and access concrete enough to inspect. They also reveal an interaction gap: the assistant should clearly distinguish a successful write from an intention to remember, and identify which memory system it used. That is an area for further refinement.

Separate access from initiative

Storage location does not settle how a memory should be used. Something can be appropriate to retrieve when I ask and inappropriate to introduce in an unrelated conversation.

I separated retrievability from permission to volunteer a memory. Sensitive memories can remain available for a direct request while being excluded from unsolicited recall. Memories carry usage policies as well as content.

This distinction became a broader design rule: knowing something does not automatically make mentioning it helpful. Access boundaries determine what an assistant can retrieve. Interaction rules determine when that knowledge belongs in the conversation.

4. Daily use: when more context made the experience worse

Health data made that distinction tangible.

I connected a smart ring, Apple Health, and manual entries so the assistants could use information about sleep and activity in everyday conversations. The early version repeatedly recited heart rate and step counts. The data was available, but the experience felt like being monitored by a dashboard.

The problem was relevance. A number could be accurate and still be the wrong thing to say.

Let context guide the response

I changed the speaking rule: say a number when it changes what I would understand or do, or when I explicitly ask for it. Otherwise, let the data provide background.

After a short night, for example, the assistant can suggest an earlier bedtime without automatically quoting a sleep score. If I ask for the evidence, the values and sources become relevant to the answer.

In my daily use, this felt less like a stream of measurements and more like context informing a conversation. The underlying readings still matter: they remain labeled by source so the assistant does not attribute a watch measurement to the ring, and live readings must arrive before the assistant can report them.

Make the evidence available when it matters

The same tension informed my Verifiable AI Health Experience. In that separate portfolio prototype, foregrounding data-quality problems initially made the interface feel like a pipeline debugging console. I reorganized it around the person’s health question, showing data limitations where they changed the interpretation.

That prototype explores how a person can verify an answer: which sources were used, what was missing, and where the evidence stops. Lakehouse explores when that information should enter an ongoing conversation. The prototype uses an illustrative dataset and is not evidence that Lakehouse’s live behavior has been validated.

Together, the two projects sharpened the same judgment: keep the evidence accessible, and give it attention in proportion to the question being answered.

What the project demonstrates

Lakehouse is a working personal system with memory connections across multiple harnesses, separate assistant histories, and explicit handoffs between group and private conversations. Building and using it changed my focus from adding capabilities to deciding how context should persist, move, and influence behavior.

My role was to translate those everyday frictions into system boundaries and interaction rules, then review the implementation against actual use. The health integration provided the clearest corrective: adding context can increase capability while reducing the quality of the experience.

The evidence so far comes from one person’s daily use. I have not established general usability results or measured continuity across model changes. The next evaluation should test concrete transitions: resuming an unfinished task in another harness, moving from group to private chat, confirming where a memory was saved, and asking for the evidence behind a contextual suggestion.

The goal is an assistant that can carry useful history forward, collaborate within clear boundaries, and make its use of that history understandable to the person living with it.