How we taught AI models to share memory instead of starting over

How we taught AI models to share memory instead of starting over

Whenever an AI model hands off a conversation, there’s a pause. Before its successor can add a single word, it has to reread the entire conversation: every message, every instruction, every line its predecessor has already worked through.

But is that pause necessary? Our research suggests it’s not.

Picture an assistant building a website. A small, fast model sketches the layout, drafts the pages, and writes the copy. Then a request comes in that needs deeper reasoning, and a stronger model has to take over.

In our tests, even between two same-size models with 27 billion parameters each, the receiving model needed 6.8 seconds to reread a 32,000-token conversation before it could start answering, and nearly 15 seconds for a 64,000-token one.

The delay occurs because each model builds its working memory in its own way, even models from the same family. The stronger model can’t reuse what the smaller one has already built, so it has to process the whole conversation from scratch. The longer the conversation, the longer the wait, and it repeats at each handover.

Our AI research team, Hostinger Research, built small translators that convert one model’s working memory into a format another model can read without having to start over. Our results, published in a new paper, show that the receiving model can pick up where the first one left off. On the longest conversations we timed, it started answering up to 47 times sooner than if it had reread the conversation itself.

“Passing a conversation as text is like describing your blood to a doctor instead of donating it,” says Povilas Kvedaras, AI researcher at Hostinger and lead author of the paper. “We hand over the memory itself, a transfusion. But every model has its own blood type. So first, we match the type.”

The approach doesn’t work equally well everywhere – one of our four main handovers fell short – and that’s part of the story too.

Here’s how we built it, what worked, and what didn’t.

What is a KV cache, and why can’t models simply share it?

When a model reads a prompt (a step called prefill), every attention layer stores information about each token, a word or piece of a word, so later tokens can look back at it. In a classic transformer (the architecture behind models like Qwen3-8B), that information is a key-value pair per token.

The key helps the model decide which earlier tokens matter, and the value carries their contributions. Together, they form the key-value (KV) cache, which a serving engine keeps between turns so the model doesn’t redo that work.

One note on terms: “KV cache” is a convenient umbrella here, but some newer models, which we cover below, don’t store conventional key-value pairs.

We wrote about the same mechanism in our AI coding models experiment, where provider-side caching cut the bill by 82%. That kind of caching works when the same model keeps going. When a different model takes over, the first model’s cache is useless to it, so the newcomer starts from scratch.

There’s a second obstacle: models don’t all store the same thing. For our paper, we read the published configurations of the newest models from the most downloaded open-weight families on Hugging Face in November 2026: DeepSeek, GLM, Kimi, Qwen, Mistral, NVIDIA’s Nemotron, MiniMax, and Meta. Within the past year, all of them except Meta have moved their newest model away from a plain key-value pair per token.

A year ago, Kvedaras says, almost every open model stored memory “the same way: one neat note per word.” Now each lab takes its own approach: “One shrink-wraps each note to a fifth of its size. One keeps no notes at all, just a single diary page it rewrites after every word. One keeps index cards saying which words are worth rereading.”

Memory typeWhat it storesSize
Grouped-query attention (GQA)A key and a value for every token and layerAbout 147 KB per token (Qwen3-8B)
Multi-head latent attention (MLA)One compressed vector per token and layerAbout 31 KB per token (Kimi-VL-A3B), roughly a fifth of Qwen3-8B’s
Linear attention: Gated DeltaNet, Kimi Delta Attention (KDA)One fixed-size running state that summarizes everything read so farAbout 44 MB per conversation, at any length (Kimi-Linear-48B’s linear-attention layers)
Sparse attention: DeepSeek Sparse Attention (DSA) and Qwen’s QSA variantA small extra index key per token, used to pick which tokens to readA 128-value index key per token, pooled into blocks of four (Qwen3.8-Flash-Next)

Per-token memory grows with the conversation, while the running state stays the same size regardless of how long the conversation gets.

Here’s a simplified picture of each memory type. Real models are more complex, and many combine several of these approaches into a single model.

These are the blood types Kvedaras has in mind, and each breaks word-by-word translation in a different way. Linear attention’s diary page can’t be produced token by token. MLA rescales each compressed note and partly rotates it to record the token’s position before storing it. And in sparse attention, the index card decides which tokens the model reads at all.

Even two models with the same memory layout, like the two Qwen models we tested, write different values into it, so they still need a translator, just a simpler one.

“A trick that only works on last year’s memory only works on last year’s models,” Kvedaras explains. “We needed to know if it works on what labs are shipping now.”

How Hostinger Research built a translator between AI models

Our translator converts the source model’s cache into the target’s own format and writes it straight into the target’s cache, as if the target had read the conversation itself. Neither model changes. We train only the small maps between them, and the translator sees only what a serving system already keeps: the cache and the token IDs, never the model’s internal states.

We started by replicating KV-Lingo, an earlier method that translates classic key-value caches between two models. On Qwen3-8B and Qwen3-4B, our version matched the smaller model’s own chat-quality score. With that baseline in place, we moved on to the harder memory types, one at a time.

We worked with two main pairs, both described in more detail in the results: Kimi-VL-A3B and Kimi-Linear-48B (the Kimi pair), and Qwen3.5-27B and Qwen3.8-27B (the Qwen pair).

Token-wise mapping – for memory stored per token, one linear map per target layer converts each token’s cached vector without context. A linear map is a single matrix multiplication, which keeps it fast. Where we read the memory matters. For memory compressed with MLA, we take the vector before the model rescales and rotates it, then let the target apply its own steps. When we trained on the engine’s stored, already-rotated version instead, exact recall for the Kimi pair collapsed beyond the 8,000-token training length: at 32,000 tokens, it fell to 25%, compared with 95% when we used the vector before rotation.

Copying or replaying the running state – a running state can’t be built token by token. When both models keep states of the same shape, as in our Qwen pair, we copy the state and apply a quick correction that adds less than 3 milliseconds. The correction costs the same at any conversation length. When the source has no running state at all, as in our Kimi pair, a map predicts what each layer would have received for every token. The target’s own machinery then replays those inputs to rebuild the state.

Recovering exact tokens – compressed memory keeps the gist of a token but not always its exact identity. In models whose first layer is an MLA layer, the vector that layer stores depends only on the token itself, so we can read the token back. We compare it against a lookup table with one row for each of the 163,840 vocabulary entries, then feed the target model’s own numeric representation of that token (its embedding) into every map. A serving system can also simply pass along the token IDs.

Predicting the index – when the target uses sparse attention and the source has no index, we predict the target’s index keys from the translated keys and the token. We tested this on one pair, in one direction, with one training run.

“The diary-style memory (Kimi’s KDA and its relatives) was the hardest,” Kvedaras says. “It keeps no notes per word, just one running summary, so you can’t translate it word by word. When the sending model never kept a diary, the receiving model has to write its own by ‘reliving’ the conversation.”

The index looked riskier at first. “Get one card slightly wrong, and the model may never open the page with the answer,” he says. In practice, the diary remained “the real bottleneck.”

What mattered was keeping the right card, not matching every card. When we swapped only the index keys into the target’s own memory, the predicted keys matched just 63–84% of the blocks its own keys picked, but the block holding the answer stayed selected more than 99% of the time. That held on the one sparse-attention pair we tested, which uses Qwen’s version of the index rather than DeepSeek’s.

Training happens in two stages. First, we fit each map in closed form – calculating it directly in one step rather than learning it gradually – on a few hundred captured conversations totaling about half a million tokens.

Then we fine-tune all the maps together so the target’s next-token predictions on translated memory match its predictions on its own memory. This approach is called self-distillation, because the target model acts as its own teacher. It needs no labels, just chat conversations from NVIDIA’s Nemotron-SFT-Instruction-Following-Chat-v2 dataset. Building a translator for one direction of our main pairs took between 2 and 8 hours of computing time on a single graphics processing unit (GPU).

To find good designs for the forward handovers (Kimi-VL to Kimi-Linear, Qwen3.5 to Qwen3.8), an AI agent proposed candidate translators as code. A locked evaluator, a read-only container with no network access, trained and scored each one with a script the agent couldn’t edit. We chose the reverse directions from a hand-built set of variants using the same evaluator, and we checked every reported result against its result file.

“It automates the experimenting,” Kvedaras says. “You work like a much bigger team, where part of the team is AI agents. First, we fix the scoring method in advance. Nobody can change it afterward, the agent included.”

The results: faster handovers without starting from scratch

The flagships we have in mind start at 300 billion parameters. A model that size fits on a multi-GPU server, but not on the one or two GPUs our experiments ran on. So we tested two hybrid pairs between 16 and 48 billion parameters, plus one sparse-attention target, all storing the same kinds of memory as the flagships:

  • Kimi-VL-A3B and Kimi-Linear-48B – a fully compressed (MLA) model and a hybrid that mixes MLA with linear attention. Together, they’re scaled-down stand-ins for flagship pairs such as Kimi-K2.6 and Kimi-K3.
  • Qwen3.5-27B and Qwen3.8-27B – two generations of hybrid models with identical memory layouts.
  • Qwen3.8-27B to Qwen3.8-Flash-Next – a narrower test, in one direction with one training run, of a target that uses sparse attention.

The two models in each pair share a tokenizer, the component that splits text into tokens. We tested handovers in both directions for the first two pairs and measured:

  • reading comprehension (CoQA)
  • reasoning (ARC-Challenge)
  • exact recall of an access code hidden in long conversations
  • chat quality, rated from 1 to 10 by Gemma-4-31B acting as a judge
  • math word problems (GSM8K), as a check outside the chat data we trained on, in the forward direction only

All runs on these pairs used NVIDIA RTX PRO 6000 GPUs, and we trained the translators on conversations up to 8,000 tokens long.

Here’s what came back on speed, measured as the time until the receiving model starts its answer:

Handover1K tokens32K tokens64K tokens
Qwen3.5-27B → Qwen3.8-27B231 ms → 61 ms (3.8×)6.82 s → 208 ms (32.8×)14.9 s → 319 ms (46.8×)
Kimi-Linear → Kimi-VL99 ms → 36 ms (2.7×)2.42 s → 203 ms (11.9×)7.56 s → 404 ms (18.7×)
Kimi-VL → Kimi-Linear558 ms → 65 ms (8.6×)2.51 s → 668 ms (3.8×)Not timed: this speed test ran on a single GPU, which ran out of memory at 64K

We calculated speed-ups from the unrounded medians of 10 runs, so dividing the rounded times can give slightly different figures. Quality tests used a different setup and did reach 64,000 tokens.

The longer the conversation, the bigger the gain, with one exception. The Kimi-VL-to-Kimi-Linear handover has to rebuild the target’s running states and look up token identities, and both of these costs grow with length.

Part of Kimi-Linear’s reread time also comes from a one-off conversion step in our research setup that a production engine wouldn’t need. Leaving that step out yields a fairer, lower speed-up: 4.3 times at 1,000 tokens and 3.3 times at 32,000 tokens.

Quality depended on direction:

  • Qwen pair, both directions – the receiving model matched its own native quality on reading comprehension, reasoning, and chat. It recalled all 12 hidden codes at 64,000 and at 128,000 tokens, 16 times the longest training conversation. The one gap: in the reverse direction, it sometimes refused to share the code (more on that below).
  • Kimi-Linear to Kimi-VL (hybrid to fully compressed) – every task stayed within the margin of error of native performance, including perfect code recall up to 64,000 tokens. Reasoning and answer accuracy were 3–4 points below native levels, within that margin, while reading comprehension scores were higher because the translated answers were shorter.
  • Kimi-VL to Kimi-Linear (fully compressed to hybrid) – the hardest direction. It matched native performance on whether the right answer appeared, but its answers ran about twice as long. Reasoning accuracy was 5 points lower, chat quality scored 5.15 against the target’s own 6.02, and math accuracy (GSM8K) landed 7.5 points below the target.
  • Qwen3.8-27B to Qwen3.8-Flash-Next (sparse attention) – after 400 training steps, the receiving model recalled 86% of hidden codes on average across conversations of 4,000 to 64,000 tokens, using predicted index keys. That compares with 87% using its own index keys and 89% when reading its own memory. At the short end of that range, 4,000 and 8,000 tokens, it still trailed the model reading its own memory.

But why does direction matter so much?

“A full model wants a notebook with one page per word. A hybrid mostly keeps a single diary page that sums up everything so far,” Kvedaras says. Going from the hybrid to the full model, “the hybrid still keeps some pages per word, and we know the exact words. That’s enough to rebuild the whole notebook.”

Going the other way, the receiving model needs a diary the sender never kept. “So the receiver has to write the diary of a trip it never took, from someone else’s photos. The mood comes through. Exact details, like a six-digit code, get blurry.”

That’s where token identity made the biggest single difference. For conversations of up to 32,000 tokens, the Kimi-VL-to-Kimi-Linear translator without it recalled the exact access code in only nine of 100 tests, even though it got about 60% of the digits right. Kvedaras compares it to someone who remembers a phone number “starts with 73”: the meaning survived, but the exact words didn’t.

The fix was to read those words back, because the first layer of Kimi-VL’s memory depends only on the word itself. With token identity, recall across conversations of up to 32,000 tokens rose to 99%. It also held at 64,000 tokens, eight times longer than any conversation we trained on: the translator recalled 10 of 12 codes when it read the tokens from the cache, and all 12 when the serving system passed the token IDs directly.

The Qwen pair was the result Kvedaras didn’t expect. “I was skeptical that we could match hybrid models at all,” he recalls. Then, on that pair, copying the diary-style memory and applying a quick correction “just worked.”

What happens when models switch back and forth?

A single handover is one thing. Real conversations can bounce between models several times, and each model keeps its own memory of the conversation.

The obvious approach is to retranslate the entire history at each switch. The problem is that parts of that history were already translated once, so they pass through the translator repeatedly, and small errors pile up. The KV-Lingo team reported this collapse, and we reproduced it on classic key-value caches before testing our hybrid pairs.

The fix, introduced by KV-Lingo, is segment-wise translation. At each switch, translate only the part of the conversation that the receiving model hasn’t seen yet, and keep everything it has already seen. Nothing gets translated twice.

We extended this to hybrid models, in which the running state must also catch up at every switch. On the Qwen pair, we copy the source’s corrected running state. On the Kimi pair, when Kimi-Linear takes over, it continues its own running state over the newly translated segment.

We compared both approaches with a full reread on 50 reading-comprehension dialogues, each with five alternating turns. We translated both directions and used a single training run per pair:

  • Kimi pair – by the fifth turn, naive retranslation dropped the answer score (CoQA F1, which measures how closely an answer’s words match the correct one) to 0.22, against 0.60 for segment-wise translation and 0.65 when each model reprocessed everything. With naive retranslation, the model answered only 26% of fifth-turn questions correctly, compared with 70% after a full reread. Segment-wise translation matched the full reread on correct answers across all turns.
  • Qwen pair – segment-wise translation tied the full reread (0.695 vs. 0.700 on average) while spending about a sixth of the time catching up, 31 ms against 201 ms in total. Naive retranslation slipped here too, to 0.55 at the fifth turn against 0.68 for segment-wise translation, mostly because answers got longer, not less accurate.
Line chart of answer quality over five alternating turns between Kimi-VL and Kimi-Linear: retranslating the whole history falls to 0.22 by turn five, while translating only what’s new holds at 0.60, close to a full reread at 0.65.

For longer conversations, a simple policy helps: translate segment by segment, but let the model reread natively once after the fourth turn. We tested this over 10 turns on the Kimi pair, with 20 dialogues translated in one direction.

Answer quality over turns 6 to 10 averaged 0.571 with the refresh, against 0.536 without it. Catching up took 10 times as long as pure segment-wise translation, 1.0 second against 0.1 in total, but less than half the time of a full reread (2.4 seconds).

The lesson for anyone building multi-model agents is to translate only what’s new. If you’re also deciding where an agent’s model, runtime, and memory should live, our guide to AI agent infrastructure decisions covers the practical side.

What else we found

Habits travel with memory – “Memory carries personality,” as Kvedaras puts it. A translated running state can carry the source model’s behavior along with its information. Qwen3.5 starts 32% of its reading-comprehension answers with a leading space, while Qwen3.8 never does. In the Qwen3.5-to-Qwen3.8 direction, a plain state copy passed that habit on in 28% of answers, and our state correction removed it.

More importantly, in the Qwen3.8-to-Qwen3.5 direction, the receiving model refused to share six of 100 access codes, deeming them “sensitive” information. Qwen3.5 never refuses them on its own, while Qwen3.8, the source model, refuses two of 100.

The same refusal appeared in all three training runs, and standard quality and accuracy scores didn’t flag it. This rests on two behaviors within a single pair of models, so it’s a signal to watch rather than a general rule, but in setups where a smaller model hands off to a flagship model, it matters.

Cached memory isn’t anonymous – a simple linear read-out recovered the exact token from Kimi-VL’s first-layer cache with 100% accuracy. It also recovered a large share of tokens from every layer of every model we tested: typically 50–80% per layer, and at least 20% even at the hardest layer. A cache, stored or translated, should be protected like the conversation itself.

A good average can hide a failure – for the sparse-attention pair, a translator with an empty running state scored close to the working one on our main similarity measure but recalled only 13% of the hidden codes, compared with 86% for the working one. Task tests caught what the average missed.

What this could mean for AI-powered website building

Routers such as RouteLLM already send easier queries to a smaller model and harder ones to a stronger one, and agent pipelines can split a single task across several models. Each time the model changes, the receiving model either starts over or works from a summary.

At Hostinger, we combine multiple models too. “One practical use case is an AI builder. We’ve noticed that delivering the best value usually requires more than one model. In production, we often combine several models, each suited to specific tasks, to get the best results,” explains Tomas Rasymas, AI research lead at Hostinger and co-author of the paper.

Choosing which model to use comes with a trade-off. Kvedaras describes it as a choice between two models: model A solves 100 of 100 tasks, while model B solves 90 of 100 tasks but costs 100 times less. “You pick A. Every time,” he says.

Translated memory opens up a third option. Every conversation starts on the cheaper model, and when it gets “out of its depth,” as Kvedaras puts it, it hands the conversation to the stronger model partway through.

The catch is the reread. With our Qwen pair, two same-size models, the receiving model needed 6.8 seconds to catch up on a 32,000-token conversation. With translated memory, the handover took 0.2 seconds, at the same quality as if it had read everything itself. “That’s the 90/10 dream: 90 conversations stay cheap, 10 get the expert, and nobody waits,” Kvedaras says.

That’s the setup from the start of this article. In Hostinger AI Builder’s Agentic mode, users build and modify websites and web apps via chat, so the conversation grows as the project progresses.

A smaller, efficient model could handle the straightforward steps, and a more capable model could take over when a request needs deeper reasoning, picking up the conversation without rereading it. Rasymas describes how that could look in practice:

“Imagine, for example, that DeepSeek-V4.1-Flash is very good at creating designs, while GLM-5.3 excels at coding. Design requests would be handled by one model, and coding requests by the other. Normally, switching models requires the new model to prefill the existing context. With KV-cache conversion, we could potentially switch without repeating that prefill step.”

“We do not want to use a frontier model for everything; we want to use the best model for the use case and optimize on that, and the KV-cache technique helps us to reduce latency of different switches,” he says.

Both examples go beyond what we’ve tested so far. Our speed numbers come from same-family pairs with similar numbers of active parameters, the share of a model’s parameters it actually uses for each token, and every pair we tested shared a tokenizer.

A handover from a cheap model to a much stronger one, or from one family to another, such as from DeepSeek to GLM, is the next step. GLM-5.3 also adds a sparse-attention index on top of compressed memory, a combination we haven’t tested, and DeepSeek’s newest memory format wasn’t part of our tests.

This isn’t a feature in the product today. It’s a research result showing that the approach is workable on the models we tested.

Limitations and what comes next

These results come with clear boundaries:

  • Model size – our two main pairs range from 16 to 48 billion parameters, with 3 to 27 billion active. The flagships are much larger. Kimi-K2.6, for example, uses 32 billion active parameters per token, about 11 times as many as Kimi-VL-A3B, the fully compressed model that served as its proxy in our tests. Whether the methods hold at 300 billion parameters and up is a hypothesis our results support, not something we’ve shown.
  • Shared tokenizers – every pair we tested uses the same tokenizer. Moving memory between model families, such as from DeepSeek to Kimi, would require a way to align tokens that we haven’t validated. KV-Lingo has tested this on a Qwen3-Mistral pair with different tokenizers, but our research doesn’t cover cross-tokenizer handovers.
  • One weaker direction – handing over from a fully compressed model to a hybrid still trails native quality on chat (5.15 vs. 6.02) and math (7.5 points), and training beyond 1,000–2,000 steps didn’t close the gap.
  • Single runs – we trained the headline handovers three times with different random seeds, which change the order in which the training data is seen, but several other results rely on a single run. These include the Kimi-Linear-to-Kimi-VL direction, the multi-turn tests, and the sparse-attention test.
  • Research timing – we measured speed in a research framework (Hugging Face Transformers), not a production serving engine. The numbers measure the time until the answer starts, not total response time.
  • Untested formats – we didn’t test sparse attention on top of MLA, as used in GLM-5.x and DeepSeek-V3.2, or DeepSeek-V4’s block-compressed memory.

“Larger models come first: our results point that way, but at that scale it’s still a hypothesis until we test it,” Kvedaras says. “Next, we want to run it inside a real production system, not just a research setup.”

Toward AI systems that share context, not just text

The interesting shift here isn’t making one model faster. It’s lowering the cost of handing work between models with different strengths.

Back in that website-building assistant, the pause at each handover doesn’t have to last seconds. For the model pairs we tested, it shrank to a fraction of a second, as long as each type of memory got its own translation method. Whether the same holds when a small model hands off to the largest frontier models is the question our next experiments need to answer.

Read the full paper on arXiv.

Karina is an award-winning journalist with over 13 years of experience in newsrooms and institutional communications across Latin America and Europe. Her work spans print and digital media, startups, NGOs, and EU-funded projects, with a focus on climate and science storytelling. As a Senior Content Writer at Hostinger, she digs into data, products, and the people behind them to find the stories and angles that matter most. Follow her on LinkedIn.