The Model Remembered a Conversation the Server Had Already Forgotten

The support bot answered with perfect confidence, citing an order number and a complaint from two days earlier. The only problem was that this session had never mentioned either one, and my application had no record of them anywhere.

I was prototyping a small support assistant on a free server with free model access through MonkeyCode, and the setup felt simple enough. The client kept a conversation array, the server persisted it to a local JSON file, and the model API received the full history on every turn. What could possibly go wrong?

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The symptom

A tester opened the bot, asked about a refund, then closed the tab. Two days later they returned, asked "what about the second option we discussed?", and the bot answered with a detailed recap of options that had never existed in this session.

My first reaction was the standard one: the model is hallucinating. Free models can be creative, and a vague follow-up question is exactly the kind of prompt that invites confident fiction.

The first suspect

I added prompt logging to see exactly what the application was sending. That is the single most useful debugging habit I have, and it is also the one I skip most often when I am in a hurry.

The log showed something interesting. The application sent an empty history array, because the server-side session file was missing, and the model still produced a coherent answer referencing old details. That meant the model was not hallucinating; it was responding to context that my code had never sent.

Wait. If the server had no history, where did the model get those details?

The log that changed everything

I logged the full request payload, not just the prompt text, and the answer appeared immediately. The client-side conversation array still contained the old turns from two days ago, and it was appending new messages to that array before sending the whole thing to the model API.

The server had restarted at some point, and the free server's filesystem had been wiped clean. The client, however, never noticed, so it kept sending the full conversation history to the model, and the model answered correctly given what it received.

The model was fine. The server was fine. The client and the server were just living in different realities.

Root cause: split-brain state

The real bug was that I had two sources of truth for conversation state, and only one of them was durable.

• The client held the authoritative conversation array in memory.
• The server mirrored it to a local JSON file for persistence.
• The free server's filesystem is ephemeral, so the mirror disappeared on restart.
• The client never detected the reset, so it kept sending stale history to the model.

That is the classic split-brain failure. Two components assume they agree, neither checks for divergence, and the system produces output that looks correct to one side and impossible to the other.

The fix

I made three changes, and the order mattered.

1. I stopped treating the server as a place to store anything. Conversation state moved to an external key-value store, keyed by session ID, with the server acting as a stateless proxy.
2. I added a session epoch to every request. The server stamps each session with a version at creation time, and the client sends that epoch along with every turn. If the server has no record of the epoch, it returns a reset signal instead of pretending everything is fine.
3. I made the client reconcile instead of trusting its own memory. When the server reports a reset, the client clears its conversation array and tells the user that the context was lost, rather than silently sending stale turns to the model.

Reproduce it yourself

Here is a minimal reproduction of the failure, so you can see the split-brain in action before you build around it.

// split-brain.js — run with: node split-brain.js
const fs = require("fs");
const path = require("path");

const HISTORY_FILE = path.join(process.cwd(), "data", "session.json");
const clientMemory = []; // the conversation array that "never lies"

function saveTurn(role, content) {
  clientMemory.push({ role, content });
  fs.mkdirSync(path.dirname(HISTORY_FILE), { recursive: true });
  fs.writeFileSync(HISTORY_FILE, JSON.stringify(clientMemory));
}

function loadFromServer() {
  // After a restart, this file simply does not exist.
  if (!fs.existsSync(HISTORY_FILE)) return [];
  return JSON.parse(fs.readFileSync(HISTORY_FILE, "utf8"));
}

saveTurn("user", "I want a refund for order 4417.");
saveTurn("assistant", "I can help with that. Which option do you prefer?");

// Simulate the free server recycling: the data directory vanishes.
fs.rmSync(path.dirname(HISTORY_FILE), { recursive: true, force: true });

// The client still holds the old turns and happily sends them.
console.log("Server history:", loadFromServer());
console.log("Client history:", clientMemory.length, "turns");

// sendToModel is pseudocode; replace it with your real API call.
sendToModel(clientMemory);

The output shows the discrepancy immediately: the server sees zero history, while the client still has two turns and will keep adding to them. That is the exact moment the model starts answering questions nobody in the current session asked.

The fixed version checks the epoch before sending anything:

async function reconcile(epoch) {
  const serverState = await fetchState(epoch); // pseudocode for your store
  if (serverState.reset) {
    clientMemory.length = 0; // drop the stale turns
    return "Session was reset. Please restate your question.";
  }
  return null;
}

Limitations and who should skip this

This approach assumes your conversation state is small enough to store externally and cheap enough to fetch on every turn. If you are building a high-throughput pipeline with thousands of concurrent sessions, a per-turn external fetch will hurt, and you should design for idempotent retries instead of reconciliation.

It also does not solve model-side context limits. If your history grows beyond the model's context window, no amount of session hygiene will help; you need summarization or retrieval, which is a different problem entirely.

And if you are just experimenting with a single user and a short session, the external store is overkill. The bug only bites when sessions outlive the server, which is exactly the case that free server instances make likely.

The reusable checklist

Next time a model answers something it should not know, walk this list before you blame the model.

• Log the exact request payload, including headers and any history arrays.
• Verify what the client sent, not what you think it sent.
• Check whether your state store survived the last deploy or restart.
• Ask whether two components hold the same state and how they detect divergence.
• Treat every server as stateless until you have proven it is not.

The model was never the liar in my story. The split-brain was, and the fix was not a better prompt; it was a better assumption about where state actually lives. If you have a similar story where the infrastructure lied and the model took the blame, I would genuinely like to hear it in the comments.