Nobody Opened Your Product to Chat With It

Most of the AI work that lands on my desk now is not a new AI product. It is an existing product, with paying customers and years of schema behind it, whose owner has been told to "add AI" before a competitor does. Sometimes the push comes from the board, sometimes from sales, sometimes from one customer on a renewal call.

The first draft is almost always the same: a chat assistant in the bottom right corner that answers questions about the product. It demos beautifully. Then usage falls off a cliff within a few weeks, because nobody opened the product to have a conversation. They opened it to get a piece of work done.

As an engineer, I find the interesting part is that the model is the easiest piece of the whole thing. The work is in the boring layers around it: which data goes in, whose permissions it runs under, where the output lands on screen, and what happens when the model is wrong or down. That is what this post is about.

Start from a job, not a model

"Where can we use AI?" is the wrong question. The useful one is: which task do users already do in this product, over and over, that is slow, tedious or error-prone?

The candidates that keep working share a shape:

• Summarizing something long that a user has to read before acting: a thread, a record history, a recording.
• Filling structured fields from something messy: an uploaded file, an email, a note, mapped into the form you already have.
• Drafting a first version of a reply or report that a human edits and approves.
• Sorting and routing an incoming item into the right queue or category.
• Finding the thing: search that understands intent, not only keywords.

Every one of those has an existing workflow you can measure today. That gives you a baseline, and a baseline is the only way you will ever know whether the feature did anything.

My own app, LectureNotes AI, is a clean example even though AI was in it from day one. Nobody records a lecture because they want to chat with a model. They want notes they can study from. The transcript, the key takeaways and the structured outline all serve that one job, and none of them is a chat window. (I wrote about what that app taught me separately.)

The data work nobody put in the estimate

Mature products were built to show data to humans, not to hand it to a model. That gap is where most of the build time goes.

Assembling context is ordinary engineering, and it is most of the job. For "summarize this customer's history", somebody has to decide which records count as history, how far back to go, which fields matter and which to leave out. In a product that has been around a few years, that data lives across several tables, a couple of services and at least one third-party tool. Retrieval quality decides answer quality far more than model choice does, which is the same thing I ran into on RAG projects.

The AI feature must obey the permissions you already have. This is the one I would underline. If a user cannot open a record, the assistant must not summarize it, quote from it, or let it leak into a search answer. The tempting shortcut is to give the AI feature its own service account with broad read access, because it is faster to build. Do not. Fetch context as the requesting user, through the same access layer the rest of the product uses.

Know where the data is allowed to go. If your customers signed contracts with data-handling terms, sending their records to a model provider may or may not be covered. Find out before the build. If it is not covered, there are workable options: zero-retention provider terms, a specific processing region, a self-hosted model, or stripping sensitive fields before the call.

Here is roughly what the "same access layer, fail quietly" shape looks like. This is illustrative, not lifted from a client codebase:

// Illustrative: the AI panel reuses the product's own access layer
// and never blocks the page it lives on.
async function getRecordSummary(user: User, recordId: string) {
  // Same call the record page already makes. If the user can't see it,
  // this throws exactly like it would for the normal UI.
  const record = await records.getForUser(user, recordId);
  const history = await activity.listForUser(user, recordId, { limit: 50 });

  const context = buildContext(record, history); // only the fields the task needs

  try {
    return await withTimeout(llm.summarize(context), 4000);
  } catch (err) {
    log.warn("summary_unavailable", { recordId, err });
    return null; // page renders without the panel
  }
}

Two things matter in there. There is no privileged path to the database, so a permission bug in the AI feature is impossible by construction rather than prevented by review. And a slow or failing provider returns null instead of hanging the screen. The product worked before the feature existed. It should keep working when the feature cannot.

Put it where the work already happens

Placement is the single biggest factor in whether anyone keeps using an AI feature. The pattern that works is to put the output inside the screen the user is already on, at the moment they need it:

• A summary panel at the top of the record they just opened.
• Form fields that arrive pre-filled from an uploaded document, visibly marked as suggested until confirmed.
• A "draft reply" button next to the existing reply box, not in a separate tool.
• A suggested category already selected in the dropdown they were about to touch.

In each case the existing workflow stays intact and the user stays in control. If the AI is wrong, they fix it the same way they would fix their own mistake. If it is right, they saved time without learning a single new thing.

A general chat assistant can earn a place later, once you know which questions people actually ask. It is almost never the right first feature.

Design for being wrong

Every AI feature will sometimes produce a bad output, so treat that as a normal code path, not an edge case.

• Suggestions should look like suggestions. Pre-filled fields stay visually distinct until a human confirms them. Nothing generated silently becomes a record of fact.
• Irreversible actions stay behind a human. Reading and suggesting is low risk. Sending, deleting, paying or changing someone else's data is a different product with different rules. I went deep on that in giving an agent write access one verb at a time.
• Give users a one-click "this is wrong". It costs almost nothing and turns every failure into a labeled example.

Cost scales with use, not seats

Normal features cost about the same whether a customer uses them once a month or all day. AI features do not. Every summary and every draft is a metered call, and your heaviest users can cost many times what your lightest ones do.

Do the arithmetic at realistic usage per active user per month before launch. If the number hurts, the fixes are architectural: generate the summary once when the record changes instead of on every page view, send only the context the task needs, and route narrow tasks to smaller models. I wrote a whole post on building AI features as if calls were metered, because they are.

Ship it like any other risky feature

1. Build a small evaluation set first. Thirty to fifty real examples from your own product, each with what a good output looks like. Run it on every prompt or model change so you can tell an improvement from a regression.
2. Release behind a feature flag: internal users, then a few friendly customers, then everyone.
3. Measure the job, not the clicks. Did time on the task drop? Did users accept the suggestion or rewrite it? Are they still using it a month later? Opening the panel once is curiosity, not adoption.
4. Keep watching after launch. Providers update models underneath you, and your data drifts.

It also pays to route every model call through one internal interface with prompts in configuration. Then trying a cheaper or better model is a config change plus an eval run, not a rewrite.

What I would actually do first

Pick one slow, repeated job. Feed the model clean context fetched under the user's own permissions. Put the output exactly where users already work, make it fail quietly, and measure whether it sticks. One feature people keep beats five they tried once.

This is adapted from a longer guide on the Null Studio blog, where we cover the commissioning side too. I am curious what others have seen: has anyone shipped the bottom-corner chat assistant and had it actually stick? What made the difference?