Building Customer Service Chatbots with LLMs

Customer service chatbots sit at the intersection of latency, accuracy, and cost. Every extra turn in a conversation increases context length, and token-based billing turns long support threads into expensive operations. A request-based pricing model removes that variable, letting engineering teams optimize for user experience instead of input length. This guide walks through the architecture, model selection, and implementation patterns for production chatbots, with examples built on Oxlo.ai.

Architecture of a Production Chatbot

A robust customer service chatbot is not just a prompt sent to an LLM. It is a pipeline: intent classification, retrieval-augmented generation (RAG) for policy lookup, a stateful memory layer, tool executors for actions like refunds or bookings, and guardrails to prevent hallucinated promises.

The LLM layer handles natural language understanding, response generation, and tool selection. Everything else is infrastructure. Keep the LLM replaceable and the state external. Store conversation history in a database or cache, and pass only the relevant window into the model. This keeps costs predictable and simplifies debugging.

Model Selection

Not every turn requires a 400B parameter model. Route simple queries to smaller, faster models and reserve large reasoning models for complex escalations.

Oxlo.ai offers 45+ models across 7 categories that fit this routing strategy:

• Llama 3.3 70B works well as a general-purpose backbone for intent detection and standard responses.
• Qwen 3 32B handles multilingual reasoning and agent workflows if your user base is global.
• DeepSeek R1 671B MoE or Kimi K2.6 are better suited for technical troubleshooting or ambiguous disputes that require deep reasoning.
• For vision-enabled support, such as analyzing user-uploaded photos of damaged products, Kimi VL A3B or Gemma 3 27B are relevant options.

Because Oxlo.ai uses request-based pricing, you can afford to keep more context in the prompt without watching token meters spin up. That matters when a customer sends a wall of text or when you include full RAG context.

Managing Context and Memory

Customer service conversations often exceed dozens of turns. Sending the entire transcript to the model on every request is simple, but it bloats latency and cost on token-based platforms.

Instead, implement a sliding window with summarization. Keep the last N messages in full, and compress older turns into a running summary. When a user asks, "What did I say my order number was?", the model still has the answer without you paying for the full transcript on every API call.

On Oxlo.ai, the economics change. Because you pay per request rather than per token, you can experiment with larger context windows and more elaborate system prompts without linear cost growth. This is especially useful for agentic chatbots that maintain long-horizon state.

Tool Use and Function Calling

A chatbot that only talks is a FAQ engine. A chatbot that acts is a support agent. Function calling lets the model invoke external tools to check order status, update tickets, or initiate returns.

Oxlo.ai supports tool use and streaming across its chat models. The API is fully OpenAI SDK compatible, so you can drop in your existing client with one URL change.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_order_status",
            "description": "Retrieve the current status of a customer order",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {"type": "string", "description": "The order identifier"}
                },
                "required": ["order_id"]
            }
        }
    }
]

def run_chat_loop(messages):
    response = client.chat.completions.create(
        model="llama-3.3-70b",  # verify exact slug in Oxlo.ai catalog
        messages=messages,
        tools=tools,
        tool_choice="auto",
        stream=True
    )

    for chunk in response:
        # Handle streaming deltas or tool calls
        delta = chunk.choices[0].delta
        if delta.tool_calls:
            yield {"type": "tool_call", "data": delta.tool_calls}
        elif delta.content:
            yield {"type": "content", "data": delta.content}

When the model emits a tool call, execute it in your backend, append the result to the message list, and send a follow-up request. This multi-turn pattern is standard, but on token-based platforms it becomes expensive fast. With Oxlo.ai, the second request costs the same flat rate as the first, regardless of how much JSON you append.

Guardrails and Structured Output

Customer service requires consistency. You cannot have the model offering refunds outside policy or hallucinating tracking numbers. Use JSON mode to constrain outputs to a schema, and validate every response before showing it to the user.

Oxlo.ai supports JSON mode and system prompts on all relevant chat models. A typical guardrail setup looks like this:

system_prompt = """You are a support agent. Follow these rules strictly:
1. Never promise a refund unless the order_id was verified by the get_order_status tool.
2. If the user is angry, acknowledge their frustration before solving.
3. Respond in JSON with keys: 'response_text', 'action', 'requires_human'."""

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_message}
    ],
    response_format={"type": "json_object"}
)

Parse the JSON, check the requires_human flag, and route to a human agent if needed. This keeps the LLM inside a deterministic boundary.

Cost Optimization at Scale

Token-based pricing penalizes long-context and agentic workloads. A customer service thread with a detailed system prompt, RAG documents, and ten turns of history can easily consume tens of thousands of tokens per request. On token-based providers, that cost scales linearly.

Oxlo.ai charges a flat rate per API request. For chatbots, this means:

• Long system prompts do not increase cost.
• Multi-turn history does not increase cost.
• RAG context injection does not increase cost.
• Tool result loops do not increase cost.

This predictability makes capacity planning simple. You know that 10,000 customer conversations equals 10,000 requests, not a variable token bill dependent on how chatty your users are. For high-volume support centers, request-based pricing can be 10-100x cheaper than token-based for long-context workloads. See the exact rates on the Oxlo.ai pricing page.

Deployment Patterns

Deploy the chatbot as a stateless service that calls Oxlo.ai. Because there are no cold starts on popular models, you get consistent latency from the first request. This matters for synchronous chat interfaces where users expect sub-second replies.

If you need higher throughput, Oxlo.ai offers a Premium plan with priority queueing and an Enterprise tier with dedicated GPUs. Start with the Free tier to prototype: it includes 60 requests per day and a 7-day full-access trial across 16+ models.

Next Steps

Build a prototype that combines RAG, function calling, and streaming. Start with Llama 3.3 70B on Oxlo.ai for general queries, and add DeepSeek R1 or Kimi K2.6 for escalation routing. Keep your state external, your prompts versioned, and your outputs validated.

You can sign up for an API key at Oxlo.ai and point your existing OpenAI SDK client to https://api.oxlo.ai/v1. The flat per-request pricing means you can focus on resolution quality instead of token counting.