From “Make It Talk” to “Make It Think”: My 10-Day Voice AI Journey
Jhansi Lakshmi Sambhana
Building my first AI voice agent through the VoiceForBharat challenge — from real-time conversations...
Building my first AI voice agent through the VoiceForBharat challenge — from real-time conversations to memory, tools, specialist handoffs, human support, and analytics.
Ten days ago, I started building my first AI voice agent.
I had never worked with voice agents or agentic AI before. I had worked with AI/ML, but building a complete system where speech, an LLM, tools, memory, real-time communication, and multiple agents work together was completely new to me.
I started with one simple question:
“How do I make an AI talk?”
I ended with a much bigger one:
“How do I make an AI system handle a real conversation responsibly?”
That journey became ExpenseMate.
💰 The Problem
Expense tracking is usually a form-filling task.
I wanted to make it conversational.
A user can simply say:
“I spent ₹500 on groceries today.”
ExpenseMate can understand the request, record the expense, and answer questions about spending.
It is designed for users who want a faster and more natural way to manage everyday expenses, especially when typing or navigating an app is inconvenient.
For the challenge, I chose the expense-management and payment-support use case, building ExpenseMate as a voice-first financial assistant with a human-support path for payment-related issues.
🎙️ From Voice to a Complete Agent
The core flow looks like:
User Speech → STT → LLM → Memory & Tools → Murf Falcon TTS → LiveKit → User
Speech-to-Text (STT) converts the user's speech into text.
Gemini understands the request, reasons about the next action, and decides when tools are needed.
Memory allows approved user information to be used in returning conversations.
Tools connect the agent to real application data and actions.
Murf Falcon converts the response back into natural speech.
LiveKit provides the real-time communication layer.
The result isn't just a chatbot that speaks.
It's an agent that can understand, act, remember, and respond.
🧩 What I Built
During the challenge, ExpenseMate grew into a complete voice-agent workflow:
🎙️ Real-time voice conversations
🧠 Consent-based user memory
💰 Expense recording and spending summaries
🛠️ Tool-based actions
🌏 English, Hindi, Telugu, and code-mixed conversations
🛡️ Financial safety guardrails
📊 Custom Call Analytics Dashboard
📞 Outbound SIP calling
🤝 Human escalation
💳 Payment Support Specialist
🔄 Specialist-agent handoff
👩💻 Human Support Center
🎨 Redesigned frontend UI
🧪 Routing tests
The frontend also reflects the agent's state, including ready, connecting, active conversation, and completed states.
🤝 When the AI Knows It Needs Help
One of my favourite parts of ExpenseMate is the payment-support flow.
If a user says:
“My refund hasn't arrived.”
the main agent doesn't pretend it can solve everything.
Instead:
Main Agent → Payment Support Specialist → Support Case → Human Support
The specialist continues the conversation, collects the necessary information, and creates a support case.
I then went beyond the basic handoff requirement and built a Human Support Center where a support agent can move the case through:
🟡 Pending Review → 🔵 In Review → 🟢 Resolved
This turned a simple specialist handoff into a complete human-in-the-loop workflow.
📊 Making the Agent Measurable
I also built a custom Call Analytics Dashboard to understand what happens beyond the conversation.
It provides visibility into:
Total calls
Successful and failed calls
Call outcomes
Recent call activity
I also integrated outbound SIP calling, allowing the voice agent to participate in real telephonic conversations and track their outcomes.
Because getting an agent to work once isn't enough.
You need to know how it performs.
The Challenge I Didn't Expect
One of my biggest challenges was Telugu voice interaction.
When I spoke Telugu quickly, some words weren't recognized correctly and the response could become slow or inaccurate.
I tested different speaking speeds, voice/recognition settings, and prompt adjustments, then repeatedly tested the conversation until the experience became more reliable.
That taught me something important:
Voice AI isn't just an LLM problem.
Speech recognition, language, latency, prompts, tools, and responses all affect the final experience.
⭐ Going Beyond “Task Completed”
I didn't want to stop once the required functionality worked.
I kept asking:
“What would make this feel more like a real product?”
That led me to add improvements beyond the core task requirements, including:
🎨 A redesigned and more polished UI
📊 A custom Call Analytics Dashboard
👩💻 A Human Support Center
🔄 Support-case lifecycle tracking
🤝 A complete specialist handoff workflow
🧪 Routing tests for different user requests
Instead of treating the ten days as ten separate tasks, I connected them into one system.
🛠️ Want to Build Your Own Voice Agent?
The basic setup is:
git clone https://github.com/Jhansi48/murf-livekit-starter.git
cd murf-livekit-starter
Then:
Clone the repository
↓ Create environment files
↓ Add API keys
↓ Install backend dependencies
↓ Install frontend dependencies
↓ Start the LiveKit agent
↓ Start the frontend
↓ Open the app and start a voice conversation
For my project, the backend uses uv and the frontend uses pnpm.
The project uses environment variables for credentials such as:
LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET
MURF_API_KEY
DEEPGRAM_API_KEY
GOOGLE_API_KEY
API keys should remain in environment files and never be committed to GitHub.
The same applies to phone numbers, caller data, OTPs, PINs, card details, UPI credentials, and other private information.
ExpenseMate — a multilingual AI voice agent for conversational expense tracking, payment support, human escalation, and call analytics.
Voice Agent Starter — Powered by Murf Falcon
Build a production voice AI agent in 5 minutes. Powered by the fastest TTS on the market - swap the system prompt to build anything from customer support to language tutors.
Why Murf Falcon
55ms model latency - fastest production TTS
130ms time-to-first-audio across 10+ global regions
$0.01/1000 characters - up to 10x cheaper than alternatives
150+ voices across 35+ languages
99.38% pronunciation accuracy
Architecture
flowchart LR
A[🎙️ User speaks] -->|audio| B[Deepgram STT]
B -->|text| C[LLM]
C -->|response text| D[Murf Falcon TTS]
D -->|audio| E[LiveKit]
E -->|stream| F[🔊 User hears]
style A fill:#444441,stroke:#888780,color:#fff
style B fill:#185FA5,stroke:#85B7EB,color:#fff
style C fill:#534AB7,stroke:#AFA9EC,color:#fff
style D fill:#0F6E56,stroke:#5DCAA5,color:#fff
style E fill:#D85A30,stroke:#F0997B,color:#fff
style F fill:#444441,stroke:#888780,color:#fff
The demo shows the complete workflow, including voice interaction, memory, multilingual conversation, specialist handoff, human support, case resolution, outbound SIP calling, and call analytics.
🚀 What's Next?
There is still a lot I would like to improve:
Better Telugu and other Indian-language recognition
More reliable fast-speech handling
Stronger specialist routing
Multi-agent handbacks
Failed-handoff recovery
Richer analytics and evaluation
Production-ready deployment
The goal is not simply to make the agent work.
It is to make it more reliable, useful, and natural in real conversations.
❤️What These 10 Days Taught Me
For someone who had never built a voice agent or worked with agentic AI before, this challenge gave me much more than a working project.
I learned how speech recognition, LLM reasoning, memory, tools, safety guardrails, specialist agents, human support, telephony, and analytics can work together as one system.
And perhaps the most valuable part was realizing that building the first version is only the beginning.
The real work is testing it with real conversations, finding where it fails, understanding why it fails, and improving the experience.
Ten days ago, I had never built a voice agent.
Today, I don't just have a working project — I have a much clearer understanding of what it takes to build, test, debug, and improve an AI agent that interacts with real people.
The biggest lesson wasn't learning how to make an AI speak.
It was learning how to make it listen, remember, act, stay safe, know its limits, and ask for help when it needs to.
That is what ExpenseMate became during these 10 days. 🚀
10 Days of Voice Agents — VoiceForBharat Edition, complete.
drive.google.com