Over the last 10 days, I built Bharat Finance Saathi, a voice-first AI assistant designed to help Indian users understand financial and government loan schemes more easily.

This project was part of 10 Days of Voice Agents — VoiceForBharat Edition, and it gave me the opportunity to go beyond building a simple chatbot. During the challenge, I worked with real-time voice communication, memory, tools, outbound calling, human escalation, analytics, and specialist agent handoffs.

More importantly, I learned that building a voice agent is not just about making an AI talk. The real challenge is making the conversation useful, natural, reliable, and easy for people to interact with.

1. The Problem I Wanted to Solve

Many first-time entrepreneurs need financial support but may not know:

• Which government scheme is suitable for them
• Whether they are eligible
• What documents they need
• How much funding they may qualify for
• What they should do next

Traditional websites and forms can be difficult to navigate, especially for users who are more comfortable speaking than typing.

So I chose the Financial Services track and built Bharat Finance Saathi as a voice-first assistant.

The idea was simple:

Instead of making users search through complicated information, let them simply ask.

The agent can communicate naturally, understand the user's request, provide relevant guidance, and involve a human when necessary.

2. What Bharat Finance Saathi Does

Bharat Finance Saathi is designed as a multilingual financial voice assistant.

Some of its main capabilities include:

• Natural voice conversations using Murf Falcon
• English and code-mixed conversations
• MUDRA eligibility checking
• Loan document guidance
• User memory
• Web/data lookup
• Outbound calling
• Human-help escalation
• Call analytics
• Specialist agent handoff

I also focused heavily on the user experience. The frontend provides a clean interface where users can easily see the agent state and interact with it without needing technical knowledge.

3. How the System Works

The basic flow of the voice agent looks like this:

User speaks → LiveKit → Gemini → Python tools → Murf Falcon → User

The user speaks through the browser.

LiveKit Agents handles the real-time audio communication between the user and the agent.

Google Gemini understands the conversation and generates the response.

When the agent needs additional information or needs to perform an action, it can use Python function tools.

Finally, Murf Falcon converts the response into natural speech and delivers it back to the user.

This combination allowed me to build the complete voice interaction instead of simply generating text responses.

4. The Technology Behind It

Backend

• Python
• LiveKit Agents
• Google Gemini API
• SQLite
• Murf Falcon

Frontend

• Next.js
• TypeScript
• React

SQLite was used for lightweight persistence and memory-related functionality.

The frontend was built with Next.js and TypeScript, while LiveKit Agents handled the real-time voice-agent workflow.

5. Important Features I Built

Memory for Returning Users

One of the early goals was to make the agent remember useful information.

For example, if a user previously mentioned their business type or loan requirement, the agent could use that information later instead of asking the same questions repeatedly.

This made the interaction feel more like a continuing conversation rather than a fresh chatbot session every time.

Financial Tools

The agent can use Python function tools to perform specific tasks instead of relying only on the LLM.

Some examples include:

• MUDRA eligibility checking
• Document checklist generation
• Web/data lookup
• User memory
• Human-help escalation
• Outbound calling

This separation between the LLM and actual tools helped make the system more practical and controlled.

Human Escalation

Not every financial question should be handled entirely by an AI agent.

I added a human-help escalation flow so that when the conversation requires human assistance, the system can create a request instead of forcing the AI to answer something outside its scope.

This was an important lesson in designing AI systems:

A good AI agent should know when it needs help.

Call Analytics Dashboard

For Day 8, I built a call analytics dashboard connected to actual call data.

It tracks:

• Total Calls
• Successful Calls
• Failed Calls
• Success Rate

The dashboard uses real call outcomes rather than hardcoded numbers.

I also made sure that sensitive information such as OTPs, PINs, account numbers, phone numbers, and full conversation transcripts is not exposed.

Specialist Agent Handoff

For Day 9, I added a dedicated Government Scheme Specialist.

The main agent handles general financial assistance, while the specialist focuses on more detailed government-scheme questions such as:

• PMEGP
• Stand-Up India
• CGTMSE
• Scheme comparisons

For example, if a user asks:

“Which is better for my manufacturing business, PMEGP or MUDRA?”

the main agent can explain that it will connect the user to the Government Scheme Specialist.

The specialist then continues the conversation without asking the user to repeat everything.

This made the architecture more modular and showed me why one agent should not try to be an expert at everything.

6. Challenges I Faced

The most valuable part of this challenge was probably the problems I encountered.

Things did not always work on the first attempt.

Outbound Calling Issues

Around Day 7, I worked on the outbound calling functionality.

The browser-based agent could work, but making the calling workflow reliable was more difficult. I had to deal with the interaction between the agent session, LiveKit, and the calling setup.

Sometimes the call did not connect as expected, which taught me that real-time calling involves much more than simply triggering a function.

Linphone Connection Problems

While testing the calling functionality, I also faced Linphone connection errors.

The agent could be running correctly while the phone-side connection was still failing.

This forced me to debug the complete chain instead of assuming that the problem was in the agent code.

It helped me understand the importance of checking:

Agent → LiveKit → SIP/Calling layer → Phone client

Slow Agent Responses

Another major challenge was response latency.

Sometimes the user finished speaking and had to wait before the agent responded.

In a normal chatbot, waiting a few seconds may not feel like a major problem. In a voice conversation, however, silence feels much more noticeable.

This made me pay more attention to:

• Prompt size
• Conversation flow
• API response time
• External service latency
• How much information the agent processes before speaking

I learned that a voice agent needs to be designed for speed as well as accuracy.

Gemini API Timeouts and Rate Limits

During testing, I also encountered Gemini API errors such as:

504 Gateway Timeout

and

DEADLINE_EXCEEDED

At one point, I also reached the API rate limit.

The important lesson was that an application can be correctly implemented and still fail because an external API is temporarily unavailable or a quota has been exceeded.

Instead of immediately changing working code, I learned to inspect the logs first and identify whether the issue was coming from my application, the network, or the external API.

Managing Multiple Features

Another challenge was combining everything without breaking existing functionality.

Over the 10 days, the project grew to include:

• Memory
• Tools
• Eligibility checks
• Document guidance
• Outbound calls
• Human escalation
• Analytics
• Specialist handoffs
• Dark/light mode
• A user-friendly frontend

Adding one feature was usually manageable.

Making sure the new feature did not break an existing one was the harder part.

This taught me to make small changes, test frequently, inspect logs, and preserve working functionality whenever possible.

7. How You Can Build Your Own Voice Agent

A basic voice agent can be understood as four main components:

1. Speech-to-Text

Converts what the user says into text.

2. LLM

Understands the request and decides what the agent should do.

3. Text-to-Speech

Converts the AI response back into natural speech.

4. Real-Time Transport

Moves audio between the user and the agent with low latency.

In my project, LiveKit Agents handles the real-time voice workflow, Google Gemini handles the LLM side, and Murf Falcon provides the voice output.

8. Running the Project

Clone the repository:

git clone https://github.com/Net-Ion/murf-livekit-voiceagent.git
cd murf-livekit-voiceagent

Install the backend dependencies and configure the required environment variables.

Your API keys should be stored in environment variables, for example:

GOOGLE_API_KEY=your_key
LIVEKIT_API_KEY=your_key
LIVEKIT_API_SECRET=your_secret
MURF_API_KEY=your_key

Never commit your .env file or API keys to GitHub.

Then start the backend using the project's existing LiveKit agent command.

For the frontend:

cd frontend
npm install
npm run dev

For the Backend :

you can run: uv run python src/agent.py dev
Open the local Next.js application in your browser and start a voice session.

You can test questions such as:

“What documents are required for a MUDRA loan?”

or:

“Which government scheme is suitable for my business?”

9. What I Would Improve Next

There is still a lot I would like to improve.

Some ideas for the next version include:

• Better Hindi and regional-language support
• Faster response times
• More robust calling and SIP handling
• Better interruption handling
• More advanced call analytics
• More government schemes
• Better error recovery
• More specialist agents
• WhatsApp integration

The goal would be to make Bharat Finance Saathi even more useful for people who prefer voice over traditional financial websites.

10. What I Learned

This challenge changed the way I think about voice AI.

At first, I thought the main challenge would be connecting an LLM with speech.

But after building the project, I realized that the difficult part is everything around the model:

conversation design, latency, memory, tools, networking, error handling, privacy, user experience, and reliability.

The problems I faced with calling, Linphone, API limits, and response delays were not just obstacles. They became some of the most useful learning experiences during the challenge.

Bharat Finance Saathi started as an idea for a financial voice assistant and gradually became a complete voice AI application with memory, tools, human escalation, analytics, and specialist handoffs.

Final Thoughts

Building Bharat Finance Saathi over these 10 days was a challenging but rewarding experience.

I learned how to build a voice agent, but more importantly, I learned how to think about voice AI as a real product.

The experience showed me that a useful AI agent should not simply answer questions. It should understand the user, remember useful context, use the right tools, know its limitations, ask for human help when necessary, and provide a simple experience.

I'm grateful to Murf AI for creating the 10 Days of Voice Agents — VoiceForBharat Edition challenge and giving me the opportunity to build, test, break, debug, and improve a real voice AI application.

🇮🇳 This is Bharat Finance Saathi.

Project

GitHub:
: https://github.com/Net-Ion/murf-livekit-voiceagent

Demo: : https://murf-livekit-voiceagent.vercel.app/