Building FinSaathi: A Voice-First Financial Assistant for Bharat

A voice-first financial assistant for Bharat, built during the 10 Days of Voice Agents — VoiceForBharat Edition.

1. The Real-World Problem: Making Financial Assistance More Conversational

Financial services and government schemes can be difficult to navigate. Users may need to understand eligibility criteria, required documents, application information, deadlines, or what to do when something goes wrong.

For many users, especially users who are more comfortable speaking in Hindi, English, or Hinglish than typing long queries, a voice interface can provide a more natural way to access assistance.

This led to the idea behind FinSaathi.

Instead of making the user navigate through multiple forms and pages, FinSaathi allows the user to simply describe what they need.

"PMJJBY ke liye main eligible hoon?"

Or:

"Mere account se ek unauthorized transaction hua hai."

The goal is not to replace banks, government portals, or human support teams. The goal is to create a conversational first layer that can understand intent, provide appropriate assistance, perform supported actions, and know when the conversation should be handed to a human or specialist.

Why Voice Matters

A text interface gives users time to edit and rewrite their questions. A voice conversation is different: the user explains their situation naturally and the system has to understand the intent in real time.

The interaction becomes:

Speak → Understand → Decide → Act → Respond

That simple loop became the foundation of FinSaathi.

2. Responsible Product Positioning: Assistant, Not Financial Authority

FinSaathi is designed as an assistance system, not as a bank employee, financial regulator, or replacement for human support.

What FinSaathi Is

• A conversational financial assistance interface.
• A voice-first way to access supported information.
• A tool-using AI agent.
• A system with persistent user data for supported workflows.
• A bridge to human support.
• A platform that can route specialized requests to specialist agents.

What FinSaathi Is Not

• A replacement for a bank.
• A replacement for a government portal.
• A system that should request passwords, PINs, OTPs, or CVVs.
• A system with unlimited access to private financial accounts.
• A guaranteed financial decision-maker.

Safety Guardrails

Financial conversations require strong safety boundaries.

FinSaathi explicitly avoids requesting sensitive authentication information such as:

• OTPs.
• PINs.
• Passwords.
• CVVs.
• Full card details.

When the agent cannot perform a requested action directly, it should communicate that limitation rather than inventing an action.

3. What Was Built: The 10-Day Evolution

The important part was that the features were not built as isolated demos. Each new capability had to work with the voice experience that already existed.

4. Technical Architecture

FinSaathi is built as a full-stack voice-agent system.

At a high level, the user speaks through the real-time communication layer. The speech is processed by the voice pipeline, the AI agent determines the next action, and the response is converted back into speech using Murf Falcon.

The agent can also access application-specific tools, memory, human escalation, analytics, and specialist workflows.

End-to-End Pipeline

1. Real-Time Transport: LiveKit manages the real-time voice connection.
2. Speech Recognition: User speech is converted into text by the configured speech-to-text layer.
3. Agent Core: The LLM-powered agent understands the request and determines the appropriate response or action.
4. Tools: The agent can invoke application-specific workflows such as eligibility checks or escalation creation.
5. Memory & Database: Relevant information is persisted in SQLite.
6. Specialist Routing: Specific conversations can be transferred to a specialist agent.
7. Human Escalation: Situations requiring human support can create a support request with a reference ID.
8. Analytics: Call outcomes are stored and surfaced through the analytics dashboard.
9. Speech Synthesis: Murf Falcon converts the generated response into natural speech.

Architecture Diagram

Technology Stack

5. Building the Agent

The first version of FinSaathi was intentionally kept simple.

Instead of trying to build every feature at once, I started with the core voice loop and gradually connected the additional services around it.

The basic voice-agent pipeline can be thought of as four major layers:

• Speech-to-Text — converts the user's speech into text.
• LLM / Agent — understands the request and decides what to do.
• Tools — allow the agent to perform real actions.
• Text-to-Speech — converts the final response back into natural speech.

LiveKit provides the real-time communication layer.

Real-Time Voice with LiveKit

The simplified flow is:

User speaks → LiveKit → Speech Recognition → AI Agent → Response → Murf Falcon → User

This makes the interaction feel much closer to a normal conversation.

The AI Agent

The AI agent is the decision-making layer of FinSaathi. Its responsibility is to understand the user's intent and determine whether the request can be answered directly or whether a tool or specialist is required.

For example:

User: "PMJJBY ke liye main eligible hoon?"

The agent can identify this as an eligibility request, collect information, invoke the eligibility workflow, store the result, and explain the outcome.

The overall loop is:

Understand → Decide → Act → Respond

6. Giving the Agent Tools

An LLM can generate useful responses, but it cannot reliably perform application-specific operations without access to tools.

For FinSaathi, tools were introduced for:

• Government-scheme information.
• Eligibility checks.
• Document requirements.
• User information.
• Human-support escalation.
• Call-outcome tracking.

A simplified tool pattern looks like this:

@function_tool
async def check_eligibility(
    scheme: str,
    user_information: dict
):
    result = perform_eligibility_check(
        scheme,
        user_information
    )

    return result

The model decides when the tool is needed, while the actual business logic remains inside the tool.

7. Memory and Persistent Data

A voice assistant becomes more useful when it can remember relevant information across conversations.

FinSaathi uses SQLite to persist information required by the application:

• User information.
• Eligibility results.
• Escalation requests.
• Call outcomes.
• Analytics data.

The architecture is:

Agent → database.py → SQLite

while dashboard requests follow:

Frontend → FastAPI → database.py → SQLite

This kept database operations in one place instead of scattering SQL logic across components.

Example Database Access Pattern

def get_escalations():
    connection = sqlite3.connect(DB_PATH)
    connection.row_factory = sqlite3.Row

    rows = connection.execute(
        "SELECT * FROM escalations ORDER BY created_at DESC"
    ).fetchall()

    connection.close()

    return [dict(row) for row in rows]

8. Human Escalation: Knowing When AI Should Step Aside

Not every financial situation should be handled entirely by an AI agent.

For a potentially unauthorized transaction, FinSaathi can provide safe guidance and ask whether the user wants human support.

The workflow is:

User reports issue → Safe guidance → User gives consent → Escalation created → Reference ID generated → Human Support Dashboard

A unique reference ID is generated for every escalation.

For example:

FS-A5323F

A stored escalation can contain:

• Request ID.
• User ID.
• Reason.
• Summary.
• What was checked.
• Urgency.
• Language.
• Preferred follow-up method.
• Status.
• Created timestamp.

Example Escalation Tool

@function_tool
async def create_escalation(
    reason: str,
    summary: str,
    urgency: str = "normal",
    preferred_followup: str = "phone"
):
    request_id = create_escalation_record(
        reason=reason,
        summary=summary,
        urgency=urgency,
        preferred_followup=preferred_followup
    )

    return request_id

9. Call Analytics

For FinSaathi, a successful call means that the intended financial workflow was completed or that an appropriate resolution or escalation was reached.

A failed call does not necessarily mean that the software crashed. A user may leave before completing an eligibility workflow.

The dashboard records:

• Total calls.
• Successful calls.
• Failed calls.

The architecture is:

Actual Call → Outcome → Database → FastAPI → Analytics Dashboard

The values come from actual browser or SIP interactions rather than hardcoded demo numbers.

Analytics Recording

record_call_outcome(
    call_id=call_id,
    outcome="SUCCESS"
)

10. Outbound Calling

FinSaathi was extended beyond incoming browser conversations.

The outbound workflow allows the system to initiate supported phone conversations.

Application Data → Outbound Logic → SIP / LiveKit → Voice Agent → User

One of the challenging parts was debugging the call lifecycle. There were situations where the AI greeting started but the call terminated before the conversation could continue.

The investigation required checking:

• Worker lifecycle.
• SIP configuration.
• Network connections.
• Agent state.
• Call state.
• Real-time transport.

This taught me that voice debugging requires looking at the complete lifecycle rather than only the AI response.

11. Specialist Agent Handoff

As FinSaathi grew, putting every workflow into one large agent became less attractive.

The solution was to introduce specialist agents.

For government-scheme conversations:

Main Agent → Government Scheme Specialist → Eligibility / Scheme Workflow → Result

The specialist can focus on:

• Government schemes.
• Eligibility.
• Benefits.
• Required documents.
• Application information.

A handoff should not feel like starting a completely new conversation. The specialist needs enough context from the previous conversation.

Context-Preserving Handoff

@function_tool
async def transfer_to_scheme_specialist():
    copied_context = self.chat_ctx.copy(
        exclude_instructions=True
    )

    specialist = GovernmentSchemeSpecialist(
        chat_ctx=copied_context
    )

    return (
        specialist,
        "Connecting you with our government scheme specialist."
    )

12. Visual Evidence From the Build

A. Main FinSaathi Interface

B. Active Voice Conversation

C. Human Support Dashboard

D. Call Analytics Dashboard

13. Verified Implementation Highlights

Example 1: Tool-Based Eligibility

@function_tool
async def check_eligibility(
    scheme: str,
    user_information: dict
):
    result = perform_eligibility_check(
        scheme,
        user_information
    )

    return result

This keeps the eligibility logic deterministic and testable while allowing the agent to decide when the workflow is needed.

Example 2: Human Escalation

@function_tool
async def create_escalation(
    reason: str,
    summary: str,
    urgency: str,
    preferred_followup: str
):
    request_id = create_escalation_record(
        reason=reason,
        summary=summary,
        urgency=urgency,
        preferred_followup=preferred_followup
    )

    return request_id

Example 3: Analytics

record_call_outcome(
    call_id=call_id,
    outcome="SUCCESS"
)

Example 4: Specialist Handoff

@function_tool
async def transfer_to_scheme_specialist():
    copied_context = self.chat_ctx.copy(
        exclude_instructions=True
    )

    return GovernmentSchemeSpecialist(
        chat_ctx=copied_context
    )

14. Hard Technical Challenges and Lessons Learned

Challenge 1: Outbound Call Lifecycle

An outbound call could start with an AI greeting and then terminate unexpectedly.

The important realization was that the generated response was not necessarily the problem.

I had to investigate the worker lifecycle, SIP configuration, network connectivity, room state, agent state, and call termination events.

Lesson: Voice debugging requires observing the entire real-time lifecycle.

Challenge 2: FastAPI Dependency and ASGI Setup

While building the dashboard API, I encountered a missing FastAPI dependency and later an ASGI loading error where Uvicorn could not find the expected app object.

The debugging process was:

Install dependency → verify import → verify module → verify app object → start Uvicorn → test endpoint

Lesson: Verify each backend layer independently before connecting it to the frontend.

Challenge 3: Database and API Separation

The database module does not need to run as a separate process.

The agent can directly use:

agent.py → database.py → SQLite

The dashboard uses:

Frontend → FastAPI → database.py → SQLite

Understanding this distinction made the architecture much clearer.

Challenge 4: Specialist Handoff

A specialist handoff is not simply a function call. The new agent needs enough context to understand the user's previous conversation while keeping its own instructions separate.

Lesson: Context preservation and instruction isolation are both important in multi-agent systems.

15. Architectural Design Decisions

Decision 1: Shared Database Layer

Choice: Keep database operations inside database.py.

Benefit: Agent tools and API routes can reuse the same data-access functions.

Decision 2: FastAPI for Dashboard APIs

Choice: Use FastAPI as the API layer.

Benefit: The frontend receives structured data while the database remains behind the backend boundary.

Decision 3: Tool-Based Actions

Choice: Use function tools for eligibility, escalation, analytics, and other supported workflows.

Benefit: The agent decides when an action is required while the application remains responsible for performing it.

Decision 4: Specialist Agents

Choice: Route domain-specific conversations to specialist agents.

Benefit: Smaller responsibilities, clearer instructions, and easier future expansion.

16. Practical Step-by-Step Build Guide

Prerequisites

You will need:

• Python with uv.
• Node.js for the frontend.
• A LiveKit project.
• A Murf API key.
• Credentials for the configured speech-to-text and LLM providers.
• SQLite.

Step 1: Clone the Repository

git clone https://github.com/NipunGoel02/murf-ai-project
cd murf-ai-project

Step 2: Configure Environment Variables

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
MURF_API_KEY=your_murf_api_key

Never commit real credentials.

Step 3: Install Backend Dependencies

cd backend
uv sync

Step 4: Start the Agent

Use the LiveKit Agents startup command defined in the repository.

Step 5: Start the Frontend

cd frontend
npm install
pnpm dev

Use the package manager and scripts defined by the repository if they differ.

Step 6: Start the Human-Support API

uv run uvicorn src.escalation_api:app --reload --port 8000

Step 7: Test a Conversation

Open the frontend and test:

1. A normal financial question.
2. A government-scheme question.
3. An eligibility workflow.
4. A human escalation.
5. A specialist handoff.
6. A completed call.
7. An incomplete call.

17. Security: Keep Secrets and Caller Data Private

A public repository should never contain:

• API keys.
• API secrets.
• SIP credentials.
• Private phone numbers.
• Caller information.
• OTPs.
• PINs.
• Passwords.
• Account numbers.
• Private database files.
• Full private conversation transcripts.

Use environment variables for secrets.

Before publishing the repository, inspect the files and Git history to ensure that secrets were not accidentally committed.

The public analytics dashboard should also avoid exposing sensitive caller information.

18. Practical Troubleshooting

Issue 1: The Agent Speaks and Then the Call Ends

Check:

• LiveKit worker logs.
• SIP state.
• Room state.
• Agent process state.
• Network connectivity.
• Call termination events.

Issue 2: FastAPI Says app Is Missing

Verify that the module exposes:

from fastapi import FastAPI

app = FastAPI()

Then start it using the correct module path:

uv run uvicorn src.escalation_api:app --reload --port 8000

Issue 3: Dashboard Data Does Not Change

Debug the complete path:

Agent → Database → API → Browser

Verify the database record first, then the API response, and finally the frontend request.

Issue 4: Specialist Does Not Have Enough Context

Make sure the handoff transfers relevant conversation context while keeping specialist instructions separate.

19. What I Would Improve Next

1. More Financial Specialists

Future specialists could cover:

• Banking support.
• Insurance.
• Loans.
• Credit cards.
• Government benefits.

2. Better Observability

The analytics system could track:

• Average call duration.
• Tool usage.
• Handoff rate.
• Escalation rate.
• Failure reasons.
• Per-workflow success rate.

3. More Robust Telephony

The SIP layer could be extended with production carrier integrations and more robust call-state handling.

4. More Indian Languages

The system could be extended to support more Indian languages and regional speech patterns.

5. Better Human Support

Human agents could receive privacy-safe context so users do not have to repeat their entire issue.

20. What I Learned

At first, a voice agent looked like:

Speech → AI → Speech

After these 10 days, I realized that a useful voice agent needs much more:

Voice + LLM + Memory + Tools + Safety + Database + Real-Time Transport + Human Escalation + Analytics + Specialist Handoffs

The biggest lesson was:

A useful voice agent is not just an AI that can talk. It is a system that can understand, act, remember, measure its performance, and know when a human or specialist should take over.

FinSaathi started as a simple voice conversation and gradually became a complete financial assistance workflow.

The most valuable part of the challenge was learning how these pieces fit together into one system.

21. Project Links

GitHub Repository: https://github.com/NipunGoel02/murf-ai-project

Built using Murf Falcon, the fastest TTS API.

22. Closing

Building FinSaathi during the 10 Days of Voice Agents — VoiceForBharat Edition was an opportunity to work with voice AI beyond simple prompt engineering.

I worked with real-time communication, speech, LLM orchestration, tools, persistent data, outbound calling, human escalation, analytics, and specialist-agent routing.

The biggest takeaway was simple:

The hardest part of a voice agent is not making it speak. The hardest part is making everything around the conversation reliable.

FinSaathi is still a prototype, but it represents the kind of voice-first system I want to continue building: conversational, useful, safe, and connected to real-world workflows.

References and Further Reading

• LiveKit Agents: https://docs.livekit.io/agents
• Murf AI: https://murf.ai/
• FastAPI: https://fastapi.tiangolo.com/
• Next.js: https://nextjs.org/docs
• Python: https://docs.python.org/3/