> ## Documentation Index
> Fetch the complete documentation index at: https://visionagents.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Voice Agents

> Build voice agents with realtime models or custom STT/LLM/TTS pipelines. Phone integration, function calling, and production deployment.

Build voice agents with swappable providers, [phone integration](/guides/calling), [function calling](/guides/mcp-tool-calling), and [production deployment](/guides/deployment) with [built-in metrics](/core/telemetry).

If you completed the [Quickstart](/introduction/quickstart), you already have a working voice agent. This page explains the two ways to build one, and when to move beyond the default setup.

There are two common approaches. The first uses a **realtime model**, a single API that handles listening and speaking end-to-end, such as [OpenAI Realtime](/integrations/realtime/openai), [Gemini Live](/integrations/realtime/gemini), or [Qwen OMNI](/integrations/realtime/qwen). The second is a **[custom pipeline](#custom-pipeline-mode)** where you pick separate STT, LLM, and TTS providers and wire them together.

| Mode                | Best for                     | You choose                         |
| ------------------- | ---------------------------- | ---------------------------------- |
| **Realtime Models** | Fastest path, lowest latency | One provider for speech in and out |
| **Custom Pipeline** | Full control over each stage | STT, LLM, and TTS independently    |

## Realtime Mode

Realtime models handle the full voice loop (speech-to-text, reasoning, and text-to-speech) over a single WebRTC or WebSocket connection. No separate STT or TTS plugins needed.

The [Quickstart](/introduction/quickstart) uses `gemini.Realtime()`. To swap providers, change one line in `agent.py`:

```python theme={null}
llm=openai.Realtime()   # OpenAI
llm=gemini.Realtime()   # Gemini
llm=qwen.Realtime()     # Qwen OMNI
```

See the integration pages for provider-specific setup: [OpenAI Realtime](/integrations/realtime/openai), [Gemini Live](/integrations/realtime/gemini), [Qwen OMNI](/integrations/realtime/qwen).

## Custom Pipeline Mode

Use a custom pipeline when you want to mix providers. For example, Deepgram for transcription, Gemini for reasoning, and Inworld for voice output. You also get control over [turn detection](/integrations/turn-detection/smart-turn) (when the agent starts and stops listening).

<Prompt description="Copy this prompt into Claude Code, Cursor, Windsurf, or any coding agent to scaffold a custom pipeline." actions={["copy", "cursor"]}>
  {`Create a Python project for a Vision Agents custom voice pipeline using uv and Python 3.12.

    Steps:
    1. Scaffold: uvx vision-agents init my-voice-agent && cd my-voice-agent
    2. Add plugins: uv add "vision-agents[deepgram,inworld]"
    3. Copy .env.example to .env and fill in: STREAM_API_KEY, STREAM_API_SECRET (from getstream.io), GOOGLE_API_KEY (from aistudio.google.com), DEEPGRAM_API_KEY (from deepgram.com), INWORLD_API_KEY (from studio.inworld.ai)
    4. Edit agent.py (created by init) to use a custom pipeline:

    from dotenv import load_dotenv
    from vision_agents.core import Agent, AgentLauncher, User, Runner
    from vision_agents.plugins import getstream, gemini, deepgram, inworld

    load_dotenv()

    def setup_llm():
      llm = gemini.LLM()

      @llm.register_function(description="Get current weather for a location")
      async def get_weather(location: str) -> dict:
          return {"temperature": "22C", "condition": "Sunny", "location": location}

      return llm

    async def create_agent(**kwargs) -> Agent:
      return Agent(
          edge=getstream.Edge(),
          agent_user=User(name="Assistant", id="agent"),
          instructions="You're a helpful voice assistant with access to tools. Be concise.",
          llm=setup_llm(),
          stt=deepgram.STT(eager_turn_detection=True),
          tts=inworld.TTS(),
      )

    async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
      call = await agent.create_call(call_type, call_id)
      async with agent.join(call):
          await agent.simple_response("Greet the user and let them know you can check the weather")
          await agent.finish()

    if __name__ == "__main__":
      Runner(AgentLauncher(create_agent=create_agent, join_call=join_call)).cli()

    5. Run with: uv run agent.py run

    Reference docs: https://visionagents.ai
    MCP server: https://visionagents.ai/mcp
    Skill.md: https://visionagents.ai/skill.md`}
</Prompt>

Or follow the steps below manually.

### 1. Add plugins

The Quickstart scaffolds with Gemini Realtime. Add STT and TTS plugins on top:

```bash theme={null}
uv add "vision-agents[deepgram,inworld]"
```

Add these keys to your `.env`:

```bash theme={null}
DEEPGRAM_API_KEY=your_deepgram_api_key
INWORLD_API_KEY=your_inworld_api_key
```

### 2. Wire up the pipeline

Replace `gemini.Realtime()` with separate `stt`, `llm`, and `tts` components in `agent.py`:

```python theme={null}
from dotenv import load_dotenv

from vision_agents.core import Agent, AgentLauncher, User, Runner
from vision_agents.plugins import getstream, gemini, deepgram, inworld

load_dotenv()


async def create_agent(**kwargs) -> Agent:
    return Agent(
        edge=getstream.Edge(),
        agent_user=User(name="Assistant", id="agent"),
        instructions="You're a helpful voice assistant.",
        llm=gemini.LLM(),
        stt=deepgram.STT(eager_turn_detection=True),
        tts=inworld.TTS(),
    )


async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
    call = await agent.create_call(call_type, call_id)
    async with agent.join(call):
        await agent.simple_response("Greet the user")
        await agent.finish()


if __name__ == "__main__":
    Runner(AgentLauncher(create_agent=create_agent, join_call=join_call)).cli()
```

Audio flows: user speaks → **STT** transcribes → **LLM** generates a response → **TTS** speaks it back.

### 3. Mix and match providers

| Component          | Options                                                          |
| ------------------ | ---------------------------------------------------------------- |
| **LLM**            | Gemini, OpenAI, OpenRouter, Anthropic, Grok, HuggingFace         |
| **STT**            | Deepgram, ElevenLabs, Fast-Whisper, Fish, Wizper                 |
| **TTS**            | Inworld, ElevenLabs, Cartesia, Deepgram, Grok, Pocket, AWS Polly |
| **Turn Detection** | Deepgram (built-in), ElevenLabs (built-in), Smart Turn, Vogent   |

## Function Calling & MCP

Custom pipelines use an LLM component (`gemini.LLM()`, not `gemini.Realtime()`), which supports tool registration:

```python theme={null}
@llm.register_function(description="Get weather for a location")
async def get_weather(location: str) -> dict:
    return {"temperature": "22C", "condition": "Sunny"}
```

Functions are automatically converted to the right format for each LLM provider. For MCP servers, external tools, and advanced patterns, see the [Function Calling & MCP guide](/guides/mcp-tool-calling).

## What's Next

<CardGroup cols={2}>
  <Card title="Phone Integration" icon="phone" href="/guides/calling">
    Connect agents to inbound and outbound phone calls
  </Card>

  <Card title="RAG Support" icon="database" href="/guides/rag">
    Add knowledge bases with Gemini FileSearch or TurboPuffer
  </Card>

  <Card title="Docker Deployment" icon="docker" href="/guides/deployment">
    Docker setup and environment configuration
  </Card>

  <Card title="Built-in HTTP Server" icon="globe" href="/guides/http-server">
    Console mode and HTTP server for running agents
  </Card>
</CardGroup>

## Examples

* [Simple Agent](/examples/simple-agent): Minimal voice agent with Deepgram STT + Inworld TTS + Gemini LLM
* [Twilio Phone Agent](/examples/twilio-phone-agent): Step-by-step inbound and outbound calls
* [Phone & RAG](/examples/phone-and-rag): Add knowledge retrieval to phone agents