Author(s): Mostafa Ibrahim

Originally published on Towards AI.

Give Your Voice Agent Hands: Tool Calling on Twilio ConversationRelay with Python

You’ve built a voice agent on Twilio ConversationRelay. It greets callers, understands what they say, and answers in a natural voice. Then a caller asks where their order is, and the agent has nothing to give them. It can explain how shipping generally works. It can apologize. What it can’t do is open your database and read back a tracking number, because nothing connects what the model says to your actual systems.

The article explains that the limitation isn’t the model, but the missing architecture: in a basic ConversationRelay loop, the model can ask to call a function (tool) but nothing executes it or returns the result. It scopes the project to building a tool-calling loop in Python with FastAPI, then walks through prerequisites (Twilio AI/ML addendum acceptance, voice-capable number, OpenAI API key, Python 3.11+, and ngrok for public access). It defines tool calling/function calling and shows how to keep the voice layer intact while adding logic inside the WebSocket server. Next, it outlines the server setup (TwiML endpoint plus a /ws WebSocket), then defines tool schemas and a mock backend (order lookup and appointment booking), and finally implements the core loop that: sends the conversation to OpenAI, runs the requested tool via a dispatch table, appends tool results to conversation history, and continues until the model returns plain text to speak. The post also discusses practical details like interim “Let me check that for you” audio to avoid dead air, history management requirements for correct tool-response sequencing, and production concerns including interruptions and compliance (PCI/HIPAA provider configuration and BAA with Twilio). It concludes by showing next steps such as adding more tools without changing the loop, streaming responses, and integrating analytics via Twilio Conversational Intelligence.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI