AI Gateway now supports audio and voice with realtime streaming

now supports audio/voice. You can add realtime voice, text to speech, and speech to text with the same calls you already use for text, image, and video, routed through AI Gateway alongside every other modality.AI Gateway

Audio launches with models from and . Each call gets the same provider routing, observability, spend controls, and bring-your-own-key support you already use for your other models.OpenAIxAI

These capabilities are in beta and available in 7.AI SDK

Realtime, speech, and transcription model are supported on .AI SDK 7

Realtime turns your app into something a user can hold a conversation with. When they speak, the model responds right away. Because it replies in the moment instead of waiting for a full turn, users can interrupt and talk over it the way they would with a person. It fits voice assistants, customer support agents, hands-free tools, and anywhere a user would rather talk than type.

What sets it apart from chaining models together is that a single realtime model hears audio and produces audio directly, instead of running a speech-to-text, then language model, then text-to-speech pipeline.

In the browser, the hook manages the WebSocket connection, microphone capture, and audio playback.useRealtime

The connection is authenticated with your AI Gateway credential, so you mint a short-lived token on the server and hand the browser only that token. Your API key never reaches the client. Add a route that mints the token:

Then connect from a client component:

The hook captures the microphone, streams the audio to the model through AI Gateway, and plays back the spoken reply. Outside the browser, you can drive the session over a WebSocket yourself with , , and . See the for that path.getWebSocketConfigserializeClientEventparseServerEventrealtime reference

A realtime session works differently from a normal model call:

Generate spoken audio from text with . Pass a voice and an output format, then write the result to a file:generateSpeech

Transcribe recordings into text with . The audio can be a buffer, a base64 string, or a URL:transcribe

Speech and transcription are complementary, so they compose. You can generate audio with one model and read it back with the other, which is a quick way to check both ends of an audio pipeline.

You can also try audio models without writing any code. Open the , click into a model, and interact with it right in the browser. Talk to a realtime model to hold a voice conversation, or send text or audio to a speech or transcription model and read or play back the result.models page

Audio calls behave like every other model call on AI Gateway. You use one API key across , see requests and usage in , apply the same , and when you need to. Adding speech to an app that already uses AI Gateway for text/images/videos can now all be done in the same place.providersobservabilitybudgets and spend limitsbring your own provider keys

Read more

Getting started

Realtime voice agents

Text to speech

Speech to text (transcription)

Playground

Routing audio through AI Gateway

More information

Inside a realtime session

• lets the server decide when the user has stopped speaking, and lets the user talk over the model to cut a reply short (barge-in), with no client-side silence timers.Turn-taking and interruptions. turnDetection: { type: 'server-vad' }

• The model emits a tool call mid-reply, you run it and return the result as a client event, and it folds the answer into what it says next without ending the turn.Tools mid-conversation.

• Realtime quickstart

• Speech quickstart (text to speech and speech to text)

• Realtime reference

• See all audio models