Baseten

LLM service implementation using Baseten’s OpenAI-compatible API

Overview

BasetenLLMService provides access to Baseten’s Model APIs and dedicated deployments through an OpenAI-compatible interface. It inherits from OpenAILLMService and supports streaming responses, function calling, and context management.

Baseten’s Model APIs offer serverless access to open-weights models including GLM, Kimi, DeepSeek, Nemotron, and gpt-oss. For dedicated deployments running on your own GPUs, point base_url to your deployment’s endpoint.

Baseten LLM API Reference

Pipecat’s API methods for Baseten integration

Example Implementation

Complete example with function calling

Baseten Documentation

Official Baseten API documentation

Baseten Console

Access models and manage API keys

Installation

To use Baseten services, install the required dependency:

uv add "pipecat-ai[baseten]"

Prerequisites

Baseten Account Setup

Before using Baseten LLM services, you need:

1. Baseten Account: Sign up at Baseten
2. API Key: Generate an API key from your account dashboard
3. Model Selection: Choose from available Model APIs or deploy your own model

Required Environment Variables

• BASETEN_API_KEY: Your Baseten API key for authentication

Configuration

api_key · str · required

Baseten API key for authentication.

base_url · str · default: "https://inference.baseten.co/v1"

Base URL for Baseten API endpoint. Use the default for serverless Model APIs, or set to your dedicated deployment’s /sync/v1 URL to use a model running on your own GPUs.

settings · BasetenLLMService.Settings · default: "None"

Runtime-configurable settings. See Settings below.

Settings

Runtime-configurable settings passed via the settings constructor argument using BasetenLLMService.Settings(...). These can be updated mid-conversation with LLMUpdateSettingsFrame. See Service Settings for details.

This service uses the same settings as OpenAILLMService. See OpenAI LLM Settings for the full parameter reference.

The default model is moonshotai/Kimi-K2.5, which streams visible content immediately for low time-to-first-word.

Usage

Basic Setup

import os
from pipecat.services.baseten import BasetenLLMService

llm = BasetenLLMService(
    api_key=os.getenv("BASETEN_API_KEY"),
    settings=BasetenLLMService.Settings(
        model="moonshotai/Kimi-K2.5",
    ),
)

With Custom Settings

from pipecat.services.baseten import BasetenLLMService

llm = BasetenLLMService(
    api_key=os.getenv("BASETEN_API_KEY"),
    settings=BasetenLLMService.Settings(
        model="moonshotai/Kimi-K2.5",
        temperature=0.7,
        max_completion_tokens=1024,
        system_instruction="You are a helpful assistant in a voice conversation. Your responses will be spoken aloud, so avoid emojis, bullet points, or other formatting that can't be spoken. Respond to what the user said in a creative, helpful, and brief way.",
    ),
)

Using a Dedicated Deployment

To use a model running on your own dedicated GPUs, point base_url to your deployment’s endpoint:

llm = BasetenLLMService(
    api_key=os.getenv("BASETEN_API_KEY"),
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
    settings=BasetenLLMService.Settings(
        model="Qwen/Qwen2.5-3B-Instruct",
    ),
)

Notes

• Model APIs vs. Dedicated Deployments: The default base_url connects to Baseten’s serverless Model APIs. For dedicated deployments on your own GPUs, set base_url to your deployment’s /sync/v1 endpoint URL.
• Token Usage Reporting: Baseten reports cumulative token usage on every streamed chunk. The service automatically accumulates these values and reports the final total once at the end of processing.
• Message Roles: Baseten supports system, user, assistant, and tool message roles. The developer role is not supported.
• OpenAI Compatibility: Baseten fully supports the OpenAI-compatible parameter set inherited from OpenAILLMService, including function calling and streaming responses.

The InputParams / params= pattern is deprecated as of v0.0.105. Use Settings / settings= instead. See the Service Settings guide for migration details.