elevenlabs-agents

$2a

Step 1: Create Agent via Dashboard or CLI

Dashboard: https://elevenlabs.io/app/conversational-ai -> Create Agent

CLI (Agents as Code):

elevenlabs agents init

elevenlabs agents add "Support Bot" --template customer-service

# Edit agent_configs/support-bot.json

elevenlabs agents push --env dev

Templates: default, minimal, voice-only, text-only, customer-service, assistant.

Configure:

• Voice -- Choose from 5000+ voices or clone

• LLM -- GPT, Claude, Gemini, or custom

• System prompt -- Use the 6-component framework below

• First message -- What the agent says when conversation starts

Step 2: Write the System Prompt

Use the 6-component framework for effective agent prompts:

1. Personality -- who the agent is:

You are [NAME], a [ROLE] at [COMPANY].

You have [EXPERIENCE]. Your traits: [LIST TRAITS].

2. Environment -- communication context:

You're communicating via [phone/chat/video].

Consider [environmental factors]. Adapt to [context].

3. Tone -- speech patterns and formality:

Tone: Professional yet warm. Use contractions for natural speech.

Avoid jargon. Keep responses to 2-3 sentences. Ask one question at a time.

4. Goal -- objectives and success criteria:

Primary Goal: Resolve customer issues on the first call.

Success: Customer verbally confirms issue is resolved.

5. Guardrails -- boundaries and ethics:

Never: provide medical/legal/financial advice, share confidential info.

Always: verify identity before account access, document interactions.

Escalation: customer requests manager, issue beyond knowledge base.

6. Tools -- available functions and when to use them:

1. lookup_order(order_id) -- Use when customer mentions an order.

2. transfer_to_supervisor() -- Use when issue requires manager approval.

Always explain what you're doing before calling a tool.

Step 3: Add Tools

Client-side tools (run in browser):

const clientTools = {

  updateCart: {

    description: "Add or remove items from the shopping cart",

    parameters: z.object({

      action: z.enum(['add', 'remove']),

      item: z.string(),

      quantity: z.number().min(1)

    }),

    handler: async ({ action, item, quantity }) => {

      const cart = getCart();

      action === 'add' ? cart.add(item, quantity) : cart.remove(item, quantity);

      return { success: true, total: cart.total, items: cart.items.length };

    }

  },

  navigate: {

    description: "Navigate user to a different page",

    parameters: z.object({ url: z.string().url() }),

    handler: async ({ url }) => { window.location.href = url; return { success: true }; }

  }

};

Server-side tools (webhooks):

{

  "name": "get_weather",

  "description": "Fetch current weather for a city",

  "url": "https://api.weather.com/v1/current",

  "method": "GET",

  "parameters": {

    "type": "object",

    "properties": {

      "city": { "type": "string", "description": "City name" }

    },

    "required": ["city"]

  },

  "headers": {

    "Authorization": "Bearer {{secret__weather_api_key}}"

  }

}

Use {{secret__key_name}} for API keys in webhook headers -- never hardcode.

MCP Tools -- CRITICAL COMPATIBILITY NOTE:

ElevenLabs labels their MCP integration as "Streamable HTTP" but does NOT support the actual MCP 2025-03-26 Streamable HTTP spec (SSE responses). ElevenLabs expects:

• Plain JSON responses (application/json), NOT SSE (text/event-stream)

• Protocol version 2024-11-05, NOT 2025-03-26

• Simple JSON-RPC over HTTP with direct JSON responses

What does NOT work:

• Official MCP SDK's createMcpHandler (returns SSE)

• Cloudflare Agents SDK McpServer.serve() (returns SSE)

• Any server returning Content-Type: text/event-stream

Working MCP server pattern for ElevenLabs:

import { Hono } from 'hono';

import { cors } from 'hono/cors';

const tools = [{

  name: "my_tool",

  description: "Tool description",

  inputSchema: {

    type: "object",

    properties: { param1: { type: "string", description: "Description" } },

    required: ["param1"]

  }

}];

async function handleMCPRequest(request, env) {

  const { id, method, params } = request;

  switch (method) {

    case 'initialize':

      return {

        jsonrpc: '2.0', id,

        result: {

          protocolVersion: '2024-11-05',  // MUST be 2024-11-05

          serverInfo: { name: 'my-mcp', version: '1.0.0' },

          capabilities: { tools: {} }

        }

      };

    case 'tools/list':

      return { jsonrpc: '2.0', id, result: { tools } };

    case 'tools/call':

      const result = await handleTool(params.name, params.arguments, env);

      return { jsonrpc: '2.0', id, result };

    default:

      return { jsonrpc: '2.0', id, error: { code: -32601, message: `Unknown: ${method}` } };

  }

}

const app = new Hono();

app.use('/*', cors({ origin: '*', allowMethods: ['GET', 'POST', 'OPTIONS'] }));

app.post('/mcp', async (c) => {

  const body = await c.req.json();

  return c.json(await handleMCPRequest(body, c.env));  // Plain JSON, NOT SSE

});

export default app;

Step 4: Add Knowledge Base (RAG)

Upload documents for the agent to reference:

• PDFs, text files, web URLs

• Configure via dashboard: Agent -> Knowledge Base -> Upload

• Or via API: POST /v1/convai/knowledge-base/upload (multipart/form-data)

• Agent automatically searches knowledge base during conversation

Step 5: Integrate SDK

React -- copy and customise assets/react-sdk-boilerplate.tsx:

import { useConversation } from '@elevenlabs/react';

const { startConversation, stopConversation, status } = useConversation({

  agentId: 'your-agent-id',

  signedUrl: '/api/elevenlabs/auth',

  clientTools,

  dynamicVariables: {

    user_name: 'John',

    account_type: 'premium',

  },

  onEvent: (event) => { /* transcript, agent_response, tool_call */ },

});

System prompt references dynamic variables as {{user_name}}.

React Native -- see assets/react-native-boilerplate.tsx

Widget embed -- see assets/widget-embed-template.html

Swift -- see assets/swift-sdk-boilerplate.swift

Step 6: Test

CLI testing:

# Run all tests for an agent

elevenlabs agents test "Support Agent"

# Add a test scenario

elevenlabs tests add "Refund Request" --template basic-llm

Test configuration:

{

  "name": "Refund Request Test",

  "scenario": "Customer requests refund for defective product",

  "user_input": "I want a refund for order #12345. The product arrived broken.",

  "success_criteria": [

    "Agent acknowledges the issue empathetically",

    "Agent asks for or uses provided order number",

    "Agent verifies order details",

    "Agent provides clear next steps or refund timeline"

  ],

  "evaluation_type": "llm"

}

Tool call testing:

{

  "name": "Order Lookup Test",

  "scenario": "Customer asks about order status",

  "user_input": "What's the status of order ORD-12345?",

  "expected_tool_call": {

    "tool_name": "lookup_order",

    "parameters": { "order_id": "ORD-12345" }

  }

}

API simulation:

const simulation = await client.agents.simulate({

  agent_id: 'agent_123',

  scenario: 'Customer requests refund',

  user_messages: [

    "I want a refund for order #12345",

    "It arrived broken",

    "Yes, process the refund"

  ],

  success_criteria: [

    "Agent shows empathy",

    "Agent verifies order",

    "Agent provides timeline"

  ]

});

console.log('Passed:', simulation.passed);

CI/CD integration:

name: Test Agent

on: [push, pull_request]

jobs:

  test:

    runs-on: ubuntu-latest

    steps:

      - uses: actions/checkout@v3

      - run: npm install -g @elevenlabs/cli

      - run: elevenlabs tests push

        env:

          ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY }}

      - run: elevenlabs agents test "Support Agent"

        env:

          ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY }}

Step 7: Deploy

# Dry run first (always)

elevenlabs agents push --env prod --dry-run

# Deploy to production

elevenlabs agents push --env prod

Multi-environment workflow:

elevenlabs agents push --env dev       # Development

elevenlabs agents push --env staging   # Staging

elevenlabs agents test "Agent Name"    # Test in staging

elevenlabs agents push --env prod      # Production

Critical Patterns

Signed URLs (Security)

Never expose API keys in client code. Use a server endpoint:

app.get('/api/elevenlabs/auth', async (req, res) => {

  const response = await fetch(

    'https://api.elevenlabs.io/v1/convai/conversation/get-signed-url',

    {

      headers: { 'xi-api-key': process.env.ELEVENLABS_API_KEY },

      body: JSON.stringify({ agent_id: 'your-agent-id' }),

      method: 'POST'

    }

  );

  const { signed_url } = await response.json();

  res.json({ signed_url });

});

Agent Versioning (A/B Testing)

Dashboard: Agent -> Versions -> Create Branch. Compare metrics, promote winner.

Post-Call Webhook

{

  "type": "post_call_transcription",

  "data": {

    "conversation_id": "conv_xyz789",

    "transcript": "...",

    "duration_seconds": 120,

    "analysis": { "sentiment": "positive", "resolution": true }

  }

}

Verify with HMAC SHA-256:

const hmac = crypto.createHmac('sha256', process.env.WEBHOOK_SECRET)

  .update(JSON.stringify(request.body)).digest('hex');

if (signature !== hmac) { /* reject */ }

Cost Optimisation

Model

Cost/1M tokens

Speed

Best For

GPT-4o

$5

Medium

Complex reasoning

GPT-4o-mini

$0.15

Fast

Most use cases

Claude Sonnet 4.5

$3

Medium

Long context

Gemini 2.5 Flash

$0.075

Fastest

Simple tasks

Start with gpt-4o-mini for all agents. Upgrade only if quality requires it.

Key savings:

• LLM caching: up to 90% on repeated prompts (enable in config)

• Prompt length: 150 tokens > 500 tokens for same instructions

• RAG over context: use knowledge base instead of stuffing system prompt

• Duration limits: set max_duration_seconds to prevent runaway conversations

• Turn mode: "patient" mode = fewer LLM calls = lower cost

CLI Quick Reference

elevenlabs auth login                              # Authenticate

elevenlabs agents init                             # Init project

elevenlabs agents add "Name" --template default    # Add agent

elevenlabs agents push --env dev                   # Deploy to dev

elevenlabs agents push --env prod --dry-run        # Preview prod deploy

elevenlabs agents push --env prod                  # Deploy to prod

elevenlabs agents pull                             # Pull from platform

elevenlabs agents test "Name"                      # Run tests

elevenlabs agents list                             # List agents

elevenlabs agents status                           # Check sync status

elevenlabs agents widget "Name"                    # Generate widget

elevenlabs tools add-webhook "Name" --config-path tool.json  # Add tool

elevenlabs tests add "Name" --template basic-llm   # Add test

Environment: ELEVENLABS_API_KEY for CI/CD.

Optional References

For specialised use cases, see:

• references/api-reference.md -- full REST API for programmatic agent management

• references/compliance-guide.md -- GDPR, HIPAA, PCI DSS, data residency

• references/workflow-examples.md -- multi-agent routing, escalation, multi-language

Asset Files

• assets/react-sdk-boilerplate.tsx -- React integration template

• assets/react-native-boilerplate.tsx -- React Native template

• assets/swift-sdk-boilerplate.swift -- Swift/iOS template

• assets/javascript-sdk-boilerplate.js -- Vanilla JS template

• assets/widget-embed-template.html -- Embeddable widget

• assets/system-prompt-template.md -- System prompt guide

• assets/agent-config-schema.json -- Config schema reference

• assets/ci-cd-example.yml -- CI/CD pipeline template