How to Build an Ability

Introduction

Custom Abilities are the cornerstone of extending OpenHome’s functionality. They allow developers to:

Add personalized features to AI agents.
Integrate third-party APIs for dynamic interactions.
Customize logic for enhanced user engagement.

This guide walks you through:

Structuring and registering an Ability.
Using CapabilityWorker for seamless I/O management.
Examples showcasing how to create powerful custom Abilities.

Adding an Ability

File Structure

Each Ability resides in its folder and requires a main.py file to define the logic.

<YourAbilityFolder>
|── __ init __.py
|── README.md
└── main.py

Example File: `main.py`

Here’s a basic template for building a new Ability:

import json
from src.agent.capability import MatchingCapability
from src.main import AgentWorker
from src.agent.capability_worker import CapabilityWorker

class YourCapability(MatchingCapability):
    worker: AgentWorker = None
    capability_worker: CapabilityWorker = None

    # Do not change following tag of register capability
    #{{register capability}}

    def call(
        self,
        worker: AgentWorker,
    ):
        # Your capability logic here
        return "Your Capability Called!"

Key Components

#{{register_capability}}: is essential.
call: Executes the Ability’s logic when triggered.

Custom API Keys (Third-Party Services)

If your Ability needs credentials for external services (for example OpenAI, SendGrid, or Twilio), configure those keys in the dashboard and read them at runtime using get_api_keys("key_name").

Setup Flow

Keys live on your account, not in the Ability. You create one in the dashboard with its value, and the Ability reads it back by name at runtime.

1. Create the key in Settings

Go to Settings → API Keys, find Third Party Keys, and click + Add. The dialog takes three things:

Key name: the exact name your code will ask for, for example n2yo_api_key.
Key value: the secret itself. It is stored masked and is never shown again in full.
Link to provider: the URL where the key comes from, so anyone using the Ability knows where to get their own.

Each saved key is listed with its provider link and a Linked Abilities field showing which Abilities use it.

The name must match what your code asks for, character for character. A mismatched name is the most common reason get_api_keys() returns nothing.

2. Read the value at runtime

openai_key = self.capability_worker.get_api_keys("openai_api_key")
if not openai_key:
    self.worker.editor_logging_handler.warning("Missing openai_api_key")

Put the name in a constant at the top of main.py so it is easy to find and change, as the API template does:

API_KEY_NAME = "your_api_key_name"   # the name this key is saved under in Settings -> API Keys

Check for several keys at once and fail with a spoken message rather than a silent error:

required_keys = ["openai_api_key", "sendgrid_api_key"]
keys = {name: self.capability_worker.get_api_keys(name) for name in required_keys}
missing = [name for name, value in keys.items() if not value]
if missing:
    await self.capability_worker.speak(
        f"I'm missing required keys: {', '.join(missing)}. "
        "Please add them in Settings, API Keys."
    )
    self.capability_worker.resume_normal_flow()
    return

3. Tell your users which keys to add

Nothing prompts a user for keys automatically, so an Ability that needs one has to say so. List each key in the Ability’s README.md, giving the exact name and where to get the value. The Orbit Ability does this well:

Go to n2yo.com/api, create a free account, copy your API Key, and add it in your OpenHome Settings → API Keys as n2yo_api_key (name must match exactly).

Then degrade gracefully. Treat a missing key as an expected state, not a crash: fall back to cached or demo data where you can, and otherwise say what is missing out loud.

Never hardcode secrets in Ability code. Always read them at runtime with get_api_keys(). Community Abilities are published as source, so anything hardcoded is public.

Reading Linked Account Tokens with `get_token()`

For providers that support OAuth-based account linking, OpenHome stores the user’s access token on the user’s behalf when they connect the account from Settings → Linked Accounts in the Dashboard. Inside an Ability, use get_token() to retrieve that token and call the provider’s API on behalf of the user without handling the OAuth flow yourself. This is different from Custom API Keys above: API keys are values the user pastes in for arbitrary third-party services, while get_token() returns an OAuth access token managed by OpenHome for a supported provider.

Signature

get_token(platform: str) -> str | None

get_token() is synchronous, do not await it.

`get_token()` parameters

Parameter	Type	Description
`platform`	`str`	The provider key. One of `"google"`, `"slack"`, `"discord"`, `"microsoft"`, or `"tesla"`.

Returns

A non-empty str access token if the user has linked the requested provider.
None or an empty value if the provider is not linked for the user. Always guard the return value before using it.

Example: token lookup with a connect-account fallback

The recommended pattern is to read the token at the start of run() and, if it is missing, speak a clear instruction telling the user where to connect the account, then exit cleanly.

async def run(self):
    token = self.capability_worker.get_token("google")

    if not token:
        await self.capability_worker.speak(
            "Your Google account isn't connected yet. "
            "Please connect it in Settings, Linked Accounts, on the OpenHome Dashboard, then try again."
        )
        self.capability_worker.resume_normal_flow()
        return

    # Token is available — call the provider's API as needed.
    # ... use `token` to call the provider API ...

    self.capability_worker.resume_normal_flow()

Substitute the provider key ("slack", "discord", "microsoft", "tesla") and the spoken instruction to match the account your Ability needs.

Reference Abilities Using `get_token()`

The following community Abilities use get_token("google") in production. Review them for end-to-end patterns covering token retrieval, API calls, and error handling:

Morning Brief: reads Gmail and Google Calendar to generate a daily summary.
Gmail Voice Assistant: voice-driven Gmail inbox management.
Google Calendar: view, create, and update calendar events by voice.
Google Tasks: manage Google Tasks lists by voice.

For the user-facing account-linking flow, see Linked Accounts in the Dashboard reference.

Making HTTP Requests

When your Ability calls an API, always route the request through self.worker.session_tasks. This lets the SDK manage the request for you: it paces outbound calls and smooths out latency, so a slow or heavy API doesn’t block your Agent or affect other Abilities. Do not call requests, httpx, or aiohttp directly, use the matching session_tasks helper instead. This applies to code you write by hand and to code generated by an AI assistant. The simplest option is get(). It works just like the requests library: give it a URL and read the response with .json().

resp = self.worker.session_tasks.get("https://api.example.com/data", timeout=5)
data = resp.json()

This is all most Abilities need. get() accepts the usual arguments (params, headers, timeout, auth) and returns a standard requests.Response.

Which helper should I use?

Most Abilities can just use get(). You only need the others if you want an async call or prefer a different library. When you do, the choice depends on how your method is written:

Normal def method → use a sync helper and call it directly, with no await: get() or httpx_get().
async def method (for example, a background-daemon loop) → use an async helper and await it: get_async(), httpx_get_async(), or aiohttp_get_async(). Async helpers don’t freeze the rest of your Ability while the request is running, so prefer them whenever your method is already async.

requests has no async version of its own, so get_async() is there to give you a requests-style request that you can await. Every library has the same helpers, so pick the one you’re comfortable with:

Library	Sync	Async
requests	`get()`	`get_async()`
httpx	`httpx_get()`	`httpx_get_async()`
aiohttp	None	`aiohttp_get_async()`

Examples

# requests (sync)
resp = self.worker.session_tasks.get("https://api.example.com/data", timeout=5)
data = resp.json()

# requests (async, inside an `async def`)
resp = await self.worker.session_tasks.get_async("https://api.example.com/data")
data = resp.json()

# httpx (sync)
resp = self.worker.session_tasks.httpx_get("https://api.example.com/data", params={"q": "test"})
data = resp.json()

# httpx (async, inside an `async def`)
resp = await self.worker.session_tasks.httpx_get_async(
    "https://api.example.com/data",
    headers={"Authorization": f"Bearer {token}"},
)
data = resp.json()

# aiohttp (async, inside an `async def`)
resp = await self.worker.session_tasks.aiohttp_get_async("https://api.example.com/data")
data = await resp.json()   # with aiohttp, reading the body is also awaited

With requests and httpx, you read the response directly with .json(), .text, and .status_code. With aiohttp, reading the body is also asynchronous, so use await resp.json() and await resp.text().

Understanding `CapabilityWorker`

The CapabilityWorker class simplifies I/O interactions, enabling:

Speech synthesis: Using text-to-speech (TTS).
Listening for user input: Capturing and processing responses.
Running interaction loops: Supporting conversational flows.

CapabilityWorker Quick Reference

Use these functions directly on self.capability_worker.

Conversation

Speak text to the user using the configured TTS.

async def speak(self, tokens: str, file_content: str = None):

Wait for a single user reply and return the transcription.

async def user_response(self):

Speak once, then wait for one reply.

async def run_io_loop(self, tokens: str):

Ask a yes/no question and return True or False.

async def run_confirmation_loop(self, tokens: str):

Wait for the user’s full transcription.

async def wait_for_complete_transcription(self):

Get full session message history.

def get_full_message_history(self):

Get the current user’s timezone.

def get_timezone(self):

Get the user’s approximate location from their IP. Returns a dict with city, region, country, postal, timezone, ip, loc, and more.

def get_region_data(self):

Get linked account access token.

def get_token(self, platform: str):

Get a custom API key value by key name.

def get_api_keys(self, key_name: str):

Append context/instructions to the active Agent prompt.

def update_personality_agent_prompt(self, prompt_addition):

Return control back to normal Agent flow when your ability is done.

def resume_normal_flow(self):

Text Generation

Return plain text from the model (no speech).

def text_to_text_response(self, prompt_text: str, history: list = [], system_prompt: str = ""):

Alternate text generation routed through OpenRouter.

def generate_ttt_using_openrouter(self, prompt_text: str, system_prompt: str = "", history: list = []):

Web-search-backed short answer.

def llm_search(self, query: str, system_prompt: str = "", history: list = []) -> str:

Tool-calling flow.

def llm_tools(self, query: str, tools: list, system_prompt: str = "", history: list = []):

File Helpers

Read a file.

async def read_file(self, file_name: str, in_ability_directory=False):

Write content to a file.

async def write_file(self, file_name: str, content: str = None, in_ability_directory=False, mode: str = "a+"):

Delete a file.

async def delete_file(self, file_name: str, in_ability_directory=False) -> bool:

Check if a file exists.

async def check_if_file_exists(self, file_name: str, in_ability_directory=False) -> bool:

List user data files.

async def get_user_data_file_names(self) -> list:

Note: Storage Scope Usage

Use in_ability_directory=False for persistent user-level storage shared across abilities.

Use in_ability_directory=True for ability-scoped data that should remain isolated within the ability session.

Context Storage (Key-Value)

def create_key(self, key: str, value: dict):
def update_key(self, key: str, value: dict):
def delete_key(self, key: str):
def get_all_keys(self):
def get_single_key(self, key: str):

Audio and Streaming

async def text_to_speech(self, prompt, voice_id):                         # speak with a specific voice for this call only
async def play_audio(self, file_content):                                 # play raw audio bytes or a file-like object
async def play_from_audio_file(self, file_name: str = None):              # play an mp3 bundled in the Ability directory
async def send_audio_data_in_stream(self, file_content, chunk_size=4096): # low-level chunked streaming, used internally by play_audio() and speak()
async def stream_init(self):                                              # open the audio stream before sending chunks
async def stream_end(self):                                               # close the stream and wait for the client to finish playback
def get_audio_recording(self):                                            # the latest mic recording as .wav bytes
def get_audio_recording_length(self):                                     # duration in seconds of the latest recording
def flush_audio_recording(self):                                          # clear the current recording before a new capture

Reach for play_from_audio_file() or play_audio() for short audio bundled with your Ability, such as a chime, a sound effect, or a recorded line. Reach for stream_music_from_url() below for the minutes-long, interruptible case.

Music Playback

For audio that runs for minutes and can be interrupted by voice, use stream_music_from_url. It plays a progressive mp3 link, blocks until playback genuinely ends, and reports why it ended. Your Ability owns only the branch on outcome. The audio pipeline, the device buffer, music mode, and the recovery afterwards are all inside the call.

async def stream_music_from_url(self, url, auth, announce=""):

Play a track:

result = await self.capability_worker.stream_music_from_url(
    url,
    "",                                   # Authorization header value
    announce="Playing Blinding Lights.")

result["outcome"]   # "finished" | "paused" | "stopped" | "unplayable" | "error"
result["position"]  # float seconds heard, for logs
result["sent"]      # int bytes delivered, for logs

Parameter	Required	What it’s for
`url`	yes	A progressive mp3 link, not an HLS or DASH manifest. Also the resume signal: the same url carries on, any other url starts a new track.
`auth`	yes	An `Authorization` header value, sent verbatim. Pass `""` where the host wants none. Positional rather than defaulted, on purpose.
`announce`	no	Spoken before playback starts, and only after the link is proven to deliver audio. Leave empty on a resume.

Returns exactly {"outcome", "position", "sent"}. position is measured at the halt instant rather than after cleanup, so it is an honest number for logs. Resuming needs nothing from it.

`outcome`	Means	Usual response
`finished`	played to the end	queue the next track, or ask what’s next
`paused`	user wants it held	ask, then resume or leave
`stopped`	user wants playback over	acknowledge, stop asking
`unplayable`	the link never delivered audio	try the next candidate track
`error`	something failed mid-stream	apologise once, do not retry the same track

While a stream is live the Ability is in music mode: it receives no transcriptions, and must not call speak() or run_io_loop(). Speak before the call or after it returns. However playback ends, including the error path, the client is audible again and the platform is listening when the call returns.

No duration is asked for, and no position is tracked. The engine reads the byte rate off the mp3’s own frame header, so a track with missing or wrong metadata still plays and resumes correctly.

Do not assume a stream url handed back by an authenticated API is self-signed. SoundCloud’s is not: without the header its CDN answers 401, and 429 once a few tracks have been tried in a row. If every track comes back unplayable, check auth first.

A resume is the same call again with the same url, which makes playback a loop. Four rules live in your code, not in the function:

Resolve the url once per track, not per pass. A fresh url reads as a new track and restarts it from the top. This is the one cost of the url being the resume signal: if a signed link expires while the user sits paused, re-resolving restarts the track.
Only "paused" loops. finished, stopped, error, and unplayable all leave.
An unclear pause reply should stop, not resume. A user who asked for silence must not get the track back because a reply did not parse.
announce on the first pass only, or every resume re-announces the title.

To end playback from your own code (a timer, a device event), set the same events the platform sets, from a task other than the one awaiting the stream:

self.capability_worker.pause_music()    # the live call returns "paused"
self.capability_worker.stop_music()     # the live call returns "stopped"

See Example 4: Music Playback for the complete Ability with the loop already written, or start from the music-template directly. Only two functions are yours to fill in:

def search_track(self, request)   # {"id", "title", "artist"}, or None if there isn't one
def stream_url(self, track)       # a fresh progressive mp3 link for that track

Everything else in that file, the pause menu, the outcome branch, the resume_normal_flow() on exit, can be left alone.

WebSocket / Device Actions

async def send_data_over_websocket(self, data_type: str = "", data: dict = {}):
async def send_interrupt_signal(self):
async def send_devkit_action(self, action: str = ""):
async def send_devkit_mqtt_action(self, topic: str = "", action: str = "", value: str = "", command: str = ""):
async def send_notification_to_ios(self, title: str = "", body: str = "", time_interval: int = 1):
async def send_agent_message_without_audio(self, value: str):

Using Specific Voice IDs for Text-to-Speech

The CapabilityWorker class supports the use of specific Voice IDs for text-to-speech (TTS) functionality. This allows you to customize the voice used for speech synthesis by specifying a Voice ID from the provided list.

Available Voice IDs

You can use any of the following Voice IDs for TTS:

{
  "voices": [
    {
      "voice_id": "21m00Tcm4TlvDq8ikWAM",
      "labels": {
        "accent": "american",
        "description": "calm",
        "age": "young",
        "gender": "female",
        "use case": "narration"
      }
    },
    {
      "voice_id": "29vD33N1CtxCmqQRPOHJ",
      "labels": {
        "accent": "american",
        "description": "well-rounded",
        "age": "middle aged",
        "gender": "male",
        "use case": "news"
      }
    },
    {
      "voice_id": "2EiwWnXFnvU5JabPnv8n",
      "labels": {
        "accent": "american",
        "description": "war veteran",
        "age": "middle aged",
        "gender": "male",
        "use case": "video games"
      }
    },
    {
      "voice_id": "5Q0t7uMcjvnagumLfvZi",
      "labels": {
        "accent": "american",
        "description": "ground reporter",
        "age": "middle aged",
        "gender": "male",
        "use case": "news"
      }
    },
    {
      "voice_id": "AZnzlk1XvdvUeBnXmlld",
      "labels": {
        "accent": "american",
        "description": "strong",
        "age": "young",
        "gender": "female",
        "use case": "narration"
      }
    },
    {
      "voice_id": "CYw3kZ02Hs0563khs1Fj",
      "labels": {
        "accent": "british-essex",
        "description": "conversational",
        "age": "young",
        "gender": "male",
        "use case": "video games"
      }
    },
    {
      "voice_id": "D38z5RcWu1voky8WS1ja",
      "labels": {
        "accent": "irish",
        "description": "sailor",
        "age": "old",
        "gender": "male",
        "use case": "video games"
      }
    },
    {
      "voice_id": "EXAVITQu4vr4xnSDxMaL",
      "labels": {
        "accent": "american",
        "description": "soft",
        "age": "young",
        "gender": "female",
        "use case": "news"
      }
    },
    {
      "voice_id": "ErXwobaYiN019PkySvjV",
      "labels": {
        "accent": "american",
        "description": "well-rounded",
        "age": "young",
        "gender": "male",
        "use case": "narration"
      }
    },
    {
      "voice_id": "GBv7mTt0atIp3Br8iCZE",
      "labels": {
        "accent": "american",
        "description": "calm",
        "age": "young",
        "gender": "male",
        "use case": "meditation"
      }
    },
    {
      "voice_id": "IKne3meq5aSn9XLyUdCD",
      "labels": {
        "accent": "australian",
        "description": "casual",
        "age": "middle aged",
        "gender": "male",
        "use case": "conversational"
      }
    },
    {
      "voice_id": "JBFqnCBsd6RMkjVDRZzb",
      "labels": {
        "accent": "british",
        "description": "raspy",
        "age": "middle aged",
        "gender": "male",
        "use case": "narration"
      }
    },
    {
      "voice_id": "LcfcDJNUP1GQjkzn1xUU",
      "labels": {
        "accent": "american",
        "description": "calm",
        "age": "young",
        "gender": "female",
        "use case": "meditation"
      }
    },
    {
      "voice_id": "MF3mGyEYCl7XYWbV9V6O",
      "labels": {
        "accent": "american",
        "description": "emotional",
        "age": "young",
        "gender": "female",
        "use case": "narration"
      }
    },
    {
      "voice_id": "N2lVS1w4EtoT3dr4eOWO",
      "labels": {
        "accent": "american",
        "description": "hoarse",
        "age": "middle aged",
        "gender": "male",
        "use case": "video games"
      }
    },
    {
      "voice_id": "ODq5zmih8GrVes37Dizd",
      "labels": {
        "accent": "american",
        "description": "shouty",
        "age": "middle aged",
        "gender": "male",
        "use case": "video games"
      }
    },
    {
      "voice_id": "SOYHLrjzK2X1ezoPC6cr",
      "labels": {
        "accent": "american",
        "description": "anxious",
        "age": "young",
        "gender": "male",
        "use case": "video games"
      }
    },
    {
      "voice_id": "TX3LPaxmHKxFdv7VOQHJ",
      "labels": {
        "accent": "american",
        "age": "young",
        "gender": "male",
        "use case": "narration",
        "description ": "neutral"
      }
    },
    {
      "voice_id": "ThT5KcBeYPX3keUQqHPh",
      "labels": {
        "accent": "british",
        "description": "pleasant",
        "age": "young",
        "gender": "female",
        "use case": "children's stories"
      }
    },
    {
      "voice_id": "TxGEqnHWrfWFTfGW9XjX",
      "labels": {
        "accent": "american",
        "description": "deep",
        "age": "young",
        "gender": "male",
        "use case": "narration"
      }
    },
    {
      "voice_id": "VR6AewLTigWG4xSOukaG",
      "labels": {
        "accent": "american",
        "description": "crisp",
        "age": "middle aged",
        "gender": "male",
        "use case": "narration"
      }
    },
    {
      "voice_id": "XB0fDUnXU5powFXDhCwa",
      "labels": {
        "accent": "english-swedish",
        "description": "seductive",
        "age": "middle aged",
        "gender": "female",
        "use case": "video games"
      }
    },
    {
      "voice_id": "Xb7hH8MSUJpSbSDYk0k2",
      "labels": {
        "accent": "british",
        "description": "confident",
        "age": "middle aged",
        "gender": "female",
        "featured": "new",
        "use case": "news"
      }
    },
    {
      "voice_id": "XrExE9yKIg1WjnnlVkGX",
      "labels": {
        "accent": "american",
        "description": "warm",
        "age": "young",
        "gender": "female",
        "use case": "audiobook"
      }
    },
    {
      "voice_id": "ZQe5CZNOzWyzPSCn5a3c",
      "labels": {
        "accent": "australian",
        "description": "calm ",
        "age": "old",
        "gender": "male",
        "use case": "news"
      }
    },
    {
      "voice_id": "Zlb1dXrM653N07WRdFW3",
      "labels": {
        "accent": "british",
        "age": "middle aged",
        "gender": "male",
        "use case": "news",
        "description ": "ground reporter "
      }
    },
    {
      "voice_id": "bVMeCyTHy58xNoL34h3p",
      "labels": {
        "accent": "american-irish",
        "description": "excited",
        "age": "young",
        "gender": "male",
        "use case": "narration"
      }
    },
    {
      "voice_id": "flq6f7yk4E4fJM5XTYuZ",
      "labels": {
        "accent": "american",
        "age": "old",
        "gender": "male",
        "use case": "audiobook",
        "description ": "orotund"
      }
    },
    {
      "voice_id": "g5CIjZEefAph4nQFvHAz",
      "labels": {
        "accent": "american",
        "age": "young",
        "gender": "male",
        "use case": "ASMR",
        "description ": "whisper"
      }
    },
    {
      "voice_id": "iP95p4xoKVk53GoZ742B",
      "labels": {
        "accent": "american",
        "description": "casual",
        "age": "middle aged",
        "gender": "male",
        "featured": "new",
        "use case": "conversational"
      }
    },
    {
      "voice_id": "jBpfuIE2acCO8z3wKNLl",
      "labels": {
        "accent": "american",
        "description": "childlish",
        "age": "young",
        "gender": "female",
        "use case": "animation"
      }
    },
    {
      "voice_id": "jsCqWAovK2LkecY7zXl4",
      "labels": {
        "accent": "american",
        "age": "young",
        "gender": "female",
        "description ": "overhyped",
        "usecase": "video games"
      }
    },
    {
      "voice_id": "nPczCjzI2devNBz1zQrb",
      "labels": {
        "accent": "american",
        "description": "deep",
        "age": "middle aged",
        "gender": "male",
        "featured": "new",
        "use case": "narration"
      }
    },
    {
      "voice_id": "oWAxZDx7w5VEj9dCyTzz",
      "labels": {
        "accent": "american-southern",
        "age": "young",
        "gender": "female",
        "use case": "audiobook ",
        "description ": "gentle"
      }
    },
    {
      "voice_id": "onwK4e9ZLuTAKqWW03F9",
      "labels": {
        "accent": "british",
        "description": "deep",
        "age": "middle aged",
        "gender": "male",
        "use case": "news presenter"
      }
    },
    {
      "voice_id": "pFZP5JQG7iQjIQuC4Bku",
      "labels": {
        "accent": "british",
        "description": "raspy",
        "age": "middle aged",
        "gender": "female",
        "use case": "narration"
      }
    },
    {
      "voice_id": "pMsXgVXv3BLzUgSXRplE",
      "labels": {
        "accent": "american",
        "description": "pleasant",
        "age": "middle aged",
        "gender": "female",
        "use case": "interactive"
      }
    },
    {
      "voice_id": "pNInz6obpgDQGcFmaJgB",
      "labels": {
        "accent": "american",
        "description": "deep",
        "age": "middle aged",
        "gender": "male",
        "use case": "narration"
      }
    },
    {
      "voice_id": "piTKgcLEGmPE4e6mEKli",
      "labels": {
        "accent": "american",
        "description": "whisper",
        "age": "young",
        "gender": "female",
        "use case": "audiobook"
      }
    },
    {
      "voice_id": "pqHfZKP75CvOlQylNhV4",
      "labels": {
        "accent": "american",
        "description": "strong",
        "age": "middle aged",
        "gender": "male",
        "use case": "documentary"
      }
    },
    {
      "voice_id": "t0jbNlBVZ17f02VDIeMI",
      "labels": {
        "accent": "american",
        "description": "raspy ",
        "age": "old",
        "gender": "male",
        "use case": "video games"
      }
    },
    {
      "voice_id": "yoZ06aMxZJJ28mfd3POQ",
      "labels": {
        "accent": "american",
        "description": "raspy",
        "age": "young",
        "gender": "male",
        "use case": "narration"
      }
    },
    {
      "voice_id": "z9fAnlkpzviPz146aGWa",
      "labels": {
        "accent": "american",
        "description": "witch",
        "age": "middle aged",
        "gender": "female",
        "use case": "video games"
      }
    },
    {
      "voice_id": "zcAOhNBS3c14rBihAFp1",
      "labels": {
        "accent": "english-italian",
        "description": "foreigner",
        "age": "young",
        "gender": "male",
        "use case": "audiobook"
      }
    },
    {
      "voice_id": "zrHiDhphv9ZnVXBqCLjz",
      "labels": {
        "accent": "english-swedish",
        "description": "childish",
        "age": "young",
        "gender": "female",
        "use case": "animation"
      }
    }
  ]
}

text_to_speech Function

The text_to_speech function converts the provided text into speech using the specified Voice ID and streams it to the user via WebSocket.

async def text_to_speech(self, text: str, voice_id: str):

`text_to_speech` parameters

text (str): The text to be converted into speech.
voice_id (str): The Voice ID to be used for speech synthesis.

Exiting an Ability

An Ability returns control to the Agent when it finishes its work by calling resume_normal_flow(). Beyond that normal completion, there are two ways to leave a Skill Ability on demand: the built-in exit phrases the SDK recognizes, and exit handling you add in your own code.

Built-in exit phrases

While a Skill Ability is active, the SDK recognizes a fixed set of spoken exit phrases. Speaking any of them ends the current Ability and returns control to the Agent’s conversation flow. This is the same outcome as calling resume_normal_flow(), but initiated by the user rather than by the Ability. This is built in and requires no additional code, so a user always has a way out of an Ability, including one that is mid-conversation or waiting for input. The recognized exit phrases are:

“open home exit”
“openhome exit”
“open home quit”
“openhome quit”
“open home, quit”
“open home, exit”
“exit open home”
“quit open home”
“exit, open home”
“quit, open home”

Note: Built-in exit phrases apply to Skill Abilities.

Handling exit in your own code

You can also end an Ability from your own code. Detect an exit condition in the user’s input, such as your own stop words, and call resume_normal_flow() to return control to the Agent.

async def run(self):
    while True:
        user_input = await self.capability_worker.user_response()

        if user_input and user_input.strip().lower() in ("stop", "cancel", "done"):
            await self.capability_worker.speak("Okay, done.")
            self.capability_worker.resume_normal_flow()
            return

        # ... handle the request ...

Advanced CapabilityWorker Functions

Audio Processing Functions

The CapabilityWorker provides comprehensive audio handling capabilities:

play_audio: Play audio content directly or file objects
play_from_audio_file: Play audio files stored in the capability directory
send_audio_data_in_stream: Stream processed audio data over WebSocket

Text Generation Functions

Multiple options for text generation:

text_to_text_response: Standard text generation with history and system prompts
generate_ttt_using_openrouter: Alternate text generation using OpenRouter
llm_search: Web-search-backed short answer
llm_tools: Tool-calling with the model

Streaming and Communication

Advanced communication features:

stream_init and stream_end: Manage audio streaming sessions
stream_music_from_url: Play a progressive mp3 link, blocking until playback genuinely ends, and report why it ended. See Music Playback for the full reference and Example 4 for a complete Ability.
send_data_over_websocket: Send custom data over WebSocket
send_interrupt_signal: Interrupt ongoing output and hand control back to user input
send_agent_message_without_audio: Send a text reply without TTS
send_devkit_action: Trigger a DevKit action
send_devkit_mqtt_action: Trigger a DevKit MQTT action to control a smart device. See Controlling MQTT Devices for the full reference and an example Ability.

Context and Session Helpers

get_timezone: Read the current user’s timezone for local-time-aware behavior
get_token: Read the access token of a user-linked provider account ("google", "slack", "discord", "microsoft", "tesla"). See Reading Linked Account Tokens with get_token() for the full reference and a run() example with a fallback message when the account isn’t connected.
get_api_keys: Read custom API key values from Settings → API Keys by key name
get_full_message_history: Read full session message history for context-aware responses
update_personality_agent_prompt: Append context/instructions to the Agent personality prompt
create_key / update_key / delete_key: Manage structured key-value context storage
get_single_key / get_all_keys: Read one or all stored context entries

Recording and Local Audio

get_audio_recording: Load the latest user recording bytes
get_audio_recording_length: Duration in seconds for the latest recording
flush_audio_recording: Clear the current recording before a new capture
play_from_audio_file: Play an audio file stored in the Ability directory

Session Task Utilities (replace raw asyncio usage)

To ensure Abilities run within the agent’s managed lifecycle, avoid using raw asyncio helpers directly.

Use self.worker.session_tasks.sleep(seconds: float) instead of asyncio.sleep(...):

async def some_task(self):
    await self.worker.session_tasks.sleep(1.5)
    await self.capability_worker.speak("Thanks for waiting!")

These helpers ensure proper cancellation, cleanup, and session scoping.

Background Daemon Entry Point (`background.py`)

Background daemons run automatically when a session starts. Use a separate background.py file with this entry signature:

def call(self, worker: AgentWorker, background_daemon_mode: bool):
    self.worker = worker
    self.background_daemon_mode = background_daemon_mode
    self.capability_worker = CapabilityWorker(self)
    self.worker.session_tasks.create(self.background_loop())

Daemon rules:

Keep daemon logic inside a continuous while True loop.
Use await self.worker.session_tasks.sleep(...) between cycles.
Do not call resume_normal_flow() inside daemon loops.
Call await self.capability_worker.send_interrupt_signal() before daemon speech/audio.

Example 1: Basic Capability

This Ability creates a daily life advisor that:

Asks the user for a problem: Initiates a conversation to gather user input.
Provides advice: Offers a solution based on user input.
Collects feedback: Captures user satisfaction with the advice.

Example 1 code

import json
from src.agent.capability import MatchingCapability
from src.main import AgentWorker
from src.agent.capability_worker import CapabilityWorker

class BasicCapability(MatchingCapability):
    worker: AgentWorker = None
    capability_worker: CapabilityWorker = None

    # Do not change following tag of register capability
    #{{register capability}}

    async def give_advice(self):
        await self.capability_worker.speak("Hi! I'm your daily life advisor. Tell me your problem.")
        user_problem = await self.capability_worker.user_response()

        solution_prompt = f"Provide a solution for: {user_problem}"
        solution = self.capability_worker.text_to_text_response(solution_prompt)

        user_feedback = await self.capability_worker.run_io_loop(
            solution + " Are you satisfied with the advice?"
        )
        await self.capability_worker.speak("Thank you for using the advisor.")
        self.capability_worker.resume_normal_flow()

    def call(self, worker: AgentWorker):
        self.worker = worker
        self.capability_worker = CapabilityWorker(self)
        self.worker.session_tasks.create(self.give_advice())

Key Functions

speak: Introduces the advisor and provides the solution.
user_response: Captures user input (e.g., their problem).
run_io_loop: Combines speaking the solution and listening for feedback.
resume_normal_flow: Resumes the agent’s default workflow after interaction.

Example 2: Weather Capability

This Ability integrates a weather API to fetch and share weather updates based on user-provided locations.

Example 2 code

import json
from src.agent.capability import MatchingCapability
from src.main import AgentWorker
from src.agent.capability_worker import CapabilityWorker

class WeatherDocsCapability(MatchingCapability):
    worker: AgentWorker = None
    capability_worker: CapabilityWorker = None

    # Do not change following tag of register capability
    #{{register capability}}
    async def first_setup(self, location: str):
        if not location:
            await self.capability_worker.speak("Which location?")
            location = await self.capability_worker.user_response()

        geolocator = Nominatim(user_agent="weather_agent")
        loc = geolocator.geocode(location)

        if loc:
            result = self.worker.session_tasks.get(
                f"https://api.open-meteo.com/v1/forecast?latitude={loc.latitude}&longitude={loc.longitude}&current=temperature_2m"
            ).json()
            weather_report = f"The temperature in {location} is {result['current']['temperature_2m']}°C."
            await self.capability_worker.speak(weather_report)
        else:
            await self.capability_worker.speak("Invalid location. Try again.")

        self.capability_worker.resume_normal_flow()

    def call(self, worker: AgentWorker):
        self.worker = worker
        self.capability_worker = CapabilityWorker(self)
        self.worker.session_tasks.create(self.first_setup(""))

Example 2 key features

External API Call: Fetches real-time weather data.
Geolocation: Validates and processes user-provided locations.
Error Handling: Provides meaningful feedback for invalid inputs.

Allowed/Disallowed Libraries and Patterns

The following imports, keywords, and patterns are not allowed in Abilities. Use the safe alternatives.

Blocked Imports and Keywords

Name	Why not allowed
redis	Direct datastore coupling and security concerns, and not portable across deployments.
user_config	Raw config access can leak or mutate global state, so use provided APIs on `CapabilityWorker`/`worker`.
print	Bypasses structured logging, and is noisy and untraceable in production. Use `editor_logging_handler`.
open (raw)	Unmanaged filesystem access, with security and portability risks. Prefer approved helpers/per-user storage.

Guidance:

Avoid direct storage/infra access. Use platform-provided helpers within CapabilityWorker/worker or request an API if needed.
Use the provided logging (editor_logging_handler) instead of prints.
For files, prefer platform abstractions and per-user capability folders, and ask for an approved helper if you need persistent storage.
For HTTP GET requests, always use the self.worker.session_tasks helpers (get, get_async, httpx_get, httpx_get_async, aiohttp_get_async). Never call requests, httpx, or aiohttp directly. See Making HTTP Requests.

Security Guidance

Avoid insecure or unsafe patterns such as runtime assert checks, exec() of dynamic code, binding servers to all interfaces, hardcoded secrets, swallowing exceptions, insecure deserialization (pickle/dill/shelve/marshal), weak hashes like MD5, or weak cipher modes (e.g., ECB). If you have a special case, request approval and an approved wrapper/utility.

Conclusion

Building Abilities in OpenHome empowers developers to create custom functionalities for AI agents. With the examples like the Basic Advisor and Weather Capability, you can:

Core Communication: Use speak, run_io_loop, and user_response for basic interactions.
Advanced Audio: Play custom audio files, and stream audio data.
Text Generation: Leverage multiple text-to-text options with history and system prompts.
Voice Customization: Use specific voice IDs for varied and engaging responses.
External APIs: Integrate third-party services for dynamic functionality.

The examples demonstrate everything from basic conversational flows to advanced audio processing and device control. The CapabilityWorker provides all the tools needed to create sophisticated, interactive Abilities.

Start creating innovative Abilities and push the boundaries of voice AI with OpenHome! 🎉

Note: For GET requests to third-party APIs, use self.worker.session_tasks.get() (see Making HTTP Requests). It takes the same arguments as the requests module’s get() and returns the same response. For other request types, use the requests module and avoid other libraries. If you need a library that isn’t available, you can request that we add it.

Example 3: Read/Write File (from `example_main.py`)

This is the simplest pattern for per-user storage.

class ReadwriteFileCapability(MatchingCapability):
    async def perform_action(self):
        user_response = await self.capability_worker.wait_for_complete_transcription()
        await self.capability_worker.speak("Writing last transcription to file.")

        if await self.capability_worker.check_if_file_exists(
            "temp_data.txt",
            in_ability_directory=False,
        ):
            await self.capability_worker.write_file(
                "temp_data.txt",
                "\n%s: %s" % (time(), user_response),
                in_ability_directory=False,
            )
        else:
            await self.capability_worker.write_file(
                "temp_data.txt",
                "%s: %s" % (time(), user_response),
                in_ability_directory=False,
            )

        file_data = await self.capability_worker.read_file(
            "temp_data.txt",
            in_ability_directory=False,
        )
        last_written_line = file_data.split("\n")[-1].split(":")[1]
        await self.capability_worker.speak("Last Written Line: %s" % last_written_line)

        self.capability_worker.resume_normal_flow()

Example 4: Music Playback

stream_music_from_url() plays a track and blocks until playback genuinely ends, then reports why in result["outcome"]. Only "paused" continues the loop, and a resume is the same call again with the same url. See the Music Playback reference above for the full parameter and outcome tables.

from src.agent.capability import MatchingCapability
from src.main import AgentWorker
from src.agent.capability_worker import CapabilityWorker

# Replace with your music service.
MUSIC_API = "https://api.example.com"


class MusicDocsCapability(MatchingCapability):
    worker: AgentWorker = None
    capability_worker: CapabilityWorker = None
    api_key: str = None

    # Do not change following tag of register capability
    #{{register capability}}
    async def play_music(self, request: str):
        if not request:
            await self.capability_worker.speak("What would you like to hear?")
            request = await self.capability_worker.user_response()

        track = self.search_track(request)

        if track:
            # Resolved once, not per pass: handing the SAME url back is what
            # resumes a paused track. Re-resolving mints a new url, which reads
            # as a new track and restarts it from the top.
            url = self.stream_url(track)
            announced = False
            while True:
                result = await self.capability_worker.stream_music_from_url(
                    url,
                    # Positional, so it cannot be forgotten. Pass "" for a host
                    # that wants no Authorization header. Do not assume a link
                    # from an authenticated API is self-signed -- some CDNs
                    # answer 401 without the token, and every track then comes
                    # back "unplayable".
                    f"Bearer {self.api_key}",
                    announce=f"Playing {track['title']}." if not announced else "",
                )
                announced = True

                # Only a pause continues the loop. finished, stopped, error and
                # unplayable all leave.
                if result["outcome"] != "paused":
                    break

                reply = await self.capability_worker.run_io_loop(
                    "Paused. Say resume or stop."
                )
                # An unclear reply stops. A user who asked for silence must not
                # get the track back because a reply didn't parse.
                if "resume" not in (reply or "").lower():
                    break

            await self.capability_worker.speak("Okay, that's it for the music.")
        else:
            await self.capability_worker.speak("I couldn't find that one. Try again.")

        self.capability_worker.resume_normal_flow()

    def search_track(self, request: str):
        result = self.worker.session_tasks.get(
            f"{MUSIC_API}/search?q={request}&limit=1"
        ).json()
        tracks = result.get("tracks") or []
        if not tracks:
            return None
        track = tracks[0]
        # No duration: the engine reads the byte rate off the mp3 frame header,
        # so a track with missing or wrong metadata still plays and resumes.
        return {
            "id": track["id"],
            "title": track["title"],
            "artist": track.get("artist", ""),
        }

    def stream_url(self, track):
        # Kept separate from search_track() so a link can be re-resolved without
        # searching again -- though the loop above resolves it only once.
        result = self.worker.session_tasks.get(
            f"{MUSIC_API}/tracks/{track['id']}/stream"
        ).json()
        return result["mp3_url"]                          # progressive mp3

    def call(self, worker: AgentWorker):
        self.worker = worker
        self.capability_worker = CapabilityWorker(self)
        self.api_key = self.capability_worker.get_api_keys("music_api_key")
        self.worker.session_tasks.create(self.play_music(""))

Example 4 key features

One await covers the whole track. The audio pipeline, the device buffer, music mode and the recovery afterwards are all inside the call, so speak() works on the very next line no matter how playback ended.
No duration, no position, no byte offset. The engine reads the byte rate off the mp3’s own frame header, so a track with missing or wrong metadata still plays and resumes correctly.
The URL is resolved once per track, outside the loop: handing the same url back is what resumes a pause, so a fresh url would restart the track instead.
auth is positional, so it cannot be forgotten. Pass "" for a host that wants no Authorization header.
An unclear pause reply stops. A user who asked for silence must not get the track back because a reply didn’t parse.

While a stream is live the Ability is in music mode: it receives no transcriptions, and must not call speak() or run_io_loop(). Speak before the call or after it returns.

​Introduction

​Adding an Ability

​File Structure

​Example File: main.py

​Key Components

​Custom API Keys (Third-Party Services)

​Setup Flow

​1. Create the key in Settings

​2. Read the value at runtime

​3. Tell your users which keys to add

​Reading Linked Account Tokens with get_token()

​Signature

​get_token() parameters

​Returns

​Example: token lookup with a connect-account fallback

​Reference Abilities Using get_token()

​Making HTTP Requests

​Which helper should I use?

​Examples

​Understanding CapabilityWorker

​CapabilityWorker Quick Reference

​Conversation

​Text Generation

​File Helpers

​Context Storage (Key-Value)

​Audio and Streaming

​Music Playback

​WebSocket / Device Actions

​Using Specific Voice IDs for Text-to-Speech

​Available Voice IDs

​text_to_speech Function

​text_to_speech parameters

​Exiting an Ability

​Built-in exit phrases

​Handling exit in your own code

​Advanced CapabilityWorker Functions

​Audio Processing Functions

​Text Generation Functions

​Streaming and Communication

​Context and Session Helpers

​Recording and Local Audio

​Session Task Utilities (replace raw asyncio usage)

​Background Daemon Entry Point (background.py)

​Example 1: Basic Capability

​Example 1 code

​Key Functions

​Example 2: Weather Capability

​Example 2 code

​Example 2 key features

​Allowed/Disallowed Libraries and Patterns

​Blocked Imports and Keywords

​Security Guidance

​Conclusion

​Example 3: Read/Write File (from example_main.py)

​Example 4: Music Playback

​Example 4 key features

Introduction

Adding an Ability

File Structure

Example File: `main.py`

Key Components

Custom API Keys (Third-Party Services)

Setup Flow

1. Create the key in Settings

2. Read the value at runtime

3. Tell your users which keys to add

Reading Linked Account Tokens with `get_token()`

Signature

`get_token()` parameters

Returns

Example: token lookup with a connect-account fallback

Reference Abilities Using `get_token()`

Making HTTP Requests

Which helper should I use?

Examples

Understanding `CapabilityWorker`

CapabilityWorker Quick Reference

Conversation

Text Generation

File Helpers

Context Storage (Key-Value)

Audio and Streaming

Music Playback

WebSocket / Device Actions

Using Specific Voice IDs for Text-to-Speech

Available Voice IDs

text_to_speech Function

`text_to_speech` parameters

Exiting an Ability

Built-in exit phrases

Handling exit in your own code

Advanced CapabilityWorker Functions

Audio Processing Functions

Text Generation Functions

Streaming and Communication

Context and Session Helpers

Recording and Local Audio

Session Task Utilities (replace raw asyncio usage)

Background Daemon Entry Point (`background.py`)

Example 1: Basic Capability

Example 1 code

Key Functions

Example 2: Weather Capability

Example 2 code

Example 2 key features

Allowed/Disallowed Libraries and Patterns

Blocked Imports and Keywords

Security Guidance

Conclusion

Example 3: Read/Write File (from `example_main.py`)

Example 4: Music Playback

Example 4 key features