Skip to main content
Every method below is accessed through self.capability_worker (the SDK) or self.worker (the Agent). This is the complete toolkit for building any Ability.

Twenty essential SDK methods

Bonus methods

delete_file() · get_audio_recording_length() · flush_audio_recording() · send_data_over_websocket() · send_devkit_action() · get_token() · get_region_data() · stream_init() / stream_end() · stream_music_from_url() · pause_music() / stop_music() / resume_music() / flush_music() · create_key() / update_key() / delete_key() / get_single_key() / get_all_keys() · update_personality_agent_prompt() · exec_local_command() · session_tasks.sleep() · session_tasks.get()

Music playback

stream_music_from_url() is for audio that runs for minutes and can be interrupted by voice. It plays a progressive mp3 link, blocks until playback genuinely ends, and reports why it ended in result["outcome"]. Your Ability owns only the branch on outcome. The audio pipeline, the device buffer, music mode, and the recovery afterwards are all inside the call. Play a track:
url and auth are both required. auth is positional rather than defaulted on purpose. Assuming a stream link from an authenticated API is self-signed is how every track ends up unplayable. Pause and resume is the same call again with the same url. No position, byte offset or duration is tracked. The engine reads the byte rate off the mp3’s own frame header, and holds the pause against the url. Only "paused" continues the loop:
To end playback from your own code (a timer, a device event), pause_music() and stop_music() set the same events the platform sets, and the live call returns "paused" or "stopped".
While a stream is live the Ability is in music mode: it receives no transcriptions, and must not call speak() or run_io_loop(). Speak before the call or after it returns. However playback ends, including the error path, the client is audible again and the platform is listening when the call returns.
See Music Playback for the full parameter and outcome tables.

OpenRouter models

Use OpenRouter (openrouter.ai) as a single API endpoint to access any model. Pick by job: fast/cheap for routing, multimodal for audio, high-quality for user-facing responses.
This table is about calling OpenRouter from inside your Ability’s code with your own key, so any model OpenRouter carries is available to you. It is separate from the TTT Platform & Model setting in the dashboard, which offers a fixed list. See Voice and Model Configuration for the models the Agent itself can be set to.
Mix models in a single Ability. Use fast/cheap (Gemini Flash, Haiku, GPT-4o-mini) for intent routing and keyword extraction. Use quality models (Claude Sonnet, GPT-4o) for user-facing spoken responses. Use multimodal (Gemini Flash/Pro) for audio analysis.

Battle-tested prompt patterns

Each prompt below is designed for voice output: short, spoken, no markdown.

1. Intent router (JSON classification)

Use with text_to_text_response(). Always strip markdown fences before parsing JSON.

2. Persona system prompt (voice character)

Use as system_prompt parameter. Keep persona prompts specific about length, format, and forbidden phrases.

3. Audio analysis: Pass 1 (general)

Use with an OpenRouter audio-capable model (Gemini Flash/Pro). Send alongside base64 WAV.

4. Audio analysis: Pass 2 (specific with context)

Inject Pass 1 results as context. The two-pass pattern hides latency while providing deep answers.

5. Conversational response (with history)

Inject accumulated analysis + full chat history. Context compounds with every turn.

6. LLM-driven time parser (alarm pattern)

Loop up to 6 rounds. If response starts with QUESTION:, ask the user and continue.

7. Grocery list extractor

Turns stream-of-consciousness rambling into structured, organized output.

8. Restart vs continue intent detection

Two-tier approach: check fast keywords first, fall back to LLM only for ambiguous input.

9. Contextual voice assistant

Inject user context (name, location, time) for natural, personalized responses.

10. Farewell / exit summary

Generate a contextual goodbye instead of a generic sign-off. Makes exits feel natural.

Architecture patterns

Ability categories

See Ability Types for the full breakdown.

File structure

main.py vs background.py


Core patterns

The loop template (multi-turn conversation)

Greet → loop (listen → process → respond) → exit on command. Most common pattern for interactive Abilities.

The two-pass analysis pattern

Pass 1 fires in background immediately (general analysis). While it runs, the Ability talks to the user. Pass 2 fires with Pass 1 context injected, answering the user’s specific question from depth.
  • Pass 1: fire-and-forget via session_tasks.create(asyncio.to_thread(run_general))
  • Talk to user while Pass 1 runs (hides 10–15s of latency)
  • Pass 2: inject Pass 1 results as context, answer the specific question
  • Each follow-up turn fires a background re-analysis, enriching future turns

The rolling window pattern (ambient audio)

For always-on audio monitoring. Continuously record, slice the last N seconds, send to model on a fixed cadence. Fire-and-forget, so never await inside the loop.
  • 10-second window, 3-second refresh cadence
  • API call fires as background task, poll loop never waits
  • Responses arrive asynchronously and log themselves

The coordination pattern (main.py + background.py)

Main writes data to persistent file storage. Background polls that file on a timer and acts on it. This is how alarms, reminders, and scheduled tasks work.
  • main.py: parse user input, write to JSON file, resume_normal_flow()
  • background.py: poll file every 15–30 seconds, check conditions, act
  • Use delete + write for JSON files, because append corrupts JSON
  • Call send_interrupt_signal() before speaking from a daemon

The pending state pattern (multi-step collection)

Track what info you’re waiting for with a dictionary. Each loop iteration checks pending state first and routes input to the correct handler.

Sandbox rules

Breaking these rules will fail the Ability scanner.
  • Never write register_capability() by hand, always use the platform tag
  • No import os, no import json at the top level outside the register block
  • No raw open(), use play_from_audio_file() for audio and the file storage API for data
  • No signal module, not even in docstrings or comments, because the scanner catches it
  • Always call resume_normal_flow() on every exit path in main.py
  • Use session_tasks.sleep() and session_tasks.create(), not raw asyncio
  • Wrap all blocking HTTP calls in asyncio.to_thread()
  • No print(), use editor_logging_handler
  • Blocked imports: redis, connection_manager, user_config, exec(), eval(), pickle

Voice UX best practices

  • Keep speak() to 1–2 sentences. This is voice, not text
  • Fill the silence: say “One sec” before any API call over 1 second
  • Read your speak() strings out loud before shipping
  • Handle messy voice input: use the LLM to extract clean data from noisy transcription
  • Offer exit at every loop iteration: check for “done”, “stop”, “quit”, etc.
  • Use run_confirmation_loop() before destructive actions (send, delete, cancel)
  • Idle detection: 1 empty response = keep going, 2 in a row = offer to leave
  • Namespace your filenames: smarthub_prefs.json not data.json
  • JSON persistence: always delete + write (append corrupts JSON)
  • API calls: always set timeout=10, wrap in try/except, speak errors to user