self.capability_worker (the SDK) or self.worker (the Agent). This is the complete toolkit for building any Ability.
Twenty essential SDK methods
Bonus methods
delete_file() · get_audio_recording_length() · flush_audio_recording() · send_data_over_websocket() · send_devkit_action() · get_token() · get_region_data() · stream_init() / stream_end() · stream_music_from_url() · pause_music() / stop_music() / resume_music() / flush_music() · create_key() / update_key() / delete_key() / get_single_key() / get_all_keys() · update_personality_agent_prompt() · exec_local_command() · session_tasks.sleep() · session_tasks.get()
Music playback
stream_music_from_url() is for audio that runs for minutes and can be interrupted by voice. It plays a progressive mp3 link, blocks until playback genuinely ends, and reports why it ended in result["outcome"]. Your Ability owns only the branch on outcome. The audio pipeline, the device buffer, music mode, and the recovery afterwards are all inside the call.
Play a track:
url and auth are both required. auth is positional rather than defaulted on purpose. Assuming a stream link from an authenticated API is self-signed is how every track ends up unplayable.
Pause and resume is the same call again with the same url. No position, byte offset or duration is tracked. The engine reads the byte rate off the mp3’s own frame header, and holds the pause against the url. Only "paused" continues the loop:
pause_music() and stop_music() set the same events the platform sets, and the live call returns "paused" or "stopped".
See Music Playback for the full parameter and outcome tables.
OpenRouter models
Use OpenRouter (openrouter.ai) as a single API endpoint to access any model. Pick by job: fast/cheap for routing, multimodal for audio, high-quality for user-facing responses.This table is about calling OpenRouter from inside your Ability’s code with your own key, so any model OpenRouter carries is available to you. It is separate from the TTT Platform & Model setting in the dashboard, which offers a fixed list. See Voice and Model Configuration for the models the Agent itself can be set to.
Battle-tested prompt patterns
Each prompt below is designed for voice output: short, spoken, no markdown.1. Intent router (JSON classification)
text_to_text_response(). Always strip markdown fences before parsing JSON.
2. Persona system prompt (voice character)
system_prompt parameter. Keep persona prompts specific about length, format, and forbidden phrases.
3. Audio analysis: Pass 1 (general)
4. Audio analysis: Pass 2 (specific with context)
5. Conversational response (with history)
6. LLM-driven time parser (alarm pattern)
QUESTION:, ask the user and continue.
7. Grocery list extractor
8. Restart vs continue intent detection
9. Contextual voice assistant
10. Farewell / exit summary
Architecture patterns
Ability categories
See Ability Types for the full breakdown.File structure
main.py vs background.py
Core patterns
The loop template (multi-turn conversation)
Greet → loop (listen → process → respond) → exit on command. Most common pattern for interactive Abilities.The two-pass analysis pattern
Pass 1 fires in background immediately (general analysis). While it runs, the Ability talks to the user. Pass 2 fires with Pass 1 context injected, answering the user’s specific question from depth.- Pass 1: fire-and-forget via
session_tasks.create(asyncio.to_thread(run_general)) - Talk to user while Pass 1 runs (hides 10–15s of latency)
- Pass 2: inject Pass 1 results as context, answer the specific question
- Each follow-up turn fires a background re-analysis, enriching future turns
The rolling window pattern (ambient audio)
For always-on audio monitoring. Continuously record, slice the last N seconds, send to model on a fixed cadence. Fire-and-forget, so neverawait inside the loop.
- 10-second window, 3-second refresh cadence
- API call fires as background task, poll loop never waits
- Responses arrive asynchronously and log themselves
The coordination pattern (main.py + background.py)
Main writes data to persistent file storage. Background polls that file on a timer and acts on it. This is how alarms, reminders, and scheduled tasks work.
main.py: parse user input, write to JSON file,resume_normal_flow()background.py: poll file every 15–30 seconds, check conditions, act- Use delete + write for JSON files, because append corrupts JSON
- Call
send_interrupt_signal()before speaking from a daemon
The pending state pattern (multi-step collection)
Track what info you’re waiting for with a dictionary. Each loop iteration checks pending state first and routes input to the correct handler.Sandbox rules
- Never write
register_capability()by hand, always use the platform tag - No
import os, noimport jsonat the top level outside the register block - No raw
open(), useplay_from_audio_file()for audio and the file storage API for data - No
signalmodule, not even in docstrings or comments, because the scanner catches it - Always call
resume_normal_flow()on every exit path inmain.py - Use
session_tasks.sleep()andsession_tasks.create(), not rawasyncio - Wrap all blocking HTTP calls in
asyncio.to_thread() - No
print(), useeditor_logging_handler - Blocked imports:
redis,connection_manager,user_config,exec(),eval(),pickle
Voice UX best practices
- Keep
speak()to 1–2 sentences. This is voice, not text - Fill the silence: say “One sec” before any API call over 1 second
- Read your
speak()strings out loud before shipping - Handle messy voice input: use the LLM to extract clean data from noisy transcription
- Offer exit at every loop iteration: check for “done”, “stop”, “quit”, etc.
- Use
run_confirmation_loop()before destructive actions (send, delete, cancel) - Idle detection: 1 empty response = keep going, 2 in a row = offer to leave
- Namespace your filenames:
smarthub_prefs.jsonnotdata.json - JSON persistence: always delete + write (append corrupts JSON)
- API calls: always set
timeout=10, wrap intry/except, speak errors to user

