Voice

Two directions, one shape. Speech in (a voice message becomes text) and speech out (text becomes a voice message) are each a declared fallback chain: the first step that answers wins, a step whose credential is missing is skipped with a warning rather than failing the boot, and declaring a chain means declaring all of it.

Neither is required. Omit both keys and you get sensible chains.

Voice in (transcription)

Every voice message your brain receives is transcribed before anything else sees it — the agent gets words, the chat gets a 🎤 quote, the transcript is cached by the file’s identity and the chain that heard it, so re-reading a nine-minute recording costs nothing. Agents can also transcribe on demand with the audio_transcribe tool (a url or a local path). Any role can hear in a chain of its own (below).

Long recordings (past ~8 000 chars of transcript, about ten minutes of speech) arrive differently: the agent’s turn carries the first ~1 500 chars plus the path of a file holding the full transcript, and the 🎤 quote posts the excerpt only. A CLI agent reads the file natively; nothing is lost, the whole text is one read away. Files over Telegram’s 20 MB bot limit — an hour-long meeting recording — are fetched over MTProto automatically when an account is authorized via /tg_auth, so they transcribe like any other voice message.

defineProject({
  transcription: {
    chain: [
      { provider: 'groq' },
      { provider: 'openai', model: 'gpt-4o-transcribe' },
      { provider: 'local', model: 'large-v3' },
    ],
  },
});

Omit transcription and you get groq → openai → deepgram → local — every step the engine knows. That is not greedy: each step drops out when its key or its binary is absent, so the default collapses to exactly what your deployment can do.

The engine knows those limits and skips a step it cannot satisfy instead of paying for the upload — you see it in the failure reason as openai (62 min over its 25 min limit), not as an opaque provider error. It also keeps most recordings under those limits in the first place: see below.

Long recordings just work

Before anything is uploaded, a file bigger than the smallest limit in your chain is converted to a 16 kHz mono opus speech track — video dropped, an hour of recording down from ~65 MB to ~5 MB. Providers that would have answered 413 Request Entity Too Large take it without chunking, and 25 MB then buys about 3.5 hours of audio.

Nothing to configure. Two things worth knowing:

• It needs ffmpeg, which is baked into the engine image. On an engine-dev workstation without it, an oversized recording still goes to the steps whose provider has no size limit (deepgram, local) — it is only refused when every step in your chain would have rejected it anyway, and then the reason says so (omnislash doctor → Audio transcoding). Everything under the limit is unaffected either way.
• A file carrying no audio track at all fails immediately, in ffmpeg’s own words, instead of walking the whole chain to collect four rejections.

Language

defineProject({
  transcription: { language: 'ru' },
});

Optional, and omitting it (auto-detect) is usually right. Declaring it is a commitment, not a hint: whisper honours it literally, so a brain that says ru and receives an English recording gets Russian letters spelling out English words. Set it only where the input really is one language — a two-language speaker is better served by auto-detect plus a prompt (below).

A speaker of two languages — prompt

Whisper picks ONE language per recording and spells everything in it: a Russian speaker practising Dutch who says «сегодня я был bij de huisarts» gets the Dutch back in Cyrillic. language cannot fix that — it commits to one language harder. What does is a prompt: text a whisper step reads as if it were the transcript so far, so it keeps writing in the same style. Write a short sample the way the speaker actually talks, both languages mixed, plus the words it would otherwise misspell:

transcription: {
  chain: [
    {
      provider: 'groq',
      prompt: 'Hoi! Сегодня я был bij de huisarts, и он сказал dat alles goed is. Inburgering, KNM.',
    },
    { provider: 'openai', prompt: 'Hoi! Сегодня я был bij de huisarts, и он сказал dat alles goed is.' },
  ],
},

• It is a hint the model imitates, not an instruction it obeys — “transcribe Dutch in Latin letters” does nothing; a sample written that way does.
• It lives on the step, like a speech step’s voice: each provider reads its own. groq and openai send it as prompt, local as --initial_prompt. deepgram takes no prompt (its API has key terms, a different thing) — the key fails the load there; nova-3 without a language already switches between its ten languages (multi), Russian and Dutch among them.
• Keep it short — whisper reads only the last ~224 tokens of a prompt; a longer one is cut from the front, silently.

Usually the mixed speaker is one role’s (a language teacher’s student), not the whole brain’s: put the prompt in that role’s own block (below).

Where it fits, it buys accuracy and latency: the whisper-family steps stop spending their first window guessing.

The engine translates the value into each provider’s own vocabulary — Deepgram’s nova-3 has no ru, it has multi, and gets that. That translation applies to every way a language can be named: what you declare here, what you declare nothing about (nova-3 defaults to English and returns an empty transcript for anything else, so it is asked for multi unless you said en), and a language an agent passes per call to audio_transcribe. A caller asking for ru has no way to know one provider’s vocabulary is closed — that is the engine’s job, on every path.

Per-role transcription

A role hears in a chain of its own: give it a transcription: block in its file — the same shape as defineProject({ transcription }):

# features/dutch-coach/dutch-coach.agent.md
---
name: dutch-coach
transcription:
  chain:
    - provider: groq
      prompt: "Hoi! Сегодня я был bij de huisarts, и он сказал dat alles goed is."
    - provider: openai
      prompt: "Hoi! Сегодня я был bij de huisarts, и он сказал dat alles goed is."
---

The role that hears is the role that answers — the same role whose voice speaks back: the role serving the chat the voice message arrives in (its bot’s DM, the group its rule names, a Secretary counterpart’s secretary.contacts route), and the calling role for audio_transcribe. A chat no rule serves, and your own messages in a Secretary chat, are heard by the brain’s chain.

• The role’s block replaces the brain’s whole — chain and language. What it leaves out is the engine’s: a role that declares a chain and no language is auto-detected even when the brain says ru. List every step it should fall back through.
• It is hot. An agent changes it with role_update (patch: { transcription: { chain: [...] } }, null to go back to the brain’s) and the next voice message is heard by it.
• A mistyped key fails the role load, as for the brain.
• Each chain keeps its own transcripts. The cache is keyed by the file and the chain (its steps, models, prompts and language), so the same voice message forwarded into another role’s chat is heard again by that role’s chain, and an edited prompt is a fresh transcription, never the old text. The chat-history lines a turn is shown (a group’s or thread’s context) are not a hearing: they show the role’s own transcript of a voice row when it has one, else whichever chain heard it.
• Earlier transcripts are kept. Everything cached before per-role chains existed was heard by the brain’s chain and stays the brain’s chain’s transcript after the upgrade; a role with its own chain hears such a file again the next time it is heard in its chat.
• model_list names every role that hears in a chain of its own (roleTranscription).

The local step

local runs whisper-ctranslate2 (or whisper) as a subprocess. There is no flag to turn it on: the binary on PATH is the declaration, and it gets there by enabling the whisper-ctranslate2 skill and rebuilding your image. Enable the skill and the step lights up; don’t, and the step is skipped with a boot warning.

It is last for a reason — a CPU transcription of a long recording takes minutes, against seconds for an API — and it is worth having for exactly that reason too: it needs no network, no key and no credit balance, so it is what answers when every provider is down, rate-limited or unpaid. One run is capped at 15 minutes.

If you want to run the CLI yourself — subtitles, --task translate, word-level timestamps — that is what the skill’s own instructions are for. Plain speech-to-text should go through audio_transcribe, which already ends in the same binary and caches what it gets.

Voice out (TTS)

Which voice your brain speaks in is one config key — and any role can speak in a voice of its own (below). It applies to both speech tools — voice_send (synthesizes and delivers a voice message — see notifications) and voice_render (renders and delivers nothing: with widget: { widgetId, name } the audio goes straight onto that open widget’s page as widget.media.<name>, the widget’s own file — widgets.md § 3; without it an agent gets a file in its artifacts directory and its path, for file_send / artifact_publish, and only a caller with no agent — a pipeline step — gets base64) — and to voice replies in chat (below). elevenlabs_sound_effect answers the same way.

defineProject({
  speech: {
    chain: [
      { provider: 'openai', voice: 'cedar', instructions: 'Unhurried and low. Warm, but not soft.' },
      { provider: 'elevenlabs', voice: 'nPczCjzI2devNBz1zQrb' },
    ],
  },
});

A voice name names a voice — it is not a secret, so it belongs in config, not env (provider keys stay in OPENAI_API_KEY / ELEVENLABS_API_KEY). Omit speech entirely and you get ElevenLabs → OpenAI with stock voices. Config is baked into your image, so swapping the brain’s voice ships with a rebuild + redeploy, not a brain-sync pull. A role’s own voice (below) does not.

Per-role voice

Two personas rarely share a voice: a warm Russian coach and a teacher who reads listening drills in standard Dutch need different timbres, languages and directions. Give a role its own speech: block in its file — the same shape as defineProject({ speech }):

# features/dutch-coach/dutch-coach.agent.md
---
name: dutch-coach
speech:
  chain:
    - provider: openai
      voice: marin
      instructions: Spreek helder, standaard Nederlands (ABN), rustig tempo, duidelijke articulatie.
---

Whenever that role’s agent is the one speaking, its chain is used instead of the brain’s: voice_send and voice_render called by the role’s agent (a heartbeat or hook running as the role included), and voice replies in a chat the role answers. Pipeline code speaks as a role by naming it on the call — callTool(ctx, 'voice_send', { text, caption: null }, { role: 'dutch-coach' }) (pipelines.md § callTool) — so a text that must be read exactly as written (a listening drill) is spoken in the role’s voice without an agent in between. Everything else — a role with no speech: block, a pipeline call that names no role, a device calling voice_render with no role — speaks in the brain’s voice.

• The role’s chain replaces the brain’s whole, exactly as the brain’s replaces the engine’s. Nothing is merged step by step: a Dutch role does not inherit a Russian fallback. List every step you want, fallback included.
• It is hot. Role files sync with brain-sync, so an agent changes a voice with role_update (patch: { speech: { chain: [...] } }, null to go back to the brain’s) and the next synthesis uses it — no rebuild.
• A mistyped key fails the role load (instruction, voiceID), the same strictness as the brain tier — a silently ignored voice is invisible until someone hears the wrong one.
• model_list names every role that speaks in a voice of its own (roleSpeech).

The voice is the role’s, never the call’s: neither tool takes a voice or an instructions argument. A voice id means something only to its own provider, and a persona that has to restate its voice on every call is a prompt rule the model must remember each turn.

Voice replies

A person who would rather listen than read switches it on in the chat itself: /settings → Voice replies. From then on every answer in that chat arrives twice — the text, as always, and right after it a voice note of the same answer. Nothing to configure and nothing for your agents to do: the engine speaks what the agent wrote, and the agent never learns that it did.

• In the answering role’s voice. A role with its own speech: block speaks its answers in it; every other role in the brain’s.
• Off by default, per chat. Each chat (and each bot in a shared group) has its own switch; an owner flips it. A chat’s topics and threads share it.
• The voice reads the answer, not the markup. Formatting is dropped, a table is read as lines, a link as its label; code blocks and bare URLs are left out — they stay in the text above.
• Text first, voice right after. The text is never held back for the synthesis; while the voice is being made the chat shows “recording a voice message”. The voice note arrives silently — the text already notified.
• Only answers. A failed turn’s ⚠️ line, cards, questions and command replies stay text. So does an answer delivered after a bot restart.
• No chain, no switch. A brain whose speech chain has no usable step refuses the switch and says why. A synthesis that fails mid-run (a provider down, credits gone) drops that one voice note; the text already arrived.
• Long answers are long voice notes, billed like any synthesis. A provider that refuses the length hands off to the next step in your chain.

Business (Secretary) chats have no settings: there the bot speaks as you, in text.

The speech chain

Synthesis falls over across providers exactly like transcription does: the first step that answers wins, and a step that fails hands off to the next one. You declare the order, because who speaks first and what voice they speak in are one decision — which is why voice lives on the step, not on the chain.

Two rules worth knowing before you write one:

• A step whose key is missing is skipped, not fatal. Declare an ElevenLabs step in a brain with no ELEVENLABS_API_KEY and you get a warning at boot and the rest of the chain. The same config works before and after you buy a key.
• Declaring a chain means declaring all of it. The engine appends no hidden last resort. A one-step chain that fails is silence — which is a legitimate choice, as long as it is the one you made.

openai

instructions is the steerable half and the reason this provider is worth the first slot: the preset picks the timbre, instructions picks the reading. Write it as direction to a voice actor — pace, weight, warmth, what to avoid — not as a description of a person:

instructions: 'Unhurried, low, plenty of air between sentences. Calm authority, ' +
  'never brisk. No broadcaster polish and no motivational lift.'

Voices are auditionable at openai.fm with your own text and your own instructions, which beats guessing from the names.

elevenlabs

ElevenLabs splits voices into premade (usable on every plan, including free) and Voice Library (paid plans only — a free account gets 402 paid_plan_required, and adding the voice to “My Voices” does not lift that).

A voice your plan can’t use fails silently. The chain treats a 402 like any other provider failure: it logs a warning and moves to the next step. You hear a voice, delivery succeeds, no tool call errors — it is just never the voice you configured. Same for a typo’d or deleted id. Nothing about a wrong voice id is visible at boot.

So verify the id before you commit it. The elevenlabs_voices tool lists what your key can actually reach, including each voice’s category — premade is the free-safe bucket. Or check it directly:

curl -s -o /dev/null -w '%{http_code}\n' \
  -X POST "https://api.elevenlabs.io/v1/text-to-speech/<VOICE_ID>" \
  -H "xi-api-key: $ELEVENLABS_API_KEY" -H 'Content-Type: application/json' \
  -d '{"text":"test","model_id":"eleven_flash_v2_5"}'
# 200 = usable · 402 = library voice on a free plan · 400/404 = no such voice

Note that ElevenLabs moves voices between those buckets over time: a premade voice can be retired into the Voice Library, at which point it starts 402’ing for free-plan brains that were happily using it.

Running out of credits looks exactly the same. The allowance is a hard stop, not an overdraft: past it every request 401s with quota_exceeded and the chain moves on. If your brain quietly stops sounding like itself mid-month, check the credit balance before checking anything else — and consider whether ElevenLabs belongs first in your chain at all, since a pay-as-you-go step above it has no cliff to fall off.