The honest state of the service and the road ahead — speech shipped, avatars staged, the operator Console next. No vaporware: every row says exactly what's live and what isn't.
Ears first, then a voice, then the live loop. Each phase shipped as its own PR.
Repo, Modal entrypoint, whisper.cpp worker on a T4, bearer-auth /transcribe. The service goes live and scales to zero.
Mic capture in the status bar → /v1/voice/memo → transcript posted to the room, audio persisted, every memo audited. Multilingual, auto-detected.
Agents transcribe over MCP; language detection surfaces on every memo, with the audio preserved so a listener can verify.
Real Piper synthesis on Modal, /synthesize, the 8-voice registry, the voice_synthesize MCP tool, and the amplitude envelope on every render. Text→audio under a second for a sentence.
Render an agent turn to speech and publish it into the LiveKit room as an audio track — reusing the A/V edge both apps already have.
Streaming STT → agent turn → streaming TTS, with voice-activity detection and barge-in. A sub-second round trip, its own project window.
The through-line: audio is generated once, the envelope is the shared clock, and every face is a function of it.
/synthesize returns the {v,rate,peaks} envelope; the avatar rides peaks[time·rate] for true amplitude sync. Running today.
MuseTalk lip-sync (MIT). Benchmarked on Modal — it works; the engineering is a realtime pipeline plus a per-agent latent cache to reach seconds-per-clip. Then a /animate endpoint.
Audio → viseme track (Rhubarb) driving a rigged 3D avatar per agent, rendered in VIAx3D and AR-in-room. The horizon this fleet can reach cheaply — it owns the 3D tooling.
Turn an opaque API into a legible, operable surface — see it, understand it, connect to it, maintain it. Served from the service at /console, behind fleet SSO.
No warm pool, no idle bill. The service wakes on a request (~a few seconds cold) and sleeps after a burst.
Piper runs on CPU burst — effectively zero marginal cost per synth. Self-hosted, no per-character API fee.
Whisper on a T4 is billed per GPU-second, only while transcribing. Modest, metered per calling app.