AI Voice Collections Agentportfolio POCSynthetic data only

Architecture

One runtime serves the browser demo, phone calls and the evaluation suite. Blue outlines mark components that use a language model; green outlines mark components that hold authority.

Callerbrowser mic / PSTNTransportWebSocket · Twilio MediaVADenergy + noise floorStreaming STTCartesia Ink / mockEnd-of-turnsemantic silence rulesBarge-ingen bump · cancel · clearUnderstandingLLM JSON + rules netPaced playbackgeneration-taggedStreaming TTSCartesia Sonic / mockRealizer + guardtemplates / LLM rephraseController.apply()authoritative stateAudit + DBPostgreSQLPolicy enginedeterministic, auditedaudio outcancelevery decisionturn texttyped proposalcheck

Language models handle language

The LLM (Cloudflare Workers AI, JSON mode) turns a caller utterance into typed ProposedActions such as {"action":"PROPOSE_PAYMENT","amount":30000,"days_from_now":14}. It never sees the date of birth or account terms. Invalid output fails closed to the deterministic parser; stop-contact and human requests are also detected by rules so a misclassification cannot drop them.

Application code owns authority

ConversationController.apply() validates every proposal through the policy engine and is the only writer of identity, disclosure, proposal, promise, stop-contact and transfer state. Replies come from a plan of approved facts; an output guard rejects amounts, dates or threats the plan did not approve.

Voice lifecycle

IDLE → PROCESSING → AGENT_SPEAKING → LISTENING → USER_SPEAKING → PROCESSING → AGENT_SPEAKING → INTERRUPTED → USER_SPEAKING → … → TRANSFER_REQUESTED | ENDED | ERROR

Transitions are validated against an explicit table and recorded with cause and timestamp. Barge-in timestamps: barge_in_detected_at, tts_cancel_requested_at, tts_stopped_at.

Implemented vs simulated vs planned

ComponentStatusNotes
Conversation controllerImplementedOwns CollectionState; applies typed proposals only after policy checks.
Policy engineImplementedIdentity-before-disclosure, envelope (min / max days / ≤ balance), no discounts, stop-contact, transfer, calling hours, attempt limits.
Promise-to-payImplementedRequires verified identity, valid terms, a fully played read-back and an explicit yes; one per session (DB unique).
Barge-inImplementedVAD sustained speech, a non-filler STT partial/final, or typed input → generation bump → TTS cancel → transport clear.
Turn detectionImplementedVAD separated from semantic end-of-turn (short answers, thinking pauses, noise, hard ceiling).
Latency instrumentationImplementedPer-stage marks per turn, p50/p95 by provider mode; mock timings labelled as such.
Browser voice / typedImplementedMic via AudioWorklet (16 kHz PCM16) when STT is configured; typed fallback otherwise.
Cloudflare / Cartesia / TwilioAdapters implementedContract-tested against local fakes; live use requires credentials (not exercised in CI).
Human transferDomain-levelDeterministic state; live PSTN <Dial> only when Twilio + transfer number are configured.
SMS / e-mailInterface + mockNotifier interface; promise confirmations go to a mock outbox.
NLU post-training (SFT)Done on synthetic dataGemma-2B + LoRA on 1,136 rows; frozen held-out exact match 0% → 90.6% (70B zero-shot 88.0%). Servable via Cloudflare BYO-LoRA (4.1 s p50, so 70B stays live); no customer data. See docs/POST_TRAINING_RESULTS.md.