voice architecture

2min

NEON's bidi voice loop is a realtime architecture, with one twist: audio out goes to the G1 chest speaker via DDS (AudioClient.PlayStream), not a PortAudio device.

signal chain

USB mic (16k mono; DJI receiver or Brio, auto-picked by VOICE_MIC_NAME)  → raw "near"
  → WebRTC AudioProcessor              AEC + NS + AGC · stream_delay_ms=120 ◄ KEY KNOB
  → ratecv 16k→24k                     (Realtime needs ≥24k in)
  → BidiAgent.run()                    ◄ briefings inject BidiTextInputEvent here
       ├ OpenAI / Nova / Gemini Live (websocket)
       └ tools (G1 + telegram + memory + …)
  → 24k assistant audio + transcripts
  → ratecv 24k→16k                     (PlayStream is 16k native)
  → G1SpeakerWriter._loop:
       PlayStream() rc=0 → slice 160-sample frames → feed ref_buf HERE
  → chest speaker emits ~120ms later (DDS roundtrip)

Critical timing

ref_buf is fed post-PlayStream, in the writer thread, not at enqueue time. The speaker is async-queued, so feeding the AEC reference on enqueue would misalign it by a variable amount (queue depth). Feeding it right after PlayStream() returns makes stream_delay_ms=120 a constant delay.

why these values

param	value	reason
stream_delay_ms	120	DDS PlayStream adds ~100-150 ms (a PortAudio device would be ~10)
vad_threshold	0.7	higher = server-VAD less likely to fire on AEC residual
silence_duration_ms	700	longer turn-end → fewer self-interrupts
ref_buf maxsize	500	caps ~5 s; drops oldest on overflow

echo troubleshooting

1. Tune stream_delay_ms first. Time g1_speak(action="say", text="ping"). Emits ~120ms later → keep 120. Almost instant → drop to 80/60 in 20ms steps.
2. Mic picked? g1_speak(action="debug") shows device_index=N rate=16000; None means the wrong default mic (VOICE_MIC_NAME).
3. ref_buf fed? g1_speak(action="status"): g1_frames_sent climbing with ref_buf_qsize at 0 means the AEC has nothing to subtract.
4. Bypass VOICE_NO_AEC=1 make voice: echo gone, tune the AEC; echo stays, the server VAD fires on speaker bleed, raise vad_threshold to 0.9 and silence_duration_ms past 1000.
5. Self-interrupting: vad_threshold 0.7 to 0.85.

cross-persona briefings

The voice agent also receives async briefings from sibling personas:

telegram_listener → voice_bridge.push("telegram", "...")
                  → _BriefingInput poll(2s) → BidiTextInputEvent → model
                  → speaks "From telegram: …" naturally

Push manually: make voice-push MSG="hi neon" or voice_say(text=..., importance=1).

mute and snooze

One flag, four switches. The kv pair voice.muted / voice.muted_until in .memory/mem.db is read and written only through tools/voice_state.py; the listener's MuteState polls it once a second and gates both the mic and the speaker, so muted means silent, briefings included. A snooze is a deadline: once passed, the flag reads as unmuted and clears itself.

switch
dashboard voice pill	snooze 15 min, 1 h, 3 h or until unmuted
voice_control tool	the agent on itself: mute, snooze minutes=, unmute, status, volume level=
make mute / unmute / voice-status	from a shell on the robot
Telegram /mute /unmute /voice	the same flag (the bot's reply still describes the old mic-only mute)

Muting is not stopping: the session stays connected and tools keep working.

two output sinks

_G1SpeakerOutput sends frames to DDS PlayStream and the AEC reference buffer; _LogOutput writes final transcripts to agent_log as the voice persona, so every other persona reads what was said.

providers

provider	env	default voice
openai	OPENAI_API_KEY	alloy (VOICE_MODEL=gpt-realtime)
nova_sonic	AWS_BEARER_TOKEN_BEDROCK / AWS creds	tiffany (us-east-1)
gemini	GOOGLE_API_KEY / GEMINI_API_KEY	Kore

Switch via VOICE_PROVIDER=... in .env, or from the dashboard Voice drawer (provider, voice and realtime model), which writes the same variables and restarts neon-voice through neon-ctl. OpenAI recommended for G1: lowest-latency websocket, best at AEC residuals, good chest-speaker quality (alloy/cedar).

moving by voice

The voice persona carries the four walking tools under the same movement policy: a spoken "walk forward" is the consent, it looks first, walks only in FSM 501 and says "done" only on moved=true.

Edit page