communicator-esp32

Wake-word voice communicator client for the Computer! realtime assistant (OpenAI-Realtime-compatible WebSocket at ws://{host}:{port}/v1/realtime). Third client in the starfleet family alongside the PC/server package (computer) and the Sailfish OS client (communicator-sailfish). Say the wake word, hear the chime, talk hands-free — server-side VAD decides turn ends, barge-in interrupts playback, and the device drops its connection after idle so it never holds a server pipeline slot.

Hardware

Waveshare ESP32-S3-AUDIO-Board: ESP32-S3 (dual-core @ 240 MHz), 16 MB flash, 8 MB octal PSRAM, Wi-Fi 2.4 GHz. Onboard ES8311 DAC (speaker) + ES7210 ADC (mic) behind an I2C control bus, WS2812 x7 RGB ring, and three user keys. USB flashing only (idf.py flash) — there is no OTA path by design.

Firmware: ESP-IDF v5.5.0 (exact; installed at ~/esp/esp-idf, toolchains in ~/.espressif). Target esp32s3, partition table partitions_sr16m.csv (16 MB layout). Managed components resolve via the IDF component registry espressif/esp-sr, espressif/esp_codec_dev, espressif/esp_websocket_client, espressif/led_strip, …; the board layer itself is a custom BSP under components/bsp).

Board pins (canonical source: components/bsp/include/bsp.h):

Signal	GPIO	Notes
I2C SDA	11	vendor wiki pinout confirmed by bus scan
I2C SCL	10	(the plan's SDA=10/SCL=11 was transposed)
I2S MCLK	12
I2S BCLK	13
I2S WS	14
Mic DIN	15	ES7210 ASDOUT (capture)
Spk DOUT	16	ES8311 DSDIN (playback)
LED ring data	38	WS2812 x7

Keys K1/K2/K3 are not native ESP32 GPIOs — they hang off a TCA9555 I2C expander at address 0x20 (EXIO9/10/11), read active-low:

• K1 — volume down (−10)
• K2 — volume up (+10)
• K3 — mic mute toggle (blocks upstream audio)

The speaker amp enable is also behind the TCA9555 (EXIO8, active-high).

Build & flash

. $HOME/esp/esp-idf/export.sh
idf.py set-target esp32s3      # once; sdkconfig.defaults pins it anyway
idf.py menuconfig              # provision Wi-Fi/server (table below)
idf.py build
idf.py -p /dev/ttyACM0 flash monitor

dialout caveat: if your user is not in the dialout group (and you don't want to re-login after adding it), wrap every port-touching command in sg:

sg dialout -c "idf.py -p /dev/ttyACM0 flash"
sg dialout -c "idf.py -p /dev/ttyACM0 monitor"

Build-only calls need no wrapper. idf.py monitor refuses non-TTY stdin in some capture setups; plain pyserial works for scripted serial logging.

Provisioning (Kconfig)

All configuration lives in main/Kconfig.projbuild → Communicator Configuration. Reflash to change network settings (no NVS persistence).

Option	Default	Meaning
WIFI_SSID	(empty)	Wi-Fi SSID
WIFI_PASSWORD	(empty)	Wi-Fi password
SERVER_HOST	192.168.0.2	computer server host
SERVER_PORT	8766	computer server port
CONNECT_TIMEOUT_SEC	5	WebSocket connect timeout
IDLE_DISCONNECT_SEC	15	Disconnect after N s idle past last server event
DEFAULT_VOLUME	60	Speaker volume 0–100 (runtime-adjustable)
BSP_SELFTEST	n	Boot-time tone + mic RMS selftest
UI_COLOR_CALIBRATE	n	Ring R→G→B channel-map probe
UI_STATE_WALK	n	Cycles all six ring states
AFE_REF_DELAY_MS	80	AEC reference delay compensation

Contributor rules — including secrets never get committed — live in AGENTS.md.

Host conformance gate

cd host/tests && python3 -m pytest -q

Must be green before any on-device network work. The suite unit-tests the wire-format helpers and runs the protocol layer against the shared conformance harness from the sibling checkout: ../computer/tests/conformance/ fake_realtime_server.py. No SDK needed — pure host pytest.

Architecture (one paragraph)

afe_task runs two FreeRTOS tasks (feed/fetch) around the ESP-SR audio frontend: WakeNet wake-word detection plus AEC reference alignment (AFE_REF_DELAY_MS). A detection wakes the app; ws_client opens the WebSocket to computer, sends session.update immediately on connect to arm server-side VAD, then shuttles events. player buffers outbound audio in a 256 KB PSRAM jitter ring (~8.2 s cap) feeding 20 ms slices and does ack accounting (exactly one output_audio.played per response). app_state is the machine IDLE → CONNECTING → LISTENING ⇄ SPEAKING → DISCONNECTING; ui_ring maps states onto the WS2812 x7 ring; buttons polls the TCA9555 keys.

Behavior notes

• Server-side VAD: the server commits utterances on silence; the device just streams mic audio while LISTENING.
• Barge-in: speaking over playback makes the server cancel and start a new response; the player drops buffered audio and any chime tail.
• Follow-up window: after each response the mic stays live until the idle watchdog fires — the device disconnects after 15 s without server events (CONFIG_IDLE_DISCONNECT_SEC), so follow-ups must land within ~15 s.
• Idle disconnect: after 15 s without server events the device disconnects itself, releasing the server pipeline slot (a second client can connect while this one sits idle).
• session.update on connect arms the server's VAD before any audio flows.
• Wake chime is local: the canonical family cue (here.wav) is embedded in the firmware and injected into the playback ring when CONNECTING starts.
• Wire contract is owned by computer: see ../computer/src/computer/api/openai_realtime/CONFORMANCE.md (canonical — this repo's docs/conformance.md and host tests track it).

Status

Ships stock Espressif wn9_hiesp ("Hi ESP") wake model. The custom "Computer" wake model is pending training; when it arrives it is a drop-in via menuconfig model selection (no code changes) — rebuild, flash, verify boot log shows the new model name.