feat: ESP32-S3 wake-word communicator firmware v1 #1

Merged

troed merged 34 commits from devel into main 2026-08-23 12:54:39 +02:00

Conversation 0 Commits 34 Files changed 44 +9510 -10

troed commented 2026-08-23 12:50:35 +02:00

Owner

Copy link

Implements docs/superpowers/specs/2026-08-22-communicator-esp32-design.md per docs/superpowers/plans/2026-08-22-communicator-esp32-firmware.md.

What's here

• Portable protocol layer (components/protocol, zero IDF deps, C99+cJSON) with host conformance suite vs computer fake_realtime_server — cd host/tests && python3 -m pytest -q = 23 passed; gate green before any on-device work
• BSP: ES8311/ES7210 via esp_codec_dev (I2S pins 12/13/14/15/16, I2C SDA11/SCL10 bus-scan-corrected), TCA9555 keys K1/K2/K3 = vol-/vol+/mute, WS2812 ring on GPIO38 (RGB fmt per vendor driver)
• esp-sr WakeNet + AEC feed/fetch tasks replacing raw mic streaming; AEC reference fed from playout path
• Wire behavior: session.update sent on connect to arm server VAD; ~100 ms input_audio_buffer.append chunks; exactly-one output_audio.played ack after done+drain (guaranteed delivery, logged); unknown/malformed events never fatal; barge-in flushes promptly
• Player: 256 KB PSRAM jitter ring (~8.2 s cap), Q8 volume gain, mono→stereo, local embedded here.wav wake chime (sox-resampled)
• State machine IDLE→CONNECTING→LISTENING⇄SPEAKING→DISCONNECTING; 15 s idle disconnect so an idle device never holds a server pipeline slot

Verification

Field-tested against the real server: hands-free "Hi ESP" wake + chime, TTS + transcripts, follow-up turns without re-wake, barge-in, volume/mute keys, idle disconnect releasing the slot (server-journal verified), clean audio. Final whole-branch review approved; deferred minors recorded in ledger.

Custom "Computer" wake model is a later drop-in (menuconfig swap); stock wn9_hiesp ships for now.

Implements docs/superpowers/specs/2026-08-22-communicator-esp32-design.md per docs/superpowers/plans/2026-08-22-communicator-esp32-firmware.md. ## What's here - **Portable protocol layer** (`components/protocol`, zero IDF deps, C99+cJSON) with host conformance suite vs `computer` fake_realtime_server — `cd host/tests && python3 -m pytest -q` = 23 passed; gate green before any on-device work - **BSP**: ES8311/ES7210 via esp_codec_dev (I2S pins 12/13/14/15/16, I2C SDA11/SCL10 bus-scan-corrected), TCA9555 keys K1/K2/K3 = vol-/vol+/mute, WS2812 ring on GPIO38 (RGB fmt per vendor driver) - **esp-sr WakeNet + AEC** feed/fetch tasks replacing raw mic streaming; AEC reference fed from playout path - **Wire behavior**: session.update sent on connect to arm server VAD; ~100 ms input_audio_buffer.append chunks; exactly-one output_audio.played ack after done+drain (guaranteed delivery, logged); unknown/malformed events never fatal; barge-in flushes promptly - **Player**: 256 KB PSRAM jitter ring (~8.2 s cap), Q8 volume gain, mono→stereo, local embedded here.wav wake chime (sox-resampled) - **State machine** IDLE→CONNECTING→LISTENING⇄SPEAKING→DISCONNECTING; 15 s idle disconnect so an idle device never holds a server pipeline slot ## Verification Field-tested against the real server: hands-free "Hi ESP" wake + chime, TTS + transcripts, follow-up turns without re-wake, barge-in, volume/mute keys, idle disconnect releasing the slot (server-journal verified), clean audio. Final whole-branch review approved; deferred minors recorded in ledger. Custom "Computer" wake model is a later drop-in (menuconfig swap); stock wn9_hiesp ships for now.

troed added 34 commits 2026-08-23 12:50:37 +02:00

docs: design spec for ESP32-S3 wake-word communicator 0b6450f792

docs(plan): communicator-esp32 firmware implementation plan 08f45c3d86

feat: IDF project scaffold for ESP32-S3-AUDIO-Board 5071c21207

feat(protocol): base64 codec + host ctypes harness 7ea69f9922

feat(protocol): event encoders + vendored cJSON bc23feb678

feat(protocol): inbound event decoder/dispatcher 0af9604c49

test(host): conformance loopback vs computer fake_realtime_server 4f12587f9a

feat(bsp): I2C/I2S bring-up + ES8311/ES7210 via esp_codec_dev 62878c5cb9

fix(bsp): propagate ES7210 bring-up error codes e7e095d701

feat(bsp): TCA9555 exio level read for user keys K1-K3 f0bdf885fe

feat(main): status ring patterns + button handling a8cea0a16d

feat(main): wifi/ws/state-machine/mic-stream end-to-end vs echo server 295e9e0316

fix(main): arm idle watchdog at LISTENING entry, drop zero-guard 92c57afbe6

feat(main): player jitter buffer, played-ack accounting, volume/mute 744df14126

fix(main): drop stray odd ring byte instead of busy-spinning 62cf46be6a

feat(main): esp-sr wakenet + AEC feed replaces raw mic stream ... 9f14d4a635

esp-sr 2.5.1 AFE (AEC + WakeNet wn9_hiesp) now owns the microphone path:
a feed task drains I2S into interleaved [mic, ref] frames and a fetch task
raises app_state_wake_detected() on 'Hi ESP' and streams enhanced audio
while LISTENING/SPEAKING. player feeds afe_ref_write() with exactly what is
on air; the reference ring compensates output-chain latency
(CONFIG_AFE_REF_DELAY_MS) and bounds staleness. mic_stream.c deleted;
K3 stays mute-only. ws_client accepts multi-second inbound deltas
(RX_MAX 256 KB, oversize frames now logged when dropped).

fix(main): raise WS handshake buffer for servers pushing post-upgrade frames 690a9fa613

test(host): skip unknown server-initiated frames in ack-sequence test b777adf617

feat(protocol): arm server VAD with session.update right after WS open b5921bc385

fix(main): size playout jitter ring for multi-second bursts in PSRAM ... a617326b03

Real backends push deltas larger than the whole old 400 ms ring
(observed: a single 22 KB delta vs 12.8 KB capacity, 135 KB responses).
Allocate 256 KB in PSRAM via heap_caps_malloc + xStreamBufferCreateStatic
so ring memory never lands in internal RAM; drop-newest stays the last
resort and the >=150 ms start-drain gate is unchanged.

fix(main): guarantee delivery of control events through the state queue ... 1a09e0d14b

The delta storm overflowed the 16-deep event queue and swallowed
EV_RESPONSE_DONE, so output_audio.played was never sent (contract
violation). Enlarge the queue to 64 and route control events (wake, WS
lifecycle, response.done, player-drained) through a bounded retry post
(100 x 20 ms) with a loud ESP_LOGE if it ever gives up; audio-delta and
other server events keep drop-newest semantics.

fix(main): idempotent ws teardown, no stop-after-close warning ... 0d1d96201f

The idle-watchdog disconnect hit an already-stopped client and logged
"Client was not started" (double teardown). Mirror client liveness in
s_running: set on successful start, cleared on DISCONNECTED (server-side
loss) and at teardown entry, so close runs exactly once. On a graceful
close the component fully stops the task itself, so only fall back to
esp_websocket_client_stop() when close() fails. Also check start()'s
return value instead of ignoring it.

fix(main): atomic ui_ring state updates, last-write-wins at 30 fps ... fd1d841f30

(s_prev, s_state) were written as a non-atomic pair from several tasks
and re-written by the animation task's ERROR auto-revert, so a torn or
resurrected state could wedge the wrong pattern on the ring. Guard both
the setters and the revert with a portMUX critical section and let the
animation task take an atomic snapshot each frame. Verified on device:
state sets arrive in order (IDLE/CONNECTING/LISTENING/SPEAKING/MUTED)
across 115 echo-server response cycles; temporary probe logging removed.

feat(main): add menuconfig-gated RGB channel-map calibration probe (default n) 77da85259d

fix(main): remap ring pixels to BGR wire order via set_px helper 38a36690d9

Revert "fix(main): remap ring pixels to BGR wire order via set_px helper" ... 544f73c19c

This reverts commit 38a36690d9.

feat(main): add menuconfig-gated semantic state-walk probe (default n) 9c8ce029ca

fix(ui): use RGB color component format per vendor LED driver ... 139a3b52c4

Vendor rgb_led_driver.c (factory_01) configures this ring as RGB; the
previous GRB setting permuted logical colors on the wire.

fix(ws): guaranteed control-json delivery + ack outcome logging f0a848dae3

feat(main): local two-tone wake chime on CONNECTING entry 11f926f587

feat(main): embed canonical here.wav as wake chime 38a0f0e7b7

fix(main): regenerate wake chime via sox resampler 70717cd404

docs: README + AGENTS (build, provisioning, conformance gate, field test) 10519889e7

docs: correct jitter-ring size and follow-up window; refresh stale header comments ... 73983bb402

- README/AGENTS: player ring is 256 KB PSRAM (~8.2 s cap), not 400 ms
- README: follow-up window bounded by IDLE_DISCONNECT_SEC=15 idle watchdog
  measured from last server event (was '~10 s after each response')
- player.h/afe_task.h: drop dead mic_stream references, describe current
  behavior; note runtime ws .buffer_size=8192 wins over CONFIG_WS_BUFFER_SIZE
  in sdkconfig.defaults

troed merged commit fd5200b2aa into main 2026-08-23 12:54:39 +02:00

troed referenced this pull request from a commit 2026-08-23 12:54:40 +02:00

Merge pull request 'feat: ESP32-S3 wake-word communicator firmware v1' (#1) from devel into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

starfleet/communicator-esp32!1

No description provided.