"Hey Grok" wake word — evaluation report
Recommended threshold: 0.8 (lowest threshold with ≤1 false accept/hour on real speech and ≥90% cross-engine recall).
At that threshold: cross-engine recall 90.3% (65/72), 0.0 false accepts/hour on 45 min of real continuous speech, 0.0/hour on 60 min of noise and music, 0/96 fires on phonetic near-misses.
Every set below was held out: the model never trained on these speakers, recordings, or noise files, and none were used to pick the recipe.
Recall (does it fire when it should)
| threshold |
0.5 |
0.6 |
0.7 |
0.75 |
0.8 |
0.9 |
| Kokoro TTS, 72 clips, 12 voices × 6 intonations, never trained on |
93.1% |
93.1% |
91.7% |
90.3% |
90.3% |
83.3% |
| Piper held-out speakers, 144 clips |
100.0% |
100.0% |
100.0% |
100.0% |
99.3% |
98.6% |
Hardest positives (peak score): bm_george_question_0.8.wav 0.01, bm_george_neutral_0.8.wav 0.01, bm_george_excited_0.8.wav 0.02, am_eric_neutral_0.8.wav 0.24, bm_george_weather_0.8.wav 0.34, am_eric_hesitant_1.25.wav 0.69
False accepts (does it stay quiet when it should)
| threshold |
0.5 |
0.6 |
0.7 |
0.75 |
0.8 |
0.9 |
| real speech, LibriSpeech, 45 min: fires/hour |
1.3 |
1.3 |
1.3 |
1.3 |
0.0 |
0.0 |
| noise + music, MUSAN, 60 min: fires/hour |
1.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
| phonetic near-misses, Kokoro, 96 clips: fires |
0 |
0 |
0 |
0 |
0 |
0 |
| phonetic near-misses, Piper held-out, 120 clips: fires |
5 |
1 |
1 |
1 |
1 |
1 |
Near-miss sentences used: "The grotesque statue stood at the entrance."; "The heyday of that band was the nineties."; "The frost cracked the old stone walkway."; "We drove through fog on the winding gravel road."; "It took me a while to grok that idea."; "It's hard to grok recursion at first glance."; "A stray cat crept along the brick wall."; "Hey Chad, let's grab lunch."
Highest-scoring near-misses: af_nicole_neg05.wav 0.08, bm_george_neg04.wav 0.07, bm_lewis_neg07.wav 0.07, af_heart_neg05.wav 0.07, bf_emma_neg04.wav 0.05, am_michael_neg07.wav 0.05
Your recordings
Never trained on. A take counts as detected if any window scores over the threshold.
| file |
length |
peak score |
fires @0.5 / @0.6 / @0.7 / @0.75 / @0.8 / @0.9 |
| ben_take00.wav |
39.6s |
1.00 |
9 / 9 / 9 / 9 / 9 / 9 |
Size and speed
| parameters |
34,209 |
| end-to-end ONNX |
180 KiB |
| classifier ONNX |
136 KiB |
| Core ML classifier (fp16) |
97 KiB |
| CPU inference, one thread |
3.67 ms per 1.5 s window |
| duty cycle at 160 ms hop |
2.3% of one core |
| ONNX vs PyTorch agreement |
5.1e-11 max abs diff |
How it was built
- Positives: 3000 synthetic utterances of "Hey Grok" (with punctuation-driven intonation and spoken continuations) across ~1050 Piper voices, plus pitch/tempo, reverb, noise and gain augmentation.
- Negatives: 2500 phrase-specific near-miss sentences, 3000 general sentences, real speech (LibriSpeech), noise and music (MUSAN), near-silence.
- Two training rounds: the first model was streamed over 15.7 h of real speech and noise it had never seen and every window it fired on (779 of them) was folded back in as a negative for the second round.
- Validation loss over training (round 2): 0.061, 0.049, 0.040, 0.039, 0.037 … 0.036.
Limitations
- Trained on synthetic speech. The cross-engine numbers are the honest estimate of recall on voices it has not heard; real microphones, rooms and accents will differ. Collect a few real recordings on your hardware and check the threshold against them before shipping.
- Single microphone, no echo cancellation, synthetic room reverb only.
- Threshold is a product decision: raise it if false accepts annoy users more than misses.