flash-turn
A tiny decider for voice assistants, English and German. One small text model answers the two questions a voice assistant has to answer all the time:
- turn: the person paused - is what they said so far a finished thought, or will they go on? "What's the weather like in Vienna tomorrow?" β finished, answer now. "If I drive to Graz tomorrow, then" β wait.
- barge: the assistant is speaking and the person says something - do they interrupt, are they just listening ("mhm", "right"), or are they talking to someone else (or is it the TV)?
73 MB, ONNX int8, ~11 ms per question on one CPU core. No GPU. It is the default decider of flash-voice, where it replaces a 4B decision model (~5 GB of GPU memory).
Results
Test sets never seen in training.
Measured on test sets never used in training; for comparison the 4B decision model vllm-d3-mini (Q8_0, llama-server), asked the same questions:
| Test | flash-turn | vllm-d3-mini |
|---|---|---|
| End of turn, decider test cases (English / German, 24 each) | 23 / 23 | 24 / 23 |
| Interjections, decider test cases (English / German, 28 each) | 26 / 25 | 24 / 18 |
| End of turn, 300 clips of Austrian speakers (speech recognition text) | 92.3 % | 87.0 % |
| End of turn, unseen sentences (English / German) | 91.5 % / 91.0 % | 90.7 % / 90.7 % |
| Time per question | 11 ms on 1 CPU thread | ~35 ms on a GPU |
| Size | 73 MB | ~5 GB |
Use
import numpy as np, onnxruntime as ort
from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer.json")
tok.enable_truncation(96, strategy="longest_first")
sess = ort.InferenceSession("flash-turn.onnx", providers=["CPUExecutionProvider"])
def logits(first, text):
e = tok.encode(first, text)
return sess.run(None, {"input_ids": np.array([e.ids], np.int64),
"attention_mask": np.array([e.attention_mask], np.int64)})[0][0]
def softmax(x):
x = np.exp(x - x.max()); return x / x.sum()
# turn: outputs 0-1 = not_finished, finished. Remove punctuation, keep only a final "?".
p = softmax(logits("turn", "If I drive to Graz tomorrow then")[:2]) # -> not finished
# barge: outputs 2-4 = interrupts, listens, other. First part: "barge " + the end of what the assistant is saying.
p = softmax(logits("barge Tomorrow will be sunny in Vienna", "Honey, where are my keys?")[2:])
Input as speech recognition hands it over: punctuation removed except a final question mark (recognisers also put
full stops after cut-off phrases, so those carry no information; ? does). For the barge question, the first part is
"barge " plus the last ~160 characters of what the assistant is saying. training/turn_train2.py has the full recipe (data: training/gen_barge.py, training/teacher_barge.py).
How it was made
- Base: distilbert-base-multilingual-cased (Apache 2.0), one classifier with five outputs; each question only uses its own (the others are masked in training). The vocabulary is cut from 119k to 38k pieces (the ones English and German text uses; tokenization of English/German text is unchanged), which shrinks the embedding.
- turn data: ~2.8 million German sentences (Mozilla Common Voice, CC0, and Tatoeba, CC BY 2.0 FR) and ~2 million English sentences (Tatoeba). Whole sentence = finished, the same sentence cut after a word inside = not finished; punctuation removed, case varied, spoken tags ("right?", "oder?") count as finished, fillers at the cut ("uh", "Γ€h") as not finished.
- barge data: ~7,600 interjections written by a local LLM in English and German for 40 topics (stop/wait, corrections, follow-up questions, new requests, "speak slower"; backchannels; talking to a partner, kids, a pet, a phone call, a visitor; TV news, sports commentators), each with a sentence the assistant is saying. A decision model (vllm-d3-mini) checked them: "someone else" examples it clearly read as addressed to the assistant were dropped - the model only says "someone else" for clear cases. Plus hand-written backchannels and 12,000 sentences addressed to a person by name ("Honey, β¦", "Max, β¦" β someone else) or to the assistant ("Hey Flash, β¦" β interrupts).
- One epoch, 3.3 million examples, ~35 minutes on an RTX 5060 Ti.
Limits
- English and German only.
- Text only: it does not hear the tone of voice. Whether a sentence is meant for someone else in the room is often impossible to tell from the words alone ("could you pass the water?"); in flash-voice longer interjections stop the assistant unless flash-turn is very sure (β₯ 80 %).
- A wrong "not finished" costs time, not the answer: the assistant waits for a longer pause.
License
Apache 2.0. Training data: Mozilla Common Voice German sentences (CC0), Tatoeba (CC BY 2.0 FR, https://tatoeba.org), and generated examples.