Inputs
A scripted recording (up to 30 seconds) and its reference text. The text is converted to expected units: phonemes for English, French, Spanish, German, Italian and Portuguese; an initial, a final and a tone for every Mandarin syllable. Sandhi contexts where two realisations are correct (3rd tone before a 3rd tone; 不 before a 4th tone) are accepted at scoring time; see the sandhi post for why 一 is not.
Alignment and per-unit evidence
A self-supervised speech encoder produces frame-level representations; forced alignment places each expected unit in time (the start_ms / end_ms you see on words, and the shared syllable interval on Mandarin components). For each unit the engine derives (a) a measure of how well the frames match the expected unit, and (b) an estimate of which unit was actually produced when it differs — the heard field. Heard values are estimates, not transcriptions.
Display grades
Grades are 0–100 and calibrated per unit against native-speaker references (score_scale.calibration = native_reference_per_unit in /v1/capabilities). A unit that native speakers produce with wide variation is graded more leniently than one they produce consistently. Word grades aggregate their units; sentence grades aggregate words plus sentence-level signals (pausing for fluency, timing regularity for rhythm, aligned-word count for completeness). Because calibration is per language, a 70 in Portuguese and a 70 in English are not the same distance from native. Grades are for display and progress; they are not accuracy percentages and not comparable across vendors.
Correction decisions
Separately from the grade, the engine emits a decision: review (evidence of an error is strong enough to correct), uncertain (some evidence, not enough to interrupt), no_correction. For English, French, Spanish, German, Italian and Portuguese the decision is made at word level and phoneme rows are supporting evidence (correction_level: word, phoneme_detail: experimental evidence in capabilities). For Mandarin with task=connected_speech, decisions are made per initial, final and tone by a separate human-judgment model trained on expert-labelled learner recordings. Decisions and grades can disagree by design: a modest grade with no_correction means "not native-like, not worth stopping for".
What we measure before a release
- Error detection on expert-labelled learners. For Mandarin, an evaluation set rated by three expert annotators at initial, final and tone level; we track the area under the ROC curve for detecting rated errors per component and per context (for example 3rd + 3rd tone sequences).
- False accusations on native speakers. The share of native units flagged
review, measured on held-out speakers from public corpora (AISHELL-1 and AISHELL-3 for Mandarin; native reference sets for the other packs). This is the number that decides whether learners trust the product, and we report it internally for every candidate model. - Robustness. Stability of scores and flags under gain changes, small time shifts, lossy encoding (AAC) and tempo changes. Tempo stretching is the known weak spot for Mandarin tone flags and is tracked as an open gap.
- Retake consistency. Agreement between two takes of the same sentence by the same learner, compared with the agreement of a commercial reference engine on the same pairs.
We do not publish a single accuracy percentage because there is no such number: detection rates depend on the error type, the context, the learner's first language and the false-accusation budget you accept. Ask us for the evaluation report for a specific pack and we will share the current figures with their denominators.
What "experimental" means
Every pack is currently labelled status: experimental. Concretely: the native calibration set behind the pack is still small (Spanish, German, Italian, Portuguese in particular), some sounds are outside the acoustic inventory and are returned as unassessed (for example the English rhotic ɝ, German ʏ, some Portuguese nasal glides), and learner score bands have not been validated against human ratings for every language. The limitations field in /v1/capabilities lists the current gaps per pack, and every response carries the model_version that scored it, so you can detect when a pack changes.
What the engine does not do
- It does not transcribe or score unscripted speech.
- It does not score lexical stress or intonation; fluency and rhythm are available on English and French only.
- It does not identify the speaker, estimate age or accent, or store audio; recordings are deleted after scoring.
Reading a response
{
"model_version": "mandarin-human-v2-…",
"scores": { "overall": 86, "pronunciation": 90, "initial": 91, "final": 89, "tone": 80, "fluency": null },
"words": [{
"word": "世", "pinyin": "shi4", "score": 68, "start_ms": 1260, "end_ms": 1640,
"phones": [
{ "dimension": "initial", "expected": "sh", "heard": "s", "score": 78, "decision": "uncertain" },
{ "dimension": "final", "expected": "i", "heard": "i", "score": 83, "decision": "no_correction" },
{ "dimension": "tone", "expected": 4, "heard": 2, "score": 62, "decision": "review" }
]
}]
}null means a dimension is not supported by the pack, never "zero". Unassessed symbols are listed per word so your UI can say "not assessed" instead of "wrong".