Step 1: know what was supposed to be said

Scripted assessment starts with the reference text. The engine converts it to a sequence of expected sounds — phonemes for most languages, initial + final + tone for each Mandarin syllable — using a pronunciation dictionary and grapheme-to-phoneme rules. This step has its own failure modes: heteronyms ("read" present vs past), numbers, foreign words, and Mandarin sandhi, where the canonical tone in the dictionary is not the tone a native speaker produces. Good engines handle those at this stage; TonePerfect, for example, accepts both realisations of a 3rd tone before another 3rd tone (see the sandhi post).

Step 2: align the audio to the expected sounds

Forced alignment finds where each expected sound starts and ends in the recording. Modern engines use a neural acoustic model — typically a self-supervised speech encoder such as wav2vec 2.0 or HuBERT with a small head on top — to produce per-frame posterior probabilities over sounds, and a dynamic-programming search to fit the expected sequence to those frames. The output is the word and phone timestamps you see as start_ms / end_ms.

Alignment is also where "completeness" comes from: if a word cannot be aligned with any confidence, it was probably skipped.

Step 3: score each sound

The classic measure is goodness of pronunciation (GOP): for each aligned segment, compare the likelihood of the expected sound against the best competing sound. A high ratio means the frames look like the expected sound and nothing else; a low ratio means another sound fits better. That "another sound" is the raw material for a heard estimate — the phoneme the engine thinks was actually produced — which is what makes feedback actionable ("you said ɑ, this needs ʌ").

GOP alone has a known weakness: it is calibrated to the acoustic model's training data, not to the learner population or to native speakers of the target language. Two things fix that.

Native-reference calibration

Score each unit relative to how native speakers score on the same unit. If native speakers of Portuguese produce a wide spread on nasal glides, a learner's modest value on that glide should not be punished as harshly as the same value on a sound natives produce consistently. TonePerfect's /v1/capabilities describes its scale as native_reference_per_unit for exactly this reason. It also means scales differ by language: a Portuguese 69 and an English 69 are not the same distance from native.

Learned heads on top of the encoder

Rather than using raw likelihood ratios, train small classifiers on labelled learner recordings to predict "is this sound acceptable?" and "which sound was produced?". This is how tone assessment works in practice: pitch contours are noisy, and a head trained on thousands of expert-labelled learner syllables outperforms hand-written contour rules.

Step 4: aggregate

Word scores are aggregates of their sounds; sentence scores are aggregates of words plus sentence-level signals — pause structure for fluency, timing regularity for rhythm, aligned-word count for completeness. The weights are a product decision, which is one reason vendors' overall scores differ more than their phone scores do.

Step 5: decide whether to say anything

This is the step most engines skip, and the one that determines whether learners trust your app. A score tells you how far a sound was from the reference; it does not tell you whether the evidence is strong enough to interrupt the learner. Thresholding a score at 60 or 70 by hand has two failure modes:

  • False accusations. Native speakers are told they mispronounced something. Any threshold picked to catch learner errors will also catch some fraction of perfectly good native speech; the question is how much. We measure that fraction on held-out native recordings (public corpora such as AISHELL-1 for Mandarin) for every model release, and it is the number we care about most, because a learner who is corrected when they were right stops believing the app.
  • Silent errors. Real errors score just above the threshold and are never mentioned.

TonePerfect returns a decision per unit — review, uncertain, no_correction — separately from the display grade. The grade is what you show; the decision is what you act on. A word can score 74 and carry no_correction, or score 80 on one component and review on another. The methodology page describes how the two are produced and what "experimental" means for each language pack.

Why two engines disagree

Different acoustic models, different reference populations, different aggregation weights, different thresholds. None of that is a bug. It does mean that comparing raw numbers across vendors is meaningless; comparing ordering and decisions on the same recordings is not. When you evaluate, use a set of recordings you know: a few native speakers, a few strong learners, a few beginners with known errors. Check that natives score high and are rarely flagged, that the known errors are found, and that the heard sounds are plausible.

What this means for your product

  • Show the grade; act on the decision.
  • Show the heard sound next to the expected one — that is the feedback the learner can use.
  • Use timestamps to let the learner hear the exact segment.
  • Expect scales to differ across languages; compare a learner to their own history, not to a universal pass mark.