What you will build

A "say it" step: the learner sees a sentence, taps record, speaks, and gets back coloured words, a highlighted sound to fix, and a button to replay the exact segment. Three parts: capture on the client, assess on the server, render the result.

1. Capture audio on the client

Browsers give you MediaRecorder; it produces WebM (Chrome, Firefox) or MP4/M4A (Safari). Both are accepted by TonePerfect, so you do not need to transcode. Keep recordings short — one sentence, under 15 seconds — and stop automatically after a silence.

JavaScript (browser)
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const recorder = new MediaRecorder(stream);
const chunks = [];
recorder.ondataavailable = e => chunks.push(e.data);
recorder.onstop = async () => {
  const blob = new Blob(chunks, { type: recorder.mimeType });
  const form = new FormData();
  form.append('audio', blob, 'take.webm');
  form.append('text', sentence);          // what the learner was asked to say
  form.append('language', 'en-US');
  form.append('attempt', attemptId);      // your own id, used for idempotency
  const result = await fetch('/api/assess', { method: 'POST', body: form }).then(r => r.json());
  render(result);
};
recorder.start();
setTimeout(() => recorder.stop(), 8000);   // or stop on silence

On iOS and Android use the platform recorder (AVAudioRecorder, MediaRecorder) and write M4A or WAV. On Android, note that some devices restart the microphone when the screen rotates or an interruption arrives; treat a recording that is too short or silent as "try again" rather than sending it.

Never call the assessment API from the client with your secret key. The browser or app posts to your backend, which adds the key and forwards the request.

2. Assess on your server

Your endpoint forwards the file, the reference text and the language, and adds two headers: the bearer key and an Idempotency-Key derived from the attempt. If the mobile network drops and the client retries, the same key returns the same result without a second charge.

Node (server)
import express from 'express';
import multer from 'multer';

const app = express();
const upload = multer({ limits: { fileSize: 10 * 1024 * 1024 } });

app.post('/api/assess', upload.single('audio'), async (req, res) => {
  const form = new FormData();
  form.append('audio', new Blob([req.file.buffer], { type: req.file.mimetype }), req.file.originalname);
  form.append('text', req.body.text);
  form.append('language', req.body.language);
  if (req.body.language === 'zh-CN') form.append('task', 'connected_speech');

  const upstream = await fetch('https://api.toneperfect.app/v1/assess', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.TONEPERFECT_API_KEY}`,
      'Idempotency-Key': `${req.user.id}-${req.body.attempt}`,
    },
    body: form,
  });
  const result = await upstream.json();
  if (!upstream.ok) return res.status(502).json({ error: result.error?.message || 'Assessment failed' });
  res.json(result);
});
Python (server)
import os, requests
from flask import Flask, request, jsonify

app = Flask(__name__)

@app.post("/api/assess")
def assess():
    audio = request.files["audio"]
    data = {"text": request.form["text"], "language": request.form["language"]}
    if data["language"] == "zh-CN":
        data["task"] = "connected_speech"
    r = requests.post(
        "https://api.toneperfect.app/v1/assess",
        headers={
            "Authorization": f"Bearer {os.environ['TONEPERFECT_API_KEY']}",
            "Idempotency-Key": f"{request.user_id}-{request.form['attempt']}",
        },
        files={"audio": (audio.filename, audio.stream, audio.mimetype)},
        data=data,
        timeout=60,
    )
    body = r.json()
    if not r.ok:
        return jsonify({"error": body.get("error", {}).get("message", "Assessment failed")}), 502
    return jsonify(body)

Handling the three outcomes

  • assessable: true — render the scores.
  • assessable: false — the recording could not be scored (silence, wrong sentence, too noisy). reason says why; no credit was charged. Show "We couldn't hear that — try again" and let them retry.
  • HTTP error — transport or service problem. Retry with the same Idempotency-Key; if the request had completed upstream you get the original result back.

3. Render feedback learners can act on

This is where most integrations go wrong: they show a number. A number is not feedback. Use the three levels the response gives you.

Colour the words, not the sentence

Map words[].score to three bands and colour each word. Keep the overall score small; it is a progress signal, not the lesson.

JavaScript (render)
function render(result) {
  if (!result.assessable) return showRetry(result.reason);
  sentenceEl.innerHTML = result.words.map((w, i) => {
    const band = w.score >= 85 ? 'good' : w.score >= 70 ? 'fair' : 'weak';
    const flag = needsReview(w) ? ' needs-review' : '';
    return `<button class="word ${band}${flag}" data-i="${i}">${w.word}</button>`;
  }).join(' ');
  overallEl.textContent = result.scores.overall;
}

// Decisions live on the word for most languages, and on each
// initial / final / tone for Mandarin. Act on decisions, display scores.
function needsReview(word) {
  return word.decision === 'review' || (word.phones || []).some(p => p.decision === 'review');
}

Explain one thing

When the learner taps a flagged word, show the sound that needs work: expected vs heard. For Mandarin, name the component — "Tone: expected 4th, heard 2nd". Pick the lowest-scoring unit with decision === 'review'; if there is none, say the word was fine and show the score.

Let them hear it

Use start_ms / end_ms to play just that word from the learner's own recording, then play your reference audio for the same word. Hearing the two back to back is worth more than any score.

JavaScript (segment playback)
function playSegment(audioEl, word) {
  audioEl.currentTime = word.start_ms / 1000;
  const stop = () => { if (audioEl.currentTime >= word.end_ms / 1000) { audioEl.pause(); audioEl.removeEventListener('timeupdate', stop); } };
  audioEl.addEventListener('timeupdate', stop);
  audioEl.play();
}

Rules we learned the hard way

  • Act on decision, not on a threshold you invented. Correcting a learner who was right costs more trust than missing an error.
  • Never show more than one correction per attempt. A wall of red is demoralising and unactionable.
  • Treat uncertain as silence. Say nothing, or "close — try once more".
  • Compare to the learner's own history, not to a universal pass mark; scales differ by language and the packs improve over time (watch model_version).
  • Make retry effortless. One tap, same sentence, new attempt id.

4. Before you ship

  • Test with a native speaker: they should score high and almost never be flagged.
  • Test with a known error (say "sunny" with "ah"): the right sound should be flagged with a plausible heard value.
  • Test a dropped connection mid-upload: the retry must return the same result and cost one credit.
  • Check GET /v1/capabilities at startup so a language that is unavailable degrades gracefully.

The full field reference is in the documentation. Language-specific notes — which sounds are covered, which are not — are on each language page.