AI Speech — Multilingual Quality Assessment

Evaluation of :ai-speech library capabilities across non-English languages on Android mobile.

Scope

1. Speech-to-Text (STT)

1.1 Script Output

All providers return native script, never romanization.

Important: Romaji output only occurs if locale is misconfigured (e.g., Locale.US while speaking Japanese). Always set correct locale.

1.2 Recognition Quality

1.3 Japanese-Specific: Kanji Homophone Problem

Spoken "かみ" (kami) could map to multiple kanji:

• Google STT uses context to disambiguate — works well in sentences, less reliable for isolated words
• Always returns kanji (correct or not), never falls back to romaji
• Gemini STT performs better at disambiguation due to stronger language model

1.4 Provider Comparison for Multilingual

Recommendation: Default to AndroidSpeechToTextProvider for real-time UX. Use GeminiSpeechToTextProvider when accuracy on CJK languages is critical or for pronunciation coaching flows.

1.5 Gotcha: Language Pack Requirement

From integrate-stt.md:

Locale must match the device's installed STT pack for AndroidSpeechToTextProvider. Unsupported locales fall back to system default and silently produce wrong-language transcripts.

Action required: Guide users to download the target language's offline speech pack in device Settings > Languages > Speech.

2. Text-to-Speech (TTS) — Word Highlight / Karaoke

2.1 Range Event Behavior

TtsEvent.Range(uttId, start, end, frameMs) emits character positions during playback.

2.2 Japanese Highlight Challenge

Japanese text has no whitespace between words:

Text:    東京に行きたいです
Ranges:  [0,3] [3,6] [6,9]  ← arbitrary character chunks, not linguistic words

To achieve word-level highlight in Japanese, a morphological analyzer (MeCab, Kuromoji, or Sudachi) is needed to segment text into words before mapping Range events.

2.3 Korean Highlight

Korean uses spaces (어절 boundaries) but spacing rules are inconsistent in informal text. TTS Range events generally align with spacing — adequate for most UX but not linguistically precise.

3. Pronunciation Scoring

3.1 SimpleWordMatchAssessor Limitations

Current tokenizer uses whitespace splitting via normalizeText(). Impact by language:

3.2 Required: Custom PronunciationAssessor for CJK

For Japanese/Korean to work properly, subclass PronunciationAssessor with language-aware tokenization:

class JapanesePronunciationAssessor(
    private val tokenizer: JapaneseTokenizer // MeCab/Kuromoji wrapper
) : PronunciationAssessor() {
    override fun assess(
        referenceText: String,
        spokenText: String,
        locale: Locale
    ): PronunciationResult {
        val refTokens = tokenizer.tokenize(referenceText)   // [東京, に, 行きたい, です]
        val spokenTokens = tokenizer.tokenize(spokenText)
        // word-level matching on morpheme tokens
    }
}

Tokenizer options:

3.3 Reading-to-Reading Matching (Japanese)

Even with proper tokenization, STT may return different kanji than reference:

Reference: 紙を買う (buy paper)
STT heard:  髪を買う (buy hair) — wrong kanji, same pronunciation

Mitigation: Convert both reference and spoken text to hiragana reading before comparison. Kuromoji provides getReading() for this:

Reference reading: かみをかう
Spoken reading:    かみをかう → 100% match ✓

4. Summary Matrix

5. Recommendations

1. Spanish: No extra work needed — all features work out of the box
2. Japanese:
   • Add Kuromoji dependency for word segmentation
   • Build JapanesePronunciationAssessor with reading-based matching
   • For TTS karaoke: pre-segment text with Kuromoji, map Range events to morpheme boundaries
3. Korean:
   • Scoring mostly works at 어절 level (space-separated units)
   • For finer granularity, add a Korean morphological analyzer (Mecab-ko or Komoran)
4. All CJK: Consider GeminiSpeechToTextProvider for higher accuracy when network available
5. Testing: Always test on real devices — emulator STT behavior differs significantly

6. Unresolved Questions

• Kuromoji dictionary size (~18MB) acceptable for mobile app bundle?
• Should reading-based matching be default behavior in PronunciationAssessor base class, or Japanese-only subclass?
• Need benchmarks: Gemini STT vs Android STT accuracy for Japanese isolated words
• ElevenLabs/Azure TTS Range event behavior for Japanese — untested, may differ from Android system TTS