# AI Speech — Multilingual Quality Assessment

Evaluation of `:ai-speech` library capabilities across non-English languages on Android mobile.

## Scope

| Language | Code | Script |
|----------|------|--------|
| Spanish | `es` | Latin |
| Japanese | `ja` | Kanji + Hiragana + Katakana |
| Korean | `ko` | Hangul |

---

## 1. Speech-to-Text (STT)

### 1.1 Script Output

All providers return **native script**, never romanization.

| Language | Input (spoken) | Android STT Output | Romaji/Romanization? |
|----------|---------------|--------------------|-----------------------|
| Japanese | "konnichiwa" | こんにちは | No — returns hiragana/kanji |
| Japanese | "nani" | 何 | No — returns kanji |
| Japanese | "Tokyo ni ikitai" | 東京に行きたい | No — mixed kanji + hiragana |
| Korean | "annyeonghaseyo" | 안녕하세요 | No — returns Hangul |
| Spanish | "como estas" | ¿Cómo estás? | N/A — already Latin script |

> **Important:** Romaji output only occurs if locale is misconfigured (e.g., `Locale.US` while speaking Japanese). Always set correct locale.

### 1.2 Recognition Quality

| Language | Quality | Tier | Notes |
|----------|---------|------|-------|
| Spanish | ⭐⭐⭐⭐⭐ | Tier-1 | Near-English accuracy. Accents/diacritics handled well |
| Korean | ⭐⭐⭐⭐ | Tier-1 | Strong support. Samsung devices have additional engine |
| Japanese | ⭐⭐⭐⭐ | Tier-1 | Good overall, kanji homophone disambiguation is main weakness |

### 1.3 Japanese-Specific: Kanji Homophone Problem

Spoken "かみ" (kami) could map to multiple kanji:

| Kanji | Meaning | Reading |
|-------|---------|---------|
| 紙 | paper | kami |
| 髪 | hair | kami |
| 神 | god | kami |

- Google STT uses context to disambiguate — works well in sentences, less reliable for isolated words
- **Always returns kanji** (correct or not), never falls back to romaji
- Gemini STT performs better at disambiguation due to stronger language model

### 1.4 Provider Comparison for Multilingual

| Aspect | `AndroidSpeechToTextProvider` | `GeminiSpeechToTextProvider` |
|--------|-------------------------------|------------------------------|
| Script output | Native (kanji, hangul, etc.) | Native |
| Partial results | Yes (live transcript) | No (final only) |
| Kanji disambiguation | Good (context-dependent) | Better (stronger LM) |
| Offline | Yes (if language pack installed) | No |
| Cost | Free | Per-request billing |
| Locale coverage | Device-dependent | All Gemini-supported |

**Recommendation:** Default to `AndroidSpeechToTextProvider` for real-time UX. Use `GeminiSpeechToTextProvider` when accuracy on CJK languages is critical or for pronunciation coaching flows.

### 1.5 Gotcha: Language Pack Requirement

From `integrate-stt.md`:

> Locale must match the device's installed STT pack for `AndroidSpeechToTextProvider`. Unsupported locales fall back to system default and silently produce wrong-language transcripts.

**Action required:** Guide users to download the target language's offline speech pack in device Settings > Languages > Speech.

---

## 2. Text-to-Speech (TTS) — Word Highlight / Karaoke

### 2.1 Range Event Behavior

`TtsEvent.Range(uttId, start, end, frameMs)` emits character positions during playback.

| Language | Range granularity | Word boundary detection | Karaoke quality |
|----------|-------------------|------------------------|-----------------|
| Spanish | Word-level | Good (spaces between words) | ✅ Works well |
| Japanese | Clause/phrase-level | Poor (no spaces in text) | ⚠️ Highlights chunks, not individual words |
| Korean | Varies | Mixed (spaces exist but less consistent) | ⚠️ Acceptable, not precise |

### 2.2 Japanese Highlight Challenge

Japanese text has **no whitespace between words**:

```
Text:    東京に行きたいです
Ranges:  [0,3] [3,6] [6,9]  ← arbitrary character chunks, not linguistic words
```

To achieve word-level highlight in Japanese, a **morphological analyzer** (MeCab, Kuromoji, or Sudachi) is needed to segment text into words before mapping Range events.

### 2.3 Korean Highlight

Korean uses spaces (어절 boundaries) but spacing rules are inconsistent in informal text. TTS Range events generally align with spacing — adequate for most UX but not linguistically precise.

---

## 3. Pronunciation Scoring

### 3.1 `SimpleWordMatchAssessor` Limitations

Current tokenizer uses **whitespace splitting** via `normalizeText()`. Impact by language:

| Language | Tokenization | Scoring behavior | Verdict |
|----------|-------------|-----------------|---------|
| Spanish | `["hola", "mundo"]` | Per-word matching ✅ | Works correctly |
| Japanese | `["何をしますか"]` (entire sentence = 1 token) | All-or-nothing (100% or 0%) | ❌ Broken |
| Korean | `["안녕하세요"]` (greeting = 1 token) | All-or-nothing for single-eojeol phrases | ⚠️ Partially broken |

### 3.2 Required: Custom `PronunciationAssessor` for CJK

For Japanese/Korean to work properly, subclass `PronunciationAssessor` with language-aware tokenization:

```kotlin
class JapanesePronunciationAssessor(
    private val tokenizer: JapaneseTokenizer // MeCab/Kuromoji wrapper
) : PronunciationAssessor() {
    override fun assess(
        referenceText: String,
        spokenText: String,
        locale: Locale
    ): PronunciationResult {
        val refTokens = tokenizer.tokenize(referenceText)   // [東京, に, 行きたい, です]
        val spokenTokens = tokenizer.tokenize(spokenText)
        // word-level matching on morpheme tokens
    }
}
```

**Tokenizer options:**

| Library | Platform | Size | Notes |
|---------|----------|------|-------|
| Kuromoji (Atilika) | JVM | ~18MB dict | Most common for Android/JVM |
| Sudachi | JVM | ~70MB+ | More accurate, larger |
| TinySegmenter | JS/Kotlin | ~5KB | Lightweight, less accurate |
| Gemini STT coaching | Cloud | 0 (API) | Offload tokenization to Gemini |

### 3.3 Reading-to-Reading Matching (Japanese)

Even with proper tokenization, STT may return different kanji than reference:

```
Reference: 紙を買う (buy paper)
STT heard:  髪を買う (buy hair) — wrong kanji, same pronunciation
```

**Mitigation:** Convert both reference and spoken text to **hiragana reading** before comparison. Kuromoji provides `getReading()` for this:

```
Reference reading: かみをかう
Spoken reading:    かみをかう → 100% match ✓
```

---

## 4. Summary Matrix

| Feature | Spanish | Japanese | Korean |
|---------|---------|----------|--------|
| STT returns native script | ✅ | ✅ Kanji/Kana | ✅ Hangul |
| STT recognition quality | Excellent | Good | Good |
| TTS word highlight (karaoke) | ✅ Works | ⚠️ Needs morphological segmentation | ⚠️ Acceptable |
| Pronunciation scoring | ✅ Works | ❌ Needs custom tokenizer + reading conversion | ⚠️ Needs custom tokenizer |
| Offline STT | ✅ | ✅ (with language pack) | ✅ (with language pack) |

## 5. Recommendations

1. **Spanish:** No extra work needed — all features work out of the box
2. **Japanese:**
   - Add Kuromoji dependency for word segmentation
   - Build `JapanesePronunciationAssessor` with reading-based matching
   - For TTS karaoke: pre-segment text with Kuromoji, map Range events to morpheme boundaries
3. **Korean:**
   - Scoring mostly works at 어절 level (space-separated units)
   - For finer granularity, add a Korean morphological analyzer (Mecab-ko or Komoran)
4. **All CJK:** Consider `GeminiSpeechToTextProvider` for higher accuracy when network available
5. **Testing:** Always test on real devices — emulator STT behavior differs significantly

## 6. Unresolved Questions

- Kuromoji dictionary size (~18MB) acceptable for mobile app bundle?
- Should reading-based matching be default behavior in `PronunciationAssessor` base class, or Japanese-only subclass?
- Need benchmarks: Gemini STT vs Android STT accuracy for Japanese isolated words
- ElevenLabs/Azure TTS Range event behavior for Japanese — untested, may differ from Android system TTS