From 137896f13f9337e05c5ba3f537ff4e7dc786b49b Mon Sep 17 00:00:00 2001 From: edsonlima2000 <30158415+edsonlima2000@users.noreply.github.com> Date: Sat, 8 Aug 2026 21:13:11 -0300 Subject: [PATCH] Add experimental multilingual speech --- AGENTS.md | 6 +- README.md | 17 +-- THIRD_PARTY_NOTICES.md | 2 +- .../edson/metaglassesnotes/MainActivity.kt | 33 ++++- .../audio/GlassesAudioService.kt | 81 ++++++----- .../speech/OfflineModelInstaller.kt | 132 ++++++++++++++---- .../metaglassesnotes/speech/SpeechLanguage.kt | 109 +++++++++++++++ .../speech/SpeechLanguageTest.kt | 36 +++++ .../java/com/whispercpp/whisper/LibWhisper.kt | 17 ++- .../lib/src/main/jni/whisper/jni.c | 7 +- 10 files changed, 357 insertions(+), 83 deletions(-) create mode 100644 app/src/main/java/com/edson/metaglassesnotes/speech/SpeechLanguage.kt create mode 100644 app/src/test/java/com/edson/metaglassesnotes/speech/SpeechLanguageTest.kt diff --git a/AGENTS.md b/AGENTS.md index f973dda..faf6bb3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,10 +13,10 @@ The parent `../AGENTS.md` applies. - Treat stable capture from the glasses microphone over Android Bluetooth communication routing as a hard go/no-go prerequisite. - Continuous listening must run in a microphone foreground service with visible Android notification and permission handling. - Keep wake-word detection, voice activity detection, speech understanding, action validation, note storage, and TTS local to the phone. -- Use the bundled Apache-2.0 Vosk Android runtime and Brazilian Portuguese small model only for grammar-constrained wake and service commands. Transcribe bounded note audio in batch with the vendored MIT-licensed `whisper.cpp` v1.9.1 runtime and multilingual Whisper `small-q5_1`, fixed to Portuguese (`pt`). Build native code only for `arm64-v8a` initially. Do not add network recognition as a fallback. -- Download the official Whisper model once on first activation, validate its pinned SHA-1, and store it in private app storage. Network access is allowed only for this explicit model acquisition; inference remains local. Removing app data may require downloading the model again. +- Use the Apache-2.0 Vosk Android runtime only for grammar-constrained wake, note-ending, cancellation, and service commands. Portuguese (`pt-BR`) uses the bundled `vosk-model-small-pt-0.3`; English (`en-US`) and Spanish (`es-ES`) download their official small Vosk models on first selection. Load only the selected language model. Transcribe bounded note audio in batch with the vendored MIT-licensed `whisper.cpp` v1.9.1 runtime and multilingual Whisper `small-q5_1`, passing the selected language (`pt`, `en`, or `es`) explicitly. Build native code only for `arm64-v8a` initially. Do not add network recognition as a fallback. +- Download official Whisper and optional Vosk language models only when required, validate pinned hashes, and store extracted models in private app storage. Network access is allowed only for this explicit model acquisition; recognition and transcription remain local. Removing app data may require downloading the models again. - Load Whisper only for bounded batch transcription, retain it for up to five minutes after its latest use to accelerate consecutive notes, then release it while waiting. Always release it when the service stops. Route acknowledgement tones explicitly to the selected Bluetooth communication device. -- Initial wake phrases are `nova anotação` and `anote para mim`. Accept the conservative recognition variants `nova notação`, `nova nota`, `ova anotação`, `ova notação`, and `ova nota`, but never accept `anotação` alone as a wake phrase. While dictating, save after four seconds of silence or when `fim da anotação`, `fim da nota`, or `encerrar anotação` is recognized; omit the ending command from the saved note. Recognition text may exist transiently in memory, but continuous audio and intermediate transcripts must not be persisted. +- Keep language definitions centralized in `SpeechLanguage`; do not mix several wake languages in one active recognizer. Portuguese remains the default and preserves its conservative variants without accepting `anotação` alone. English and Spanish are the first optional languages and must remain labeled experimental until physically validated by fluent speakers. Add German, Italian, Chinese, Japanese, or Korean only with a compatible local model, natural phrases reviewed by a fluent speaker, pinned download integrity, and tests. While dictating, save after four seconds of silence or a configured language-specific ending phrase; omit the ending command from the saved note. Recognition text may exist transiently in memory, but continuous audio and intermediate transcripts must not be persisted. - Keep continuous audio in a bounded in-memory buffer. Do not persist audio except during an explicit diagnostic recording or when the user explicitly chooses an audio note. - Validate AI-produced actions through deterministic application rules before storing or executing them. - Store notes locally. External export or synchronization must be optional and require a separate product decision. diff --git a/README.md b/README.md index 24471bb..eb9f6ec 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ Core transcription does not use ChatGPT, a backend, or a hosted speech service. ### Highlights -- Brazilian Portuguese voice commands and transcription; +- selectable Brazilian Portuguese voice commands plus experimental English and Spanish support; - on-device wake phrase detection with Vosk; - local batch transcription with multilingual Whisper `small-q5_1`; - private local note storage, selection, deletion, and Android sharing; @@ -26,10 +26,11 @@ A arquitetura pode ser adaptada para outras marcas e modelos que disponibilizem ## Funcionamento 1. Ative o modo de anotações no aplicativo. -2. Diga **“Nova anotação”** ou **“Anote para mim”**. -3. Aguarde o sinal sonoro enviado pela rota Bluetooth e dite a nota. -4. Diga **“Fim da anotação”**, **“Fim da nota”** ou **“Encerrar anotação”**, ou permaneça em silêncio por quatro segundos. -5. O texto é salvo no armazenamento privado do aplicativo e o áudio transitório é descartado. +2. Selecione o idioma da fala. Português é o padrão validado; inglês e espanhol são opcionais e ainda precisam de validação física por falantes desses idiomas. +3. Diga **“Nova anotação”** ou **“Anote para mim”** em português, **“New note”** em inglês ou **“Nueva nota”** em espanhol. +4. Aguarde o sinal sonoro enviado pela rota Bluetooth e dite a nota. +5. Use um comando de encerramento do idioma selecionado ou permaneça em silêncio por quatro segundos. +6. O texto é salvo no armazenamento privado do aplicativo e o áudio transitório é descartado. As anotações podem ser selecionadas individualmente ou todas de uma vez. A seleção pode ser excluída após confirmação ou compartilhada como um único texto, com uma linha em branco entre as notas, pelo seletor padrão do Android (incluindo Samsung Notes, WhatsApp e ChatGPT quando disponíveis). @@ -40,12 +41,12 @@ O pipeline é híbrido para equilibrar precisão, latência e bateria: - Vosk com vocabulário restrito permanece ativo para detectar os comandos; - Whisper `small-q5_1`, executado pelo `whisper.cpp`, transcreve somente a fala delimitada; - o Whisper é carregado sob demanda, reutilizado por até cinco minutos entre notas e depois liberado para reduzir a memória em espera; -- o idioma do Whisper é fixado em português (`pt`); +- o idioma selecionado é passado explicitamente ao Whisper (`pt`, `en` ou `es`); - a inferência do Whisper ocorre em lote e fora da thread principal; - o código nativo usa otimização `-O3` e até seis núcleos para reduzir a latência das notas; - nenhum áudio ou texto é enviado para transcrição na nuvem. -Na primeira ativação, o aplicativo precisa de internet para baixar aproximadamente 190 MB do modelo oficial `ggml-small-q5_1.bin`. O arquivo é validado por SHA-1 e armazenado no diretório privado do aplicativo. Após isso, o fluxo funciona offline. Limpar os dados ou desinstalar o aplicativo remove o modelo e exige um novo download. +Na primeira ativação, o aplicativo precisa de internet para baixar aproximadamente 190 MB do modelo oficial `ggml-small-q5_1.bin`. Português usa o modelo Vosk incluído no APK. Na primeira seleção de inglês ou espanhol, o respectivo modelo Vosk oficial, com cerca de 40 MB, também é baixado. Todos os downloads são validados por hash e os modelos ficam no diretório privado do aplicativo. Após isso, o idioma instalado funciona offline. Limpar os dados ou desinstalar o aplicativo remove os modelos e exige novos downloads. ## Diagnóstico @@ -55,7 +56,7 @@ O botão **Gravar teste** cria deliberadamente um WAV no armazenamento privado p - Kotlin e Jetpack Compose/Material 3; - JVM 17, `minSdk 31`, `compileSdk 36` e `targetSdk 36`; -- Vosk Android com `vosk-model-small-pt-0.3`; +- Vosk Android com `vosk-model-small-pt-0.3`, `vosk-model-small-en-us-0.15` ou `vosk-model-small-es-0.42`; - `whisper.cpp` v1.9.1 compilado para `arm64-v8a`; - Whisper multilíngue `small-q5_1`; - NDK `25.2.9519653` e CMake `3.22.1`. diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 9357a79..9db05ed 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -2,7 +2,7 @@ ## Vosk -The application uses `com.alphacephei:vosk-android` and the `vosk-model-small-pt-0.3` Portuguese speech-recognition model. Both are distributed by Alpha Cephei under the Apache License 2.0. +The application uses `com.alphacephei:vosk-android` and the official `vosk-model-small-pt-0.3`, `vosk-model-small-en-us-0.15`, and `vosk-model-small-es-0.42` speech-recognition models. They are distributed by Alpha Cephei under the Apache License 2.0. Portuguese is bundled; English and Spanish are downloaded on first selection and stored privately on the device. - Runtime: https://github.com/alphacep/vosk-api - Model: https://alphacephei.com/vosk/models diff --git a/app/src/main/java/com/edson/metaglassesnotes/MainActivity.kt b/app/src/main/java/com/edson/metaglassesnotes/MainActivity.kt index e0413c7..34779b7 100644 --- a/app/src/main/java/com/edson/metaglassesnotes/MainActivity.kt +++ b/app/src/main/java/com/edson/metaglassesnotes/MainActivity.kt @@ -43,6 +43,8 @@ import com.edson.metaglassesnotes.audio.AudioMode import com.edson.metaglassesnotes.audio.GlassesAudioService import com.edson.metaglassesnotes.notes.LocalNote import com.edson.metaglassesnotes.notes.LocalNoteRepository +import com.edson.metaglassesnotes.speech.SpeechLanguage +import com.edson.metaglassesnotes.speech.SpeechLanguagePreferences import java.text.DateFormat import java.util.Date @@ -60,6 +62,7 @@ class MainActivity : ComponentActivity() { override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) AudioCaptureController.update { it.copy(notes = LocalNoteRepository(this).list()) } + var selectedLanguage by mutableStateOf(SpeechLanguagePreferences.get(this)) setContent { MaterialTheme { Surface(modifier = Modifier.fillMaxSize()) { @@ -72,6 +75,11 @@ class MainActivity : ComponentActivity() { play = ::playLastRecording, deleteNotes = ::deleteNotes, shareNotes = ::shareNotes, + selectedLanguage = selectedLanguage, + selectLanguage = { language -> + SpeechLanguagePreferences.set(this, language) + selectedLanguage = language + }, ) } } @@ -144,6 +152,8 @@ private fun NotesScreen( play: () -> Unit, deleteNotes: (List) -> Unit, shareNotes: (List) -> Unit, + selectedLanguage: SpeechLanguage, + selectLanguage: (SpeechLanguage) -> Unit, ) { val state by AudioCaptureController.state.collectAsStateWithLifecycle() var selectedIds by remember { mutableStateOf(emptySet()) } @@ -183,6 +193,27 @@ private fun NotesScreen( Text(stringResource(R.string.app_name), style = MaterialTheme.typography.headlineMedium) Text("Anotações privadas pelos seus óculos", style = MaterialTheme.typography.titleMedium) } + item { + Card(modifier = Modifier.fillMaxWidth()) { + Column(Modifier.padding(16.dp), verticalArrangement = Arrangement.spacedBy(8.dp)) { + Text("Idioma da fala", style = MaterialTheme.typography.titleMedium) + Text( + "O idioma selecionado define o modelo do gatilho e da transcrição.", + style = MaterialTheme.typography.bodySmall, + ) + SpeechLanguage.entries.forEach { language -> + OutlinedButton( + onClick = { selectLanguage(language) }, + enabled = state.mode == AudioMode.IDLE, + modifier = Modifier.fillMaxWidth(), + ) { + val marker = if (language == selectedLanguage) "✓ " else "" + Text("$marker${language.displayName}") + } + } + } + } + } item { Card(modifier = Modifier.fillMaxWidth()) { Column(Modifier.padding(20.dp), verticalArrangement = Arrangement.spacedBy(10.dp)) { @@ -208,7 +239,7 @@ private fun NotesScreen( } item { Text( - "Com o modo ativo, diga “Nova anotação”, aguarde o sinal e fale. O áudio é processado localmente e descartado.", + "Com o modo ativo, diga “${selectedLanguage.primaryWakePhrase}”, aguarde o sinal e fale. O áudio é processado localmente e descartado.", style = MaterialTheme.typography.bodySmall, ) } diff --git a/app/src/main/java/com/edson/metaglassesnotes/audio/GlassesAudioService.kt b/app/src/main/java/com/edson/metaglassesnotes/audio/GlassesAudioService.kt index 54d4297..37f5dd4 100644 --- a/app/src/main/java/com/edson/metaglassesnotes/audio/GlassesAudioService.kt +++ b/app/src/main/java/com/edson/metaglassesnotes/audio/GlassesAudioService.kt @@ -22,6 +22,8 @@ import com.edson.metaglassesnotes.MainActivity import com.edson.metaglassesnotes.R import com.edson.metaglassesnotes.notes.LocalNoteRepository import com.edson.metaglassesnotes.speech.OfflineModelInstaller +import com.edson.metaglassesnotes.speech.SpeechLanguage +import com.edson.metaglassesnotes.speech.SpeechLanguagePreferences import com.edson.metaglassesnotes.speech.WhisperModelManager import com.whispercpp.whisper.WhisperContext import java.io.File @@ -51,25 +53,6 @@ class GlassesAudioService : Service() { private const val MAX_PROBE_MS = 60_000L private const val NOTE_SILENCE_MS = 4_000L private const val WHISPER_IDLE_TIMEOUT_MS = 5 * 60_000L - private val WAKE_PHRASES = - listOf( - "nova anotacao", - "nova notacao", - "nova nota", - "ova anotacao", - "ova notacao", - "ova nota", - "anote para mim", - "criar anotacao", - "fazer anotacao", - ) - private val NOTE_END_PHRASES = - listOf("fim da anotacao", "fim da nota", "encerrar anotacao") - private const val COMMAND_GRAMMAR = - "[\"nova anotação\", \"nova anotacao\", \"nova notação\", \"nova nota\", " + - "\"ova anotação\", \"ova anotacao\", \"ova notação\", \"ova nota\", " + - "\"anote para mim\", \"criar anotação\", \"fazer anotação\", " + - "\"encerrar anotações\", \"encerrar anotacoes\", \"[unk]\"]" } private val scope = CoroutineScope(SupervisorJob() + Dispatchers.IO) @@ -154,6 +137,7 @@ class GlassesAudioService : Service() { return } + val language = SpeechLanguagePreferences.get(this) val whisperModel = if (mode == AudioMode.HANDS_FREE) { try { @@ -194,7 +178,8 @@ class GlassesAudioService : Service() { recorder.startRecording() when (mode) { AudioMode.PROBE -> captureProbe(recorder, minimum, routeName) - AudioMode.HANDS_FREE -> captureHandsFree(recorder, minimum, routeName, whisperModel!!) + AudioMode.HANDS_FREE -> + captureHandsFree(recorder, minimum, routeName, whisperModel!!, language) AudioMode.IDLE -> Unit } } catch (error: Exception) { @@ -247,13 +232,19 @@ class GlassesAudioService : Service() { minimum: Int, routeName: String, whisperModel: File, + language: SpeechLanguage, ) { - val modelDirectory = OfflineModelInstaller.install(this) + val modelDirectory = + OfflineModelInstaller.install(this, language) { progress -> + val message = "Preparando Vosk (${language.displayName}): $progress%" + AudioCaptureController.update { it.copy(message = message) } + updateNotification(message) + } var whisper: WhisperContext? = null var lastWhisperUseAt = 0L try { Model(modelDirectory.absolutePath).use { model -> - Recognizer(model, SAMPLE_RATE.toFloat(), COMMAND_GRAMMAR).use { commandRecognizer -> + Recognizer(model, SAMPLE_RATE.toFloat(), language.commandGrammar).use { commandRecognizer -> Recognizer(model, SAMPLE_RATE.toFloat()).use { noteRecognizer -> val samples = ShortArray(minimum / 2) var waitingForWake = true @@ -261,7 +252,7 @@ class GlassesAudioService : Service() { var noteAudio = ShortAudioBuffer() var latestText = "" var lastWakeDiagnostic = "" - setWaitingState(routeName) + setWaitingState(routeName, language) while (audioJob != null) { val count = recorder.read(samples, 0, samples.size, AudioRecord.READ_BLOCKING) @@ -286,11 +277,11 @@ class GlassesAudioService : Service() { lastWakeDiagnostic = recognized } when { - normalized.contains("encerrar anotacoes") -> { + matchesPhrase(normalized, language.stopServicePhrases) -> { AudioCaptureController.update { it.copy(message = "Encerrando modo de anotações") } break } - WAKE_PHRASES.any(normalized::contains) -> { + matchesPhrase(normalized, language.wakePhrases) -> { waitingForWake = false latestText = "" lastWakeDiagnostic = "" @@ -300,7 +291,7 @@ class GlassesAudioService : Service() { noteRecognizer.reset() playBluetoothTone(success = true) AudioCaptureController.update { - it.copy(message = "Fale a anotação", partialText = "") + it.copy(message = language.speakMessage, partialText = "") } updateNotification("Ouvindo a nova anotação") } @@ -313,7 +304,7 @@ class GlassesAudioService : Service() { } else { noteAudio.append(samples, count) AudioCaptureController.update { it.copy(partialText = latestText) } - val endPhraseDetected = NOTE_END_PHRASES.any(normalize(latestText)::endsWith) + val endPhraseDetected = endsWithPhrase(latestText, language.endPhrases) val shouldFinish = noteDetector.process(samples.copyOf(count), count * 1_000L / SAMPLE_RATE) if (endPhraseDetected || shouldFinish) { @@ -328,15 +319,20 @@ class GlassesAudioService : Service() { } val finalText = removeNoteEndPhrase( - activeWhisper.transcribeData(noteAudio.toFloatArray(), false) + activeWhisper.transcribeData( + noteAudio.toFloatArray(), + language.whisperCode, + false, + ), + language, ) lastWhisperUseAt = SystemClock.elapsedRealtime() - handleNote(finalText) + handleNote(finalText, language) noteRecognizer.reset() commandRecognizer.reset() latestText = "" waitingForWake = true - setWaitingState(routeName) + setWaitingState(routeName, language) } } } @@ -348,14 +344,14 @@ class GlassesAudioService : Service() { } } - private fun handleNote(text: String) { + private fun handleNote(text: String, language: SpeechLanguage) { val normalized = normalize(text) if (text.isBlank()) { playBluetoothTone(success = false) AudioCaptureController.update { it.copy(message = "Não entendi a anotação", partialText = "") } return } - if (normalized == "cancelar" || normalized.endsWith(" cancelar")) { + if (endsWithPhrase(normalized, language.cancelPhrases)) { playBluetoothTone(success = false) AudioCaptureController.update { it.copy(message = "Anotação cancelada", partialText = "") } return @@ -371,15 +367,16 @@ class GlassesAudioService : Service() { } } - private fun removeNoteEndPhrase(text: String): String { + private fun removeNoteEndPhrase(text: String, language: SpeechLanguage): String { val normalized = normalize(text) - val ending = NOTE_END_PHRASES.firstOrNull(normalized::endsWith) ?: return text.trim() + val ending = language.endPhrases.map(::normalize).firstOrNull(normalized::endsWith) + ?: return text.trim() val wordsToRemove = ending.split(' ').size return text.trim().split(Regex("\\s+")).dropLast(wordsToRemove).joinToString(" ").trim() } - private fun setWaitingState(routeName: String) { - val message = "Aguardando “Nova anotação”" + private fun setWaitingState(routeName: String, language: SpeechLanguage) { + val message = language.waitingMessage AudioCaptureController.update { it.copy(routeName = routeName, message = message, partialText = "") } @@ -393,10 +390,18 @@ class GlassesAudioService : Service() { } .getOrDefault("") + private fun matchesPhrase(normalizedText: String, phrases: List): Boolean = + phrases.map(::normalize).any(normalizedText::contains) + + private fun endsWithPhrase(text: String, phrases: List): Boolean { + val normalizedText = normalize(text) + return phrases.map(::normalize).any(normalizedText::endsWith) + } + private fun normalize(text: String): String = - Normalizer.normalize(text.lowercase(Locale("pt", "BR")), Normalizer.Form.NFD) + Normalizer.normalize(text.lowercase(Locale.ROOT), Normalizer.Form.NFD) .replace("\\p{Mn}+".toRegex(), "") - .replace("[^a-z0-9 ]".toRegex(), " ") + .replace("[^\\p{L}\\p{N} ]".toRegex(), " ") .replace("\\s+".toRegex(), " ") .trim() diff --git a/app/src/main/java/com/edson/metaglassesnotes/speech/OfflineModelInstaller.kt b/app/src/main/java/com/edson/metaglassesnotes/speech/OfflineModelInstaller.kt index ae13e30..1bd9f33 100644 --- a/app/src/main/java/com/edson/metaglassesnotes/speech/OfflineModelInstaller.kt +++ b/app/src/main/java/com/edson/metaglassesnotes/speech/OfflineModelInstaller.kt @@ -2,44 +2,124 @@ package com.edson.metaglassesnotes.speech import android.content.Context import java.io.File +import java.net.HttpURLConnection +import java.net.URL +import java.security.MessageDigest import java.util.zip.ZipInputStream object OfflineModelInstaller { - private const val ASSET_NAME = "vosk-model-small-pt-0.3.zip" - private const val MODEL_FOLDER = "vosk-model-small-pt-0.3" private const val READY_MARKER = ".ready" - fun install(context: Context): File { + fun install( + context: Context, + language: SpeechLanguage, + onProgress: (Int) -> Unit = {}, + ): File { val modelsRoot = File(context.filesDir, "models").apply { mkdirs() } - val modelDirectory = File(modelsRoot, MODEL_FOLDER) - if (File(modelDirectory, READY_MARKER).isFile) return modelDirectory + val modelDirectory = File(modelsRoot, language.voskModelFolder) + if (File(modelDirectory, READY_MARKER).readTextOrNull() == language.code) return modelDirectory - val staging = File(modelsRoot, "$MODEL_FOLDER-staging") + val archive = + language.bundledAsset?.let { assetName -> + File(modelsRoot, "${language.voskModelFolder}.asset.zip").also { destination -> + destination.delete() + context.assets.open(assetName).use { input -> + destination.outputStream().buffered().use(input::copyTo) + } + onProgress(100) + } + } ?: downloadModel(modelsRoot, language, onProgress) + + val staging = File(modelsRoot, "${language.voskModelFolder}-staging") staging.deleteRecursively() staging.mkdirs() - context.assets.open(ASSET_NAME).use { source -> - ZipInputStream(source.buffered()).use { zip -> - while (true) { - val entry = zip.nextEntry ?: break - val relative = entry.name.removePrefix("$MODEL_FOLDER/") - if (relative.isBlank()) continue - val destination = File(staging, relative).canonicalFile - require(destination.path.startsWith(staging.canonicalPath + File.separator)) { - "Entrada inválida no modelo offline" - } - if (entry.isDirectory) destination.mkdirs() - else { - destination.parentFile?.mkdirs() - destination.outputStream().buffered().use { output -> zip.copyTo(output) } + try { + extractArchive(archive, staging, language.voskModelFolder) + modelDirectory.deleteRecursively() + check(staging.renameTo(modelDirectory)) { "Não foi possível instalar o modelo Vosk" } + File(modelDirectory, READY_MARKER).writeText(language.code) + return modelDirectory + } finally { + staging.deleteRecursively() + archive.delete() + } + } + + private fun downloadModel( + modelsRoot: File, + language: SpeechLanguage, + onProgress: (Int) -> Unit, + ): File { + val modelUrl = checkNotNull(language.modelUrl) + val expectedSha256 = checkNotNull(language.modelSha256) + val temporary = File(modelsRoot, "${language.voskModelFolder}.download") + temporary.delete() + val connection = URL(modelUrl).openConnection() as HttpURLConnection + connection.instanceFollowRedirects = true + connection.connectTimeout = 20_000 + connection.readTimeout = 60_000 + connection.connect() + check(connection.responseCode in 200..299) { + "Falha ao baixar modelo Vosk: HTTP ${connection.responseCode}" + } + val total = connection.contentLengthLong + try { + connection.inputStream.buffered().use { input -> + temporary.outputStream().buffered().use { output -> + val buffer = ByteArray(1024 * 1024) + var downloaded = 0L + while (true) { + val count = input.read(buffer) + if (count < 0) break + output.write(buffer, 0, count) + downloaded += count + if (total > 0) onProgress((downloaded * 100 / total).toInt().coerceIn(0, 100)) } - zip.closeEntry() } } + check(sha256(temporary) == expectedSha256) { + "O modelo Vosk baixado falhou na validação" + } + return temporary + } catch (error: Exception) { + temporary.delete() + throw error + } finally { + connection.disconnect() } - modelDirectory.deleteRecursively() - check(staging.renameTo(modelDirectory)) { "Não foi possível instalar o modelo offline" } - File(modelDirectory, READY_MARKER).writeText("vosk-model-small-pt-0.3") - return modelDirectory } -} + private fun extractArchive(archive: File, staging: File, modelFolder: String) { + val stagingRoot = staging.canonicalPath + File.separator + ZipInputStream(archive.inputStream().buffered()).use { zip -> + while (true) { + val entry = zip.nextEntry ?: break + val relative = entry.name.removePrefix("$modelFolder/") + if (relative.isBlank()) continue + val destination = File(staging, relative).canonicalFile + require(destination.path.startsWith(stagingRoot)) { "Entrada inválida no modelo offline" } + if (entry.isDirectory) destination.mkdirs() + else { + destination.parentFile?.mkdirs() + destination.outputStream().buffered().use { output -> zip.copyTo(output) } + } + zip.closeEntry() + } + } + } + + private fun sha256(file: File): String { + val digest = MessageDigest.getInstance("SHA-256") + file.inputStream().buffered().use { input -> + val buffer = ByteArray(1024 * 1024) + while (true) { + val count = input.read(buffer) + if (count < 0) break + digest.update(buffer, 0, count) + } + } + return digest.digest().joinToString("") { byte -> "%02x".format(byte) } + } + + private fun File.readTextOrNull(): String? = runCatching { readText() }.getOrNull() +} diff --git a/app/src/main/java/com/edson/metaglassesnotes/speech/SpeechLanguage.kt b/app/src/main/java/com/edson/metaglassesnotes/speech/SpeechLanguage.kt new file mode 100644 index 0000000..f9ebac7 --- /dev/null +++ b/app/src/main/java/com/edson/metaglassesnotes/speech/SpeechLanguage.kt @@ -0,0 +1,109 @@ +package com.edson.metaglassesnotes.speech + +import android.content.Context + +enum class SpeechLanguage( + val code: String, + val displayName: String, + val whisperCode: String, + val voskModelFolder: String, + val bundledAsset: String?, + val modelUrl: String?, + val modelSha256: String?, + val wakePhrases: List, + val endPhrases: List, + val stopServicePhrases: List, + val cancelPhrases: List, + val waitingMessage: String, + val speakMessage: String, +) { + PORTUGUESE( + code = "pt-BR", + displayName = "Português (Brasil)", + whisperCode = "pt", + voskModelFolder = "vosk-model-small-pt-0.3", + bundledAsset = "vosk-model-small-pt-0.3.zip", + modelUrl = null, + modelSha256 = null, + wakePhrases = + listOf( + "nova anotação", + "nova notação", + "nova nota", + "ova anotação", + "ova notação", + "ova nota", + "anote para mim", + "criar anotação", + "fazer anotação", + ), + endPhrases = listOf("fim da anotação", "fim da nota", "encerrar anotação"), + stopServicePhrases = listOf("encerrar anotações"), + cancelPhrases = listOf("cancelar"), + waitingMessage = "Aguardando “Nova anotação”", + speakMessage = "Fale a anotação", + ), + ENGLISH( + code = "en-US", + displayName = "English (US)", + whisperCode = "en", + voskModelFolder = "vosk-model-small-en-us-0.15", + bundledAsset = null, + modelUrl = + "https://alphacephei.com/vosk/models/vosk-model-small-en-us-0.15.zip", + modelSha256 = "30f26242c4eb449f948e42cb302dd7a686cb29a3423a8367f99ff41780942498", + wakePhrases = listOf("new note", "take a note", "create a note", "make a note"), + endPhrases = listOf("end note", "finish note", "save note"), + stopServicePhrases = listOf("stop notes", "stop note mode"), + cancelPhrases = listOf("cancel"), + waitingMessage = "Waiting for “New note”", + speakMessage = "Speak your note", + ), + SPANISH( + code = "es-ES", + displayName = "Español", + whisperCode = "es", + voskModelFolder = "vosk-model-small-es-0.42", + bundledAsset = null, + modelUrl = "https://alphacephei.com/vosk/models/vosk-model-small-es-0.42.zip", + modelSha256 = "09b239888f633ef2f0b4e09736e3d9936acfd810bc65d53fad45261762c6511f", + wakePhrases = listOf("nueva nota", "anota para mí", "crear una nota", "hacer una nota"), + endPhrases = listOf("fin de la nota", "terminar nota", "guardar nota"), + stopServicePhrases = listOf("cerrar notas", "detener notas"), + cancelPhrases = listOf("cancelar"), + waitingMessage = "Esperando “Nueva nota”", + speakMessage = "Di tu nota", + ); + + val commandGrammar: String + get() = + (wakePhrases + endPhrases + stopServicePhrases + cancelPhrases + "[unk]") + .joinToString(prefix = "[", postfix = "]") { phrase -> + "\"${phrase.replace("\\", "\\\\").replace("\"", "\\\"")}\"" + } + + val primaryWakePhrase: String + get() = wakePhrases.first() + + companion object { + fun fromCode(code: String?): SpeechLanguage = entries.firstOrNull { it.code == code } ?: PORTUGUESE + } +} + +object SpeechLanguagePreferences { + private const val PREFERENCES = "speech_language" + private const val KEY_LANGUAGE = "selected_language" + + fun get(context: Context): SpeechLanguage = + SpeechLanguage.fromCode( + context.getSharedPreferences(PREFERENCES, Context.MODE_PRIVATE) + .getString(KEY_LANGUAGE, SpeechLanguage.PORTUGUESE.code) + ) + + fun set(context: Context, language: SpeechLanguage) { + context.getSharedPreferences(PREFERENCES, Context.MODE_PRIVATE) + .edit() + .putString(KEY_LANGUAGE, language.code) + .apply() + } +} diff --git a/app/src/test/java/com/edson/metaglassesnotes/speech/SpeechLanguageTest.kt b/app/src/test/java/com/edson/metaglassesnotes/speech/SpeechLanguageTest.kt new file mode 100644 index 0000000..d7b3e78 --- /dev/null +++ b/app/src/test/java/com/edson/metaglassesnotes/speech/SpeechLanguageTest.kt @@ -0,0 +1,36 @@ +package com.edson.metaglassesnotes.speech + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertTrue +import org.junit.Test + +class SpeechLanguageTest { + @Test + fun supportedLanguagesHaveUniqueCodesAndModelFolders() { + assertEquals(SpeechLanguage.entries.size, SpeechLanguage.entries.map { it.code }.toSet().size) + assertEquals( + SpeechLanguage.entries.size, + SpeechLanguage.entries.map { it.voskModelFolder }.toSet().size, + ) + } + + @Test + fun everyLanguageProvidesConstrainedCommands() { + SpeechLanguage.entries.forEach { language -> + assertTrue(language.wakePhrases.isNotEmpty()) + assertTrue(language.endPhrases.isNotEmpty()) + assertTrue(language.stopServicePhrases.isNotEmpty()) + assertTrue(language.cancelPhrases.isNotEmpty()) + assertTrue(language.commandGrammar.startsWith("[")) + assertTrue(language.commandGrammar.endsWith("]")) + assertTrue(language.commandGrammar.contains("\"[unk]\"")) + assertFalse(language.wakePhrases.any { it.isBlank() }) + } + } + + @Test + fun unknownPreferenceFallsBackToPortuguese() { + assertEquals(SpeechLanguage.PORTUGUESE, SpeechLanguage.fromCode("unknown")) + } +} diff --git a/third_party/whisper.cpp/examples/whisper.android/lib/src/main/java/com/whispercpp/whisper/LibWhisper.kt b/third_party/whisper.cpp/examples/whisper.android/lib/src/main/java/com/whispercpp/whisper/LibWhisper.kt index 37ae0e9..078785e 100644 --- a/third_party/whisper.cpp/examples/whisper.android/lib/src/main/java/com/whispercpp/whisper/LibWhisper.kt +++ b/third_party/whisper.cpp/examples/whisper.android/lib/src/main/java/com/whispercpp/whisper/LibWhisper.kt @@ -16,11 +16,15 @@ class WhisperContext private constructor(private var ptr: Long) { Executors.newSingleThreadExecutor().asCoroutineDispatcher() ) - suspend fun transcribeData(data: FloatArray, printTimestamp: Boolean = true): String = withContext(scope.coroutineContext) { + suspend fun transcribeData( + data: FloatArray, + language: String = "pt", + printTimestamp: Boolean = true, + ): String = withContext(scope.coroutineContext) { require(ptr != 0L) val numThreads = WhisperCpuConfig.preferredThreadCount Log.d(LOG_TAG, "Selecting $numThreads threads") - WhisperLib.fullTranscribe(ptr, numThreads, data) + WhisperLib.fullTranscribe(ptr, numThreads, data, language) val textCount = WhisperLib.getTextSegmentCount(ptr) return@withContext buildString { for (i in 0 until textCount) { @@ -134,7 +138,12 @@ private class WhisperLib { external fun initContextFromAsset(assetManager: AssetManager, assetPath: String): Long external fun initContext(modelPath: String): Long external fun freeContext(contextPtr: Long) - external fun fullTranscribe(contextPtr: Long, numThreads: Int, audioData: FloatArray) + external fun fullTranscribe( + contextPtr: Long, + numThreads: Int, + audioData: FloatArray, + language: String, + ) external fun getTextSegmentCount(contextPtr: Long): Int external fun getTextSegment(contextPtr: Long, index: Int): String external fun getTextSegmentT0(contextPtr: Long, index: Int): Long @@ -177,4 +186,4 @@ private fun cpuInfo(): String? { Log.w(LOG_TAG, "Couldn't read /proc/cpuinfo", e) null } -} \ No newline at end of file +} diff --git a/third_party/whisper.cpp/examples/whisper.android/lib/src/main/jni/whisper/jni.c b/third_party/whisper.cpp/examples/whisper.android/lib/src/main/jni/whisper/jni.c index e773862..188fdbe 100644 --- a/third_party/whisper.cpp/examples/whisper.android/lib/src/main/jni/whisper/jni.c +++ b/third_party/whisper.cpp/examples/whisper.android/lib/src/main/jni/whisper/jni.c @@ -163,11 +163,13 @@ Java_com_whispercpp_whisper_WhisperLib_00024Companion_freeContext( JNIEXPORT void JNICALL Java_com_whispercpp_whisper_WhisperLib_00024Companion_fullTranscribe( - JNIEnv *env, jobject thiz, jlong context_ptr, jint num_threads, jfloatArray audio_data) { + JNIEnv *env, jobject thiz, jlong context_ptr, jint num_threads, jfloatArray audio_data, + jstring language_str) { UNUSED(thiz); struct whisper_context *context = (struct whisper_context *) context_ptr; jfloat *audio_data_arr = (*env)->GetFloatArrayElements(env, audio_data, NULL); const jsize audio_data_length = (*env)->GetArrayLength(env, audio_data); + const char *language = (*env)->GetStringUTFChars(env, language_str, NULL); // The below adapted from the Objective-C iOS sample struct whisper_full_params params = whisper_full_default_params(WHISPER_SAMPLING_GREEDY); @@ -176,7 +178,7 @@ Java_com_whispercpp_whisper_WhisperLib_00024Companion_fullTranscribe( params.print_timestamps = false; params.print_special = false; params.translate = false; - params.language = "pt"; + params.language = language; params.n_threads = num_threads; params.offset_ms = 0; params.no_context = true; @@ -192,6 +194,7 @@ Java_com_whispercpp_whisper_WhisperLib_00024Companion_fullTranscribe( whisper_print_timings(context); } (*env)->ReleaseFloatArrayElements(env, audio_data, audio_data_arr, JNI_ABORT); + (*env)->ReleaseStringUTFChars(env, language_str, language); } JNIEXPORT jint JNICALL