Learn any Android task once. Replay it with your voice.
SAAR is an on-device Android assistant that watches you perform a task once (like ordering groceries), learns the abstract workflow, and replays it whenever you ask — with different items, quantities, or addresses — all via voice command.
Built with Flutter + native Kotlin for a hackathon.
- How It Works
- Features
- Tech Stack
- Project Structure
- Setup & Installation
- Running the App
- Granting Permissions
- Testing Your First Flow
- Architecture
- Flow JSON Schema
- Credential Guard
- Role Ontology
- Configuration
- Known Limitations
- License
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ 1. TEACH │ │ 2. LEARN │ │ 3. REPLAY │
│ │ │ │ │ │
│ "Teach me to │────▶│ SAAR-NLU │────▶│ "Order milk │
│ order on │ │ converts │ │ from Zepto" │
│ Zepto" │ │ your taps │ │ │
│ │ │ into abstract│ │ SAAR replays │
│ You perform │ │ flow steps │ │ with "milk" │
│ the task │ │ with roles │ │ as the item │
└──────────────┘ └──────────────┘ └──────────────┘
- Teach — Tap the mic, say "teach me to order on Zepto", then switch to Zepto and perform the task normally. SAAR records every tap, type, and scroll via Android's Accessibility Service.
- Learn — When you tap "Stop & Save", the raw action trace is abstracted into a generalised flow with parametrised slots (item name, quantity, address) entirely on-device.
- Replay — Next time, say "order 2 kg rice from Zepto". SAAR matches the utterance to the learned flow using its local NLU, fills in the slots, and executes step-by-step — stopping automatically before any payment/credential screen.
| Feature | Status |
|---|---|
| Voice-triggered teach & command | ✅ |
| One-shot learning from demonstration | ✅ |
| Abstract flows over element roles (not coordinates) | ✅ |
| Parametrised slots (item, quantity, address) | ✅ |
| Local Intent classification | ✅ |
| Paraphrase handling ("buy groceries" = "order food") | ✅ |
| Fail-closed credential guard (dual Kotlin + Dart) | ✅ |
| Popup/dialog auto-dismiss during replay | ✅ |
| Scroll-to-find target elements | ✅ |
| Ask-when-stuck clarification dialogs | ✅ |
| Noise filtering during teach (system UI, duplicates) | ✅ |
| Manual STOP kill-switch during execution | ✅ |
| Flow library — view, expand, delete saved flows | ✅ |
| Session logging (teach & replay history) | ✅ |
| Layer | Technology |
|---|---|
| UI & app logic | Flutter 3.x (Dart), Material 3 |
| State management | Provider (ChangeNotifier) |
| Screen automation | Native Kotlin AccessibilityService |
| Flutter ↔ Kotlin bridge | MethodChannel + EventChannel |
| On-device ASR | speech_to_text plugin |
| Local NLU Pipeline | SAAR-NLU (Regex baseline / Future ONNX) |
| Local storage | sqflite (flows + session logs) |
SAAR/
├── pubspec.yaml # Flutter dependencies
├── README.md # This file
│
├── android/
│ └── app/src/main/
│ ├── AndroidManifest.xml # Permissions + service declaration
│ ├── res/xml/
│ │ └── accessibility_service_config.xml # Accessibility service config
│ └── kotlin/com/lesgo/saar/
│ ├── SaarAccessibilityService.kt # UI tree capture, gestures, credential guard
│ └── MainActivity.kt # MethodChannel + EventChannel bridge
│
├── lib/
│ ├── main.dart # Entry point, Provider
│ ├── app_controller.dart # Central state machine
│ │
│ ├── models/
│ │ ├── flow.dart # Flow, FlowStep, Slot
│ │ ├── ui_node.dart # Accessibility tree node
│ │ ├── action_trace_event.dart # Teach-session event
│ │ └── role_ontology.dart # UI roles + heuristic matcher
│ │
│ ├── services/
│ │ ├── accessibility_bridge.dart # Typed wrapper over native channels
│ │ ├── asr_service.dart # Speech-to-text push-to-talk
│ │ ├── flow_store.dart # SQLite CRUD
│ │ ├── flow_matcher.dart # Flow matching logic
│ │ ├── flow_synthesizer.dart # Action trace → abstract flow
│ │ ├── replay_engine.dart # Step executor with guard + adaptation
│ │ ├── credential_guard.dart # Fail-closed sensitive field check
│ │ └── saar_nlu.dart # On-device natural language understanding
│ │
│ └── screens/
│ ├── home_screen.dart # Mic button, status, flow list
│ ├── teach_screen.dart # Recording indicator, Stop & Save
│ ├── replay_screen.dart # Step progress, STOP button, clarification
│ ├── flow_library_screen.dart # List / expand / delete flows
│ ├── report_screen.dart # Execution reports UI
│ └── settings_screen.dart # Accessibility toggle
│
└── test/
└── models_test.dart
└── services_test.dart
- Flutter 3.13+ with Dart 3.13+
- Android SDK (API level 24+ / Android 7.0+)
- Android device or emulator with Accessibility support
git clone https://github.com/Lesgo-HQ/SAAR.git
cd SAARflutter pub getflutter analyzeflutter run# Debug APK (faster build, larger size)
flutter build apk --debug
# Release APK
flutter build apk --release- Launch SAAR → you'll see a red banner: "Accessibility service disabled"
- Tap ENABLE
- Toggle ON and tap Allow on the confirmation dialog
- Return to SAAR — the banner should disappear and the status shows green
The first time you tap the mic button, Android will prompt for microphone permission. Tap Allow.
- Open SAAR
- Tap the mic button 🎤
- Say: "teach me to search on Amazon"
- SAAR will show "Recording your actions..." and switch you to teach mode
- Switch to Amazon
- Perform the task: tap the search bar → type "headphones" → tap search → tap a result
- Switch back to SAAR
- Tap the red Stop & Save button
- Tap the mic button 🎤
- Say: "search for wireless earbuds on Amazon"
- SAAR will match your utterance to the learned flow, fill slots, and execute step-by-step.
- Navigate to any app's login page (with a password field visible)
- Try to replay a flow — SAAR will immediately halt with a safety message.
SAAR operates entirely on-device with zero cloud dependencies at runtime, ensuring maximal privacy and security.
flowchart TD
User([User]) -->|Speech| ASR[ASR System]
ASR -->|Text Query| NLU[SAAR-NLU]
NLU -->|ParsedIntent| FM[Flow Matcher]
FM -->|Flow ID| SE[Script Engine]
SE -->|Flow Steps| AS[Accessibility Service]
AS -->|Inject Events| App[Target App]
AS -->|Screen Nodes| SE
The credential guard runs as a dual-layer, deterministic, fail-closed check before every single dispatched action.
- Runs in Dart before calling the bridge.
- Runs natively in Kotlin before gesture dispatch.
- Regex-based NLU Baseline: The current NLU is based on regex (
saar-nlu-lite). It is not a trained neural model. - Generalization: Cross-app generalization is currently an architectural design rather than a physically demonstrated capability.
- ASR Network Dependency: ASR relies on Android's native
SpeechRecognizer, which may require network. - No Overlay UI: Currently, there is no floating overlay UI to pause/resume.
- Trace Quality: Flow synthesis quality relies heavily on clean action traces during teaching.
- Domain Restriction: Largely limited to e-commerce patterns currently.
Built for hackathon use. See repository for license details.