Buddy iOS Agent

On-device iOS AI assistant starter built for voice sessions, reminders, notes, widgets, and App Intents with Gemma-ready architecture.

GitHub Search Keywords: ios ai assistant, gemma ios app, swiftui voice assistant, on device ai, app intents, live activities, ios productivity app, mediapipe gemma

Documentation · Architecture · Contributing · Security · Authors · Workflows

Buddy is an App-Store-safe starter repo for building a custom on-device iOS assistant with Gemma. The app focuses on daily productivity tasks like note capture, reminder creation, quick task summaries, and lightweight voice interaction on modern iPhones.

This repo intentionally does not claim unsupported iOS behavior like a permanent system overlay or continuous 24/7 background listening. Instead, it implements the closest compliant experience: a SwiftUI app, App Intents, widget entry points, a Live Activity during active sessions, local storage, and an on-device Gemma adapter wired for MediaPipe LLM Inference.

What ships in v1

• SwiftUI app scaffold with conversation, onboarding, settings, and quick capture flows
• BuddyCore framework for assistant models, prompt assembly, wake phrase matching, response decoding, and JSON-backed stores
• App Intents for Talk to Buddy, Quick Note, and New Reminder
• Widget + Live Activity extension
• EventKit-backed reminder creation and local file-backed notes
• MediaPipe-ready Gemma engine adapter with a preview fallback for local development
• GitHub-ready docs, templates, CI, and release checklist

Repo layout

BuddyApp/
  Sources/
    BuddyApp/        SwiftUI app + iOS services
    BuddyIntents/    App Intents shared by app and widget
    BuddyShared/     Shared ActivityKit types
    BuddyWidget/     Widget + Live Activity extension
  Support/           plist and entitlement files
  Tests/             app unit and UI tests
BuddyCore/
  Sources/BuddyCore/ platform-neutral assistant domain logic
  Tests/BuddyCoreTests/
docs/
.github/
project.yml
Podfile

Product constraints

• Wake phrase detection is scoped to an active voice session.
• Voice capture starts only after user action from the app, widget, or shortcut.
• The assistant name defaults to Buddy and can be renamed.
• Notes stay local on device.
• Reminders are written into Apple Reminders after permission is granted.
• Model assets are not tracked in git.

Setup

1. Install Xcode 15 or later on macOS.
2. Install XcodeGen and CocoaPods.
3. Clone this repo.
4. Run xcodegen generate.
5. Run pod install.
6. Place your local Gemma model in Models/.
7. Open BuddyApp.xcworkspace in Xcode.
8. Build and run on a recent iPhone or iOS 17+ simulator.

See docs/setup-models.md, docs/architecture.md, and docs/roadmap.md for details.

Model setup

The default integration target is a small on-device Gemma variant suited to phone-class hardware:

• FunctionGemma 270M as the action-routing default
• MediaPipe LLM Inference on iOS via the MediaPipeTasksGenAI and MediaPipeTasksGenAIC CocoaPods

Keep model assets outside git in Models/. The app falls back to a preview engine when no local model file is present so the UI and flows can still be developed.

CI

GitHub Actions will:

• generate the Xcode project with XcodeGen
• install CocoaPods dependencies
• fail if tracked model binaries appear under Models/
• run unit and UI test targets on an iOS simulator

Limitations

• No system-wide overlay on top of other apps
• No always-listening background hotword detector
• No server-hosted inference in v1
• No long-term autonomous memory

References

• Apple background execution modes
• Apple EventKit access guidance
• Google Gemma mobile integrations
• Google MediaPipe LLM Inference for iOS