diff --git a/README.md b/README.md index 6da838b68d..6f6270d327 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,8 @@ The library ships with a curated set of pre-exported models covering computer vision, language models, text-to-speech, transcription, and more — all available in our [Hugging Face collection](https://huggingface.co/software-mansion/collections) and ready to use out of the box. You can also bring your own `.pte` models and plug them into existing pipelines or build entirely custom ones from scratch. +To export your own PyTorch models to `.pte`, use [ExecuTorch Export Recipes](https://github.com/software-mansion-labs/executorch-export-recipes), our export harness with model recipes and agent skills. + To explore all on-device capabilities in an interactive showcase app, check out the [React Native ExecuTorch Gallery](https://github.com/software-mansion-labs/react-native-executorch-gallery). [![npm version](https://img.shields.io/npm/v/react-native-executorch?color=00008B)](https://www.npmjs.com/package/react-native-executorch) @@ -54,6 +56,19 @@ To explore all on-device capabilities in an interactive showcase app, check out ## Quickstart +> [!TIP] +> **Using an AI coding agent?** The consumer skill helps your agent choose ready-made models and task +> hooks, integrate a custom `.pte`, or build a custom TypeScript pipeline. +> +> Install the consumer skill from the Software Mansion skills repository: +> +> ```bash +> npx skills add software-mansion-labs/skills -s react-native-executorch +> ``` +> +> See [AI Tools](https://docs.swmansion.com/react-native-executorch/docs/next/fundamentals/ai-tools) +> for model-selection guidance and an example prompt. + ### 1. Installation Install `react-native-executorch` alongside its required peer dependencies: @@ -70,8 +85,8 @@ pnpm add react-native-executorch react-native-worklets react-native-blob-util > React Native ExecuTorch requires: > - **New Architecture** enabled > - **React Native 0.83+** or **Expo SDK 55+** with [Development Builds](https://docs.expo.dev/develop/development-builds/introduction/) (**Expo Go is not supported** due to custom C++ native libraries) -> - **`react-native-worklets` 0.10 or newer** (`>=0.10.0 <0.13.0`) -> - **iOS 17.0+** / **Android 13+** (`minSdkVersion` >= 26) +> - **`react-native-worklets` 0.10 or newer** (`>=0.10.0 <0.14.0`) +> - **iOS 17.0+** / **Android 8.0+** (`minSdkVersion` >= 26) > - **`react-native-worklets/plugin`** as the last entry in your Babel `plugins` > > Downloads of our pre-exported models send an anonymous download event; call `setTelemetryEnabled(false)` to opt out. @@ -112,6 +127,10 @@ export function App() { ## Documentation +**Building with an AI coding agent?** See [AI Tools](https://docs.swmansion.com/react-native-executorch/docs/next/fundamentals/ai-tools) +for model-selection guidance and an example prompt. +Choose a pre-exported model with a task hook, or bring your own `.pte` and build a custom TypeScript pipeline. + Full documentation, guides, architecture deep dives, and API references are available at: **[docs.swmansion.com/react-native-executorch](https://docs.swmansion.com/react-native-executorch/)** diff --git a/docs/docs/01-fundamentals/01-getting-started.md b/docs/docs/01-fundamentals/01-getting-started.md index 2c0abb2b24..a7548d3ecb 100644 --- a/docs/docs/01-fundamentals/01-getting-started.md +++ b/docs/docs/01-fundamentals/01-getting-started.md @@ -16,6 +16,8 @@ keywords: description: 'Get started with React Native ExecuTorch — high-performance, privacy-first on-device AI inference for React Native.' --- + + import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; @@ -33,6 +35,19 @@ computer vision, language models, text-to-speech, transcription, and more — al and ready to use out of the box. You can also bring your own models and plug them into existing pipelines or build entirely custom ones from scratch. +:::tip Using an AI coding agent? +The consumer skill helps your agent choose ready-made models and task hooks, integrate a custom +`.pte`, or build a custom TypeScript pipeline. + +Install the consumer skill from the Software Mansion skills repository: + +```bash +npx skills add software-mansion-labs/skills -s react-native-executorch +``` + +See [AI Tools](./05-ai-tools.md) for model-selection guidance and an example prompt. +::: + ## What is ExecuTorch? [ExecuTorch](https://executorch.ai) is a PyTorch Core project — Meta's on-device @@ -86,7 +101,7 @@ React Native ExecuTorch requires: - **New Architecture** enabled - **React Native 0.83+** or **Expo SDK 55+** with [Development Builds](https://docs.expo.dev/develop/development-builds/introduction/) (**Expo Go is not supported** due to custom C++ native libraries) - **`react-native-worklets` 0.10 or newer** (`>=0.10.0 <0.14.0`) -- **iOS 17.0+** / **Android 13+** (`minSdkVersion` >= 26); a bare app's +- **iOS 17.0+** / **Android 8.0+** (`minSdkVersion` >= 26); a bare app's `Podfile` needs `platform :ios, '17.0'` - **`tar` on `PATH`** — the install-time download extracts with it; macOS, Linux and Windows 10 1803+ all ship one. Windows and Linux hosts can build for diff --git a/docs/docs/01-fundamentals/05-ai-tools.md b/docs/docs/01-fundamentals/05-ai-tools.md new file mode 100644 index 0000000000..0e88189e23 --- /dev/null +++ b/docs/docs/01-fundamentals/05-ai-tools.md @@ -0,0 +1,51 @@ +--- +title: AI Tools +slug: /fundamentals/ai-tools +description: 'Use React Native ExecuTorch with a coding agent: choose ready-to-use models and hooks, bring your own .pte, and retrieve documentation for the installed version.' +--- + +# AI Tools + +React Native ExecuTorch gives your coding agent two starting points: + +- **Use a ready-made task.** Choose a [pre-exported model](./04-supported-models.md) and a task hook + for chat, vision, speech or embeddings. The task handles model loading, preprocessing, inference + and postprocessing; the hook manages downloads and React lifecycle. +- **Bring your own model.** [Export a compatible `.pte`](../03-core-and-advanced/07-exporting-custom-models.md), + inspect its schema, and reuse a built-in task if its contract fits. Otherwise build a custom + TypeScript pipeline using [models and tensors](../03-core-and-advanced/02-models-and-tensors.md), + [schema validation](../03-core-and-advanced/03-schema-validation.md) and + [worklets](../03-core-and-advanced/06-worklets-and-threading.md). + +These paths use the same runtime. You can start with a shipped model and replace it with your own +without rewriting the application when the input/output contract stays compatible. + +## Consumer skill + +The consumer skill teaches an agent how to choose between these paths, prepare a React Native app, +handle native resources, and verify model behavior. Install it from the +[Software Mansion skills repository](https://github.com/software-mansion-labs/skills). + +```bash +npx skills add software-mansion-labs/skills -s react-native-executorch +``` + +The skill is optional; the library and its documentation work independently of skill installation. + +For existing apps, use documentation matching the installed react-native-executorch version. + +## A prompt to start with + +> Create a new React Native app using the latest stable react-native-executorch release and its +> matching documentation. Help me build [describe the feature]. Choose a pre-exported model and task +> hook if one fits; otherwise use +> [ExecuTorch Export Recipes](https://github.com/software-mansion-labs/executorch-export-recipes) +> to export my custom model and build the mobile pipeline. Configure the required native dependencies +> and verify inference on the target device. + +For a custom model, provide its PyTorch definition or exported `.pte`, sample inputs and expected +outputs, preprocessing/tokenizer assets, and target devices. Use +[ExecuTorch Export Recipes](https://github.com/software-mansion-labs/executorch-export-recipes) and its +agent skills for the export step, then follow +[Exporting Custom Models](../03-core-and-advanced/07-exporting-custom-models.md) to integrate and verify +the result in your app. diff --git a/docs/docs/03-core-and-advanced/07-exporting-custom-models.md b/docs/docs/03-core-and-advanced/07-exporting-custom-models.md index 34dc6656cd..bf15bce46b 100644 --- a/docs/docs/03-core-and-advanced/07-exporting-custom-models.md +++ b/docs/docs/03-core-and-advanced/07-exporting-custom-models.md @@ -16,30 +16,60 @@ keywords: # Exporting Custom Models -React Native ExecuTorch runs any ExecuTorch `.pte` file, not just the models in +React Native ExecuTorch runs compatible ExecuTorch `.pte` files, including models outside our [HuggingFace collection](https://huggingface.co/software-mansion/collections). Getting your own model running takes two things: -1. **Export it to `.pte`** with a hardware backend — the standard ExecuTorch - workflow. +1. **Export it to `.pte`** for your target backend using + [ExecuTorch Export Recipes](https://github.com/software-mansion-labs/executorch-export-recipes). 2. **Integrate it** — either feed it into a built-in pipeline by matching that pipeline's contract, or build your own pipeline around it, where you define the contract. Both come down to the model's schema. -The export step is plain ExecuTorch and is documented thoroughly upstream, so this -page links out for it and focuses on the second part — integrating a custom model, -into a built-in pipeline or one you build yourself — which is where the -library-specific work lives. +The exporter provides model recipes, pinned environments and agent skills for the +PyTorch → ExecuTorch step. This guide connects that workflow to integrating the +result in a built-in pipeline or one you build yourself. + +## Choose your integration level + +| Starting point | Integration | +| -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | +| A supported task and a model from the catalog fit your goal | Use a [pre-exported model and task hook](../01-fundamentals/04-supported-models.md) | +| Your own model matches a built-in task's schema and preprocessing contract | Supply a custom configuration to that task's hook or factory | +| Your model needs different inputs, outputs or orchestration | Build a custom TypeScript pipeline on the model, tensor and worklet primitives | + +You can mix these approaches in one application. A custom model does not have to be contributed to the +library registry, and a custom pipeline generally does not require app-specific C++. + +## Keep export and mobile integration in agreement + +Carry these artifacts from PyTorch through export to the app: + +- The `.pte` and every required tokenizer, vocabulary, label map or other auxiliary asset. +- Exported method names, ordered inputs/outputs, data types, layouts, shapes and dynamic bounds. +- Preprocessing and output-decoding rules, such as normalization, image resize/padding or sample rate. +- The export/runtime version, backend requirements, and representative inputs with reference outputs. + +Use an ExecuTorch export version and operators compatible with the native runtime shipped by the +installed React Native ExecuTorch package. Match its enabled backends and target hardware; a `.pte` +extension or successful schema inspection alone does not establish execution compatibility. See +[Native Libraries](./08-native-libraries.md) for app backend selection. + +Preserve that contract when exporting. Compare exported outputs against the PyTorch reference before +moving to mobile, with tolerances appropriate to the selected precision/backend. ## Exporting to `.pte` -Producing the `.pte` is the standard PyTorch → ExecuTorch path: -[`torch.export`](https://docs.pytorch.org/docs/2.14/user_guide/torch_compiler/export.html), -then lowering to a backend (XNNPACK, Core ML, Vulkan, …) with -[`to_edge_transform_and_lower`](https://docs.pytorch.org/executorch/stable/export-to-executorch-api-reference.html#executorch.exir.to_edge_transform_and_lower). -It is not specific to this library, so follow the official -[ExecuTorch documentation](https://docs.pytorch.org/executorch/stable/index.html) -for the details, including which backend to target for your model and platform. +Use [ExecuTorch Export Recipes](https://github.com/software-mansion-labs/executorch-export-recipes), +our export harness, model recipes and agent skills. Its +[README](https://github.com/software-mansion-labs/executorch-export-recipes/blob/docs/public-repo-foundations/README.md) +covers setup and exporting; its repository instructions and skills guide agents through adding recipes, +choosing a backend and validating outputs. Follow those instructions for your model and target device, +and check compatibility with the native runtime shipped by your installed React Native ExecuTorch package. + +For the underlying PyTorch → ExecuTorch APIs, see the official +[ExecuTorch documentation](https://docs.pytorch.org/executorch/stable/index.html). +Once validated, bring the `.pte` and the assets listed above into your React Native app. What matters here is the **contract** the resulting `.pte` exposes: the input and output shapes, data types, and any dynamic dimensions or dimension relationships. @@ -359,6 +389,10 @@ Before wiring a custom model into a screen, confirm two things on the device: Once both check out, load the model into the target pipeline and run it. +Compare a known inference result with the PyTorch/exported reference on the actual target backend. +Also verify preprocessing, repeated calls, cancellation where supported, and resource cleanup. Export, +inspection, a TypeScript build and device inference establish different parts of the workflow. + ## Where to go next - [Schema Validation](./03-schema-validation.md) — the contract a custom model must satisfy, in depth. diff --git a/docs/docs/05-other/01-compatibility.mdx b/docs/docs/05-other/01-compatibility.mdx index 0f75b0e720..b3674e4c5f 100644 --- a/docs/docs/05-other/01-compatibility.mdx +++ b/docs/docs/05-other/01-compatibility.mdx @@ -138,7 +138,7 @@ it. On SDK 55 and 56, install both: ```bash -npm install react-native-worklets@^0.10.0 react-native-reanimated@^4.5.0 +npm install react-native-worklets@^0.10.0 react-native-reanimated@~4.5.0 ``` SDK 54 is React Native 0.81, below what worklets 0.10 accepts, so no