From 72878d3282617d65eb2366312bf83216fd480e9c Mon Sep 17 00:00:00 2001 From: Bartosz Hanc Date: Mon, 2 Nov 2026 17:17:56 +0100 Subject: [PATCH 01/11] docs: add consumer agent skill and version-aware documentation --- README.md | 8 +- docs/README.md | 37 ++++- .../01-fundamentals/01-getting-started.md | 6 +- docs/docs/01-fundamentals/05-ai-tools.md | 79 +++++++++ .../07-exporting-custom-models.md | 35 +++- docs/docusaurus.config.js | 9 +- docs/scripts/postbuild.mjs | 6 +- docs/scripts/tasks/agent-docs.mjs | 116 +++++++++++++ docs/scripts/tasks/fix-llms-txt.mjs | 38 ----- docs/scripts/tasks/llms-txt-models.mjs | 101 ------------ .../src/components/Hero/StartScreen/index.tsx | 10 +- docs/src/theme/DocItem/Metadata/index.js | 23 ++- skills/react-native-executorch/SKILL.md | 83 ++++++++++ .../references/custom-models.md | 154 ++++++++++++++++++ .../references/ready-made-tasks.md | 92 +++++++++++ .../references/setup.md | 68 ++++++++ .../references/troubleshooting.md | 29 ++++ 17 files changed, 737 insertions(+), 157 deletions(-) create mode 100644 docs/docs/01-fundamentals/05-ai-tools.md create mode 100644 docs/scripts/tasks/agent-docs.mjs delete mode 100644 docs/scripts/tasks/fix-llms-txt.mjs delete mode 100644 docs/scripts/tasks/llms-txt-models.mjs create mode 100644 skills/react-native-executorch/SKILL.md create mode 100644 skills/react-native-executorch/references/custom-models.md create mode 100644 skills/react-native-executorch/references/ready-made-tasks.md create mode 100644 skills/react-native-executorch/references/setup.md create mode 100644 skills/react-native-executorch/references/troubleshooting.md diff --git a/README.md b/README.md index 6da838b68d..4cf7a8b117 100644 --- a/README.md +++ b/README.md @@ -70,8 +70,8 @@ pnpm add react-native-executorch react-native-worklets react-native-blob-util > React Native ExecuTorch requires: > - **New Architecture** enabled > - **React Native 0.83+** or **Expo SDK 55+** with [Development Builds](https://docs.expo.dev/develop/development-builds/introduction/) (**Expo Go is not supported** due to custom C++ native libraries) -> - **`react-native-worklets` 0.10 or newer** (`>=0.10.0 <0.13.0`) -> - **iOS 17.0+** / **Android 13+** (`minSdkVersion` >= 26) +> - **`react-native-worklets` 0.10 or newer** (`>=0.10.0 <0.14.0`) +> - **iOS 17.0+** / **Android 8.0+** (`minSdkVersion` >= 26) > - **`react-native-worklets/plugin`** as the last entry in your Babel `plugins` > > Downloads of our pre-exported models send an anonymous download event; call `setTelemetryEnabled(false)` to opt out. @@ -112,6 +112,10 @@ export function App() { ## Documentation +**Building with an AI coding agent?** See [AI Tools](https://docs.swmansion.com/react-native-executorch/docs/next/fundamentals/ai-tools) +for the consumer skill, version selection, and [Markdown documentation](https://docs.swmansion.com/react-native-executorch/llms.txt). +Choose a pre-exported model with a task hook, or bring your own `.pte` and build a custom TypeScript pipeline. + Full documentation, guides, architecture deep dives, and API references are available at: **[docs.swmansion.com/react-native-executorch](https://docs.swmansion.com/react-native-executorch/)** diff --git a/docs/README.md b/docs/README.md index 44d4895430..3e346df5d4 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,5 +1,7 @@ # React Native ExecuTorch Documentation + + This directory contains the source code for the documentation website of React Native ExecuTorch, built with [Docusaurus](https://docusaurus.io/). ## Getting Started @@ -80,19 +82,44 @@ We use Docusaurus versioning to "freeze" documentation for older releases. ### Creating a New Version -When you release a new npm version (e.g., `1.0.0`), you must snapshot the documentation. Run this command from the `docs/` folder: +When introducing a documentation release line (for example, `1.0.x`), snapshot the documentation. +Run this command from the `docs/` folder after generating the current API reference with `yarn build`: ```bash -yarn docs:version 1.0.0 +yarn docs:version 1.0.x ``` ### Why is this important? -1. It copies the current contents of `docs/` (including the **currently generated API reference**) into `versioned_docs/version-1.0.0`. -2. This ensures that if the TypeScript API changes in the future, the docs for v1.0.0 remain accurate to that version. +1. It copies the current contents of `docs/` (including the **currently generated API reference** and + generated Supported Models page) into `versioned_docs/version-1.0.x`. +2. It pins GitHub `/blob/main/` links to the snapshot commit. Later TypeScript changes affect Next, + not the snapshot. Correct a released line's documentation in its own snapshot when needed. +3. Set `lastVersion` in `docusaurus.config.js` when that line becomes the default released docs. + Existing release lines are listed in `versions.json`; do not create a snapshot for every npm patch. ### Working on "Next" -Edit files in `docs/` as usual. These changes appear in the "Next" version of the site (the default view), reflecting the current `main` branch. +Edit files in `docs/` as usual. They appear under `/docs/next/`, labeled **Next**, and reflect the +development source. The default `/docs/` routes currently serve the `0.11.x` snapshot selected by +`lastVersion`; older lines retain `/docs//` routes. A development package version such +as `1.0.0` does not change the default documentation line automatically. + +## Agent documentation + +The LLM plugin emits `.md` representations for all documentation versions, including API symbols. +`scripts/postbuild.mjs` then reads Docusaurus's generated `globalData.json` version/route metadata and +creates one compact `/llms.txt` router for both integration paths and documentation versions. +Model catalogs remain in each version's Supported Models page; the stable index never reads the +development model registry. Every Markdown page identifies its documentation version. + +The post-build step fails when a documented page or generated index target is missing/empty. The normal +`yarn build` runs this validation, including in documentation CI. Verify the stable, Next and historical +routes when changing the plugin or Docusaurus, because the generator depends on its route metadata. + +The consumer skill under `../skills/react-native-executorch/` is prepared for a separate PR to +`software-mansion-labs/skills`. Publish that skill before changing the AI Tools page's pending status +to advertise an available installation. It is separate from the internal `.agents/skills` and is not +bundled into the npm package. ## Troubleshooting diff --git a/docs/docs/01-fundamentals/01-getting-started.md b/docs/docs/01-fundamentals/01-getting-started.md index 2c0abb2b24..6bfccd9fa1 100644 --- a/docs/docs/01-fundamentals/01-getting-started.md +++ b/docs/docs/01-fundamentals/01-getting-started.md @@ -16,6 +16,8 @@ keywords: description: 'Get started with React Native ExecuTorch — high-performance, privacy-first on-device AI inference for React Native.' --- + + import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; @@ -33,6 +35,8 @@ computer vision, language models, text-to-speech, transcription, and more — al and ready to use out of the box. You can also bring your own models and plug them into existing pipelines or build entirely custom ones from scratch. +Building with a coding agent? See [AI Tools](./05-ai-tools.md) for consumer guidance, version selection, and Markdown documentation. + ## What is ExecuTorch? [ExecuTorch](https://executorch.ai) is a PyTorch Core project — Meta's on-device @@ -86,7 +90,7 @@ React Native ExecuTorch requires: - **New Architecture** enabled - **React Native 0.83+** or **Expo SDK 55+** with [Development Builds](https://docs.expo.dev/develop/development-builds/introduction/) (**Expo Go is not supported** due to custom C++ native libraries) - **`react-native-worklets` 0.10 or newer** (`>=0.10.0 <0.14.0`) -- **iOS 17.0+** / **Android 13+** (`minSdkVersion` >= 26); a bare app's +- **iOS 17.0+** / **Android 8.0+** (`minSdkVersion` >= 26); a bare app's `Podfile` needs `platform :ios, '17.0'` - **`tar` on `PATH`** — the install-time download extracts with it; macOS, Linux and Windows 10 1803+ all ship one. Windows and Linux hosts can build for diff --git a/docs/docs/01-fundamentals/05-ai-tools.md b/docs/docs/01-fundamentals/05-ai-tools.md new file mode 100644 index 0000000000..ff7a628aaf --- /dev/null +++ b/docs/docs/01-fundamentals/05-ai-tools.md @@ -0,0 +1,79 @@ +--- +title: AI Tools +slug: /fundamentals/ai-tools +description: 'Use React Native ExecuTorch with a coding agent: choose ready-to-use models and hooks, bring your own .pte, and retrieve documentation for the installed version.' +--- + +# AI Tools + +React Native ExecuTorch gives your coding agent two starting points: + +- **Use a ready-made task.** Choose a [pre-exported model](./04-supported-models.md) and a task hook + for chat, vision, speech or embeddings. The task handles model loading, preprocessing, inference + and postprocessing; the hook manages downloads and React lifecycle. +- **Bring your own model.** [Export a compatible `.pte`](../03-core-and-advanced/07-exporting-custom-models.md), + inspect its schema, and reuse a built-in task if its contract fits. Otherwise build a custom + TypeScript pipeline using [models and tensors](../03-core-and-advanced/02-models-and-tensors.md), + [schema validation](../03-core-and-advanced/03-schema-validation.md) and + [worklets](../03-core-and-advanced/06-worklets-and-threading.md). + +These paths use the same runtime. You can start with a shipped model and replace it with your own +without rewriting the application when the input/output contract stays compatible. + +## Consumer skill + +The consumer skill teaches an agent how to choose between these paths, prepare a React Native app, +handle native resources, and verify model behavior. It uses the same distribution approach as the +TypeGPU skill in the [Software Mansion skills repository](https://github.com/software-mansion-labs/skills). + +The React Native ExecuTorch skill is being prepared for that repository. Once published, install it with: + +```bash +npx skills add software-mansion-labs/skills -s react-native-executorch +``` + +Until it is published, give your agent the versioned documentation links below. The skill is optional; +the library and its documentation work independently of skill installation. + +## Select the documentation version + +Inspect the version installed in the application, including its lockfile. Documentation is organized by +release line, not by every npm patch: + +- `/react-native-executorch/docs/...` is the **default released line**, selected by the site's version menu. +- `/react-native-executorch/docs/next/...` is **Next**, the unreleased development documentation. +- `/react-native-executorch/docs//...` contains older snapshots, for example `0.10.x`. + +Use the version menu or the [agent documentation index](https://docs.swmansion.com/react-native-executorch/llms.txt) to find the +matching line. The default line changes as releases advance. Do not combine an older installed package +with examples from Next, or assume a development checkout's package version is the current npm release. + +## Markdown and focused context + +Each documentation page, including API reference pages, has a Markdown representation. Append `.md` +to its URL, keeping the version segment. For example: + +```text +https://docs.swmansion.com/react-native-executorch/docs/fundamentals/getting-started.md +https://docs.swmansion.com/react-native-executorch/docs/next/fundamentals/getting-started.md +``` + +Use **Copy Page → View Markdown** for the page you are reading. The +[root index](https://docs.swmansion.com/react-native-executorch/llms.txt) links the default release, Next and historical +documentation. Start with a relevant guide and fetch API details as needed. + +## A prompt to start with + +```text +Help me build with react-native-executorch. First inspect my installed package version, +React Native/Expo setup, and target platform, then use the matching documentation. +Determine whether a pre-exported model and task hook fits my goal. If I have a custom +PyTorch or .pte model, inspect its input/output contract and choose either an existing +task or a custom TypeScript pipeline. Verify export/runtime/backend compatibility, +resource cleanup, and a representative inference result on the target device. +``` + +For a custom model, provide its PyTorch definition or exported `.pte`, sample inputs and expected +outputs, preprocessing/tokenizer assets, and target devices. The current export path uses the official +ExecuTorch tooling linked from [Exporting Custom Models](../03-core-and-advanced/07-exporting-custom-models.md). +An agent should use only published, documented exporter commands. diff --git a/docs/docs/03-core-and-advanced/07-exporting-custom-models.md b/docs/docs/03-core-and-advanced/07-exporting-custom-models.md index 34dc6656cd..c2f23efa49 100644 --- a/docs/docs/03-core-and-advanced/07-exporting-custom-models.md +++ b/docs/docs/03-core-and-advanced/07-exporting-custom-models.md @@ -16,7 +16,7 @@ keywords: # Exporting Custom Models -React Native ExecuTorch runs any ExecuTorch `.pte` file, not just the models in +React Native ExecuTorch runs compatible ExecuTorch `.pte` files, including models outside our [HuggingFace collection](https://huggingface.co/software-mansion/collections). Getting your own model running takes two things: @@ -31,6 +31,35 @@ page links out for it and focuses on the second part — integrating a custom mo into a built-in pipeline or one you build yourself — which is where the library-specific work lives. +## Choose your integration level + +| Starting point | Integration | +| -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | +| A supported task and a model from the catalog fit your goal | Use a [pre-exported model and task hook](../01-fundamentals/04-supported-models.md) | +| Your own model matches a built-in task's schema and preprocessing contract | Supply a custom configuration to that task's hook or factory | +| Your model needs different inputs, outputs or orchestration | Build a custom TypeScript pipeline on the model, tensor and worklet primitives | + +You can mix these approaches in one application. A custom model does not have to be contributed to the +library registry, and a custom pipeline generally does not require app-specific C++. + +## Keep export and mobile integration in agreement + +Carry these artifacts from PyTorch through export to the app: + +- The `.pte` and every required tokenizer, vocabulary, label map or other auxiliary asset. +- Exported method names, ordered inputs/outputs, data types, layouts, shapes and dynamic bounds. +- Preprocessing and output-decoding rules, such as normalization, image resize/padding or sample rate. +- The export/runtime version, backend requirements, and representative inputs with reference outputs. + +Use an ExecuTorch export version and operators compatible with the native runtime shipped by the +installed React Native ExecuTorch package. Match its enabled backends and target hardware; a `.pte` +extension or successful schema inspection alone does not establish execution compatibility. See +[Native Libraries](./08-native-libraries.md) for app backend selection. + +Whether exporting manually or using a project-provided harness, preserve that same contract. Follow +only the harness's published commands and version pins. Compare exported outputs against the PyTorch +reference before moving to mobile, with tolerances appropriate to the selected precision/backend. + ## Exporting to `.pte` Producing the `.pte` is the standard PyTorch → ExecuTorch path: @@ -359,6 +388,10 @@ Before wiring a custom model into a screen, confirm two things on the device: Once both check out, load the model into the target pipeline and run it. +Compare a known inference result with the PyTorch/exported reference on the actual target backend. +Also verify preprocessing, repeated calls, cancellation where supported, and resource cleanup. Export, +inspection, a TypeScript build and device inference establish different parts of the workflow. + ## Where to go next - [Schema Validation](./03-schema-validation.md) — the contract a custom model must satisfy, in depth. diff --git a/docs/docusaurus.config.js b/docs/docusaurus.config.js index f659bbee2f..cad81ebe52 100644 --- a/docs/docusaurus.config.js +++ b/docs/docusaurus.config.js @@ -81,11 +81,16 @@ const config = { ({ markdown: { enableFiles: true, - excludeRoutes: ['**/react-native-executorch/search', '**/api-reference/**'], - includeVersionedDocs: false, + excludeRoutes: ['**/react-native-executorch/search'], + includeVersionedDocs: true, + includeGeneratedIndex: false, relativePaths: false, }, llmsTxt: { + // Postbuild replaces the generated index with one compact version-aware router. + // Markdown pages retain the site's existing release/Next routes. + includeVersionedDocs: true, + includeGeneratedIndex: false, siteTitle: 'React Native ExecuTorch', siteDescription: "React Native ExecuTorch is Software Mansion's library for running AI models on-device in React Native, built on Meta's ExecuTorch (PyTorch Edge). One package covers LLMs and VLMs (streaming chat, tool calling, thinking mode, constrained sampling), speech (speech-to-text, text-to-speech, voice activity detection), computer vision, image generation and inpainting, and local RAG (text, image and ColBERT embeddings), fully offline on iOS and Android. Unlike bindings to a single runtime, which leave model export, tokenizers and pre/post-processing to the app, it ships a registry of pre-exported models with typed hooks for each task and per-device benchmarks, and still loads any custom ExecuTorch .pte.", diff --git a/docs/scripts/postbuild.mjs b/docs/scripts/postbuild.mjs index f5219215a3..c9b8b1b0ac 100644 --- a/docs/scripts/postbuild.mjs +++ b/docs/scripts/postbuild.mjs @@ -1,7 +1,6 @@ import { join, dirname } from 'path'; import { fileURLToPath } from 'url'; -import { fixLLMsTxtDuplicatedPaths } from './tasks/fix-llms-txt.mjs'; -import { addModelsToLLMsTxt } from './tasks/llms-txt-models.mjs'; +import { generateAgentDocs } from './tasks/agent-docs.mjs'; const __filename = fileURLToPath(import.meta.url); const __dirname = dirname(__filename); @@ -9,8 +8,7 @@ const BUILD_DIR = join(__dirname, '..', 'build'); async function postbuild() { try { - fixLLMsTxtDuplicatedPaths(BUILD_DIR); - addModelsToLLMsTxt(BUILD_DIR); + generateAgentDocs(BUILD_DIR); console.log('All post-build tasks completed successfully.\n'); } catch (error) { console.error('\nPost-build script failed:', error); diff --git a/docs/scripts/tasks/agent-docs.mjs b/docs/scripts/tasks/agent-docs.mjs new file mode 100644 index 0000000000..87b3485dd8 --- /dev/null +++ b/docs/scripts/tasks/agent-docs.mjs @@ -0,0 +1,116 @@ +import { readFileSync, writeFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { createRequire } from 'node:module'; +import { fileURLToPath } from 'node:url'; + +const docsDir = join(dirname(fileURLToPath(import.meta.url)), '..', '..'); +const config = createRequire(import.meta.url)(join(docsDir, 'docusaurus.config.js')); + +// Use the routes Docusaurus actually built. The default release has no version segment, while +// Next and older snapshots do. The development registry is not the released model inventory. +export function generateAgentDocs(buildDir) { + const globalData = JSON.parse(readFileSync(join(docsDir, '.docusaurus/globalData.json'), 'utf8')); + const versions = globalData['docusaurus-plugin-content-docs']?.default?.versions; + if (!versions?.length) throw new Error('No built documentation versions found'); + const stable = versions.find((version) => version.isLast); + const next = versions.find((version) => version.name === 'current'); + if (!stable || stable.name === 'current' || !next) { + throw new Error('Expected released default documentation and Next'); + } + + const outputPath = (route) => { + if ( + !route.startsWith(config.baseUrl) || + route.startsWith(`${config.baseUrl}${config.baseUrl.slice(1)}`) + ) { + throw new Error(`Invalid documentation base path: ${route}`); + } + return join(buildDir, route.slice(config.baseUrl.length)); + }; + const markdownRoute = (route) => `${route.replace(/\/$/, '')}.md`; + const entry = (route, title) => `- [${title}](${config.url}${route})`; + const status = (version) => + version.name === 'current' + ? 'Unreleased development documentation. Verify API availability in the installed package.' + : version.isLast + ? 'Default released documentation.' + : 'Historical release-line documentation.'; + + // Missing Markdown, including an API symbol or stable snapshot page, fails the normal docs build. + // Add provenance to generated files only; source snapshots remain unchanged. + let pages = 0; + for (const version of versions) { + for (const doc of version.docs) { + // Generated category navigation has no source guide and is excluded by the Markdown plugin. + if (/^\/?category\//.test(doc.id)) continue; + const route = markdownRoute(doc.path); + const file = outputPath(route); + const content = readFileSync(file, 'utf8'); + if (!content.trim()) throw new Error(`Empty agent documentation: ${route}`); + const note = `> Documentation version: ${version.label}. ${status(version)}\n`; + const body = content.replace(/^> Documentation version:[^\n]*\n\n/, ''); + writeFileSync(file, `${note}\n${body}`); + pages++; + } + } + + const docLink = (version, id, title) => { + const doc = version.docs.find((candidate) => candidate.id === id); + if (!doc) throw new Error(`${version.label} is missing required agent guide: ${id}`); + return entry(markdownRoute(doc.path), title); + }; + const root = + [ + '# React Native ExecuTorch', + '', + '> Run AI on-device in React Native: choose ready-to-use tasks and pre-exported models, or bring', + '> your own compatible ExecuTorch .pte and build a custom mobile pipeline in TypeScript.', + '', + `Default documentation: ${stable.label}. Inspect the app's installed package version first.`, + 'Documentation is snapshotted by release line, not by every npm patch. Next describes development', + 'code. Keep the version segment when fetching Markdown, and follow one version at a time.', + '', + `## Released documentation (${stable.label})`, + '', + docLink(stable, stable.mainDocId, 'Installation and requirements'), + docLink(stable, 'fundamentals/supported-models', 'Pre-exported models and task hooks'), + docLink( + stable, + 'core-and-advanced/exporting-custom-models', + 'Custom models: export, inspect and integrate' + ), + docLink( + stable, + 'core-and-advanced/models-and-tensors', + 'Custom pipelines: model and tensor primitives' + ), + docLink(stable, 'core-and-advanced/schema-validation', 'Model contracts and dynamic shapes'), + docLink(stable, 'api-reference/index', 'API reference and symbol links'), + '', + '## Development documentation (Next)', + '', + 'Use these pages for development/nightly APIs after checking the installed version.', + '', + docLink(next, 'fundamentals/ai-tools', 'AI tools and consumer skill'), + docLink(next, next.mainDocId, 'Next installation and requirements'), + docLink(next, 'fundamentals/supported-models', 'Next model catalog'), + docLink(next, 'core-and-advanced/exporting-custom-models', 'Next custom model workflow'), + docLink(next, 'api-reference/index', 'Next API reference'), + '', + '## Historical documentation', + '', + ...versions + .filter((version) => version !== stable && version !== next) + .map((version) => docLink(version, version.mainDocId, version.label)), + ].join('\n') + '\n'; + + // Keep one small root index. Validate links locally; no dependence on external service uptime. + for (const match of root.matchAll(/\]\((https?:\/\/[^)]+)\)/g)) { + const target = new URL(match[1]); + if (!readFileSync(outputPath(decodeURIComponent(target.pathname)), 'utf8').trim()) { + throw new Error(`Empty agent index target: ${target}`); + } + } + writeFileSync(join(buildDir, 'llms.txt'), root); + console.log(`Validated ${pages} versioned Markdown pages and generated the root agent index.`); +} diff --git a/docs/scripts/tasks/fix-llms-txt.mjs b/docs/scripts/tasks/fix-llms-txt.mjs deleted file mode 100644 index 1cf7e9f85b..0000000000 --- a/docs/scripts/tasks/fix-llms-txt.mjs +++ /dev/null @@ -1,38 +0,0 @@ -import { existsSync, readFileSync, writeFileSync } from 'fs'; -import { join, dirname } from 'path'; -import { fileURLToPath } from 'url'; - -export { fixLLMsTxtDuplicatedPaths }; - -const FILES_TO_FIX = ['llms.txt']; -const DUPLICATED_PATH = '/react-native-executorch/react-native-executorch/'; -const FIXED_PATH = '/react-native-executorch/'; - -function fixLLMsTxtDuplicatedPaths(buildDir) { - console.log('Running LLMS.txt fix script...'); - console.log(`Looking in: ${buildDir}`); - - FILES_TO_FIX.forEach((fileName) => { - const filePath = join(buildDir, fileName); - - if (existsSync(filePath)) { - try { - const content = readFileSync(filePath, 'utf8'); - - if (content.includes(DUPLICATED_PATH)) { - const fixedContent = content.replaceAll(DUPLICATED_PATH, FIXED_PATH); - - writeFileSync(filePath, fixedContent, 'utf8'); - console.log(`Fixed URLs in ${fileName}`); - } else { - console.info(`No broken URLs found in ${fileName}`); - } - } catch (err) { - console.error(`Could not process ${fileName}:`, err); - process.exit(1); - } - } else { - console.warn(`File ${fileName} not found`); - } - }); -} diff --git a/docs/scripts/tasks/llms-txt-models.mjs b/docs/scripts/tasks/llms-txt-models.mjs deleted file mode 100644 index 07997b271c..0000000000 --- a/docs/scripts/tasks/llms-txt-models.mjs +++ /dev/null @@ -1,101 +0,0 @@ -import { existsSync, readFileSync, writeFileSync } from 'fs'; -import { join, dirname } from 'path'; -import { createRequire } from 'module'; -import { fileURLToPath } from 'url'; -import { - BACKEND_LABELS, - BACKEND_PLATFORMS, - readModelRegistry, - readDocSlug, - byPrecision, -} from './model-registry.mjs'; -import { readAddedIn } from './model-history.mjs'; - -export { addModelsToLLMsTxt }; - -const __dirname = dirname(fileURLToPath(import.meta.url)); -const config = createRequire(import.meta.url)(join(__dirname, '..', '..', 'docusaurus.config.js')); -const DOCS_URL = `${config.url}${config.baseUrl}docs`; - -// The plugin writes llms.txt from the docs pages alone, so assistants reading -// it learn neither which release is current nor which models ship. This adds -// both, ahead of the page index: a summary at the top of the header quote and -// a list of every registry model. -function addModelsToLLMsTxt(buildDir) { - console.log('Adding the model registry to llms.txt...'); - const filePath = join(buildDir, 'llms.txt'); - if (!existsSync(filePath)) { - console.warn('File llms.txt not found'); - return; - } - - const tasks = readModelRegistry(); - const lines = readFileSync(filePath, 'utf8').split('\n'); - if (lines.includes('## Supported Models')) { - console.info('llms.txt already lists the models'); - return; - } - - const quote = lines.findIndex((line) => line.startsWith('>')); - const firstSection = lines.findIndex((line) => line.startsWith('## ')); - if (quote === -1 || firstSection === -1) { - throw new Error('llms.txt has no header quote or no sections to insert the models before'); - } - - const addedIn = readAddedIn(tasks.flatMap((task) => task.models.map((model) => model.path))); - lines.splice(firstSection, 0, ...renderModels(tasks, addedIn), ''); - lines.splice(quote, 0, ...renderSummary(tasks, addedIn), '>'); - - writeFileSync(filePath, lines.join('\n'), 'utf8'); - console.log('Added the model registry to llms.txt'); -} - -function renderSummary(tasks, addedIn) { - const lastVersion = config.presets[0][1].docs.lastVersion; - const models = tasks.flatMap((task) => task.models); - const backends = [...new Set(models.flatMap((m) => m.backends))]; - const backendText = Object.keys(BACKEND_LABELS) - .filter((b) => backends.includes(b)) - .map((b) => `${BACKEND_LABELS[b]} (${BACKEND_PLATFORMS[b]})`) - .join(', '); - const precisions = [...new Set(models.flatMap((m) => m.precisions))].sort(byPrecision); - const today = new Date().toISOString().slice(0, 10); - - const latestMinor = `v${lastVersion.replace(/\.x$/, '.')}`; - const added = addedIn - ? models.filter((m) => addedIn.get(m.path)?.startsWith(latestMinor)).map((m) => `\`${m.path}\``) - : []; - - return [ - `> Latest stable release: v${lastVersion} (npm \`react-native-executorch\`); this file was generated ${today}.`, - `> ${models.length} pre-exported models across ${tasks.length} tasks, listed under "Supported Models" below.`, - `> Backends: ${backendText}. Precisions: ${precisions.join(', ')}.`, - ...(added.length > 0 ? [`> New in v${lastVersion}: ${added.join(', ')}.`] : []), - ]; -} - -function renderModels(tasks, addedIn) { - const lines = [ - '## Supported Models', - '', - `Generated from the library's \`models\` registry. Full table: [Supported Models](${DOCS_URL}${readDocSlug('01-fundamentals/04-supported-models.md')}). Any other ExecuTorch \`.pte\` can be loaded too.`, - '', - ]; - for (const task of tasks) { - lines.push( - `### ${task.title} ([guide](${DOCS_URL}${readDocSlug(task.doc)}), [benchmarks](${DOCS_URL}/benchmarks#${task.bench}))`, - '' - ); - if (task.note) lines.push(task.note, ''); - for (const model of task.models) { - const parts = [model.description]; - if (model.modalities.length > 0) parts.push(`Inputs: text, ${model.modalities.join(', ')}.`); - if (model.variants.length > 0) parts.push(`Variants: ${model.variants.join(', ')}.`); - parts.push(`Backends: ${model.backends.map((b) => BACKEND_LABELS[b]).join(', ')}.`); - if (addedIn) parts.push(`Added in ${addedIn.get(model.path) ?? 'the next release'}.`); - lines.push(`- \`${model.path}\`: ${parts.join(' ')}`); - } - lines.push(''); - } - return lines.slice(0, -1); -} diff --git a/docs/src/components/Hero/StartScreen/index.tsx b/docs/src/components/Hero/StartScreen/index.tsx index e22097ffe9..32a68d9670 100644 --- a/docs/src/components/Hero/StartScreen/index.tsx +++ b/docs/src/components/Hero/StartScreen/index.tsx @@ -15,8 +15,8 @@ const StartScreen = () => { ExecuTorch

- Declarative way to run AI models and LLMs in React Native on - device, powered by ExecuTorch. + Declarative way to run AI models and LLMs in React Native on device, powered by + ExecuTorch.

{() => } @@ -28,6 +28,12 @@ const StartScreen = () => { title="Get started" /> +

+ + Build with an AI coding agent + + {' — choose a ready-made task or bring your own model.'} +

diff --git a/docs/src/theme/DocItem/Metadata/index.js b/docs/src/theme/DocItem/Metadata/index.js index d3ad5244a6..5c209df2d4 100644 --- a/docs/src/theme/DocItem/Metadata/index.js +++ b/docs/src/theme/DocItem/Metadata/index.js @@ -1,3 +1,24 @@ +// cspell:ignore describedby import { DocItemMetadata } from '@swmansion/t-rex-ui'; +import Head from '@docusaurus/Head'; +import useDocusaurusContext from '@docusaurus/useDocusaurusContext'; +import { useDoc } from '@docusaurus/plugin-content-docs/client'; -export default DocItemMetadata; +export default function Metadata() { + const { metadata } = useDoc(); + const { siteConfig } = useDocusaurusContext(); + + return ( + <> + + + + + + + ); +} diff --git a/skills/react-native-executorch/SKILL.md b/skills/react-native-executorch/SKILL.md new file mode 100644 index 0000000000..715e66879a --- /dev/null +++ b/skills/react-native-executorch/SKILL.md @@ -0,0 +1,83 @@ +--- +name: react-native-executorch +description: >- + Build React Native apps with react-native-executorch: use pre-exported AI models and task hooks, + integrate custom ExecuTorch .pte models, or build TypeScript inference pipelines from PyTorch model + contracts. Use for setup, model selection, inference, export integration and troubleshooting; + repository contribution workflows use the library's internal skills. +--- + +# React Native ExecuTorch + +Run AI on-device in React Native with a choice of integration levels: + +1. **Ready-made task and model:** choose a pre-exported configuration from `models` and a `use` + hook. Downloads, preprocessing, inference, postprocessing and React resource lifecycle are handled. +2. **Custom model, existing task:** supply your own `.pte` and associated assets to a task whose + input/output contract matches. A model need not be added to the shipped registry to use it in an app. +3. **Custom model and pipeline:** use `Model`, `Tensor`, `schema`, domain operations and worklets to + build the pipeline in TypeScript. Most integrations need no app-specific C++ code. + +Treat custom models as a first-class path. Choose based on the user's model, inputs/outputs and desired +control; do not restrict a custom architecture to the shipped task catalog or reimplement a supported +task unnecessarily. + +## Version and environment first + +The examples in this skill target **0.11.x**. Inspect the app's installed version and lockfile, +React Native/Expo version, target platforms, architecture, and relevant peers before choosing APIs. +Do not upgrade the app solely to fit these examples. + +Use the [documentation index](https://docs.swmansion.com/react-native-executorch/llms.txt) to select the +matching release line. Docusaurus routes have specific meanings: + +- `/react-native-executorch/docs/...` is the default **released** line; check its version badge because + that line changes as releases advance. +- `/react-native-executorch/docs/next/...` is **unreleased development** documentation. +- `/react-native-executorch/docs//...` contains older snapshots, such as `0.10.x`. + +Append `.md` to a page URL after removing any trailing slash, preserving its version segment. Read the +matching API reference or installed declarations for exact signatures. For older versions, consult their +migration docs before applying examples. Older `useLLM` examples and model constants may use a different +API; do not combine them with `useLLMChatSession` or the nested registry shown here. + +## Load references for the selected path + +| Task | Read | +| ----------------------------------------------------------------------------------- | ----------------------------------------------------- | +| Install, configure Expo/bare RN, resolve downloads or manage lifecycle | [setup.md](references/setup.md) | +| Select pre-exported models and use hooks for chat, vision, speech or retrieval | [ready-made-tasks.md](references/ready-made-tasks.md) | +| Export a PyTorch model, inspect a `.pte`, reuse a task, or implement a new pipeline | [custom-models.md](references/custom-models.md) | +| Diagnose setup, schema, execution, memory or output problems | [troubleshooting.md](references/troubleshooting.md) | + +For custom models, read `custom-models.md` before selecting an integration. Fetch the relevant +version's detailed schema/backend/task docs as needed; do not load every API page or model catalog. + +## Working approach + +- Start by identifying the task and input/output contract. The ready-made catalog covers language + models, vision-language chat, OCR, detection, segmentation, classification, depth, speech, embeddings, + retrieval and more. Consult the version's Supported Models page for exact available configurations. +- Prefer a hook inside React components. Use `download` plus `create` when manual lifetime or + execution outside React is needed. Use primitives when the task contract does not fit. +- Model configurations are objects: `models.llm.LFM2_5_1_2B.DEFAULT`, not accessor calls. Nested model + families have additional keys. Verify identifiers and backends rather than guessing names. +- Hooks expose `isReady`, `error`, `downloadProgress` (0–100), and methods that are undefined before + loading finishes. Handle readiness and async errors, serialize conflicting runs, and stop work before + disposing resources. Imperative pipelines require explicit `dispose()`. +- A `.pte` alone may be insufficient. Tokenizers, chat templates, labels, normalization, audio formats, + tensor layouts and preprocessing/postprocessing are part of the model's contract. +- Match the exported ExecuTorch format/operators and backend to the runtime shipped in the installed + library. Inspect registered backends on the device; a successful download or schema inspection does + not establish executable inference. +- For a supplied export harness, inspect its actual documentation and version pins before running it. + No automatic exporter command is assumed by this skill. Preserve the contract between export and app + integration so a published harness can later provide the `.pte`, assets and schema directly. + +## Finish with evidence + +Typecheck the app and run its relevant checks. For native changes, build a development app. Validate a +representative inference on the target device and compare custom-model outputs with a PyTorch reference +using appropriate tolerances. Report which checks ran; compilation, fake-runtime tests and schema +inspection are distinct from actual inference. Do not claim a model works on a platform merely because +its export or TypeScript code succeeded. diff --git a/skills/react-native-executorch/references/custom-models.md b/skills/react-native-executorch/references/custom-models.md new file mode 100644 index 0000000000..561409d6d3 --- /dev/null +++ b/skills/react-native-executorch/references/custom-models.md @@ -0,0 +1,154 @@ +# Bring your own model and pipeline + +Use this path when the user has PyTorch weights, a fine-tuned architecture, a custom `.pte`, or a task +outside the ready-made catalog. The library can orchestrate compatible ExecuTorch models entirely in +TypeScript; contributing a new model to the library registry is optional. + +## Preserve the contract from PyTorch to mobile + +Before exporting, collect: + +- The model definition, weights, evaluation mode and representative inputs/expected outputs. +- Method names, input/output ordering, data types, shapes, layouts and dynamic shape bounds. +- Normalization, resize/padding, tokenization, sampling rate, labels and output decoding requirements. +- Target devices/backends and the ExecuTorch runtime version used by the installed library's native + artifacts. The JavaScript package version is not the ExecuTorch version. + +The currently documented path is PyTorch `torch.export`, ExecuTorch lowering to a selected backend, +then serialization to `.pte`. Follow the version-compatible upstream export docs linked from the +library's Exporting Custom Models guide. Unsupported operators, delegates or serialization versions +require an export/runtime change; renaming a file to `.pte` does not convert it. + +If the user supplies an auto-exporter harness, read its real README/CLI and pinned dependencies. Use +its actual outputs rather than inventing flags or treating an unpublished tool as installed. An export +handoff should include the `.pte`, auxiliary assets, schema/method contract, preprocessing metadata, +runtime/backend requirements and reference inputs/outputs. These same artifacts drive mobile integration +whether produced manually or by a future published harness. + +Compare exported-model outputs to the PyTorch reference before debugging the app. Account for +quantization tolerances and backend-specific numerical differences. + +## Inspect before integrating + +`inspectModel` accepts a URL or absolute local path, resolves the model and returns its schema/backends. +For local inputs, use readable filesystem paths, not URI schemes. + +```typescript +import { inspectModel, getRegisteredBackends } from 'react-native-executorch'; + +export async function inspectCustomModel(modelPath: string) { + const inspection = await inspectModel(modelPath); + return { ...inspection, registeredBackends: getRegisteredBackends() }; +} +``` + +Verify the required backend is enabled in the app and that the export matches the shipped runtime. +Inspection can succeed even when delegated execution cannot run. Some model types also need tokenizer, +vocabulary, chat-template, label or phonemizer assets. + +## Reuse an existing task when the contract fits + +Find the task's allowed schema and configuration type in its versioned API reference or readable +TypeScript source. Supply a custom config to the hook/factory, including **all** required assets and +preprocessing options. Do not assume that replacing a `.pte` path leaves tokenizer or normalization +settings correct. + +For example, a classifier config contains `modelPath` and `modelOpts` with preprocessing and labels. +The label count must agree with the output dimension. Match the full task contract or adapt the export; +`SCHEMA_MISMATCH` identifies a disagreement rather than a check to disable. + +## Build a custom TypeScript pipeline + +This complete 0.11.x example assumes a user-supplied model with `forward(float32[1,4]) → float32[1,2]`. +The dimensions are an example contract, not a universal model format. It accepts already-prepared +features; add the user's preprocessing and output interpretation explicitly for a real model. + +```typescript +import { + createResourceScope, + loadModel, + RnExecuTorchError, + schema, + tensor, + wrapAsync, +} from 'react-native-executorch'; + +export async function createFeaturePipeline(config: { modelPath: string }) { + const scope = createResourceScope(); + const dispose = scope.dispose; + try { + const model = scope.track(await wrapAsync(loadModel)(config.modelPath)); + schema.validateSpec(model.schema, { + features: schema.method('forward', [schema.f32(1, 4)], [schema.f32(1, 2)]), + }); + const input = scope.track(tensor('float32', [1, 4])); + const output = scope.track(tensor('float32', [1, 2])); + + const runWorklet = (features: Float32Array) => { + 'worklet'; + if (features.length !== 4) { + throw RnExecuTorchError('INVALID_ARGUMENT', 'Expected four input features'); + } + input.setData(features); + model.execute('forward', [input], [output]); + return output.getData(new Float32Array(output.numel)); + }; + + return { run: wrapAsync(runWorklet), dispose }; + } catch (error) { + dispose(); + throw error; + } +} +``` + +The factory loads a **local** path. For a remote model, call `download({ modelPath: url })` first. +Imperative callers `await pipeline.run(features)` and dispose in `finally` after execution settles. +Calls share scratch tensors, so serialize them. Reuse the allocated tensors across runs; copy results +into ordinary JS typed arrays rather than leaking native resource handles. + +Inside React, the same factory can use the library's download/lifecycle hooks: + +```typescript +import { useModel, useResourceDownload } from 'react-native-executorch'; +import { createFeaturePipeline } from './createFeaturePipeline'; + +export function useFeaturePipeline(config: { modelPath: string }) { + const { resource, downloadProgress, downloadError } = useResourceDownload(config); + const { model: pipeline, error } = useModel(createFeaturePipeline, resource); + return { + isReady: !!pipeline, + downloadProgress, + error: downloadError ?? error, + run: pipeline?.run, + }; +} +``` + +Do not call the factory on every render. Keep heavy execution on the worklet runtime and reusable +helpers worklet-compatible. Most custom models need no new native operator; use existing CV/math/domain +ops for preprocessing and postprocessing before considering native extensions. + +## Dynamic shapes + +Plain strings such as `'H'` in `schema.f32(1, 3, 'H', 'W')` bind unknown-at-authoring-time **static** +dimensions. Runtime-varying dimensions use `schema.DynamicDim('L')`, with supported range/enum domains. +Repeated dynamic symbols constrain domains, not equality of runtime values; declare required runtime +constraints separately. + +A dynamic export needs a `get_model_schema` companion method containing the concrete domain and +constraint information; plain ExecuTorch method metadata can expose only fixed bounds. Follow the +library's JSON schema and validate the payload during export. Allocate buffers according to the task's +validated domains; do not silently substitute symbolic names for dynamic allocation logic. + +## Verify the complete handoff + +Check schema/backend inspection, one known input, expected output within tolerance, repeated execution, +construction-failure cleanup, disposal and the actual target platform/backend. For images/audio/text, +compare preprocessing/tokenizer behavior as well as output numbers. Distinguish successful export, +successful inspection and successful device inference in the result you report. + +- [Exporting Custom Models](https://docs.swmansion.com/react-native-executorch/docs/core-and-advanced/exporting-custom-models) +- [Models & Tensors](https://docs.swmansion.com/react-native-executorch/docs/core-and-advanced/models-and-tensors) +- [Schema Validation](https://docs.swmansion.com/react-native-executorch/docs/core-and-advanced/schema-validation) +- [Worklets & Threading](https://docs.swmansion.com/react-native-executorch/docs/core-and-advanced/worklets-and-threading) diff --git a/skills/react-native-executorch/references/ready-made-tasks.md b/skills/react-native-executorch/references/ready-made-tasks.md new file mode 100644 index 0000000000..17773b59cd --- /dev/null +++ b/skills/react-native-executorch/references/ready-made-tasks.md @@ -0,0 +1,92 @@ +# Ready-made models and tasks + +The catalog is a useful first choice when its tasks and quality fit the user's goal. It is not the +boundary of the library: for a different model or task, read `custom-models.md`. + +## Model selection + +Use the installed version's `models` registry and Supported Models page. `.DEFAULT` is a model config +object choosing a platform-appropriate backend among that model's published variants. It is not a +function. Explicit backend variants must actually exist and be enabled in the app. + +Representative 0.11.x entry points: + +| Goal | Hook and model configuration | +| --------------------- | ------------------------------------------------------------------------ | +| Streaming chat | `useLLMChatSession(models.llm.LFM2_5_1_2B.DEFAULT)` | +| OCR | `useOpticalCharacterRecognizer(models.ocr.PADDLE.PPOCRV6_SMALL.DEFAULT)` | +| Classification | `useClassifier(models.classification.EFFICIENTNET_V2_S.DEFAULT)` | +| Text embeddings | `useTextEmbedder(models.textEmbeddings.ALL_MINILM_L6_V2.DEFAULT)` | +| Whisper transcription | `useSpeechToText(models.speechToText.WHISPER.EN.TINY.DEFAULT)` | + +The catalog also includes object/keypoint detection, segmentation, depth, image embeddings, style +transfer, inpainting, text-to-image, text-to-speech, VAD, privacy filtering and late-interaction retrieval. +Choose exact registry keys from the matching documentation. Check per-device latency/memory benchmarks, +model license, languages, modalities, output semantics and required assets before choosing a model. + +## Streaming chat + +This component targets 0.11.x and uses the session API. The stop button interrupts the active turn; +the send handler waits for completion before allowing another one. The hook owns resource cleanup. + +```tsx +import { useState } from 'react'; +import { Button, Text, View } from 'react-native'; +import { models, useLLMChatSession } from 'react-native-executorch'; + +export function LocalChat() { + const [text, setText] = useState(''); + const [busy, setBusy] = useState(false); + const [generationError, setGenerationError] = useState(''); + const session = useLLMChatSession(models.llm.LFM2_5_1_2B.DEFAULT); + + const send = async () => { + if (!session.isReady || !session.sendMessage || busy) return; + setBusy(true); + setText(''); + setGenerationError(''); + try { + await session.sendMessage('Explain on-device AI in one sentence.', (token) => { + setText((previous) => previous + token); + }); + } catch (error) { + setGenerationError(String(error)); + } finally { + setBusy(false); + } + }; + + return ( + + {session.error?.message ?? generationError} + {text} +