Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 44 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,12 +48,13 @@

- 🎯 Run AI models directly in the browser - no server required!
- ⚡ WebGPU acceleration for blazing fast inference
- 🔄 Seamless switching between MLC and Transformers engines
- 🔄 Seamless switching between MLC, Transformers, Flare, and Demucs engines
- 📦 Pre-configured popular models ready to use
- 🛠️ Easy-to-use API for text generation and more
- 🔧 Web Worker support for non-blocking UI performance
- 📊 Structured output generation with JSON schemas
- 🎙️ Speech recognition and text-to-speech capabilities
- 🎵 Audio source separation (Demucs) - isolate vocals, drums, bass, and other stems
- 💾 Built-in database support for storing conversations and embeddings


Expand Down Expand Up @@ -163,6 +164,37 @@ audioContext.decodeAudioData(audioBuffer, (buffer) => {
});
```

### Audio Source Separation (Demucs)
```javascript
import { DemucsEngine } from '@browserai/browserai/demucs';

const engine = new DemucsEngine();
await engine.loadModel({ /* htdemucs config */ });

// Separate an AudioBuffer into stems
const result = await engine.separate(audioBuffer, {
shifts: 1, // Time-shift augmentation passes (higher = better quality, slower)
overlap: 0.25, // Segment overlap ratio
});

// result.sources contains: drums, bass, other, vocals (each as AudioBuffer)
const vocals = result.sources['vocals'];
```

### Flare Engine (GGUF Models via WASM)
```javascript
const ai = new BrowserAI();

// Load a GGUF model via the Flare engine
await ai.loadModel('llama-3.2-1b-flare');

// Generate text — same API as MLC/Transformers
const response = await ai.generateText('Explain quantum computing briefly');

// Optional: Load a LoRA adapter
await ai.loadAdapter({ url: 'https://example.com/adapter.safetensors' });
```

## 🔧 Supported Models

More models will be added soon. Request a model by creating an issue.
Expand Down Expand Up @@ -197,6 +229,15 @@ More models will be added soon. Request a model by creating an issue.
- Whisper-small-all (Speech Recognition)
- Kokoro-TTS (Text-to-Speech)

### Flare Models (GGUF via WASM)
- SmolLM2-135M-Instruct (Q8_0, Q4_K_M)
- SmolLM2-360M-Instruct (Q8_0)
- Qwen2.5-0.5B-Instruct (Q4_K_M)
- Llama-3.2-1B-Instruct (Q8_0, Q4_K_M)

### Demucs Models (Audio Source Separation)
- HTDemucs (4-stem: drums, bass, other, vocals)

## 🗺️ Enhanced Roadmap

### Phase 1: Foundation
Expand Down Expand Up @@ -238,6 +279,8 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file

- [MLC AI](https://github.com/mlc-ai/mlc-llm) for their incredible mode compilation library and support for webgpu runtime and xgrammar
- [Hugging Face](https://huggingface.co/) and [Xenova](https://github.com/xenova) for their Transformers.js library, licensed under Apache License 2.0. The original code has been modified to work in a browser environment and converted to TypeScript.
- [Facebook Research](https://github.com/facebookresearch/demucs) for Demucs, the state-of-the-art music source separation model.
- [Aspect](https://github.com/aspect-build/flare) for the Flare WASM inference engine enabling GGUF model support in the browser.
- All our contributors and supporters!

---
Expand Down
2 changes: 1 addition & 1 deletion examples/audio-demo/package.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"name": "browser-ai-demo",
"name": "browser-ai-audio-demo",
"private": true,
"version": "0.0.0",
"type": "module",
Expand Down
2 changes: 1 addition & 1 deletion examples/chat-demo/package.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"name": "browser-ai-demo",
"name": "browser-ai-chat-demo",
"private": true,
"version": "0.0.0",
"type": "module",
Expand Down
6 changes: 3 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -65,13 +65,13 @@
},
"type": "module",
"devDependencies": {
"@types/jest": "^29.5.14",
"@types/jest": "^30.0.0",
"@typescript-eslint/eslint-plugin": "^8.20.0",
"@typescript-eslint/parser": "^8.20.0",
"@webgpu/types": "^0.1.53",
"eslint": "^9.18.0",
"jest": "^29.7.0",
"jest-environment-jsdom": "^29.7.0",
"jest": "^30.3.0",
"jest-environment-jsdom": "^30.3.0",
"prettier": "^3.4.2",
"ts-jest": "^29.2.5",
"tsup": "^8.3.5",
Expand Down
2 changes: 1 addition & 1 deletion src/core/database/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ export class DatabaseImpl<T extends Storable> implements Database<T> {
const available = Object.keys(vectorStoreFactories);
throw new Error(
available.length === 0
? `Vector store support is not yet implemented. No vector store backends are available.`
? 'Vector store support is not yet implemented. No vector store backends are available.'
: `Vector store type "${type}" not supported. Available types: ${available.join(', ')}`,
);
}
Expand Down
12 changes: 6 additions & 6 deletions src/core/llm/browserai.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -185,14 +185,14 @@ describe('BrowserAI — engine registry sanity', () => {
// This is a smoke test that no two registries have collided and broken
// the spread order — if, say, demucs-models.json has a key that shadows
// an mlc-models.json key, loading the older entry would silently break.
// eslint-disable-next-line @typescript-eslint/no-require-imports
const mlc = require('../../config/models/mlc-models.json') as Record<string, ModelConfig>;
// eslint-disable-next-line @typescript-eslint/no-require-imports
const transformers =
require('../../config/models/transformers-models.json') as Record<string, ModelConfig>;
// eslint-disable-next-line @typescript-eslint/no-require-imports
const demucs = require('../../config/models/demucs-models.json') as Record<string, ModelConfig>;
// eslint-disable-next-line @typescript-eslint/no-require-imports
const flare = require('../../config/models/flare-models.json') as Record<string, ModelConfig>;

const mlcKeys = new Set(Object.keys(mlc));
Expand All @@ -215,15 +215,15 @@ describe('BrowserAI — engine registry sanity', () => {
});

test('every Flare model config has engine: "flare"', () => {
// eslint-disable-next-line @typescript-eslint/no-require-imports
const flare = require('../../config/models/flare-models.json') as Record<string, FlareConfig>;
for (const cfg of Object.values(flare)) {
expect(cfg.engine).toBe('flare');
}
});

test('every Demucs model config has engine: "demucs"', () => {
// eslint-disable-next-line @typescript-eslint/no-require-imports
const demucs =
require('../../config/models/demucs-models.json') as Record<string, DemucsConfig>;
for (const cfg of Object.values(demucs)) {
Expand Down
4 changes: 2 additions & 2 deletions src/engines/flare-engine-wrapper.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,9 @@ describe('flare-models.json', () => {
const flareKeys = new Set(Object.keys(flareModels));
// Intentionally import lazily to avoid the cross-engine tests depending
// on a specific MLC registry shape.
// eslint-disable-next-line @typescript-eslint/no-require-imports
const mlcModels = require('../config/models/mlc-models.json') as Record<string, MLCConfig>;
// eslint-disable-next-line @typescript-eslint/no-require-imports
const demucsModels = require('../config/models/demucs-models.json') as Record<string, DemucsConfig>;
for (const k of flareKeys) {
expect(mlcModels[k]).toBeUndefined();
Expand Down
Loading