Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 30 additions & 11 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,12 @@ jobs:
- name: Build native engines offline
run: python tools/build_native.py --inference-dir .cache/akiki-inference

- name: Build the local TEN-VAD library for runtime tests
run: |
cargo build --offline --locked --release --lib -p akiki-ten-vad-engine \
--manifest-path .cache/akiki-inference/Cargo.toml \
--target-dir .cache/native-target

- name: Build portable wheel and source archive
run: python -m build --no-isolation

Expand Down Expand Up @@ -127,21 +133,32 @@ jobs:
import importlib.util
from pathlib import Path
import akiki
from akiki.audio.vad import VAD, Engine, FFTPrecision, ModelPrecision
from akiki.decision import _load_library
import numpy as np

assert importlib.util.find_spec("onnxruntime") is None
assert importlib.util.find_spec("torch") is None
with akiki.VAD() as vad:
assert vad.engine is akiki.Engine.AKIKI
assert vad.model_precision is akiki.ModelPrecision.INT8
assert vad.fft_precision is akiki.FFTPrecision.FP32
with VAD() as vad:
assert vad.engine is Engine.AKIKI
assert vad.model_precision is ModelPrecision.INT8
assert vad.fft_precision is FFTPrecision.FP32
assert Path(vad._backend.lib._name).parent == Path(akiki.__file__).parent / "native"
assert len(vad.push(np.zeros(256 * 3 + 17, np.float32))) == 3
assert _load_library()[1].parent == Path(akiki.__file__).parent / "native"
print("Native wheel: bundled Akiki engines, INT8, base dependencies only")
PY

- name: Test both VAD models on both CPU runtimes from the installed wheel
env:
AKIKI_TEST_VAD_RUNTIMES: "1"
run: |
python="$RUNNER_TEMP/akiki-native/bin/python"
wheels=(dist/akiki-*-manylinux_2_34_x86_64.whl)
"$python" -m pip install "${wheels[0]}[onnx,ten-vad]" pytest
AKIKI_TEN_VAD_LIBRARY="$PWD/.cache/native-target/release/libakiki_ten_vad_engine.so" \
"$python" -m pytest -q -o pythonpath= tests/test_vad_runtimes.py

- name: Test portable wheel with the ONNX extra
run: |
version="$(python verify_dist.py --version)"
Expand All @@ -151,7 +168,7 @@ jobs:
"$RUNNER_TEMP/akiki-onnx/bin/python" -I - <<'PY'
import importlib.util
import numpy as np
from akiki import Engine, ModelPrecision, VAD
from akiki.audio.vad import VAD, Engine, ModelPrecision

assert importlib.util.find_spec("torch") is None
with VAD() as vad:
Expand Down Expand Up @@ -253,6 +270,7 @@ jobs:
from pathlib import Path
import akiki
import numpy as np
from akiki.audio.vad import VAD, Engine, ModelPrecision
from akiki.decision import _load_library

assert akiki.__version__ == os.environ["AKIKI_EXPECTED_VERSION"]
Expand All @@ -262,8 +280,8 @@ jobs:
assert {p.name for p in native.iterdir()} == {
"libakiki_vad_engine.so", "libakiki_minilm_engine.so"
}
with akiki.VAD() as vad:
assert vad.engine is akiki.Engine.AKIKI and vad.model_precision is akiki.ModelPrecision.INT8
with VAD() as vad:
assert vad.engine is Engine.AKIKI and vad.model_precision is ModelPrecision.INT8
assert Path(vad._backend.lib._name).parent == native
assert len(vad.push(np.zeros(256 * 3 + 17, np.float32))) == 3
assert _load_library()[1].parent == native
Expand Down Expand Up @@ -398,15 +416,16 @@ jobs:
from pathlib import Path
import akiki
import numpy as np
from akiki.audio.vad import VAD, Engine, FFTPrecision, ModelPrecision
from akiki.decision import _load_library

assert importlib.util.find_spec('onnxruntime') is None
assert importlib.util.find_spec('torch') is None
native = Path(akiki.__file__).parent / 'native'
with akiki.VAD() as vad:
assert vad.engine is akiki.Engine.AKIKI
assert vad.model_precision is akiki.ModelPrecision.INT8
assert vad.fft_precision is akiki.FFTPrecision.FP32
with VAD() as vad:
assert vad.engine is Engine.AKIKI
assert vad.model_precision is ModelPrecision.INT8
assert vad.fft_precision is FFTPrecision.FP32
assert Path(vad._backend.lib._name).parent == native
assert len(vad.push(np.zeros(256 * 3 + 17, np.float32))) == 3
assert _load_library()[1].parent == native
Expand Down
74 changes: 55 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,13 +115,13 @@ PyTorch is the second VAD compatibility fallback.

## Python usage

Use `akiki.vad` for VAD and `akiki.decision` for decision models.
Use `akiki.audio.vad` for VAD and `akiki.decision` for decision models.
Use `akiki.vision.object_detection` for object detection.
The training and teacher modules support the decision-model workflow:

| Module | Main interface | Supporting types |
| --- | --- | --- |
| `akiki.vad` | `VAD` for speech detection | `SpeechFrame`, `Engine`, `ModelPrecision`, `FFTPrecision` |
| `akiki.audio.vad` | `VAD` for speech detection | `SpeechFrame`, `Model`, `Engine`, `ModelPrecision`, `FFTPrecision` |
| `akiki.vision.object_detection` | `ObjectDetector` for object detection | `Detection` |
| `akiki.decision` | `DecisionModel` for specialist decisions | `BaseModel`, `Specialist`, `AffineLayer`, `TrainedSpecialist`, `SpecialistEvaluation`, `TrainingAdvice`, `list_base_models`, `Classification`, `ModelPrecision` |
| `akiki.training` | Classifier training with a private companion | The installed companion's API, including `LocalModel`, `ModelFamily`, and `Trainer` |
Expand All @@ -130,16 +130,17 @@ The training and teacher modules support the decision-model workflow:
Import the primary interfaces from their modules:

```python
from akiki.vad import VAD
from akiki.audio import vad
from akiki.decision import DecisionModel, Specialist
from akiki.vision import object_detection
```

Import configuration enums and result types from the relevant module.
VAD and decision inference have distinct `ModelPrecision` enums.
Their supported formats differ.
`MiniLM` is the lower-level interface for direct embeddings and classification
with the currently supported MiniLM/BERT family.
The existing `akiki.VAD` and `akiki.MiniLM` shortcuts remain available.
The `akiki.MiniLM` shortcut remains available.
The existing top-level result types and VAD enums remain available as
compatibility aliases. Import `training` explicitly when its optional companion
package is installed; the base package does not load it.
Expand All @@ -150,7 +151,9 @@ modules remain accessible but are omitted from that list.
The decision-model module is named `akiki.decision`. Existing imports from
`akiki.minilm` remain available for compatibility and use the same classes
and precision enum. Named top-level compatibility imports remain available.
Wildcard imports expose only `VAD`, `MiniLM`, `vad`, `decision`, `vision`, and `__version__`.
`akiki.vad` and `akiki.VAD` are deprecated and will be removed in version 0.3.0.
Use `from akiki.audio import vad` and call `vad.VAD()`.
Wildcard imports expose only `VAD`, `MiniLM`, `audio`, `decision`, `vision`, and `__version__`.

## Object detection

Expand Down Expand Up @@ -189,39 +192,72 @@ Akiki accepts mono audio sampled at 16,000 Hz.

```python
import soundfile as sf
from akiki import VAD
from akiki.audio import vad

audio, sample_rate = sf.read("recording.wav", dtype="float32")

with VAD() as vad:
for frame in vad.push(audio, sample_rate=sample_rate):
with vad.VAD() as detector:
for frame in detector.push(audio, sample_rate=sample_rate):
print(frame.start_seconds, frame.end_seconds, frame.probability, frame.speech)
```

For live audio, call `vad.push()` repeatedly with successive audio buffers.
For live audio, call `detector.push()` repeatedly with successive audio buffers.
Buffers can have any length; incomplete frames are retained for the next call.
Call `vad.reset()` before starting a new recording. A final incomplete frame is
Call `detector.reset()` before starting a new recording. A final incomplete frame is
not padded or scored.

Use `VAD(threshold=0.6)` to set the speech decision threshold. Every frame still
Use `vad.VAD(threshold=0.6)` to set the speech decision threshold. Every frame still
includes its probability, regardless of the threshold.

The public API includes docstrings for editor hints and interactive help:

```python
import akiki

help(akiki) # Overview of vad, decision, training, and teacher
help(akiki.vad) # Speech detection, configuration, and result types
help(akiki) # Overview of audio, decision, training, and teacher
help(akiki.audio.vad) # Speech detection, configuration, and result types
help(akiki.decision) # Decision models, specialists, and result types
help(akiki.decision.DecisionModel) # Base-model selection and specialist inference
help(akiki.decision.Specialist.train) # Training parameters and saved exports
help(akiki.VAD) # Configuration, attributes, and streaming example
help(akiki.VAD.__init__) # Constructor parameters and defaults
help(akiki.VAD.push) # Audio input, buffering, results, and errors
help(akiki.vad.SpeechFrame) # Result fields and timestamp units
help(akiki.audio.vad.VAD) # Configuration, attributes, and streaming example
help(akiki.audio.vad.VAD.__init__) # Constructor parameters and defaults
help(akiki.audio.vad.VAD.push) # Audio input, buffering, results, and errors
help(akiki.audio.vad.SpeechFrame) # Result fields and timestamp units
```

### Select the VAD model

`vad.VAD()` and `vad.VAD(model=vad.Model.AKIKI_VAD)` load the bundled Akiki-VAD model.

To use TEN-VAD, install `akiki[ten-vad]` and set `AKIKI_TEN_VAD_LIBRARY` to
your local Akiki TEN-VAD shared library. You can also pass `library_path`.

```python
from akiki.audio import vad

with vad.VAD(
model=vad.Model.TEN_VAD,
model_path="snapshot/src/onnx_model/ten-vad.onnx",
) as detector:
frames = detector.push([0.0] * 256)
```

Keep the matching `src/coeff.h` in the snapshot. For a standalone ONNX file,
keep `coeff.h` next to the file. The coefficients supply the audio frontend.
TEN-VAD requires mono 16 kHz samples in `[-1, 1]` and runs in FP32.

Omit `model_path` to use the supported snapshot already in the Hugging Face
cache. Set `HF_HUB_OFFLINE=1` and configure `HF_HUB_CACHE` if you use a
custom cache directory. [TEN-VAD model on Hugging Face](https://huggingface.co/TEN-framework/ten-vad)

The first load prepares the model for the local Akiki TEN-VAD library. Later
loads reuse the prepared data in the Akiki cache. The Akiki engine runs the
network by default.

To run the network in ONNX Runtime, install `akiki[onnx,ten-vad]` and pass
`engine=vad.Engine.ONNX`. TEN-VAD uses the same local Akiki library to
extract audio features for both engines.

### Command line

```bash
Expand Down Expand Up @@ -320,7 +356,7 @@ Choose another precision when creating the detector. For example, to use the
original FP32 weights:

```python
from akiki.vad import VAD, Engine, ModelPrecision
from akiki.audio.vad import VAD, Engine, ModelPrecision

with VAD(engine=Engine.AKIKI, model_precision=ModelPrecision.FP32) as vad:
frames = vad.push(audio, sample_rate=sample_rate)
Expand All @@ -342,7 +378,7 @@ akiki vad recording.wav --model-precision fp32
| `ModelPrecision.INT8_FP32_INPUT` | INT8 weights; first projection keeps FP32 activations |

The Python VAD API requires `ModelPrecision` and `FFTPrecision` enum members,
both exported from `akiki.vad`. Omit `model_precision` or pass `None` to use the
both exported from `akiki.audio.vad`. Omit `model_precision` or pass `None` to use the
selected engine's default. CLI precision options retain their lowercase names.

The FFT stays FP32: `fft_precision` accepts only `FFTPrecision.FP32`. Recurrent state,
Expand Down
10 changes: 7 additions & 3 deletions docs/releasing.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ The public PyPI description comes from `README.md`. Keep it focused on package
features and usage. Build and publishing instructions belong here.

Akiki provides VAD and decision models.
The primary classes are `akiki.vad.VAD` and `akiki.decision.DecisionModel`.
The primary classes are `akiki.audio.vad.VAD` and `akiki.decision.DecisionModel`.
Current native decision inference supports MiniLM/BERT exports.
The library filenames below retain the name of that implementation.

Expand All @@ -23,6 +23,10 @@ Its tests use public dependencies and synthetic local checkpoints.
Private engine and training checks run on `main` and trusted manual builds.
Fork PR jobs do not receive private checkout credentials.

The trusted Linux x86-64 build tests Akiki-VAD and TEN-VAD on the Akiki engine
and ONNX Runtime from an installed wheel. It builds the TEN-VAD library from
the pinned engine checkout. The tests generate the TEN-VAD ONNX fixture locally.

Use version tags to record releases and release candidates.
For example, use `v0.2.1` for version `0.2.1` and `v0.2.2rc1` for `0.2.2rc1`.
Do not create an RC branch or release branch for each version.
Expand Down Expand Up @@ -195,9 +199,9 @@ target platforms before adding those wheels to `dist/`.
Validate the Markdown and metadata with `twine check` for every new artifact.
Test a clean install with both portable and platform wheels available and
confirm that `VAD().engine is Engine.AKIKI`, loading the installed library
(`Engine` is exported from `akiki`).
(`Engine` is exported from `akiki.audio.vad`).
Confirm that `VAD().model_precision is ModelPrecision.INT8` and
`VAD().fft_precision is FFTPrecision.FP32` (both enums are exported from `akiki.vad`),
`VAD().fft_precision is FFTPrecision.FP32` (both enums are exported from `akiki.audio.vad`),
and that `akiki.decision._load_library()` loads the installed MiniLM
library. When no precision is specified, ONNX and PyTorch compatibility engines use
FP32; an explicit non-FP32 request must require the Akiki engine.
Expand Down
8 changes: 4 additions & 4 deletions docs/style.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ ONNX and PyTorch are VAD compatibility fallbacks.
Do not use the implementation language as the engine name.

Describe the two primary interfaces as VAD and decision models.
Use `akiki.vad.VAD` and `akiki.decision.DecisionModel` in API overviews.
Use `akiki.audio.vad.VAD` and `akiki.decision.DecisionModel` in API overviews.
Describe MiniLM/BERT as the family supported by the current decision-model runtime.
Use `MiniLM` for direct access to that family.
Keep model discovery, training support, and inference support separate.
Expand All @@ -36,7 +36,7 @@ Use separate enum classes when parameters accept different sets of choices.
For example, VAD model precision and FFT precision have different domains:

```python
from akiki.vad import VAD, FFTPrecision, ModelPrecision
from akiki.audio.vad import VAD, FFTPrecision, ModelPrecision

with VAD(
model_precision=ModelPrecision.INT8,
Expand All @@ -46,7 +46,7 @@ with VAD(
```

VAD and decision models also have distinct model precision domains. Use
`akiki.vad.ModelPrecision` with `VAD` and `akiki.decision.ModelPrecision` with
`akiki.audio.vad.ModelPrecision` with `VAD` and `akiki.decision.ModelPrecision` with
`DecisionModel` or `MiniLM`.
Do not share enum types just because some serialized values match.

Expand Down Expand Up @@ -117,7 +117,7 @@ Check the executable examples in the public inference docstrings:

```bash
HF_HUB_OFFLINE=1 python -m pytest -q --doctest-modules \
src/akiki/__init__.py src/akiki/vad.py src/akiki/decision.py src/akiki/hub.py
src/akiki/__init__.py src/akiki/audio/vad.py src/akiki/decision.py src/akiki/hub.py
```

VAD examples require a working local engine. Examples with placeholder text-model
Expand Down
3 changes: 2 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ dependencies = ["numpy>=1.24", "soundfile>=0.12"]
huggingface = ["huggingface-hub>=0.24"]
onnx = ["onnxruntime>=1.17"]
pytorch = ["torch>=2.1"]
ten-vad = ["huggingface-hub>=0.24", "onnx>=1.16"]
training = ["torch>=2.1", "transformers>=4.40,<6", "safetensors>=0.4"]

[project.urls]
Expand All @@ -45,7 +46,7 @@ include-package-data = false

[tool.setuptools.packages.find]
where = ["src"]
include = ["akiki", "akiki.vision"]
include = ["akiki", "akiki.audio", "akiki.vision"]
namespaces = false

[tool.setuptools.package-data]
Expand Down
Loading