Skip to content

api-reference.md

Latest commit

 

History

History
95 lines (63 loc) · 3.38 KB

api-reference.md

File metadata and controls

95 lines (63 loc) · 3.38 KB
Raw

API reference

The public surface is what import polyaudio exposes. Everything below is imported from the top-level package unless noted.

Running an evaluation

evaluate(model, tasks, *, strict=False, limit=None, seed=None, references=None) -> SuiteResult

Convenience wrapper around Evaluator. Runs model over tasks and returns a SuiteResult.

class Evaluator(*, strict=False, limit=None, seed=None)

  • evaluate_task(model, task, *, reference=None) -> TaskResult
  • evaluate(model, tasks, *, references=None) -> SuiteResult

strict re-raises the first task error instead of capturing it. limit caps samples per task. references maps task name → a sequence of Sample used to fit the model before evaluating that task.

Suites

load_suite(path) -> SuiteConfig

Parse a TOML suite file. SuiteConfig provides:

  • build_tasks() -> list[Task]
  • build_references() -> dict[str, list[Sample]]
  • build_model(override=None) -> Model
  • seed: int

Tasks

class Task

Base class. Construct a concrete type with Task(name, manifest, *, metrics=None, primary_metric=None, labels=None, ...).

Built-in types (also registered in TASKS under the given key):

Class Key Output kind Default metrics
ClassificationTask classification multiclass accuracy, macro_f1
MultiLabelTask multilabel multilabel map, macro_roc_auc
ASRTask asr sequence wer, cer
RegressionTask regression regression mae, rmse, pearson

@register_task("name") registers a custom subclass.

Models

class Model

Abstract base. Implement predict(spec, samples) -> list[Prediction]; optionally override fit(spec, samples) -> None.

Built-ins (registered in MODELS):

Class Key Output kinds Needs fit?
RandomModel random all regression only
ConstantModel constant all no
MajorityClassModel majority multiclass yes
PriorModel prior multiclass, multilabel yes
MeanRegressor mean regression yes

@register_model("name") registers a custom model class/factory.

Metrics

get_metric(key) -> Metric, register_metric(key, output_kinds, *, higher_is_better=True)

Metric is a callable (targets, predictions, spec) -> float carrying its key, output_kinds and higher_is_better flag. The bare functions are also importable from polyaudio.metrics (accuracy, macro_f1, word_error_rate, mean_average_precision, mean_absolute_error, pearson_corr, ...).

Reporting

  • format_table(suite) -> str
  • format_markdown(suite) -> str
  • to_json(suite, *, indent=2) -> str
  • compare(suites, *, markdown=False) -> str

Data & types

  • Dataset, read_records(path)
  • Sample, AudioRef, TaskSpec, TaskResult, SuiteResult, OutputKind
  • seed_everything(seed)

Exceptions

All descend from PolyAudioError: RegistryError, ManifestError, TaskLoadError, MetricError, SuiteConfigError.