Skip to content

Collect reproducible memory and retrieval benchmarks #1

Description

@initial-d

me_fasttext is now packaged with a benchmark protocol in docs/benchmark_protocol.md. The next useful step is to collect reproducible reports across corpora, languages, and hardware.

Useful reports should include:

  • corpus manifest: language, document count, token count, preprocessing, train/eval split
  • hardware manifest: CPU, memory, storage, OS, compiler
  • build and training commands
  • baselines used, ideally original FastText and me_fasttext
  • memory, artifact size, cold load time, warm query latency, and task quality

The most useful comparison is not a single speed number. It is whether exact trie-based subword ids plus compact mmap serving reduce deployment cost while preserving OOV behavior and downstream quality.

Please use the benchmark issue template when sharing results.

Metadata

Metadata

Assignees

No one assigned

    Labels

    benchmarkExternal or internal benchmark reportshelp-wantedGood external contribution opportunity

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions