freqwords

freqwords is a Node.js library for exact, profile-aware corpus analysis. It accepts complete text, incremental text, or Node readable streams and returns word frequencies, n-grams, lexical statistics, concordances, and optional language-aware views.

Use it to explore survey responses, support tickets, articles, transcripts, books, logs containing natural language, or any other text corpus.

Requirements and installation

freqwords requires Node.js 20 or newer and supports both ESM and CommonJS.

npm install freqwords

// ESM
import { analyze } from "freqwords";

// CommonJS
const { analyze } = require("freqwords");

Quick start: find the most frequent words

Suppose you want a quick overview of the vocabulary in a short collection of travel notes:

import { analyze } from "freqwords";

const result = analyze("Blue sky, blue ocean. Clear sky.", {
  profile: "english",
});

console.log(result.entries());

Output:

[
  { term: "blue", count: 2 },
  { term: "sky", count: 2 },
  { term: "clear", count: 1 },
  { term: "ocean", count: 1 },
]

Terms are ordered by descending count, then alphabetically when counts are equal. Analysis is case-insensitive and Unicode-normalized by default, so Blue and blue are counted together.

The result also describes the corpus:

console.log(result.statistics());

{
  tokens: 6,
  types: 4,
  hapaxLegomena: 2,
  lexicalDiversity: 0.6666666666666666,
}

• tokens is the total number of words.
• types is the number of distinct words.
• hapaxLegomena counts words that occur exactly once.
• lexicalDiversity is types / tokens.

Understanding result views

The raw view always contains every accepted token. Enabling stop words or stemming creates additional interpretations without destroying the original counts:

View	Meaning	Available when
raw	Normalized tokens as written	Always
filtered	Raw tokens without stop words	stopWords is enabled
stemmed	Raw tokens reduced to stems	stemming is enabled
stemmedFiltered	Stop words removed, then remaining tokens stemmed	Both are enabled

This is useful when the same corpus needs both an auditable transcription and a topic-oriented summary.

const result = analyze("The cats are running; the cat runs.", {
  profile: "english",
  stopWords: true,
  stemming: true,
});

console.log(result.entries());
console.log(result.entries("filtered"));
console.log(result.entries("stemmedFiltered"));

Representative output:

// Raw
[
  { term: "the", count: 2 },
  { term: "are", count: 1 },
  { term: "cat", count: 1 },
  { term: "cats", count: 1 },
  { term: "running", count: 1 },
  { term: "runs", count: 1 },
]

// Filtered
[
  { term: "cat", count: 1 },
  { term: "cats", count: 1 },
  { term: "running", count: 1 },
  { term: "runs", count: 1 },
]

// Stemmed and filtered
[
  { term: "cat", count: 2 },
  { term: "run", count: 2 },
]

Stop-word filtering and stemming are opt-in. stemming: true uses the bundled Snowball stemmer for English, Portuguese, Spanish, French, German, or Italian. A custom profile must provide its own stemmer function.

To extend a built-in stop-word list with domain-specific terms, pass an iterable:

const result = analyze(customerFeedback, {
  profile: "english",
  stopWords: ["product", "company", "customer"],
});

console.log(result.entries("filtered", 10));

The iterable is combined with the profile's bundled list. Here, the second argument to entries limits the ranked output to the ten most frequent terms; the exact full frequency map is still retained.

Discover recurring phrases with n-grams

Single words reveal vocabulary; n-grams reveal phrases. For example, a search team may want to identify concepts repeated in user research:

const result = analyze(
  "Machine learning improves search. Machine learning finds patterns.",
  {
    profile: "english",
    ngramSizes: [2, 3],
  },
);

console.log(result.ngramEntries(2, "raw", 3));

[
  { term: "machine learning", count: 2, size: 2 },
  { term: "finds patterns", count: 1, size: 2 },
  { term: "improves search", count: 1, size: 2 },
]

Call ngramEntries(3) for trigrams or ngrams(2) for the exact ReadonlyMap<string, number>. N-grams are counted across incremental and stream chunks exactly as if the entire corpus had been supplied at once.

N-grams are also available for linguistic views:

const result = analyze("The cat runs and the cats run.", {
  profile: "english",
  stopWords: true,
  stemming: true,
  ngramSizes: 2,
});

console.log(result.ngramEntries(2, "stemmedFiltered"));

[
  { term: "cat run", count: 2, size: 2 },
  { term: "run cat", count: 1, size: 2 },
]

Choose a language profile

Profiles determine which alphabet and locale rules are valid for the corpus. The built-in profiles are:

• latin, the default generic Latin-script profile;
• english or en;
• portuguese, pt, or pt-br;
• spanish or es;
• french or fr;
• german or de;
• italian or it.

For example, Portuguese accents and joined apostrophes are normalized and counted correctly:

const result = analyze("Água limpa, água doce; d’água.", {
  profile: "portuguese",
});

console.log(result.entries());

[
  { term: "água", count: 2 },
  { term: "d'água", count: 1 },
  { term: "doce", count: 1 },
  { term: "limpa", count: 1 },
]

A constrained profile discards a whole word when it contains an unsupported letter. It never turns café into the misleading English fragment caf, for example.

Define a custom alphabet

Custom profiles are useful for constructed languages, historical alphabets, controlled-vocabulary exercises, or corpora where foreign words must be excluded:

import { analyze, createProfile } from "freqwords";

const hawaiian = createProfile({
  name: "hawaiian",
  locale: "en",
  alphabet: "aāeēiīoōuūhklmnpw",
});

const result = analyze("aloha world mahalo", {
  profile: hawaiian,
});

console.log(result.entries());

[
  { term: "aloha", count: 1 },
  { term: "mahalo", count: 1 },
]

world is discarded because r and d are outside the chosen alphabet.

Analyze large files as streams

Use analyzeStream when a corpus should not be loaded into one large string. This is appropriate for books, transcript exports, or large batches already represented as a readable stream:

import { createReadStream } from "node:fs";
import { analyzeStream } from "freqwords";

const corpus = createReadStream("./support-tickets.txt");

const result = await analyzeStream(corpus, {
  profile: "english",
  stopWords: true,
  ngramSizes: [2, 3],
  topK: 20,
});

console.log(result.entries("filtered"));
console.log(result.ngramEntries(2, "filtered"));

topK: 20 becomes the default limit for ranked helpers such as entries(). Pass an explicit limit to override it. Frequency maps remain exact and complete.

Byte sequences, Unicode characters, words, and n-grams may cross arbitrary stream chunk boundaries; the output remains identical to analyze(fullText, options).

Stream errors reject the returned promise:

try {
  const result = await analyzeStream(createReadStream("./corpus.txt"));
  console.log(result.entries("raw", 25));
} catch (error) {
  console.error("Corpus analysis failed:", error);
}

Feed text incrementally

createAnalyzer is useful when text arrives from several in-memory sources or is produced over time:

import { createAnalyzer } from "freqwords";

const analyzer = createAnalyzer({
  profile: "english",
  ngramSizes: 2,
});

analyzer.write("first batch of ");
analyzer.write("text and a second ");
const result = analyzer.end("batch of text");

console.log(result.entries());

write returns the analyzer, so calls may also be chained. end() flushes the last token and returns the immutable analysis interface. Calling write() after end() is an error, while result() returns the finalized result after end() has been called.

Inspect words in context

Frequency tells you what is common, but not how a word is used. Enable bounded context retention to build concordances:

const result = analyze(
  "Search should be fast. Better search should also be relevant.",
  {
    profile: "english",
    context: {
      window: 2,
      maxTokens: 50_000,
    },
  },
);

console.log(result.concordance("search"));

[
  {
    term: "search",
    before: [],
    after: ["should", "be"],
    index: 0,
  },
  {
    term: "search",
    before: ["fast", "better"],
    after: ["should", "also"],
    index: 5,
  },
]

Override the configured window for one query with result.concordance("search", { window: 5 }).

Context retention is disabled unless context is supplied. At most context.maxTokens normalized source tokens are kept; frequencies continue across the whole corpus. Check result.toJSON().context.truncated before treating a concordance as complete.

Choose an output format

The result interface supports several consumers without forcing one representation:

const result = analyze("red blue red", { profile: "english" });

result.frequencies();
// Map(2) { "red" => 2, "blue" => 1 }

result.entries();
// [{ term: "red", count: 2 }, { term: "blue", count: 1 }]

result.toObject();
// { red: 2, blue: 1 }

frequencies() and ngrams() return defensive ReadonlyMap copies. Ranked entries are deterministic. toObject() is convenient for application code or simple JSON encoding.

For storage, APIs, snapshots, or sending results to another process, use the versioned interchange representation:

const summary = result.toJSON();
console.log(JSON.stringify(summary, null, 2));

Abbreviated output:

{
  "schemaVersion": 1,
  "profile": "english",
  "options": {
    "normalize": true,
    "caseSensitive": false,
    "ngramSizes": [],
    "stopWords": false,
    "stemming": false
  },
  "views": {
    "raw": {
      "terms": [
        { "term": "red", "count": 2 },
        { "term": "blue", "count": 1 }
      ],
      "ngrams": {},
      "statistics": {
        "tokens": 3,
        "types": 2,
        "hapaxLegomena": 1,
        "lexicalDiversity": 0.6666666666666666
      }
    }
  },
  "context": {
    "retainedTokens": 0,
    "truncated": false
  }
}

The complete result contains one entry under views for each enabled interpretation.

Common options

Option	Default	Purpose
profile	"latin"	Select a built-in name or custom profile
locale	Profile locale	Override locale-aware lowercasing
normalize	true	Normalize tokens to Unicode NFC
caseSensitive	false	Preserve uppercase/lowercase distinctions
keepApostrophes	true	Keep apostrophes joining two valid letters
keepHyphens	true	Keep hyphens joining two valid letters
ngramSizes	[]	Count one or more exact n-gram sizes of at least 2
stopWords	Disabled	Enable profile words or provide an iterable
stemming	Disabled	Enable a profile stemmer or provide a function
context	Disabled	Retain a bounded normalized-token prefix
topK	Unlimited	Set the default ranked-output limit

All frequency and n-gram counts are exact. Memory grows with the number of distinct terms and enabled n-grams. Context source retention is separately bounded by context.maxTokens; ranked-output limits do not make counting approximate.

Development

npm install
npm test
npm run typecheck
npm run benchmark
npm run pack:check

The test suite builds and validates both ESM and CommonJS entry points.