Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 31 additions & 35 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,32 +9,32 @@
[![License](https://img.shields.io/github/license/skyoo2003/acor.svg)](LICENSE)
[![Sponsor](https://img.shields.io/badge/sponsor-GitHub-pink)](https://github.com/sponsors/skyoo2003)

ACOR stores a shared Aho-Corasick pattern dictionary in Redis and exposes it
through a Go library and CLI. Multiple application instances can update the
same dictionary at runtime while preset engines serve matches from local memory.
ACOR keeps one Aho-Corasick dictionary in Redis and reaches it from a Go library, a CLI,
and an experimental server module. Every application instance shares that dictionary,
updates it at runtime, and matches against a local copy with no Redis I/O on the hot path.

Typical uses include content filtering, keyword extraction, intrusion detection,
search highlighting, and real-time text classification.
Typical uses: content filtering, keyword extraction, intrusion detection, search
highlighting, real-time text classification.

## Highlights

- **Shared state** — every application instance uses the same Redis-backed dictionary
- **Shared state** — every instance reads the same Redis-backed dictionary
- **Runtime updates** — Pub/Sub invalidation, with optional polling for missed messages
- **Fast reads** — preset engines match locally without Redis I/O on the hot path
- **Flexible deployment** — standalone Redis, Sentinel, Cluster, Ring, and Valkey
- **Complete matching API** — occurrences, positions, sets, streams, batches, and parallel matching
- **Fast reads** — preset engines match locally, 0 round trips
- **Topologies** — standalone Redis, Sentinel, Cluster, Ring, and Valkey
- **Full matching API** — occurrences, positions, sets, streams, batches, parallel scans

## Installation
## Install

ACOR requires Go 1.25 or newer and Redis 3.0 or newer, or Valkey 7.2 or newer.
Requires Go 1.25+ and Redis 3.0+ or Valkey 7.2+.

```sh
go get github.com/skyoo2003/acor/pkg/acor@latest
```

## Quick Start
## Quick start

Start Redis locally, then create a matcher:
Start Redis locally, then:

<!-- doccheck -->
```go
Expand Down Expand Up @@ -69,29 +69,35 @@ func main() {
}
```

New collections use the optimized V2 Redis schema by default. `PresetBalanced`
is the recommended starting point when reads should run locally.
New collections use the optimized V2 Redis schema.

## Choosing a Preset
## Choosing a preset

| Goal | Preset |
| ---- | ------ |
| General-purpose speed and memory | `PresetBalanced` |
| Highest matching throughput | `PresetSpeed` |
| Lowest memory usage | `PresetMemoryEfficient` |

Redis remains the source of truth for every preset. See the
[preset guide](docs/content/guides/preset-engine.md) for trade-offs and the
[Redis-backed engine guide](docs/content/guides/redis-backed-engine.md) for
multi-instance invalidation safety.
Redis stays the source of truth for every preset. Trade-offs are in the
[preset guide](docs/content/guides/preset-engine.md); multi-instance invalidation is in the
[Redis-backed engine guide](docs/content/guides/redis-backed-engine.md).

## Large dictionaries (V3)

`OpenVersioned` opens a separate V3 collection with leased snapshots, expected-version
writes, and background engine replacement. V1/V2 APIs are unaffected. See the
[V3 guide](docs/content/reference/versioned.md), its
[performance report](docs/content/reference/versioned-performance.md), and
[bounded text processing](docs/content/reference/text-processing.md) for `Scan`,
`MaskText`, and `ReplaceText`.

## Documentation

Everything past this point lives on the
[documentation site](https://skyoo2003.github.io/acor/): Redis topologies, the matching and
streaming API, batch and parallel guides, the schema-V2 migration and benchmarks, deployment
and troubleshooting, the `acor` CLI, the experimental server module, and what the `v1` line
promises. Package signatures are on
The [documentation site](https://skyoo2003.github.io/acor/) covers Redis topologies, the
matching and streaming API, batch and parallel guides, the V2 schema and benchmarks,
deployment and troubleshooting, the `acor` CLI, the experimental server module, and what
the `v1` line promises. Package signatures are on
[pkg.go.dev](https://pkg.go.dev/github.com/skyoo2003/acor/pkg/acor).

## Project
Expand All @@ -103,13 +109,3 @@ promises. Package signatures are on
## License

[Apache License 2.0](LICENSE) — Copyright 2016-2026 Sungkyu Yoo

### Large versioned dictionaries

Use `OpenVersioned` for V3 snapshots, atomic expected-version updates and background
engine replacement. V1/V2 APIs remain compatible. See the [V3 guide](docs/content/reference/versioned.md)
and [performance report](docs/content/reference/versioned-performance.md).

R2 reuses unchanged downloaded buckets and reduces sparse-engine build memory.
R3 adds bounded `Scan`, `MaskText`, and `ReplaceText` with original byte/rune spans;
see [text processing](docs/content/reference/text-processing.md).
5 changes: 5 additions & 0 deletions changes/unreleased/20260906-docs-condense-and-dedupe.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: Documentation
body: "Condensed the documentation site and removed duplicated pages. `preset-engine` and `redis-backed-engine` each carried a Quick Start, an API Reference block, and a preset table; they now split by question — which preset to pick, versus how the engine stays in sync across instances. `reference/api` repeated the Create/Add/Find/Info/Flush/Close surface in a second preset section and gave a one-line heading to each method; methods are tables now, with prose reserved for the contracts that need it. Eight pages carried hand-written nav footers the theme already renders, `monitoring` listed the same Grafana panels twice, and `deployment` repeated the topology and health-check wiring from other pages. Three defects surfaced: `extending/custom-storage` linked to `guides/presets/`, which does not exist; `cli/_index` and `getting-started/installation` disagreed on the command count; and `reference/api` omitted `FindSetContext` from its context-variants list. Performance and verification reports keep every data table unchanged. No documented behavior changed."
time: 2026-09-06T23:30:00+09:00
custom:
Issue: "249"
8 changes: 4 additions & 4 deletions docs/content/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,11 @@ func main() {
</a>
<a class="doc-card" href="guides/">
<strong>Guides</strong>
<span>Use batch, parallel, and Redis-backed engines.</span>
<span>Batch, parallel, and preset-optimized engines.</span>
</a>
<a class="doc-card" href="reference/">
<strong>API Reference</strong>
<span>Review public APIs and storage schemas.</span>
<strong>Reference</strong>
<span>Public API, storage schemas, and measurements.</span>
</a>
<a class="doc-card" href="operations/">
<strong>Operations</strong>
Expand All @@ -62,7 +62,7 @@ func main() {
</a>
<a class="doc-card" href="cli/">
<strong>CLI</strong>
<span>Drive a collection from the shell, twenty commands.</span>
<span>Drive a collection from the shell.</span>
</a>
<a class="doc-card" href="extending/">
<strong>Extending</strong>
Expand Down
47 changes: 17 additions & 30 deletions docs/content/cli/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,30 +5,21 @@ weight: 6

# CLI

`acor` is the third way into the same collection: one binary, twenty commands, every one of
them a shell over the library. It is the entry point for the things a program should not have
to be written for — seeding a dictionary, checking what is in one, running a migration,
grepping a log against keywords that live in Redis.

> **The CLI is not covered by the `v1` compatibility promise.** Quoting
> [Compatibility](../reference/compatibility/#what-is-not-covered) directly:
>
> **The CLI.** Flags, output format, and exit codes of the `acor` command can change in
> any release. If you need a stable contract, call the library rather than parsing CLI
> output.
>
> The library (`pkg/acor`) is covered; this is the surface that is not.

## Install
`acor` is the third way into the same collection: one binary, every command a shell over
the library. It is for the things a program should not have to be written for — seeding a
dictionary, checking what is in one, running a migration, grepping a log against keywords
that live in Redis.

> **The CLI is not covered by the `v1` compatibility promise.** Flags, output format, and
> exit codes can change in any release. If you need a stable contract, call the library
> rather than parsing CLI output. See
> [Compatibility](../reference/compatibility/#what-is-covered).

```bash
go install github.com/skyoo2003/acor/cmd/acor@latest
```

Full instructions, including verifying the install, are on
[Getting Started → Installation](../getting-started/installation/#cli-installation).

## The twenty commands
## The commands

| Group | Commands |
| ----- | -------- |
Expand All @@ -39,17 +30,13 @@ Full instructions, including verifying the install, are on
| Migrate | `migrate`, `migrate-rollback` |
| Dictionary (V3) | `dictionary list\|diff\|replace\|status\|copy-v2\|prune` |

Flags are deliberately not tabulated here. `acor --help` prints the command list followed by
the flag set's own defaults, so each flag's description lives exactly once — next to where the
flag is registered — and cannot drift out of step with a copy on this page.
Flags are deliberately not tabulated here. `acor --help` prints the command list followed
by the flag set's own defaults, so each flag's description lives exactly once — next to
where the flag is registered — and cannot drift out of step with a copy on this page.

`acor version` is the one command that needs no Redis. It prints the version stamped in at
`acor version` is the one command needing no Redis. It prints the version stamped in at
release build time, or `dev` for a binary you built yourself.

## Sections

- [Commands](commands/) - Option ordering, batch modes, the four matching shapes, parallel chunking, and when the local cache earns its memory

## Navigation

← [Server](../server/) | [Extending](../extending/) →
[Commands](commands/) covers what those one-line flag descriptions cannot carry: option
ordering, batch modes, the matching shapes, parallel chunking, and when the local cache
earns its memory.
67 changes: 29 additions & 38 deletions docs/content/cli/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,30 +5,24 @@ weight: 1

# Commands

`acor --help` prints the command list and every flag with its default. This page covers the
behavior those one-line flag descriptions cannot carry: the ordering rule, what each batch
mode does on failure, how the matching commands differ from one another, and when the local
cache is worth its memory.
`acor --help` prints every command and flag with its default. This page covers what those
one-line descriptions cannot carry.

## Options come before the command

CLI options must appear before the command. Batch commands accept keywords as
arguments, or `-` as the only argument to read one keyword per line from stdin:
Batch commands take keywords as arguments, or `-` as the only argument to read one keyword
per line from stdin:

```bash
acor -addr localhost:6379 -batch-mode transactional add-many foo bar "hello world"
printf 'foo\nbar\n' | acor -addr localhost:6379 remove-many -
```

`best-effort` is the default and reports per-keyword failures in JSON while
returning success; `transactional` fails the command if the whole batch cannot
be committed.
`best-effort` is the default: it reports per-keyword failures in JSON and still exits
successfully. `transactional` fails the command if the whole batch cannot be committed.

## Matching

Beyond `find` and `find-index`, the matching commands cover the set, span, and
presence shapes of the same scan:

```bash
acor -addr localhost:6379 find-set "he is him"
acor -addr localhost:6379 contains "he is him"
Expand All @@ -37,19 +31,17 @@ acor -addr localhost:6379 -match-kind leftmost-longest -whole-word \
```

`find-set` reports each keyword once, `contains` stops at the first match, and
`find-matches` reports each occurrence with its rune span in scan order.
`-match-kind` and `-whole-word` apply only to `find-matches`.
`find-matches` reports each occurrence with its rune span in scan order. `-match-kind` and
`-whole-word` apply to `find-matches` only.

`-whole-word` assumes a script that separates words with spaces or punctuation.
In scripts written without inter-word boundaries (CJK, Thai, …) every adjacent
character counts as a word character, so nearly every match is treated as
mid-word and dropped — scan such text without `-whole-word`, or use the library's
`MatchOptions.WordRune` to supply your own boundary rule.
`-whole-word` assumes a script that separates words with spaces or punctuation. In scripts
written without word boundaries (CJK, Thai, …) every adjacent character counts as a word
character, so nearly every match is treated as mid-word and dropped — scan such text
without `-whole-word`, or use the library's `MatchOptions.WordRune`.

## Parallel matching

Parallel matching accepts a text argument, or `-` to read the complete text
from stdin. `word`, `sentence`, and `line` chunk boundaries are available:
Takes a text argument, or `-` to read the whole text from stdin:

```bash
acor -addr localhost:6379 -workers 8 -chunk-size 10000 \
Expand All @@ -59,11 +51,13 @@ acor -addr localhost:6379 -workers 8 \
find-index-parallel - < document.txt
```

`-boundary` takes `word`, `sentence`, or `line`.

## Local cache and presets

Use `-cache` with the normal V2 engine, or select a Redis-backed local preset
engine. Preset mode already keeps a local engine, so `-cache` and `-preset`
cannot be combined:
`-cache` enables the local cache on the normal V2 engine; `-preset` selects a Redis-backed
local engine instead. Preset mode already keeps a local engine, so the two cannot be
combined:

```bash
acor -addr localhost:6379 -cache find-parallel - < document.txt
Expand All @@ -72,23 +66,20 @@ acor -addr localhost:6379 -preset balanced \
-invalidation-poll-interval 30s find-parallel - < document.txt
```

Available presets are `speed`, `balanced`, and `memory-efficient`; `none` is
the compatibility-preserving default. Preset mode requires an explicit Redis
address and does not support `suggest`, `suggest-index`, or migration commands.
The local cache is most useful for parallel matching, where every chunk shares
one CLI process; a one-shot `find` invocation has no later lookup to reuse it.
Presets are `speed`, `balanced`, and `memory-efficient`; `none` is the
compatibility-preserving default. Preset mode requires an explicit Redis address and
supports neither `suggest`/`suggest-index` nor the migration commands.

The trade-offs behind each preset are in
The cache pays off for parallel matching, where every chunk shares one CLI process. A
one-shot `find` has no later lookup to reuse it. Preset trade-offs are in
[Guides → Preset-Optimized Engine](../../guides/preset-engine/).

## Versioned dictionaries

Use `acor -name new-v3-name dictionary list|diff|replace|status|copy-v2|prune`.
Replacement and copying require `--expected-version`; empty replacements and
empty V2 sources require `--allow-empty`. Diff and replace read a JSON string
array from stdin. See the [V3 guide](../../reference/versioned/) for pagination,
case policy, cutover and command examples.

## Navigation
```bash
acor -name new-v3-name dictionary list|diff|replace|status|copy-v2|prune
```

← [CLI](../) | [Extending](../../extending/) →
Replacement and copying require `--expected-version`; empty replacements and empty V2
sources require `--allow-empty`. Diff and replace read a JSON string array from stdin.
Pagination, case policy, and cutover: [V3 guide](../../reference/versioned/#cli).
11 changes: 2 additions & 9 deletions docs/content/extending/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,5 @@ weight: 7

# Extending

Extend ACOR with custom functionality.

## Sections

- [Custom Storage](custom-storage/) - Why Redis is the only backend, and how to test without one

## Navigation

← [CLI](../cli/)
- [Custom Storage](custom-storage/) — why Redis is the only backend, and how to test
without one
Loading