Search Tools

A suite of Go CLI tools for assessing Elasticsearch/OpenSearch clusters and recommending Aiven for OpenSearch migration plans.

Tools

Tool	Purpose
cluster-ingest	High-performance bulk data loader (local files or URL streaming)
cluster-simulate	Generate realistic live traffic against a cluster
cluster-assess	Continuously observe cluster workload and produce a signed profile
cluster-recommend	Generate Aiven plan recommendations from a profile (PDF, JSON, or text)
cluster-mock	Serve a configurable mock cluster for testing the assess/discover layer

Workflow

# 1. Load data into the cluster (if needed for testing)
cluster-ingest \
  --host https://es.example.com:9200 \
  --auth basic --user admin --pass secret \
  --config configs/github-archive.json \
  --url-pattern "https://data.gharchive.org/{date}-{hour}.json.gz" \
  --start 2024-01-01 --end 2024-01-07 \
  --batch-size 5000 --workers 8

# 2. Generate continuous workload (leave running)
cluster-simulate \
  --host https://es.example.com:9200 \
  --auth basic --user admin --pass secret \
  --mode balanced --write-rate 500 --read-rate 200 --workers 8

# 3. Assess the cluster (runs until Ctrl+C or --duration)
cluster-assess \
  --host https://es.example.com:9200 \
  --auth basic --user admin --pass secret \
  --duration 30m --output profile.json

# 4. Generate recommendation
cluster-recommend \
  --profile profile.json \
  --project my-project --cloud google-us-central1 \
  --token $AIVEN_TOKEN \
  --pdf recommendation.pdf --page-size letter

Supported Source Clusters

• Elasticsearch 7.x, 8.x (self-managed)
• Elastic Cloud
• OpenSearch 1.x, 2.x, 3.x (self-managed)
• AWS Managed OpenSearch Service
• Aiven for OpenSearch

Auto-detected from the cluster's root endpoint response.

Authentication

All cluster-facing tools default to basic auth. Available methods:

--auth basic --user <user> --pass <pass>       # Basic (default)
--auth apikey --apikey <key>                   # API key
--auth cert --cert <path> --key <path>         # mTLS
--auth none                                    # No auth (must be explicit)

TLS options:

--ca <path>        # Trust a custom CA certificate (all cluster-facing tools)
--insecure         # Skip TLS verification (cluster-ingest, cluster-simulate)

cluster-ingest

Bulk loads data from local files or streaming URLs with config-driven routing, field extraction, and time bucketing.

# From local files
cluster-ingest --source-pattern "/path/to/data/*.json.gz" --config ingest.json ...

# Stream from URLs (no disk needed)
cluster-ingest --url-pattern "https://example.com/{date}-{hour}.json.gz" \
  --start 2024-01-01 --end 2024-01-07 --config ingest.json ...

Add --require-validation md5 (or crc32c) to abort the run if any remote source file is missing its checksum or fails verification. Checksums come from the GCS x-goog-hash response header; in strict mode all files are HEAD-checked before any data is downloaded. Applies to --url-pattern mode only. Without the flag, files are still validated against their md5 best-effort (mismatches are warned and skipped).

Ingest Config

{
  "index_pattern": "{type}-{created_at|daily}",
  "include_types": ["PushEvent", "PullRequestEvent"],
  "route_field": "type",
  "time_field": "created_at",
  "template": {"shards": 3, "replicas": 1},
  "mappings": {
    "PushEvent": {
      "include": ["id", "created_at"],
      "flatten": {"actor.login": "user_login", "repo.name": "repo_name"}
    }
  }
}

cluster-simulate

Generates realistic read/write traffic based on the cluster's actual data.

cluster-simulate \
  --host https://... --auth basic --user admin --pass secret \
  --mode write-heavy --write-rate 500 --read-rate 50 --workers 8 \
  --burst-multiplier 3

Modes: write-heavy, read-heavy, balanced

Bursting: --burst-multiplier N randomly spikes traffic to Nx the baseline rate at unpredictable intervals, simulating real-world traffic patterns.

Introspects the cluster to build queries matching actual field types and index structures. Runs until Ctrl+C or --duration.

cluster-assess

Continuously observes cluster workload metrics and produces a signed profile.

cluster-assess \
  --host https://... --auth basic --user admin --pass secret \
  --duration 30m --output profile.json

Collects: cluster topology, node specs, all index metadata/mappings, ILM/ISM policies, templates, plugins, vector search configuration (k-NN engine, dimensions, method, space type, memory stats), authentication configuration (security enabled, auth methods, TLS), ML workloads (anomaly detectors, trained models, data frame analytics, connectors), document sampling statistics. Continuously observes indexing rates, search rates, heap pressure, GC collection rates, thread pool rejections.

Classifies workload as: write-heavy, write-dominant, balanced, read-dominant, read-heavy, or idle.

Additional flags:

• --upload <email> — upload the profile to Aiven, associated with the given email address
• --environment <aws|gcp|azure|on-prem> — override the auto-detected cluster environment
• --sample-percent <n> — percentage of documents to sample per shard (default 1)
• --sample-max <n> — maximum documents to sample per shard (default 1000)
• --burst-factor <n> — override the observed burst factor (must be > 1.0; default 1.5 when no monitoring data)

The profile is anonymized by default using a random salt. A mapping file is written locally for looking up original index names when needed (profile_idx_map.json in the current directory, or <output>_idx_map.json when using --output). Retain this file — it is the only way to reverse anonymized names. It should not be shared externally.

Detects cloud environment via ipinfo.io IP lookup (GCP, AWS, Azure) when node IPs are public or connection IP is identifiable.

Recording and Replay (VCR)

Record all API responses for test replay:

cluster-assess \
  --host https://... --auth basic --user admin --pass secret \
  --duration 30m \
  --record ./fixtures/my-cluster.jsonl \
  --output /dev/null

Replay a recording instantly (no live cluster needed):

cluster-assess \
  --host http://dummy:9200 \
  --auth none \
  --replay ./fixtures/my-cluster.jsonl.gz \
  --output profile.json

Replay matches responses by method+path, uses the recorded duration automatically, and completes in under 1 second regardless of original recording length. Files can be gzipped for storage.

Sample Output

{
  "version": "1.0",
  "generated_at": "2026-05-10T19:25:47Z",
  "tool_version": "0.1.0",
  "source": {
    "type": "self-managed",
    "distribution": "elasticsearch",
    "version": "8.19.15",
    "cluster_name": "es-test-cluster",
    "cluster_uuid": "0oHX0i5mSLedVDkHqrDckg",
    "environment": {
      "inferred": "on-prem"
    }
  },
  "nodes": {
    "total": 3,
    "by_role": {"data": 3, "ingest": 3, "master": 3},
    "topologies": [
      {
        "roles": ["data", "ingest", "master"],
        "count": 3,
        "heap_gb": 2,
        "cpu_cores": 2,
        "ram_gb": 4,
        "disk_gb": 78.5,
        "os": "Debian GNU/Linux 12 (bookworm)",
        "jvm_version": "26"
      }
    ]
  },
  "indices": {
    "total_count": 28,
    "total_docs": 27989958,
    "total_size_bytes": 10130287449,
    "total_primary_shards": 84,
    "total_replica_shards": 84
  },
  "workload": {
    "classification": {
      "pattern": "write-dominant",
      "read_ratio": 0.22,
      "write_ratio": 0.78,
      "burst_factor": 0
    },
    "indexing": {"docs_per_sec": 999.9},
    "search": {"queries_per_sec": 287.3, "fetch_per_sec": 88.4},
    "memory_pressure": {
      "avg_heap_used_percent": 36.4,
      "max_heap_used_percent": 61.7
    }
  },
  "authentication": {
    "security_enabled": true,
    "tls_enabled": true,
    "auth_methods": ["native", "ldap"]
  },
  "vector_search": {
    "enabled": true,
    "plugin_version": "2.11.0",
    "indices": [
      {
        "name": "embeddings",
        "fields": [
          {
            "name": "content_vector",
            "dimension": 768,
            "engine": "faiss",
            "method": "hnsw",
            "space_type": "l2",
            "parameters": {"ef_construction": 256, "m": 16}
          }
        ]
      }
    ],
    "stats": {
      "total_graph_memory_bytes": 1073741824,
      "graph_count": 12,
      "cache_capacity_reached": false,
      "circuit_breaker_triggered": false
    }
  },
  "provisioning_assessment": {
    "overall": "mixed",
    "signals": [
      {"resource": "heap", "status": "over-provisioned", "detail": "avg 36%, peak 62%"},
      {"resource": "shard_density", "status": "under-provisioned", "detail": "28.0 shards/GB heap"}
    ]
  }
}

cluster-recommend

Generates sizing recommendations from a cluster profile using live Aiven plan data.

# Text output
cluster-recommend --profile profile.json --project my-project --cloud aws-us-east-1 \
  --token $AIVEN_TOKEN

# PDF report
cluster-recommend --profile profile.json --project my-project --cloud google-us-central1 \
  --token $AIVEN_TOKEN --pdf report.pdf --page-size A4

# JSON output
cluster-recommend --profile profile.json --project my-project --cloud google-us-central1 \
  --token $AIVEN_TOKEN --json

# Pipe directly from assess (no intermediate file)
cluster-assess --host https://... --auth basic --user admin --pass secret \
  --duration 30m | \
  cluster-recommend --profile - --project my-project --cloud aws-us-east-1 \
  --token $AIVEN_TOKEN

Sizing Strategies

• --strategy workload (default) — sizes to observed throughput and peaks
• --strategy resources — replicates source hardware capacity

--explain adds Claude-generated commentary explaining the recommendation (Claude API integration is not yet wired up).

Output

• Recommended standard plan with monthly cost
• Custom plan recommendation (contact Aiven sales for pricing)
• Alternative plans with binding constraint notes
• ILM-to-ISM conversion analysis (Direct/Partial/Manual/Unsupported)
• ML workload migration notes (anomaly detection, trained models, data frame analytics, connectors)
• Vector search compatibility flags (engine warmup, memory pressure, circuit breaker)
• Authentication incompatibility flags (LDAP, Kerberos, AD not supported on Aiven)
• BYOC (Bring Your Own Cloud) eligibility for AWS/GCP targets
• Migration notes and incompatibilities
• PDF with Aiven branding, QR code, page numbers, and professional formatting

cluster-mock

Serves a configurable in-memory OpenSearch/Elasticsearch cluster that the discover layer can assess — useful for testing cluster-assess without a live cluster. It exposes discovery endpoints (/, /_cluster/*, /_cat/indices, /_nodes, /{index}/_settings, /{index}/_mapping, /_plugins/_knn/stats, and distribution-gated ILM/ISM, security, and ML endpoints) with self-consistent stats.

# Default 3-node cluster on :9200
cluster-mock

# Custom topology and indices, as Elasticsearch, with fault injection
cluster-mock \
  --config cluster.json \
  --distribution elasticsearch --version 8.15.0 \
  --listen :9201 \
  --avg-latency 20ms --latency-variance 10ms \
  --error-rate 0.05 --seed 42

Flags:

• --config <path> — JSON cluster config (topology + indices); default is a 3-node cluster
• --distribution <opensearch|elasticsearch> — override the reported distribution
• --version <string> — override the reported version string
• --listen <addr> — listen address (default :9200)
• --error-rate <0..1> — probability of injecting a 503 on each request
• --avg-latency <dur> / --latency-variance <dur> — artificial latency and jitter
• --seed <n> — RNG seed for deterministic latency/error injection (default 1)
• --knn-circuit-breaker — report the k-NN circuit breaker as triggered
• --bulk-rejection-rate <0..1> — fraction of bulk items rejected with a per-item 429 (HTTP 200)

Install

make

Binaries are written to bin/.

Test

go test ./...

Using Recorded Fixtures in Tests

client, _ := discover.NewClient(discover.ClientConfig{
    Host:       "http://dummy:9200",
    Auth:       discover.AuthNone,
    ReplayFile: "./fixtures/es-gke-4node.jsonl.gz",
})