GraphBench — CognoDB Cloud Benchmark

GraphBench is a reproducible benchmark harness for the Wexa AI take-home assignment.

Scope: CognoDB Cloud is the only database under test. The official neo4j-driver Node.js package is used only as the Bolt-compatible client driver. Neo4j is not a benchmark target.

What is implemented

• SNAP Wiki-Vote dataset (7,115 nodes / 103,689 directed relationships)
• Dataset checksum and manifest validation
• Deterministic workload-node selection
• CognoDB schema and index preparation
• Batched dataset loading with ingestion throughput
• 1-hop, 2-hop and 3-hop traversal latency
• Point lookup
• Indexed/filtered lookup
• Aggregation
• 100+ measured iterations after warm-up
• p50, p95, mean, min, max and standard deviation
• Mixed read/write benchmark at 1 / 10 / 40 concurrent clients
• Configurable read/write mix (default 70/30)
• Runtime/platform/footprint metadata
• Raw result JSON
• Markdown + CSV report generation
• Deterministic comparison helper
• Optional Groq analysis through the OpenAI-compatible SDK
• Unit tests
• No credentials in source control

Architecture

                    SNAP Wiki-Vote
                          |
                          v
                  Dataset Validator
                          |
                          v
                    CognoDB Loader
                          |
                    neo4j-driver
                          |
                          v
                    CognoDB Cloud
                          |
             +------------+------------+
             |                         |
             v                         v
       Read Benchmark             Mixed Benchmark
       1/2/3-hop                   1/10/40 clients
       point lookup                read/write
       filtered lookup
       aggregation
             |                         |
             +------------+------------+
                          v
                    Raw Result JSON
                          |
                          v
                 Statistical Analyzer
                          |
             +------------+------------+
             |                         |
             v                         v
       Markdown/CSV report          Groq AI
                                  (optional)

The benchmark engine is deterministic and factual. AI is strictly downstream of verified measurements.

Setup

npm install
copy .env.example .env

Set:

COGNODB_URI=bolt+s://<instance-id>.databases.cognodb.cloud
COGNODB_USERNAME=cognodb
COGNODB_PASSWORD=<password>
COGNODB_REGION=<region>
COGNODB_TIER=c0

GROQ_API_KEY=<optional>
GROQ_MODEL=<optional>

Never commit .env.

Validate the environment

npm run build
npm test
npm run benchmark:validate

Validate the final dataset

npm run benchmark:dataset

The manifest records the source, expected size, actual size and SHA-256. The repository also contains data/edges.txt as a tiny development fixture; it is not valid final benchmark data.

If the dataset must be downloaded again:

npm run dataset:download
npm run benchmark:dataset

Load CognoDB

npm run dataset:load -- --file data/wiki-Vote.txt --output results/load-wiki-vote.json

The timed ingestion window begins after connectivity, schema preparation and GraphBench-owned data cleanup. The load result records:

• wall-clock load time
• nodes/second
• relationships/second
• batch count
• dataset checksum
• runtime and platform metadata

Read benchmark

npm run benchmark:run -- \
  --file data/wiki-Vote.txt \
  --iterations 100 \
  --warmup 20 \
  --sample-nodes 100 \
  --output results/wiki-vote-read.json

Required workloads:

Category	Workload
Traversal	1-hop
Traversal	2-hop
Traversal	3-hop
Lookup	point lookup
Lookup	indexed/filtered lookup
Aggregation	relationship-type count

Raw samples are preserved. p50/p95 are calculated by code, never by an LLM.

Mixed benchmark

Wexa highlights a concurrency sweep, so run:

npm run benchmark:mixed -- --file data/wiki-Vote.txt --concurrency 1 --duration 30 --warmup 10 --read-percent 70 --output results/mixed-c1.json
npm run benchmark:mixed -- --file data/wiki-Vote.txt --concurrency 10 --duration 30 --warmup 10 --read-percent 70 --output results/mixed-c10.json
npm run benchmark:mixed -- --file data/wiki-Vote.txt --concurrency 40 --duration 30 --warmup 10 --read-percent 70 --output results/mixed-c40.json

The mixed benchmark reports total/read/write QPS, actual achieved read percentage, errors, and p50/p95 latency.

Writes go to a dedicated GraphBenchWriteProbe label so the Wiki-Vote topology is not modified by the mixed test.

Footprint

npm run benchmark:footprint -- --output results/footprint.json

Observable values are recorded. Unsupported managed-service metrics are explicitly reported as null / not observable; they are not guessed.

Generate the report

After running the read and mixed benchmarks:

npm run report -- \
  results/load-wiki-vote.json \
  results/wiki-vote-read.json \
  results/mixed-c1.json \
  results/mixed-c10.json \
  results/mixed-c40.json \
  results/footprint.json \
  --output results/report.md \
  --csv results/report.csv

The report generator only reads executed JSON results. It never fabricates missing values.

AI analysis with Groq

Optional:

npm run ai:analyze -- \
  results/wiki-vote-read.json \
  results/mixed-c1.json \
  results/mixed-c10.json \
  results/mixed-c40.json \
  --output results/ai-analysis.json

Groq uses:

new OpenAI({
  apiKey: process.env.GROQ_API_KEY,
  baseURL: "https://api.groq.com/openai/v1",
});

The AI receives verified benchmark evidence only. It cannot access CognoDB, credentials, connection URIs, or the live benchmark path.

AI must:

• distinguish observation from hypothesis;
• avoid invented implementation details;
• avoid causal claims without evidence;
• include confidence and caveats;
• never alter benchmark measurements.

If Groq is unavailable, deterministic analysis still runs.

Compare runs

npm run benchmark:compare -- results/mixed-c1.json results/mixed-c10.json results/mixed-c40.json

Reproducibility checklist

Before publishing:

• clean CognoDB dataset load
• dataset checksum recorded
• CognoDB tier and region recorded
• client OS/Node/CPU/RAM recorded
• same client machine used for every run
• warm-up completed
• >=100 read measurements
• 1/10/40 concurrency completed
• failures and caveats retained
• footprint values marked observable/not observable
• raw JSON preserved
• report generated from raw JSON
• AI analysis reviewed by a human
• no secrets committed

Important assignment limitation

The Wexa assignment requests CognoDB plus at least four other graph databases. This repository intentionally follows the user's current scope decision and benchmarks CognoDB only. Before final submission, this scope must be reconciled with the assignment requirement if Wexa expects competitor platforms.

Do not fabricate competitor results. If competitor benchmarking is added later, reuse the adapter interface and the same workload/reporting contract.

Current status

Engineering phases complete for the CognoDB-only scope.

Remaining work before submission is execution and evidence, not scaffold implementation:

1. run the final dataset load;
2. run the read benchmark;
3. run 1/10/40 mixed benchmarks;
4. collect footprint;
5. generate report;
6. run/review Groq analysis;
7. add real observations, charts and caveats to the public README;
8. reconcile the CognoDB-only scope with Wexa's explicit multi-platform requirement.

AI Analysis: Hybrid BM25 + Vector Retrieval

GraphBench includes a bounded RAG-style analysis layer for benchmark interpretation.

Benchmark JSON
      |
      v
Deterministic evidence extraction
      |
      +----------+ 
      |          |
      v          v
    BM25     TF-IDF Vector
      |          |
      +----+-----+
           v
     Hybrid Retrieval
     0.45 BM25 + 0.55 Vector
           |
           v
       Top-K evidence
           |
           v
          Groq

The benchmark never calls Groq during measurement. The AI layer receives only deterministic summaries and retrieved evidence. This avoids large prompts such as sending every raw latency sample and makes the analysis resilient to provider token limits.

The vector component is a deterministic local TF-IDF cosine index. It is intentionally not described as neural embeddings. BM25 provides exact lexical retrieval for terms such as p95, TRAVERSE_3_HOP, and concurrency=40, while vector similarity provides a second retrieval signal.

Run:

npm run ai:analyze -- results/wiki-vote-read.json results/mixed-c1.json results/mixed-c10.json results/mixed-c40.json --output results/ai-analysis.json

The output contains the retrieval strategy, selected evidence, deterministic analysis, and optional Groq analysis.

Live dashboard deployment

GraphBench includes a lightweight read-only dashboard intended for Railway/container deployment. It presents checked-in benchmark results and AI insights without exposing benchmark execution publicly.

npm install
npm run build
npm start

See docs/deployment.md for Railway deployment and security guidance.

Web Control Center

GraphBench includes a production-oriented operator dashboard. Start it with npm run web during development or npm start after npm run build.

Pages:

• Dashboard — health, dataset state, latest runs, active task
• Dataset — download, inspect, validate, load
• Benchmarks — read, mixed, smoke, footprint
• Runs — inspect, compare, report, AI analyze, delete artifacts
• Reports — report and generated charts
• AI Analysis — hybrid BM25 + TF-IDF retrieval and Groq results
• Settings — runtime configuration and session admin token

For production, set GRAPHBENCH_ADMIN_TOKEN. The browser never receives CognoDB or Groq credentials.