Skip to content

TESTING.md

Latest commit

 

History

History
321 lines (229 loc) · 14.1 KB

TESTING.md

File metadata and controls

321 lines (229 loc) · 14.1 KB
Raw

Testing Guide

Fast Validation (before commit)

# Quick: run only contract tests
pnpm exec jest --config jest.config.js tests/mcp/mcp-tools-contract.test.ts --runInBand

# Medium: run all tests without coverage (fast)
pnpm exec jest --config jest.config.js --no-coverage --runInBand

# Full: run all tests with coverage (takes ~2min)
pnpm test

Full Validation (before PR)

pnpm run lint
pnpm run build
pnpm test
pnpm run verify

Current baseline: 96 test files, 1311 tests (all passing).

The baseline was bumped after the 2026-07 security hardening pass, which added tests for the config sandbox, shell-less spawn, interpreter-escape blocking, SSRF/DNS rebinding, cache path validation, and export sanitization.

Test Suite Map

Tests are organized in domain subdirectories mirroring src/. Use the commands below to run a specific domain in isolation:

Suite Run command Covers
Audit pnpm exec jest --config jest.config.js tests/audit/ --runInBand Stop gate, integrity, task context, coordinator, long-task
Compaction pnpm exec jest --config jest.config.js tests/compaction/ --runInBand Rolling, watermarks, reinjection, guard, entropy, textrank
Retrieval pnpm exec jest --config jest.config.js tests/retrieval/ --runInBand Hybrid search, ranking, chunking, RAG, LLM ranker, multi-session, conflict resolver
MCP pnpm run test:mcp Protocol, tool contracts, error handling, handler tests
Storage pnpm exec jest --config jest.config.js tests/storage/ --runInBand SQLite, FTS5, vector, CJK, model-switch, event-origin
Plugin pnpm run test:plugin Install, uninstall, config, adapter, RAG subagent
KG pnpm exec jest --config jest.config.js tests/kg/ --runInBand Entity CRUD, relation, graph search, MCP integration
Embedding pnpm run test:embedding sqlite-vec backend, presets, providers, edge cases
Security pnpm exec jest --config jest.config.js tests/security/ --runInBand Firewall, sanitize, FTS5 escape, API key headers
TUI pnpm exec jest --config jest.config.js tests/tui/ --runInBand Settings panel features, plugin callbacks, dimension validation
Utils pnpm exec jest --config jest.config.js tests/utils/ --runInBand Logger, memory monitor, perf tracker
CLI pnpm exec jest --config jest.config.js tests/cli/ --runInBand Log viewer command
Property pnpm exec jest --config jest.config.js tests/property/ --runInBand RRF+search invariants (fuzzing)
Scenarios pnpm exec jest --config jest.config.js tests/scenarios/ --runInBand Lifecycle, logging e2e, conflict, bridge
Load pnpm run test:load 10k event performance, perf budget (manual, excluded from CI)

You can also run individual test files:

pnpm exec jest --config jest.config.js tests/storage/sqlite-backend.test.ts
pnpm exec jest --config jest.config.js tests/mcp/mcp-tools-contract.test.ts

Writing a New Test

Step-by-Step

  1. Choose the right directory. Most tests are flat in tests/<name>.test.ts; tests with multiple related files use subdirectories (tests/embedding/, tests/load/, tests/scenarios/, tests/security/, tests/tui/, tests/cli/, tests/utils/).

  2. Create the test file and follow the naming convention:

    // tests/retrieval/ranking-coverage.test.ts
import { createTestDb } from '../test-utils';

describe('RankingEngine', () => {
  const { db, cleanup } = createTestDb();

  afterAll(cleanup);

  it('should rank user messages above assistant messages', () => {
    // Arrange: insert events, configure engine
    // Act: call the function under test
    // Assert: verify output
  });

  it('should handle empty input gracefully', () => {
    // Verify error paths as well
  });
});

  3. Use createTestDb() from tests/test-utils.ts for tests that need a real SQLite database with schema initialized. It creates an isolated database in the OS temp directory with automatic WAL/journal cleanup.

  4. Use SqliteBackend(':memory:') for pure in-memory tests that do not need schema or persistence.

  5. For MCP tool tests, import listTools() and callTool() from src/mcp/router.ts:

    import { callTool } from '../../src/mcp/router';
const result = await callTool('memorySearch', { query: 'test', mode: 'keyword' });

  6. Follow the naming pattern: describe('<Subsystem>', () => { it('should <expected behavior>', () => {}) }).

  7. Verify both success and failure paths. Every new feature should have at least one test for the happy path and one for edge cases.

  8. Run the new test in isolation first:

    pnpm exec jest --config jest.config.js tests/retrieval/ranking-coverage.test.ts --runInBand

Example: Adding a Storage Test

// tests/storage/event-integrity.test.ts
import { createTestDb } from '../test-utils';

describe('Event Integrity', () => {
  const { db, cleanup } = createTestDb();

  afterAll(cleanup);

  it('should assign monotonically increasing sequence IDs', () => {
    const seq1 = db.insertEvent({
      session_id: 'test-session',
      event_type: 'user_message',
      actor: 'user',
      timestamp: Date.now(),
      content: 'Hello',
    });
    const seq2 = db.insertEvent({
      session_id: 'test-session',
      event_type: 'assistant_message',
      actor: 'claude',
      timestamp: Date.now(),
      content: 'Hi',
    });
    expect(seq2).toBeGreaterThan(seq1);
  });

  it('should reject events with invalid event_type', () => {
    expect(() => {
      db.insertEvent({
        session_id: 'test-session',
        event_type: 'invalid_type' as any,
        actor: 'user',
        timestamp: Date.now(),
        content: 'test',
      });
    }).toThrow();
  });
});

Test Environment Setup

createTestDb() Helper

tests/test-utils.ts exports createTestDb() for tests that need a fully initialized SQLite database:

export interface TestDb {
  db: SqliteBackend;
  cleanup: () => void;
}
export function createTestDb(): TestDb;

It creates a database in os.tmpdir()/cc-test-dbs/ with a unique timestamp+random filename, runs the full schema migration, and returns a cleanup function that closes the database and removes WAL/SHM/journal files. Use createTestDb() when your test needs the full schema (FTS5 tables, constraints, decisions, KG).

In-Memory Databases

For lightweight tests that do not need persistence or schema, use SqliteBackend(':memory:'):

const db = new SqliteBackend(':memory:');

In-memory databases bypass the filesystem entirely. They are suitable for pure logic tests but cannot test schema-dependent features like FTS5 triggers or foreign key constraints.

When to Use Which

Scenario Use
Tests involving FTS5 search, constraints, or the full schema createTestDb()
Tests that need to verify file-based behavior (WAL, SHM) createTestDb()
Pure logic tests, mock-heavy tests SqliteBackend(':memory:')
Tests that need to open the same DB from multiple handles createTestDb()
Performance tests with large datasets createTestDb()

Mock Patterns

Firewall Engine

Tests that depend on the firewall typically instantiate a real FirewallEngine with a test database:

import { FirewallEngine } from '../../src/firewall/engine';
import { SqliteBackend } from '../../src/storage/sqlite-backend';

describe('FirewallEngine', () => {
  let backend: SqliteBackend;
  let firewall: FirewallEngine;

  beforeEach(() => {
    backend = new SqliteBackend(':memory:');
    firewall = new FirewallEngine(backend);
  });

  it('should block forbidden npm commands', async () => {
    const result = await firewall.check('bash', { command: 'npm install react' });
    expect(result.allowed).toBe(false);
  });
});

Embedding Provider Mock

For tests that exercise search or retrieval without a real embedding service, create a lightweight mock:

function makeMockProvider(dim = 2) {
  return {
    embed: jest.fn().mockResolvedValue([new Array(dim).fill(0.1)]),
    getDimension: () => dim,
    getProviderName: () => 'mock',
  };
}

// Used in HybridSearchEngine tests
const se = new HybridSearchEngine(db, makeMockProvider());

The DeterministicEmbeddingProvider in src/embedding/registry.ts is also available for tests that need consistent vector outputs without network access.

Module-Level Mocks

For tests that isolate from heavy dependencies, use jest.mock() at the top of the test file:

jest.mock('../../src/mcp/state', () => ({
  ensureStorage: jest.fn(),
  ensureVectorReady: jest.fn().mockResolvedValue(undefined),
}));

Always restore mocks in afterEach when using shared state.

Coverage Thresholds

Configured in jest.config.js:

  • Global: branches >= 25%, functions >= 30%, lines >= 40%, statements >= 40%

The single global floor avoids a Jest 29 pitfall where coverageThreshold keys are interpreted as file path matches, not globs, causing per-glob overrides to silently no-op. A single global floor is robust across Jest patch releases and CI matrix jobs.

The four modules with intentionally lower coverage (src/audit/ohmyopenagent-adapter.ts, src/compaction/decay.ts, src/compaction/guard.ts, src/retrieval/graph-search.ts) are tracked as follow-up work. Raise the global floor only with targeted tests that prove the new floor across all four metrics simultaneously.

CI Matrix

CI (ci.yml) runs fail-fast: false across 6 cells:

os: [ubuntu-latest, windows-latest, macos-latest]
node-version: ['18', '22']
exclude: windows-latest x node 18

Node 22 is the recommended contributor runtime (pinned by .nvmrc). Node 18 is the lowest supported floor (declared by package.json engines.node).

The windows-latest x Node 18 combination is excluded because better-sqlite3@12.11.1 does not ship a Windows prebuilt binary for the Node 18 ABI. GitHub-hosted Windows runners lack a compatible Visual Studio C++ toolchain for node-gyp rebuilds. Windows contributors on Node 18 must install the "Desktop development with C++" Visual Studio workload or upgrade to Node 22.

Each matrix cell runs: pnpm install, pnpm rebuild better-sqlite3, pnpm tsc --noEmit, pnpm run lint, pnpm run build, and pnpm test (excluding tests/load). Coverage is uploaded to Codecov. A gating job depends on the full matrix passing.

Integration Tests

MCP Protocol Tests

tests/mcp/mcp-protocol.test.ts and tests/mcp/mcp-tools-contract.test.ts verify the MCP server at the protocol level. These tests:

  • Call listTools() and verify all 6 tools are present with correct schemas
  • Call callTool() for each tool and verify response structure matches ToolResult
  • Test error responses for invalid tool names and bad arguments
  • Verify CJK search routing behaves correctly

Run MCP integration tests:

pnpm run test:mcp

Lifecycle Scenario Tests

tests/scenarios/ contains end-to-end workflow tests that simulate real usage:

Test Simulates
lifecycle-scenario.test.ts Full session: start, prompt, tools, compaction, stop
logging-e2e.test.ts File logging across a complete workflow
conflict-scenario.test.ts Plugin conflict detection and resolution
bridge-scenario.test.ts Context bridging after model switches

These tests create real databases, insert realistic event sequences, and verify that all subsystems (compaction, firewall, audit, logging) work together correctly.

Run scenario tests:

pnpm exec jest --config jest.config.js tests/scenarios/ --runInBand

Load Tests

The tests/load/ suite (10k-event performance + perf budget) is excluded from the default CI run because it inserts 10000 events into a real SQLite WAL database and measures p99 latency. In shared CI runners it can exceed the 120s Jest timeout and produce flaky failures caused by runner contention.

Run load tests manually before releases or whenever storage/compaction performance might have shifted:

pnpm run test:load        # tests/load only, 600s per-test timeout, no coverage
pnpm run test:all         # all tests including load, 600s per-test timeout, with coverage

CI excludes tests/load via --testPathIgnorePatterns='tests/load'. If you change CI behavior to include load tests, raise testTimeout to at least 600000 ms and gate by runner class.

Native Dependency Notes

better-sqlite3 is a native module. On Windows, ensure Visual Studio Build Tools are installed. On macOS/Linux, ensure python3 and C++ build tools are available. CI rebuilds better-sqlite3 explicitly with pnpm rebuild better-sqlite3.