frontevals

A minimal, vitest-native evals library for LLM applications.

Features

• Vitest-native: Works seamlessly inside your existing vitest test suite
• Simple API: Just evalSuite() or evalTest() - no custom CLI needed
• Built-in metrics: exact(), contains(), startsWith(), endsWith(), jsonValid(), hasKeys()
• LLM-as-judge: G-Eval style custom criteria evaluation via OpenAI
• Autoevals integration: Re-exports popular scorers like Factuality, Levenshtein, etc.
• Multi-run support: Run cases multiple times with pass rate thresholds for non-deterministic tasks
• Pretty output: Console tables showing scores and pass/fail status

Installation

npm install @frontsail_ai/frontevals

Quick Start

Basic Usage

import { describe, it, expect } from 'vitest';
import { evalSuite } from '@frontsail_ai/frontevals';
import { exact, contains } from '@frontsail_ai/frontevals/metrics';

describe('Greeting Evals', () => {
  it('passes all cases', async () => {
    const result = await evalSuite({
      name: 'greet function',
      data: [
        { input: 'Alice', expected: 'Hello, Alice!' },
        { input: 'Bob', expected: 'Hello, Bob!' },
      ],
      task: (name) => `Hello, ${name}!`,
      scorers: [exact(), contains('Hello')],
    });

    expect(result.summary.passRate).toBe(1);
  });
});

Even Simpler: evalTest Helper

import { evalTest } from '@frontsail_ai/frontevals/vitest';
import { exact } from '@frontsail_ai/frontevals/metrics';

describe('String Utils', () => {
  evalTest('uppercase works', {
    data: [
      { input: 'hello', expected: 'HELLO' },
      { input: 'world', expected: 'WORLD' },
    ],
    task: (s) => s.toUpperCase(),
    scorers: [exact()],
  });
});

API Reference

evalSuite(config)

Run an evaluation suite and return structured results.

interface EvalSuiteConfig<TInput, TOutput> {
  name: string;                    // Suite name for display
  data: EvalData<TInput>[];        // Test cases (or async function returning them)
  task: (input: TInput) => TOutput | Promise<TOutput>;  // Function to evaluate
  scorers: Scorer<TOutput>[];      // Scoring functions
  runs?: number;                   // Times to run each case (default: 1)
  threshold?: number;              // Pass threshold 0-1 (default: 1.0)
}

interface EvalData<TInput> {
  input: TInput;
  expected?: unknown;
  name?: string;                   // Custom display name
}

evalTest(name, config, threshold?)

Vitest helper that creates a test case automatically.

evalTest('test name', {
  data: [...],
  task: (input) => output,
  scorers: [...],
}, 0.9);  // Optional threshold (default: 1.0)

Built-in Metrics

Import from frontevals/metrics:

Metric	Description
exact()	Exact string match
contains(str)	Output contains substring
startsWith(str)	Output starts with prefix
endsWith(str)	Output ends with suffix
jsonValid()	Output is valid JSON
hasKeys(keys)	Output object has required keys

Example

import { exact, contains, jsonValid, hasKeys } from '@frontsail_ai/frontevals/metrics';

const result = await evalSuite({
  name: 'API Response',
  data: [{ input: 'query', expected: '{"status":"ok"}' }],
  task: async (q) => await api.call(q),
  scorers: [
    jsonValid(),
    hasKeys(['status']),
    contains('ok'),
  ],
});

G-Eval (LLM-as-Judge)

Use custom criteria evaluated by an LLM:

import { gEval } from '@frontsail_ai/frontevals/metrics';

const result = await evalSuite({
  name: 'Support Bot',
  data: [
    { input: 'My order is late', expected: 'Apologize and offer help' },
  ],
  task: async (q) => await bot.respond(q),
  scorers: [
    gEval({ criteria: 'Is the response empathetic and helpful?' }),
    gEval({
      name: 'actionable',
      criteria: 'Does the response provide a clear next step?',
      threshold: 0.8,
    }),
  ],
});

gEval Options

gEval({
  name?: string;           // Scorer name (default: 'gEval')
  criteria: string;        // Plain language criteria
  steps?: string[];        // Optional evaluation steps
  threshold?: number;      // Pass threshold (default: 0.7)
  model?: string;          // OpenAI model (default: 'gpt-4o-mini')
})

Note: Requires OPENAI_API_KEY environment variable.

Autoevals Integration

Re-exported scorers from the autoevals library:

import { Factuality, Levenshtein } from '@frontsail_ai/frontevals/autoevals';

const result = await evalSuite({
  name: 'Chatbot Quality',
  data: [
    { input: 'What is TypeScript?', expected: 'A typed superset of JavaScript' },
  ],
  task: async (q) => await chatbot.answer(q),
  scorers: [Factuality, Levenshtein],
});

Available: Factuality, Levenshtein, ClosedQA, Battle, Humor, Security, Summary, Translation

Multiple Runs (Non-Deterministic Tasks)

For LLM tasks with variable outputs, run multiple times and set a pass threshold:

const result = await evalSuite({
  name: 'Chatbot Consistency',
  data: [
    { input: 'Explain recursion', expected: 'A clear explanation' },
  ],
  task: async (q) => await chatbot.answer(q),
  scorers: [gEval({ criteria: 'Is the explanation clear?' })],
  runs: 5,          // Run each case 5 times
  threshold: 0.8,   // 80% of runs must pass
});

console.log(result.results[0].passRate); // e.g., 0.8

Console Output

Single Run Mode

┌─────────────────────────────────────────────────────────────────┐
│                     greet function                              │
├──────────┬────────────────┬───────┬──────────┬─────────┬────────┤
│ Case     │ Output         │ exact │ contains │ Score   │ Status │
├──────────┼────────────────┼───────┼──────────┼─────────┼────────┤
│ Alice    │ Hello, Alice!  │ 1.00  │ 1.00     │ 1.00    │ ✓ PASS │
│ Bob      │ Hello, Bob!    │ 1.00  │ 1.00     │ 1.00    │ ✓ PASS │
├──────────┴────────────────┴───────┴──────────┴─────────┴────────┤
│ Summary: 2/2 passed (100%)                                      │
└─────────────────────────────────────────────────────────────────┘

Multiple Runs Mode

┌───────────────────────────────────────────────────────────────────────┐
│                     Chatbot Consistency (5 runs, 80% threshold)       │
├────────────────────┬─────────┬───────────┬──────────┬─────────────────┤
│ Case               │ gEval   │ Runs      │ PassRate │ Status          │
├────────────────────┼─────────┼───────────┼──────────┼─────────────────┤
│ Explain recursion  │ 0.85    │ 4/5       │ 80%      │ ✓ PASS          │
│ What is a closure  │ 0.60    │ 3/5       │ 60%      │ ✗ FAIL (< 80%)  │
├────────────────────┴─────────┴───────────┴──────────┴─────────────────┤
│ Summary: 1/2 cases passed | 7/10 total trials passed (70%)            │
└───────────────────────────────────────────────────────────────────────┘

Result Types

interface SuiteResult {
  name: string;
  results: EvalResult[];
  summary: {
    total: number;
    passed: number;
    failed: number;
    passRate: number;
    totalTrials: number;
    byScorer: Record<string, { avgScore: number; passRate: number }>;
  };
  runs: number;
  threshold: number;
}

interface EvalResult {
  name: string;
  input: unknown;
  expected?: unknown;
  trials: TrialResult[];
  passRate: number;
  pass: boolean;
}

interface TrialResult {
  trialIndex: number;
  output: unknown;
  scores: Array<{ name: string; score: number; pass: boolean }>;
  pass: boolean;
}

Custom Scorers

Create custom scorers matching the autoevals signature:

import type { Scorer } from '@frontsail_ai/frontevals';

const lengthScorer: Scorer<string> = ({ output, expected }) => {
  const diff = Math.abs(output.length - String(expected).length);
  const score = Math.max(0, 1 - diff / 100);
  return { score, name: 'length' };
};

// Async scorers are supported
const apiScorer: Scorer<string> = async ({ output }) => {
  const result = await externalApi.evaluate(output);
  return { score: result.score, name: 'api', metadata: result };
};

---

Development

Setup

git clone <repo-url>
cd frontevals
npm install

Scripts

npm test          # Run tests
npm run test:watch # Run tests in watch mode
npm run build     # Build TypeScript

Running Tests with Coverage

npx vitest run --coverage

Project Structure

frontevals/
├── src/
│   ├── index.ts          # Main exports
│   ├── types.ts          # TypeScript interfaces
│   ├── eval-suite.ts     # Core evalSuite function
│   ├── reporter.ts       # Console table output
│   ├── vitest.ts         # evalTest helper
│   ├── autoevals.ts      # Re-exports from autoevals
│   └── metrics/
│       ├── index.ts      # Built-in metrics
│       └── geval.ts      # G-Eval LLM-as-judge
├── tests/
│   ├── eval-suite.test.ts
│   ├── metrics.test.ts
│   ├── geval.test.ts
│   ├── reporter.test.ts
│   ├── vitest-helper.test.ts
│   ├── exports.test.ts
│   └── output.test.ts
├── package.json
├── tsconfig.json
└── vitest.config.ts

Adding New Metrics

1. Create scorer function in src/metrics/index.ts or a new file
2. Export from src/metrics/index.ts
3. Add tests in tests/metrics.test.ts

Example:

// src/metrics/index.ts
export function regex(pattern: RegExp): Scorer<unknown> {
  return ({ output }): ScoreResult => {
    const score = pattern.test(String(output)) ? 1 : 0;
    return { score, name: 'regex' };
  };
}

Dependencies

• autoevals: LLM evaluation metrics
• console-table-printer: Pretty console tables
• openai: Required for gEval

Peer Dependencies

• vitest: >=1.0.0

License

MIT