Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: Bug report
description: Report unexpected Evalon behavior.
title: "bug: "
labels: [bug]
body:
- type: markdown
attributes:
value: "Remove secrets, API keys, and private trace data before submitting."
- type: textarea
id: behavior
attributes:
label: What happened?
validations:
required: true
- type: textarea
id: reproduce
attributes:
label: Minimal reproduction
description: Include code and commands needed to reproduce the problem.
validations:
required: true
- type: input
id: versions
attributes:
label: Evalon and Python versions
validations:
required: true
18 changes: 18 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
name: Feature request
description: Suggest an improvement to Evalon.
title: "feat: "
labels: [enhancement]
body:
- type: textarea
id: problem
attributes:
label: Problem
description: What problem would this solve?
validations:
required: true
- type: textarea
id: proposal
attributes:
label: Proposed solution
validations:
required: true
10 changes: 10 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
## Summary

Describe the change and its user-visible effect.

## Verification

- [ ] `uv run python -m compileall -q evalon`
- [ ] `uv build`
- [ ] Public documentation is updated when needed.
- [ ] No credentials, local trace data, or build output is included.
10 changes: 10 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
version: 2
updates:
- package-ecosystem: uv
directory: /
schedule:
interval: weekly
- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly
27 changes: 27 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: CI

on:
pull_request:
push:
branches: [main]

permissions:
contents: read

jobs:
quality:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v5
with:
version: latest
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install dependencies
run: uv sync --group dev
- name: Compile package
run: uv run python -m compileall -q evalon
- name: Build distribution
run: uv build
30 changes: 30 additions & 0 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Contributor Covenant Code of Conduct

## Our pledge

We pledge to make participation in our community a harassment-free experience
for everyone, regardless of age, body size, disability, ethnicity, gender
identity and expression, level of experience, education, socio-economic
status, nationality, personal appearance, race, religion, or sexual identity
and orientation.

## Our standards

Examples of behavior that contributes to a positive environment include
demonstrating empathy and kindness, respecting differing opinions and
experiences, giving and accepting constructive feedback, and taking
responsibility for mistakes.

Unacceptable behavior includes harassment, discriminatory language, personal
attacks, publishing private information, or other conduct inappropriate in a
professional setting.

## Enforcement

Report unacceptable behavior through the repository's private security
advisory reporting flow. Project maintainers may remove, edit, or reject
contributions and may temporarily or permanently ban contributors for behavior
that violates this Code of Conduct.

This Code of Conduct is adapted from the Contributor Covenant, version 2.1:
https://www.contributor-covenant.org/version/2/1/code_of_conduct.html.
35 changes: 35 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Contributing to Evalon

Thanks for contributing. For substantial changes, open an issue first so the
problem and intended scope can be agreed before implementation begins.

## Development setup

Evalon requires Python 3.11 or later and uses uv.

```bash
git clone https://github.com/sidmanale643/evalon.git
cd evalon
uv sync --group dev
```

Before opening a pull request, run:

```bash
uv run python -m compileall -q evalon
uv build
```

## Pull requests

- Keep each pull request focused on one change.
- Add or update user documentation when behavior changes.
- Do not include credentials, local trace databases, build output, or generated
dependency directories.
- Explain what you verified in the pull request description.

## Reporting bugs

Use the bug-report template and include a minimal reproduction, the Evalon and
Python versions, and the command or code that failed. Remove secrets and
private trace data before submitting an issue.
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2026 Evalon contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
34 changes: 34 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,23 @@
# Evalon

[![CI](https://github.com/sidmanale643/evalon/actions/workflows/ci.yml/badge.svg)](https://github.com/sidmanale643/evalon/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

Evalon adds local tracing, debugging, and evaluation to Python agents. Traces
are stored in SQLite on your machine and can be inspected in the built-in
terminal UI.

## What is in this repository?

```text
evalon/ Python package: tracing, storage, provider wrappers, evals, and TUI
docs/ Architecture and release documentation
.github/ CI, issue forms, dependency updates, and pull-request template
```

Evalon is intentionally local-first. It records trace data to SQLite by
default and does not require a hosted account or service.

## First trace

Create a UV project and install Evalon:
Expand Down Expand Up @@ -165,3 +179,23 @@ The browser dashboard remains available when a web view is more convenient:
```bash
uv run evalon-dashboard
```

## Development

Evalon supports Python 3.11 and later and uses [uv](https://docs.astral.sh/uv/).

```bash
git clone https://github.com/sidmanale643/evalon.git
cd evalon
uv sync --group dev
uv run python -m compileall -q evalon
uv build
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for the contribution workflow,
[docs/architecture.md](docs/architecture.md) for the data model, and
[SECURITY.md](SECURITY.md) to report vulnerabilities privately.

## License

Evalon is released under the [MIT License](LICENSE).
16 changes: 16 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Security policy

## Supported versions

Security fixes are applied to the latest version on the default branch.

## Reporting a vulnerability

Do not report suspected vulnerabilities in public issues. Use the repository's
private security advisory reporting flow:

https://github.com/sidmanale643/evalon/security/advisories/new

Include a description of the issue, affected versions, reproduction steps, and
any suggested mitigation. We will acknowledge reports within seven days and
keep you informed while investigating.
46 changes: 46 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Evalon architecture

Evalon is a local-first observability and evaluation library for Python agent
applications. The public API is exposed from `evalon/__init__.py`; it creates
and manages a process-local `EvalonClient`.

## Trace model

The local SQLite store uses this hierarchy:

```text
Session
Trace
Span
Event
Metric
```

- A session groups traces from one agent lifecycle.
- A trace represents one observed agent run or explicit trace context.
- A span represents a nested LLM, tool, or custom operation.
- Events retain chronological detail, while metrics retain aggregate numeric
values such as latency, tokens, calls, errors, and cost.

`contextvars` keeps active traces and sessions isolated across concurrent async
work, so callers do not need to pass trace handles through their application.

## Modules

| Path | Responsibility |
| --- | --- |
| `evalon/client.py` | Client lifecycle, trace/session contexts, and query facade |
| `evalon/trace.py` | Trace, span, event, metric, and session data models |
| `evalon/storage.py` | SQLite persistence and JSONL compatibility storage |
| `evalon/providers.py` | OpenAI, Anthropic, and OpenRouter instrumentation wrappers |
| `evalon/tools.py` | Sync and async tool instrumentation |
| `evalon/evals/` | Dataset-backed evaluation models, checks, and runners |
| `evalon/tui/` | Local terminal trace inspector |

## Storage and privacy

`evalon.init()` writes to `~/.evalon/evalon-runs.sqlite` unless given an
explicit output path or `EVALON_DB`. SQLite is the default storage format;
`.jsonl` outputs are retained for compatibility. Trace inputs and outputs may
contain sensitive application data, so do not commit local databases or paste
private traces into public issues.
13 changes: 13 additions & 0 deletions docs/releasing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Releasing Evalon

This repository uses a single Python package at the repository root.

1. Update the version in `pyproject.toml`.
2. Run `uv lock`, `uv run ruff check evalon`, and `uv build`.
3. Review the contents of `dist/` and install the wheel in a clean environment
before publishing it.
4. Create a GitHub release with concise user-facing notes and publish the
validated package through the selected package registry.

Do not publish local SQLite files, `.env` files, or recordings that contain
customer or provider data.
17 changes: 17 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,18 @@ version = "0.1.0"
description = "Lightweight tracing and evaluation harness for Python agents"
readme = "README.md"
requires-python = ">=3.11"
license = { text = "MIT" }
authors = [{ name = "Evalon contributors" }]
classifiers = [
"Development Status :: 3 - Alpha",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Topic :: Software Development :: Debuggers",
]
dependencies = [
"httpx>=0.27.0",
"litellm>=1.91.0",
Expand All @@ -14,6 +26,11 @@ dependencies = [
"textual>=5.0",
]

[project.urls]
Homepage = "https://github.com/sidmanale643/evalon"
Repository = "https://github.com/sidmanale643/evalon"
Issues = "https://github.com/sidmanale643/evalon/issues"

[project.optional-dependencies]
providers = [
"anthropic>=0.40.0",
Expand Down
Loading
Loading