Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@ jobs:
python-version: "3.12"
- name: Install
run: pip install -e ".[dev]"
- name: Require the umbra-core detection engine
- name: Require the signetry-core detection engine
run: |
# The release benchmark must run against a umbra-core that has the SAST
# The release benchmark must run against a signetry-core that has the SAST
# engine. Fail loudly if it is missing (a real release misconfiguration).
python -c "from umbra_core import scan_repository" || {
echo "::error::Installed umbra-core lacks scan_repository; pin a umbra-core with the detection engine."
python -c "from signetry_core import scan_repository" || {
echo "::error::Installed signetry-core lacks scan_repository; pin a signetry-core with the detection engine."
exit 1
}
- name: Regenerate the head-to-head table
Expand All @@ -35,11 +35,11 @@ jobs:
{
echo "<!-- Generated by .github/workflows/benchmark.yml — do not edit by hand. -->"
echo ""
umbra-eval corpus --markdown
signetry-eval corpus --markdown
} > docs/BENCHMARK.md
umbra-eval corpus --json > docs/BENCHMARK.json
- name: Gate on parity (Umbra must keep 100% recall, 0 false positives)
run: umbra-eval corpus --min-recall 1.0 --max-fp 0
signetry-eval corpus --json > docs/BENCHMARK.json
- name: Gate on parity (Signetry must keep 100% recall, 0 false positives)
run: signetry-eval corpus --min-recall 1.0 --max-fp 0
- uses: actions/upload-artifact@v7
with:
name: detection-benchmark
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/cla.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ jobs:
path-to-document: "https://github.com/Signetry/eval/blob/main/CLA.md"
branch: "cla-signatures"
allowlist: "dependabot[bot],bkd-dotcom"
custom-notsigned-prcomment: "Thanks for your contribution! Before it can be merged, please read the **[Contributor License Agreement](https://github.com/Signetry/eval/blob/main/CLA.md)** — Umbra is **All Rights Reserved**, and by signing you assign copyright/ownership of your contribution to the Owner (you may not use, sell, or commercialize it yourself). To agree, reply with exactly:"
custom-notsigned-prcomment: "Thanks for your contribution! Before it can be merged, please read the **[Contributor License Agreement](https://github.com/Signetry/eval/blob/main/CLA.md)** — Signetry is **All Rights Reserved**, and by signing you assign copyright/ownership of your contribution to the Owner (you may not use, sell, or commercialize it yourself). To agree, reply with exactly:"
custom-pr-sign-comment: "I have read the CLA Document and I hereby sign the CLA"
custom-allsigned-prcomment: "All contributors have signed the CLA. ✅"
lock-pullrequest-aftermerge: false
34 changes: 17 additions & 17 deletions .github/workflows/eval.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,38 +18,38 @@ jobs:
python-version: ${{ matrix.python-version }}
- name: Install
run: |
# umbra-core is installed from its source repo (a git+https dependency in
# signetry-core is installed from its source repo (a git+https dependency in
# pyproject.toml); it is All Rights Reserved and not on PyPI.
pip install -e ".[dev]"
- name: Ruff
run: ruff check umbra_eval/ tests/
run: ruff check signetry_eval/ tests/
- name: Tests
run: pytest -q
- name: Publish the adversarial report (artifact)
if: always()
run: |
umbra-eval run --markdown > eval-report.md || true
umbra-eval run --json > eval-report.json || true
signetry-eval run --markdown > eval-report.md || true
signetry-eval run --json > eval-report.json || true
- name: Detection head-to-head (corpus vs committed Opus 4.8 capture)
if: always()
run: |
# Regenerates the head-to-head table. The competitor baseline is the
# committed capture in umbra_eval/detection/captures/. Umbra runs live
# committed capture in signetry_eval/detection/captures/. Signetry runs live
# (deterministic, offline). The --min-recall/--max-fp gate activates only
# when the installed umbra-core has the detection engine (scan_repository);
# against an older PyPI umbra-core it degrades to a skip so this repo's CI
# when the installed signetry-core has the detection engine (scan_repository);
# against an older PyPI signetry-core it degrades to a skip so this repo's CI
# is green until the engine ships, then enforces 100% recall / 0 FP.
if python -c "from umbra_core import scan_repository" 2>/dev/null; then
umbra-eval corpus --markdown > detection-headtohead.md
umbra-eval corpus --json > detection-headtohead.json
umbra-eval corpus --min-recall 1.0 --max-fp 0
if python -c "from signetry_core import scan_repository" 2>/dev/null; then
signetry-eval corpus --markdown > detection-headtohead.md
signetry-eval corpus --json > detection-headtohead.json
signetry-eval corpus --min-recall 1.0 --max-fp 0
else
echo "umbra-core detection engine not available in the installed version; skipping head-to-head gate." | tee detection-headtohead.md
echo "signetry-core detection engine not available in the installed version; skipping head-to-head gate." | tee detection-headtohead.md
fi
- uses: actions/upload-artifact@v7
if: always()
with:
name: umbra-eval-report-${{ matrix.python-version }}
name: signetry-eval-report-${{ matrix.python-version }}
path: |
eval-report.md
eval-report.json
Expand All @@ -76,18 +76,18 @@ jobs:
pip install semgrep || echo "::warning::semgrep install failed; layer will be unavailable."
- name: Corpus with Semgrep layer (report-only)
run: |
if python -c "from umbra_core import scan_repository" 2>/dev/null; then
if python -c "from signetry_core import scan_repository" 2>/dev/null; then
# --semgrep enables the optional layer; no --min-recall/--max-fp gate here.
umbra-eval corpus --semgrep --markdown > detection-semgrep.md || true
umbra-eval corpus --semgrep --json > detection-semgrep.json || true
signetry-eval corpus --semgrep --markdown > detection-semgrep.md || true
signetry-eval corpus --semgrep --json > detection-semgrep.json || true
echo "=== corpus with Semgrep layer ==="; cat detection-semgrep.md || true
else
echo "detection engine unavailable; skipping." | tee detection-semgrep.md
fi
- uses: actions/upload-artifact@v7
if: always()
with:
name: umbra-eval-semgrep-coverage
name: signetry-eval-semgrep-coverage
path: |
detection-semgrep.md
detection-semgrep.json
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/reviewer.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
script: |
const fs = require('fs'); const path = require('path');
const out = process.env.OUT || '';
let body = 'Umbra Reviewer: no review was produced.';
let body = 'Signetry Reviewer: no review was produced.';
try { body = fs.readFileSync(path.join(out, 'comment.md'), 'utf8'); } catch (e) {}
const marker = '<!-- umbra-reviewer -->';
body = marker + '\n' + body;
Expand Down
14 changes: 13 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,20 @@
# Changelog — umbra-eval
# Changelog — signetry-eval

Follows [Keep a Changelog](https://keepachangelog.com/) / [SemVer](https://semver.org/).
Until `1.0.0` the public API may change between minor versions.

## [Unreleased]

### Changed — Signetry rename (breaking)

- Renamed the distribution `umbra-eval` → `signetry-eval` and the import package
`umbra_eval` → `signetry_eval`. The console command is now `signetry-eval`.
- All `UMBRA_*` environment variables are now `SIGNETRY_*`; the config directory
is `.signetry/` (was `.umbra/`). Product/brand prose updated to **Signetry**.
- Core dependency renamed and pinned: `signetry-core @ git+https://github.com/Signetry/core@v0.6.0`
(was `umbra-core @ ...@v0.5.4`). Imports moved `umbra_core` → `signetry_core`.
- No backward-compatibility fallbacks are provided.

## [0.2.2] — 2026-07-30

### Changed
Expand Down
4 changes: 2 additions & 2 deletions CLA.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Umbra Contributor License Agreement (CLA)
# Signetry Contributor License Agreement (CLA)

**Copyright (c) 2026 Binay Dalai. All rights reserved.**

Thank you for your interest in contributing to Umbra ("the Project"), owned by
Thank you for your interest in contributing to Signetry ("the Project"), owned by
Binay Dalai ("the Owner"). This Contributor License Agreement ("Agreement")
governs any contribution you submit. You must agree to it before your contribution
can be accepted.
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ I have read the CLA Document and I hereby sign the CLA
```

Your acceptance is recorded in `signatures/cla.json`. A PR **cannot be merged** until
the CLA is signed. Umbra is **not open source** (All Rights Reserved) — by signing
the CLA is signed. Signetry is **not open source** (All Rights Reserved) — by signing
you assign copyright/ownership of your contribution to the Owner and gain no right to
use, sell, or commercialize it yourself.

Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTORS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Contributors

Umbra is **owned by Binay Dalai** and is **All Rights Reserved** (not open source).
Signetry is **owned by Binay Dalai** and is **All Rights Reserved** (not open source).
The people below are gratefully acknowledged for contributions accepted under the
[Contributor License Agreement](CLA.md).

Expand Down
64 changes: 32 additions & 32 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
<div align="center">

# umbra-eval
# signetry-eval

> **Copyright (c) 2026 Binay Dalai. All rights reserved.**
> This repository is strictly for viewing and contributing to the original project. You may not use, copy, modify, distribute, or commercialize this code for your own personal or commercial projects without explicit written permission. Only the original author retains the right to use and monetize this project.


**The public adversarial evaluation suite for [Umbra](https://github.com/Signetry/signetry).**
**The public adversarial evaluation suite for [Signetry](https://github.com/Signetry/signetry).**

Measures **attack success rate (ASR)** and **utility-under-defense** for coding-agent
threats — governed by the same [`umbra-core`](https://github.com/Signetry/core)
threats — governed by the same [`signetry-core`](https://github.com/Signetry/core)
pipeline production uses.

[![Source-available](https://img.shields.io/badge/source-available-informational.svg)](CLA.md)
Expand All @@ -19,7 +19,7 @@ pipeline production uses.

---

Umbra's honest claim is never "prompt injection solved." It is **bounded +
Signetry's honest claim is never "prompt injection solved." It is **bounded +
quarantined + dual-verified + receipted**: the governed run keeps the attacker's
objective out of the *admitted* change and caps authority on the evidence, while
the legitimate task still ships with a signed receipt. This suite measures exactly
Expand All @@ -33,8 +33,8 @@ API keys, reproducible in CI.

For each scenario, two conditions:

- **Ungoverned** — the agent/tooling runs with no Umbra checkpoint.
- **Governed** — the same run passes through `umbra-core` (`run_admission` /
- **Ungoverned** — the agent/tooling runs with no Signetry checkpoint.
- **Governed** — the same run passes through `signetry-core` (`run_admission` /
`admit_extension`), exactly as production uses it.

…and two questions: **did the attack succeed?** (ASR) and **did the benign task
Expand All @@ -44,27 +44,27 @@ utility**.

## Detection benchmark (head-to-head)

Beyond the adversarial suite, umbra-eval runs a **public detection benchmark** that
scores Umbra's SAST engine against LLM security scanners on a shared, provenance-
Beyond the adversarial suite, signetry-eval runs a **public detection benchmark** that
scores Signetry's SAST engine against LLM security scanners on a shared, provenance-
cited corpus — **52 cases across 7 languages** (Python, JavaScript, Go, Java, Ruby,
PHP, C#) in six families (public/OWASP, academic/CWE, crafted, hard cross-file
taint, multilang, cross-file-lang), with safe decoys for false-positive measurement.

| Scanner | Recall | False positives | Cost |
|---|:---:|:---:|---|
| **umbra-core** (deterministic) | **100%** (42/42) | **0** | free · offline · reproducible |
| **signetry-core** (deterministic) | **100%** (42/42) | **0** | free · offline · reproducible |
| claude-code-security-review (Claude Opus 4.8) | 90% (38/42) | 0 | paid per scan · non-deterministic |
| @openai/codex-security | not run¹ | — | paid per scan |

¹ Competitor scores replay a committed capture; a tool that wasn't run is shown as
`not run`, never scored as zero.

Umbra reaches this on the **deterministic, offline, free** layer — same result every
Signetry reaches this on the **deterministic, offline, free** layer — same result every
run. The optional **Semgrep** layer (`--semgrep`) broadens coverage but can add
false positives (its generic rules don't model every sanitizer); Umbra's
false positives (its generic rules don't model every sanitizer); Signetry's
deterministic engine is **0-FP** on the corpus, including sanitizer-aware cases
where Semgrep is not — so the Semgrep layer is opt-in and non-gating. Detection
parity is table stakes; the **governance** Umbra adds on top (earned authority,
parity is table stakes; the **governance** Signetry adds on top (earned authority,
injection quarantine, independent verifier, signed receipts) is what the scanners
don't attempt, and is measured by the adversarial suite below.

Expand All @@ -80,32 +80,32 @@ don't attempt, and is measured by the adversarial suite below.
## Run it

```bash
pip install -e . # pulls umbra-core from its source repo (not on PyPI)

umbra-eval run # human summary (ASR / utility)
umbra-eval run --markdown # publishable report
umbra-eval run --json # machine-readable
umbra-eval run --category ipi # one threat category
umbra-eval list # list scenarios

umbra-eval corpus # detection head-to-head (recall, FP, by-language)
umbra-eval corpus --markdown # publishable comparison table
umbra-eval corpus --semgrep # add the optional Semgrep layer (report-only)
umbra-eval realrepo # live scan of real vulnerable repos
pip install -e . # pulls signetry-core from its source repo (not on PyPI)

signetry-eval run # human summary (ASR / utility)
signetry-eval run --markdown # publishable report
signetry-eval run --json # machine-readable
signetry-eval run --category ipi # one threat category
signetry-eval list # list scenarios

signetry-eval corpus # detection head-to-head (recall, FP, by-language)
signetry-eval corpus --markdown # publishable comparison table
signetry-eval corpus --semgrep # add the optional Semgrep layer (report-only)
signetry-eval realrepo # live scan of real vulnerable repos
```

`umbra-eval run` exits non-zero if the defense did not hold on **every** adversarial
`signetry-eval run` exits non-zero if the defense did not hold on **every** adversarial
scenario, so it doubles as a CI regression guard.

```python
from umbra_eval import run_all
from signetry_eval import run_all
report = run_all()
print(report.overall()) # asr_ungoverned, asr_governed, utility_governed, …
```

## Interpreting the results

- **ASR (governed) → low** means Umbra kept the tested attacks out of the admitted
- **ASR (governed) → low** means Signetry kept the tested attacks out of the admitted
change. It is **not** proof of coverage against unseen phrasings — a fixed pattern
detector can be paraphrased around. New adversarial phrasings belong in this repo
as new scenarios (adaptive red-teaming), and the curve is published honestly.
Expand All @@ -116,23 +116,23 @@ print(report.overall()) # asr_ungoverned, asr_governed, utility_governe

## Design

- No governance logic lives here — it is imported from `umbra-core`. This repo only
- No governance logic lives here — it is imported from `signetry-core`. This repo only
*poses attacks* and *scores outcomes*.
- Scenarios implement a tiny `Scenario` protocol (`run() -> ScenarioResult`) and are
registered in [`umbra_eval/scenarios`](umbra_eval/scenarios).
registered in [`signetry_eval/scenarios`](signetry_eval/scenarios).
- Add a scenario: model the attack, run it ungoverned + governed via the harness,
and return a `ScenarioResult`.

## Contributing

**Source-available, PRs welcome** (not open source; All Rights Reserved). Contribute under the [CLA](CLA.md) — you're **credited** ([CONTRIBUTORS.md](CONTRIBUTORS.md)) but gain no ownership or right to use/sell it. Start at the [good-first-issues board](https://github.com/Signetry/signetry/issues/10). The best contribution is **a new test case**. Add a detection
**corpus case** (`umbra_eval/detection/corpus/`, with a cited `provenance`; SAFE
**corpus case** (`signetry_eval/detection/corpus/`, with a cited `provenance`; SAFE
decoys must stay 0 false positives) or an **adversarial scenario**
(`umbra_eval/scenarios/`). See [CONTRIBUTING.md](CONTRIBUTING.md) and
(`signetry_eval/scenarios/`). See [CONTRIBUTING.md](CONTRIBUTING.md) and
[CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md). Every case ships with a test; CI gates the
benchmark on 100% recall / 0 FP.

Part of the Umbra platform — see the [umbrella overview](https://github.com/Signetry/signetry).
Part of the Signetry platform — see the [umbrella overview](https://github.com/Signetry/signetry).

## License

Expand Down
2 changes: 1 addition & 1 deletion docs/BENCHMARK.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[
{
"name": "umbra-core",
"name": "signetry-core",
"ran": true,
"recall": 1.0,
"detected": 42,
Expand Down
Loading