Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions Veyron-v1.2-full/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
MIT License

Copyright (c) 2026 Veyron contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files, to deal in the Software
without restriction, including without limitation the rights to use, copy,
modify, merge, publish, distribute, sublicense, and/or sell copies of the
Software, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND.
125 changes: 125 additions & 0 deletions Veyron-v1.2-full/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
<div align="center">

<img src="assets/banner-v1.2.png" alt="Veyron — evidence-first checkpoint forensics" width="100%">

# Veyron

**Evidence-first checkpoint forensics for model weights.**

Inspect what tensors support. Mark the rest unknown.

[Python 3.10+](pyproject.toml) · [Safetensors-first](#safety) · [MIT-style license](LICENSE)

</div>

---

## What it does

Veyron analyzes a checkpoint without trusting its model card, tokenizer, or config. It inventories tensors and produces an evidence-backed report about:

- tensor inventory, parameter count, dtypes, numerical profiles, and cryptographic file hash;
- transformer-like structural clues, layers, hidden width, attention layout, and FFN expansion;
- adapter/LoRA and merge-name clues, always with explicit limits;
- conservative checkpoint-to-checkpoint similarity evidence;
- blind scoring against independently revealed configuration facts.

It does **not** claim to recover the original dataset, training script, optimizer, exact loss, or provenance from final weights. Those are not uniquely encoded in a checkpoint.

## Quick start

```bash
git clone <repository-url>
cd Veyron-v1-full
python -m venv .venv
.venv\Scripts\python -m pip install -e ".[dev]"
```

```bash
# Weight-only report
.venv\Scripts\python -m veyron analyze path\to\model.safetensors --out report.json

# Compare aligned tensors in two checkpoints
.venv\Scripts\python -m veyron compare model-a.safetensors model-b.safetensors

# Blind fetch → strip metadata → analyze → reveal config → score
.venv\Scripts\python -m veyron.interactive
```

Windows users can also double-click `run_veyron.bat` for the interactive workflow.

## Supported input

| Input | Support | Notes |
|---|---|---|
| `.safetensors` | Full | Recommended non-executable format. |
| Sharded Safetensors directory/index | Full | Pass the directory or `*.safetensors.index.json`. |
| `.pt`, `.pth`, `.bin` state dict | Safe-mode support | Loaded with PyTorch `weights_only=True`. |
| Legacy pickle checkpoint | Trusted-file opt-in | Requires `--allow-unsafe-pickle`; see safety warning. |
| Quantized/custom architecture | Inventory + cautious inference | Findings may be incomplete; validate externally. |

## Safety

Treat checkpoints from the internet as untrusted. Safetensors is the preferred format. Legacy PyTorch checkpoints can contain pickle payloads, which may execute code during deserialization. Veyron refuses the unsafe fallback by default; only use this option for a file you fully trust:

```bash
python -m veyron analyze trusted-legacy.pt --allow-unsafe-pickle
```

## Reading a report

Each finding includes a `type`, `value`, `confidence`, and human-readable `evidence`. Confidence is an estimate of how strongly the observed tensors support that limited claim—not a probability that the model has a particular name, source, or training history.

For large tensors, distribution statistics and spectral summaries use deterministic bounded sampling. This keeps analysis practical and reproducible; checkpoint SHA-256 remains a complete file hash.

## Architecture clues in v1.2

Veyron recognizes common naming and geometry signals for split or fused QKV attention, MLP/FFN blocks, gated MLPs, repeated layer indices, embeddings, norms, and output heads. A geometry clue is stronger than naming alone, but neither is a substitute for verified configuration.

The implementation is deliberately modular:

```text
checkpoint.py input loading, safe boundaries, bounded numerical work
structure.py architecture evidence and confidence-bearing findings
fingerprint.py reproducible report signatures and layer profiles
hypotheses.py limited forensic hypotheses and stated boundaries
similarity.py name/shape-aligned comparison metrics
blind_test.py metadata-isolated real-model workflow
blind_score.py checkable scoring only
evaluation.py repeatable local benchmark manifests
```

## Evaluation instead of demos

The synthetic benchmark is a smoke test, not a claim of real-world accuracy:

```bash
python -m veyron benchmark
```

For meaningful measurement, create a local manifest from independently labelled checkpoints, then run:

```bash
python -m veyron evaluate benchmark\evaluation.example.json
```

The analyzer receives only the checkpoint; `ground_truth` is used only after findings are produced. Build a suite spanning decoder-only, encoder-only, encoder-decoder, vision, adapter, quantized, and non-transformer controls. Track exact accuracy, false positives, abstentions, per-family performance, and confidence calibration.

## Development

```bash
python -m pytest -q
ruff check veyron tests
```

GitHub Actions runs the test suite for pushes and pull requests. Contributions should add a regression test for new architecture rules and lower confidence—or return no finding—when evidence is weak.

## Roadmap

- **v1.2:** safe loading, shards, bounded analysis, richer geometry evidence, evaluation manifests.
- **Next:** calibrated confidence curves, more families and quantization-aware metrics, tensor-level diff summaries, optional trusted config normalization.
- **Never a default claim:** exact dataset, source code, or single-model genealogy from weights alone.

## License

See [LICENSE](LICENSE).
Binary file added Veyron-v1.2-full/assets/banner-v1.2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Veyron-v1.2-full/assets/banner.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
39 changes: 39 additions & 0 deletions Veyron-v1.2-full/benchmark/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Veyron Benchmark

The benchmark is deliberately separated into two tracks.

## Track A: controlled synthetic models

Veyron creates tiny checkpoints where the ground truth is known exactly. The analyzer receives only the weights.

This is the current automated test.

## Track B: real open models

For a real model, store ground truth in a separate private file and run Veyron without loading it.

Recommended fields:

- architecture
- layer count
- hidden size
- tokenizer family
- documented training objective
- documented training stages
- known training domains
- known parameter count

Do not treat undocumented dataset composition as ground truth.

## Metrics

Use:

- exact accuracy
- partial accuracy
- macro accuracy across properties
- confidence calibration
- false-positive rate
- unknown/rejection rate

A forensic system should be rewarded for saying "unknown" when evidence is insufficient.
Loading