diff --git a/Veyron-v1.2-full/LICENSE b/Veyron-v1.2-full/LICENSE new file mode 100644 index 0000000..1a5a535 --- /dev/null +++ b/Veyron-v1.2-full/LICENSE @@ -0,0 +1,14 @@ +MIT License + +Copyright (c) 2026 Veyron contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files, to deal in the Software +without restriction, including without limitation the rights to use, copy, +modify, merge, publish, distribute, sublicense, and/or sell copies of the +Software, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND. diff --git a/Veyron-v1.2-full/README.md b/Veyron-v1.2-full/README.md new file mode 100644 index 0000000..448d8eb --- /dev/null +++ b/Veyron-v1.2-full/README.md @@ -0,0 +1,125 @@ +
+ +Veyron — evidence-first checkpoint forensics + +# Veyron + +**Evidence-first checkpoint forensics for model weights.** + +Inspect what tensors support. Mark the rest unknown. + +[Python 3.10+](pyproject.toml) · [Safetensors-first](#safety) · [MIT-style license](LICENSE) + +
+ +--- + +## What it does + +Veyron analyzes a checkpoint without trusting its model card, tokenizer, or config. It inventories tensors and produces an evidence-backed report about: + +- tensor inventory, parameter count, dtypes, numerical profiles, and cryptographic file hash; +- transformer-like structural clues, layers, hidden width, attention layout, and FFN expansion; +- adapter/LoRA and merge-name clues, always with explicit limits; +- conservative checkpoint-to-checkpoint similarity evidence; +- blind scoring against independently revealed configuration facts. + +It does **not** claim to recover the original dataset, training script, optimizer, exact loss, or provenance from final weights. Those are not uniquely encoded in a checkpoint. + +## Quick start + +```bash +git clone +cd Veyron-v1-full +python -m venv .venv +.venv\Scripts\python -m pip install -e ".[dev]" +``` + +```bash +# Weight-only report +.venv\Scripts\python -m veyron analyze path\to\model.safetensors --out report.json + +# Compare aligned tensors in two checkpoints +.venv\Scripts\python -m veyron compare model-a.safetensors model-b.safetensors + +# Blind fetch → strip metadata → analyze → reveal config → score +.venv\Scripts\python -m veyron.interactive +``` + +Windows users can also double-click `run_veyron.bat` for the interactive workflow. + +## Supported input + +| Input | Support | Notes | +|---|---|---| +| `.safetensors` | Full | Recommended non-executable format. | +| Sharded Safetensors directory/index | Full | Pass the directory or `*.safetensors.index.json`. | +| `.pt`, `.pth`, `.bin` state dict | Safe-mode support | Loaded with PyTorch `weights_only=True`. | +| Legacy pickle checkpoint | Trusted-file opt-in | Requires `--allow-unsafe-pickle`; see safety warning. | +| Quantized/custom architecture | Inventory + cautious inference | Findings may be incomplete; validate externally. | + +## Safety + +Treat checkpoints from the internet as untrusted. Safetensors is the preferred format. Legacy PyTorch checkpoints can contain pickle payloads, which may execute code during deserialization. Veyron refuses the unsafe fallback by default; only use this option for a file you fully trust: + +```bash +python -m veyron analyze trusted-legacy.pt --allow-unsafe-pickle +``` + +## Reading a report + +Each finding includes a `type`, `value`, `confidence`, and human-readable `evidence`. Confidence is an estimate of how strongly the observed tensors support that limited claim—not a probability that the model has a particular name, source, or training history. + +For large tensors, distribution statistics and spectral summaries use deterministic bounded sampling. This keeps analysis practical and reproducible; checkpoint SHA-256 remains a complete file hash. + +## Architecture clues in v1.2 + +Veyron recognizes common naming and geometry signals for split or fused QKV attention, MLP/FFN blocks, gated MLPs, repeated layer indices, embeddings, norms, and output heads. A geometry clue is stronger than naming alone, but neither is a substitute for verified configuration. + +The implementation is deliberately modular: + +```text +checkpoint.py input loading, safe boundaries, bounded numerical work +structure.py architecture evidence and confidence-bearing findings +fingerprint.py reproducible report signatures and layer profiles +hypotheses.py limited forensic hypotheses and stated boundaries +similarity.py name/shape-aligned comparison metrics +blind_test.py metadata-isolated real-model workflow +blind_score.py checkable scoring only +evaluation.py repeatable local benchmark manifests +``` + +## Evaluation instead of demos + +The synthetic benchmark is a smoke test, not a claim of real-world accuracy: + +```bash +python -m veyron benchmark +``` + +For meaningful measurement, create a local manifest from independently labelled checkpoints, then run: + +```bash +python -m veyron evaluate benchmark\evaluation.example.json +``` + +The analyzer receives only the checkpoint; `ground_truth` is used only after findings are produced. Build a suite spanning decoder-only, encoder-only, encoder-decoder, vision, adapter, quantized, and non-transformer controls. Track exact accuracy, false positives, abstentions, per-family performance, and confidence calibration. + +## Development + +```bash +python -m pytest -q +ruff check veyron tests +``` + +GitHub Actions runs the test suite for pushes and pull requests. Contributions should add a regression test for new architecture rules and lower confidence—or return no finding—when evidence is weak. + +## Roadmap + +- **v1.2:** safe loading, shards, bounded analysis, richer geometry evidence, evaluation manifests. +- **Next:** calibrated confidence curves, more families and quantization-aware metrics, tensor-level diff summaries, optional trusted config normalization. +- **Never a default claim:** exact dataset, source code, or single-model genealogy from weights alone. + +## License + +See [LICENSE](LICENSE). diff --git a/Veyron-v1.2-full/assets/banner-v1.2.png b/Veyron-v1.2-full/assets/banner-v1.2.png new file mode 100644 index 0000000..90f7118 Binary files /dev/null and b/Veyron-v1.2-full/assets/banner-v1.2.png differ diff --git a/Veyron-v1.2-full/assets/banner.png b/Veyron-v1.2-full/assets/banner.png new file mode 100644 index 0000000..593e283 Binary files /dev/null and b/Veyron-v1.2-full/assets/banner.png differ diff --git a/Veyron-v1.2-full/assets/terminal_screenshot.png b/Veyron-v1.2-full/assets/terminal_screenshot.png new file mode 100644 index 0000000..e42d473 Binary files /dev/null and b/Veyron-v1.2-full/assets/terminal_screenshot.png differ diff --git a/Veyron-v1.2-full/benchmark/README.md b/Veyron-v1.2-full/benchmark/README.md new file mode 100644 index 0000000..49c25ca --- /dev/null +++ b/Veyron-v1.2-full/benchmark/README.md @@ -0,0 +1,39 @@ +# Veyron Benchmark + +The benchmark is deliberately separated into two tracks. + +## Track A: controlled synthetic models + +Veyron creates tiny checkpoints where the ground truth is known exactly. The analyzer receives only the weights. + +This is the current automated test. + +## Track B: real open models + +For a real model, store ground truth in a separate private file and run Veyron without loading it. + +Recommended fields: + +- architecture +- layer count +- hidden size +- tokenizer family +- documented training objective +- documented training stages +- known training domains +- known parameter count + +Do not treat undocumented dataset composition as ground truth. + +## Metrics + +Use: + +- exact accuracy +- partial accuracy +- macro accuracy across properties +- confidence calibration +- false-positive rate +- unknown/rejection rate + +A forensic system should be rewarded for saying "unknown" when evidence is insufficient. diff --git a/Veyron-v1.2-full/benchmark/demo_report.json b/Veyron-v1.2-full/benchmark/demo_report.json new file mode 100644 index 0000000..d3d0b6a --- /dev/null +++ b/Veyron-v1.2-full/benchmark/demo_report.json @@ -0,0 +1,785 @@ +{ + "veyron_version": "1.0.0", + "checkpoint": { + "path": "/mnt/data/Veyron-v1-full/benchmark/demo_tiny_transformer.pt", + "sha256": "68ee44931b99ab38061eed5c822bef0a1910754f9520ddef55bca854bb9e88d7" + }, + "scope": { + "input": "model checkpoint weights", + "method": "weight-only forensic analysis", + "claims_are_probabilistic": true + }, + "structure": { + "parameter_count": 37024, + "parameter_count_billions": 3.7024e-05, + "tensor_count": 21, + "dtype_counts": { + "torch.float32": 21 + } + }, + "fingerprint": "0594587ef01c5ce7aee0ae735d26c56c8ca603ee9b554aead9a332694596b6e0", + "aggregate_profile": { + "tensor_mean_average": 0.23811219373532194, + "tensor_std_average": 0.015292635364901452, + "tensor_sparsity_average": 0.0, + "tensor_count": 21 + }, + "layer_profiles": { + "0": { + "tensor_mean_average": 0.22243271501873904, + "tensor_std_average": 0.0156148262321949, + "tensor_sparsity_average": 0.0, + "tensor_count": 9 + }, + "1": { + "tensor_mean_average": 0.22205608018379686, + "tensor_std_average": 0.015641436187757388, + "tensor_sparsity_average": 0.0, + "tensor_count": 9 + } + }, + "findings": [ + { + "type": "architecture_family", + "value": "transformer-like", + "confidence": 0.94, + "evidence": "Attention and feed-forward projection tensors detected." + }, + { + "type": "attention", + "value": "attention projections detected", + "confidence": 0.96, + "evidence": "Query/key/value-like tensors detected." + }, + { + "type": "feed_forward", + "value": "MLP/FFN detected", + "confidence": 0.95, + "evidence": "Feed-forward projection tensors detected." + }, + { + "type": "normalization", + "value": "normalization detected", + "confidence": 0.93, + "evidence": "Normalization-like tensors detected." + }, + { + "type": "embedding", + "value": "input embedding detected", + "confidence": 0.92, + "evidence": "Embedding-like tensors detected." + }, + { + "type": "output_head", + "value": "output head detected", + "confidence": 0.78, + "evidence": "Output-head-like tensors detected." + }, + { + "type": "layer_count_estimate", + "value": 2, + "confidence": 0.92, + "evidence": "Layer indices 0..1 were detected." + }, + { + "type": "hidden_size_clue", + "value": 32, + "confidence": 0.97, + "evidence": "Repeated tensor geometry suggests this hidden dimension." + } + ], + "hypotheses": [ + { + "type": "training_objective_clue", + "value": "causal-language-model-like architecture", + "confidence": 0.58, + "evidence": "Decoder-style attention projection naming is compatible with causal LM architectures.", + "warning": "Architecture alone cannot prove the training objective." + } + ], + "limitations": { + "exact_training_code_recovery": false, + "exact_dataset_recovery": false, + "reason": "Final weights are not a unique encoding of the original pipeline." + }, + "tensors": [ + { + "name": "model.embed_tokens.weight", + "shape": [ + 64, + 32 + ], + "dtype": "torch.float32", + "numel": 2048, + "mean": -2.204e-05, + "std": 0.01976296, + "l1": 32.38649368, + "l2": 0.8943699, + "sparsity": 0.0, + "spectrum": { + "rank_estimate": 32, + "top_singular_values": [ + 0.2610379755496979, + 0.24911601841449738, + 0.231333389878273, + 0.22352159023284912, + 0.2204521894454956, + 0.20722313225269318, + 0.2024136632680893, + 0.19301246106624603, + 0.19150756299495697, + 0.18798133730888367, + 0.18305747210979462, + 0.16931694746017456, + 0.16567924618721008, + 0.16416208446025848, + 0.1569017618894577, + 0.1542295664548874 + ], + "top16_energy_ratio": 0.8006479144096375, + "max_singular": 0.2610379755496979, + "min_singular": 0.05202833563089371 + } + }, + { + "name": "model.norm.weight", + "shape": [ + 32 + ], + "dtype": "torch.float32", + "numel": 32, + "mean": 1.0, + "std": 0.0, + "l1": 32.0, + "l2": 5.65685415, + "sparsity": 0.0, + "spectrum": null + }, + { + "name": "lm_head.weight", + "shape": [ + 64, + 32 + ], + "dtype": "torch.float32", + "numel": 2048, + "mean": -2.104e-05, + "std": 0.02007602, + "l1": 32.95693588, + "l2": 0.90853757, + "sparsity": 0.0, + "spectrum": { + "rank_estimate": 32, + "top_singular_values": [ + 0.25880250334739685, + 0.24680602550506592, + 0.24245265126228333, + 0.22859056293964386, + 0.2210100293159485, + 0.20890043675899506, + 0.20321764051914215, + 0.20010118186473846, + 0.19444184005260468, + 0.18602079153060913, + 0.17880941927433014, + 0.17589356005191803, + 0.17093060910701752, + 0.16537827253341675, + 0.15744997560977936, + 0.15090397000312805 + ], + "top16_energy_ratio": 0.7901285290718079, + "max_singular": 0.25880250334739685, + "min_singular": 0.04559216648340225 + } + }, + { + "name": "model.layers.0.self_attn.q_proj.weight", + "shape": [ + 32, + 32 + ], + "dtype": "torch.float32", + "numel": 1024, + "mean": 0.00013592, + "std": 0.02056096, + "l1": 16.82253456, + "l2": 0.65796494, + "sparsity": 0.0, + "spectrum": { + "rank_estimate": 32, + "top_singular_values": [ + 0.2362489402294159, + 0.2061099410057068, + 0.1975964903831482, + 0.17648683488368988, + 0.17326222360134125, + 0.16889289021492004, + 0.1540328860282898, + 0.15035401284694672, + 0.14342868328094482, + 0.1318710595369339, + 0.12820670008659363, + 0.1266356110572815, + 0.1129952073097229, + 0.10570994764566422, + 0.10241647809743881, + 0.09552592039108276 + ], + "top16_energy_ratio": 0.894756555557251, + "max_singular": 0.2362489402294159, + "min_singular": 0.009930930100381374 + } + }, + { + "name": "model.layers.0.self_attn.k_proj.weight", + "shape": [ + 32, + 32 + ], + "dtype": "torch.float32", + "numel": 1024, + "mean": 0.00121781, + "std": 0.02031519, + "l1": 16.84071159, + "l2": 0.65125304, + "sparsity": 0.0, + "spectrum": { + "rank_estimate": 32, + "top_singular_values": [ + 0.22921504080295563, + 0.195582777261734, + 0.1920713484287262, + 0.18172049522399902, + 0.1790980100631714, + 0.17322774231433868, + 0.15699242055416107, + 0.14898408949375153, + 0.14167015254497528, + 0.13852980732917786, + 0.12475979328155518, + 0.12165340036153793, + 0.1128900796175003, + 0.1086130440235138, + 0.10144903510808945, + 0.09541624039411545 + ], + "top16_energy_ratio": 0.9034842848777771, + "max_singular": 0.22921504080295563, + "min_singular": 0.00010428672976559028 + } + }, + { + "name": "model.layers.0.self_attn.v_proj.weight", + "shape": [ + 32, + 32 + ], + "dtype": "torch.float32", + "numel": 1024, + "mean": -0.00010278, + "std": 0.01968879, + "l1": 16.21482468, + "l2": 0.63004977, + "sparsity": 0.0, + "spectrum": { + "rank_estimate": 32, + "top_singular_values": [ + 0.2104261815547943, + 0.1939268708229065, + 0.1887388825416565, + 0.18329216539859772, + 0.16889075934886932, + 0.15442141890525818, + 0.15010446310043335, + 0.13963934779167175, + 0.13540713489055634, + 0.13145527243614197, + 0.12467069178819656, + 0.11944103240966797, + 0.11101634800434113, + 0.10492831468582153, + 0.10048770159482956, + 0.09421221166849136 + ], + "top16_energy_ratio": 0.8898533582687378, + "max_singular": 0.2104261815547943, + "min_singular": 0.001545454142615199 + } + }, + { + "name": "model.layers.0.self_attn.o_proj.weight", + "shape": [ + 32, + 32 + ], + "dtype": "torch.float32", + "numel": 1024, + "mean": -0.00012362, + "std": 0.01975565, + "l1": 16.12402534, + "l2": 0.63219309, + "sparsity": 0.0, + "spectrum": { + "rank_estimate": 32, + "top_singular_values": [ + 0.21760955452919006, + 0.19418102502822876, + 0.1916414499282837, + 0.1801762580871582, + 0.16590988636016846, + 0.16119280457496643, + 0.15056884288787842, + 0.13963229954242706, + 0.13832399249076843, + 0.12621410191059113, + 0.1230510026216507, + 0.11982365697622299, + 0.11262456327676773, + 0.1038922518491745, + 0.09602104127407074, + 0.09173879772424698 + ], + "top16_energy_ratio": 0.8897486925125122, + "max_singular": 0.21760955452919006, + "min_singular": 0.002018279628828168 + } + }, + { + "name": "model.layers.0.mlp.gate_proj.weight", + "shape": [ + 128, + 32 + ], + "dtype": "torch.float32", + "numel": 4096, + "mean": 0.00042495, + "std": 0.02002821, + "l1": 65.65867615, + "l2": 1.28209364, + "sparsity": 0.0, + "spectrum": { + "rank_estimate": 32, + "top_singular_values": [ + 0.3189573287963867, + 0.30892711877822876, + 0.30385518074035645, + 0.2956056594848633, + 0.2858678698539734, + 0.28150585293769836, + 0.27505865693092346, + 0.27377012372016907, + 0.2685698866844177, + 0.2594478130340576, + 0.2555551528930664, + 0.2458757907152176, + 0.23608040809631348, + 0.23045213520526886, + 0.22863076627254486, + 0.22072140872478485 + ], + "top16_energy_ratio": 0.7078219056129456, + "max_singular": 0.3189573287963867, + "min_singular": 0.11582174897193909 + } + }, + { + "name": "model.layers.0.mlp.up_proj.weight", + "shape": [ + 128, + 32 + ], + "dtype": "torch.float32", + "numel": 4096, + "mean": 4.603e-05, + "std": 0.02027502, + "l1": 66.15716553, + "l2": 1.29760468, + "sparsity": 0.0, + "spectrum": { + "rank_estimate": 32, + "top_singular_values": [ + 0.35045939683914185, + 0.3255588114261627, + 0.3223556578159332, + 0.31269291043281555, + 0.2916091978549957, + 0.28772544860839844, + 0.2767041325569153, + 0.2679687738418579, + 0.2651161551475525, + 0.25345075130462646, + 0.25042465329170227, + 0.2410230189561844, + 0.23772594332695007, + 0.2287580817937851, + 0.22431524097919464, + 0.22310462594032288 + ], + "top16_energy_ratio": 0.7191925048828125, + "max_singular": 0.35045939683914185, + "min_singular": 0.12387420237064362 + } + }, + { + "name": "model.layers.0.mlp.down_proj.weight", + "shape": [ + 32, + 128 + ], + "dtype": "torch.float32", + "numel": 4096, + "mean": 0.00029612, + "std": 0.01990963, + "l1": 65.01066589, + "l2": 1.27435732, + "sparsity": 0.0, + "spectrum": { + "rank_estimate": 32, + "top_singular_values": [ + 0.33648279309272766, + 0.3237459063529968, + 0.31083908677101135, + 0.29287347197532654, + 0.28822043538093567, + 0.28035053610801697, + 0.2699081003665924, + 0.2683209776878357, + 0.2670164704322815, + 0.2488020658493042, + 0.2424982190132141, + 0.23897609114646912, + 0.23132219910621643, + 0.22868219017982483, + 0.22227966785430908, + 0.21455924212932587 + ], + "top16_energy_ratio": 0.7126330137252808, + "max_singular": 0.33648279309272766, + "min_singular": 0.12128545343875885 + } + }, + { + "name": "model.layers.0.input_layernorm.weight", + "shape": [ + 32 + ], + "dtype": "torch.float32", + "numel": 32, + "mean": 1.0, + "std": 0.0, + "l1": 32.0, + "l2": 5.65685415, + "sparsity": 0.0, + "spectrum": null + }, + { + "name": "model.layers.0.post_attention_layernorm.weight", + "shape": [ + 32 + ], + "dtype": "torch.float32", + "numel": 32, + "mean": 1.0, + "std": 0.0, + "l1": 32.0, + "l2": 5.65685415, + "sparsity": 0.0, + "spectrum": null + }, + { + "name": "model.layers.1.self_attn.q_proj.weight", + "shape": [ + 32, + 32 + ], + "dtype": "torch.float32", + "numel": 1024, + "mean": 0.0001531, + "std": 0.02030103, + "l1": 16.67596054, + "l2": 0.64965147, + "sparsity": 0.0, + "spectrum": { + "rank_estimate": 32, + "top_singular_values": [ + 0.2217450737953186, + 0.21148225665092468, + 0.19786235690116882, + 0.18450801074504852, + 0.1744852364063263, + 0.169240802526474, + 0.16226784884929657, + 0.14828558266162872, + 0.13883931934833527, + 0.13239799439907074, + 0.12617547810077667, + 0.12153036147356033, + 0.11604709923267365, + 0.10424105077981949, + 0.09913771599531174, + 0.0906808078289032 + ], + "top16_energy_ratio": 0.9107402563095093, + "max_singular": 0.2217450737953186, + "min_singular": 0.004732949193567038 + } + }, + { + "name": "model.layers.1.self_attn.k_proj.weight", + "shape": [ + 32, + 32 + ], + "dtype": "torch.float32", + "numel": 1024, + "mean": -0.00158827, + "std": 0.02063735, + "l1": 16.92158508, + "l2": 0.66234821, + "sparsity": 0.0, + "spectrum": { + "rank_estimate": 32, + "top_singular_values": [ + 0.21688427031040192, + 0.21095259487628937, + 0.19919222593307495, + 0.192270889878273, + 0.18223361670970917, + 0.17117026448249817, + 0.15852481126785278, + 0.1560555398464203, + 0.14691664278507233, + 0.12792837619781494, + 0.12418526411056519, + 0.12059338390827179, + 0.11115242540836334, + 0.10545597225427628, + 0.09788542985916138, + 0.0962623730301857 + ], + "top16_energy_ratio": 0.8899276852607727, + "max_singular": 0.21688427031040192, + "min_singular": 0.0012693445896729827 + } + }, + { + "name": "model.layers.1.self_attn.v_proj.weight", + "shape": [ + 32, + 32 + ], + "dtype": "torch.float32", + "numel": 1024, + "mean": 5.475e-05, + "std": 0.01935806, + "l1": 15.8880167, + "l2": 0.61946046, + "sparsity": 0.0, + "spectrum": { + "rank_estimate": 32, + "top_singular_values": [ + 0.2129388004541397, + 0.1934627741575241, + 0.18055810034275055, + 0.17735636234283447, + 0.16980288922786713, + 0.15406015515327454, + 0.14518336951732635, + 0.141634002327919, + 0.13609077036380768, + 0.12622663378715515, + 0.11808968335390091, + 0.11565671116113663, + 0.11063455045223236, + 0.10557441413402557, + 0.09966851025819778, + 0.09095299988985062 + ], + "top16_energy_ratio": 0.896202564239502, + "max_singular": 0.2129388004541397, + "min_singular": 0.004288932774215937 + } + }, + { + "name": "model.layers.1.self_attn.o_proj.weight", + "shape": [ + 32, + 32 + ], + "dtype": "torch.float32", + "numel": 1024, + "mean": 0.00021325, + "std": 0.01985698, + "l1": 16.25664711, + "l2": 0.63546014, + "sparsity": 0.0, + "spectrum": { + "rank_estimate": 32, + "top_singular_values": [ + 0.21090711653232574, + 0.189397931098938, + 0.18313287198543549, + 0.17919038236141205, + 0.1736418604850769, + 0.15917976200580597, + 0.1555134654045105, + 0.14691254496574402, + 0.1355028748512268, + 0.13057710230350494, + 0.12626129388809204, + 0.11918602883815765, + 0.11329563707113266, + 0.10652194917201996, + 0.10239198058843613, + 0.09363105893135071 + ], + "top16_energy_ratio": 0.8822228908538818, + "max_singular": 0.21090711653232574, + "min_singular": 0.0002839589142240584 + } + }, + { + "name": "model.layers.1.mlp.gate_proj.weight", + "shape": [ + 128, + 32 + ], + "dtype": "torch.float32", + "numel": 4096, + "mean": -3.81e-05, + "std": 0.02027398, + "l1": 66.10839844, + "l2": 1.29753697, + "sparsity": 0.0, + "spectrum": { + "rank_estimate": 32, + "top_singular_values": [ + 0.3283180296421051, + 0.31859228014945984, + 0.3106992244720459, + 0.29476919770240784, + 0.29031896591186523, + 0.2845909297466278, + 0.2784532904624939, + 0.27211427688598633, + 0.26520678400993347, + 0.2595491409301758, + 0.25731271505355835, + 0.24730610847473145, + 0.23970484733581543, + 0.23876844346523285, + 0.23051492869853973, + 0.2236088216304779 + ], + "top16_energy_ratio": 0.708156168460846, + "max_singular": 0.3283180296421051, + "min_singular": 0.1257970631122589 + } + }, + { + "name": "model.layers.1.mlp.up_proj.weight", + "shape": [ + 128, + 32 + ], + "dtype": "torch.float32", + "numel": 4096, + "mean": -0.00021367, + "std": 0.02010669, + "l1": 65.53160095, + "l2": 1.28690064, + "sparsity": 0.0, + "spectrum": { + "rank_estimate": 32, + "top_singular_values": [ + 0.330271452665329, + 0.3091115653514862, + 0.30595266819000244, + 0.29672157764434814, + 0.291510671377182, + 0.2854555547237396, + 0.28204965591430664, + 0.266812801361084, + 0.2627468407154083, + 0.2571239769458771, + 0.2513292729854584, + 0.24686089158058167, + 0.24107429385185242, + 0.2390625774860382, + 0.23271022737026215, + 0.2230675220489502 + ], + "top16_energy_ratio": 0.7136714458465576, + "max_singular": 0.330271452665329, + "min_singular": 0.12683720886707306 + } + }, + { + "name": "model.layers.1.mlp.down_proj.weight", + "shape": [ + 32, + 128 + ], + "dtype": "torch.float32", + "numel": 4096, + "mean": -7.634e-05, + "std": 0.02023883, + "l1": 66.12194824, + "l2": 1.29529417, + "sparsity": 0.0, + "spectrum": { + "rank_estimate": 32, + "top_singular_values": [ + 0.32859206199645996, + 0.3221568465232849, + 0.311693012714386, + 0.30527186393737793, + 0.28748008608818054, + 0.28671887516975403, + 0.27965277433395386, + 0.2742823362350464, + 0.26963409781455994, + 0.2590295076370239, + 0.25270769000053406, + 0.24990372359752655, + 0.24681851267814636, + 0.2406005710363388, + 0.23449179530143738, + 0.21794000267982483 + ], + "top16_energy_ratio": 0.7198734879493713, + "max_singular": 0.32859206199645996, + "min_singular": 0.1216326579451561 + } + }, + { + "name": "model.layers.1.input_layernorm.weight", + "shape": [ + 32 + ], + "dtype": "torch.float32", + "numel": 32, + "mean": 1.0, + "std": 0.0, + "l1": 32.0, + "l2": 5.65685415, + "sparsity": 0.0, + "spectrum": null + }, + { + "name": "model.layers.1.post_attention_layernorm.weight", + "shape": [ + 32 + ], + "dtype": "torch.float32", + "numel": 32, + "mean": 1.0, + "std": 0.0, + "l1": 32.0, + "l2": 5.65685415, + "sparsity": 0.0, + "spectrum": null + } + ] +} \ No newline at end of file diff --git a/Veyron-v1.2-full/benchmark/demo_run.txt b/Veyron-v1.2-full/benchmark/demo_run.txt new file mode 100644 index 0000000..e9ee051 --- /dev/null +++ b/Veyron-v1.2-full/benchmark/demo_run.txt @@ -0,0 +1,35 @@ +VEYRON v1 +============================================================ +Parameters : 37,024 (0.0000B) +Tensors : 21 +Fingerprint: 0594587ef01c5ce7aee0ae735d26c56c... + +Findings: + [94%] architecture_family: transformer-like + [96%] attention: attention projections detected + [95%] feed_forward: MLP/FFN detected + [93%] normalization: normalization detected + [92%] embedding: input embedding detected + [78%] output_head: output head detected + [92%] layer_count_estimate: 2 + [97%] hidden_size_clue: 32 + +Hypotheses: + [58%] training_objective_clue: causal-language-model-like architecture + +Boundary: + Exact training script: NOT CLAIMED + Exact dataset: NOT CLAIMED + +Report: /mnt/data/Veyron-v1-full/benchmark/demo_report.json + +Spreadsheet runtime warmup failed during python startup +Traceback (most recent call last): + File "/tmp/tmp.L2TH2Y5coc/artifact_tool_v2-2.8.22/artifact_tool/patches/warm_spreadsheet_runtime_on_startup.py", line 26, in warm_spreadsheet_runtime_on_startup + File "/tmp/tmp.L2TH2Y5coc/artifact_tool_v2-2.8.22/artifact_tool/spreadsheet_warmup.py", line 785, in warm_spreadsheet_runtime + File "/tmp/tmp.L2TH2Y5coc/artifact_tool_v2-2.8.22/artifact_tool/spreadsheet_warmup.py", line 720, in _warm_feature_flows + File "/tmp/tmp.L2TH2Y5coc/artifact_tool_v2-2.8.22/artifact_tool/spreadsheet_warmup.py", line 704, in _warm_collaboration_flows + File "/tmp/tmp.L2TH2Y5coc/artifact_tool_v2-2.8.22/artifact_tool/generated/interface/models.py", line 32317, in hydrate_crdt_from_proto + File "/tmp/tmp.L2TH2Y5coc/artifact_tool_v2-2.8.22/artifact_tool/rpc/remote.py", line 749, in __call__ + File "/tmp/tmp.L2TH2Y5coc/artifact_tool_v2-2.8.22/artifact_tool/rpc/client.py", line 150, in call +artifact_tool.rpc.client.RemoteError: hydrateCrdtFromProto requires an empty collaborative document. diff --git a/Veyron-v1.2-full/benchmark/demo_tiny_transformer.pt b/Veyron-v1.2-full/benchmark/demo_tiny_transformer.pt new file mode 100644 index 0000000..f346488 Binary files /dev/null and b/Veyron-v1.2-full/benchmark/demo_tiny_transformer.pt differ diff --git a/Veyron-v1.2-full/benchmark/evaluation.example.json b/Veyron-v1.2-full/benchmark/evaluation.example.json new file mode 100644 index 0000000..fc3c08b --- /dev/null +++ b/Veyron-v1.2-full/benchmark/evaluation.example.json @@ -0,0 +1,16 @@ +{ + "name": "local independent-label suite", + "cases": [ + { + "id": "replace-with-a-real-model", + "checkpoint": "../models/model.safetensors", + "ground_truth": { + "available": true, + "architecture_family": "transformer-like", + "layers": 12, + "hidden_size": 768, + "is_causal_lm": true + } + } + ] +} diff --git a/Veyron-v1.2-full/benchmark/last_run.json b/Veyron-v1.2-full/benchmark/last_run.json new file mode 100644 index 0000000..5261637 --- /dev/null +++ b/Veyron-v1.2-full/benchmark/last_run.json @@ -0,0 +1,15 @@ +{ + "benchmark": "Veyron synthetic weight-only benchmark v1", + "ground_truth": { + "architecture_family": "transformer-like", + "layers": 3, + "hidden_size": 32, + "parameter_count": 53472 + }, + "checks": { + "architecture_family": true, + "layers": true, + "hidden_size": true + }, + "accuracy": 1.0 +} diff --git a/Veyron-v1.2-full/benchmark/pytest.txt b/Veyron-v1.2-full/benchmark/pytest.txt new file mode 100644 index 0000000..7070220 --- /dev/null +++ b/Veyron-v1.2-full/benchmark/pytest.txt @@ -0,0 +1,13 @@ +... [100%] +3 passed in 2.10s + +Spreadsheet runtime warmup failed during python startup +Traceback (most recent call last): + File "/tmp/tmp.L2TH2Y5coc/artifact_tool_v2-2.8.22/artifact_tool/patches/warm_spreadsheet_runtime_on_startup.py", line 26, in warm_spreadsheet_runtime_on_startup + File "/tmp/tmp.L2TH2Y5coc/artifact_tool_v2-2.8.22/artifact_tool/spreadsheet_warmup.py", line 785, in warm_spreadsheet_runtime + File "/tmp/tmp.L2TH2Y5coc/artifact_tool_v2-2.8.22/artifact_tool/spreadsheet_warmup.py", line 720, in _warm_feature_flows + File "/tmp/tmp.L2TH2Y5coc/artifact_tool_v2-2.8.22/artifact_tool/spreadsheet_warmup.py", line 704, in _warm_collaboration_flows + File "/tmp/tmp.L2TH2Y5coc/artifact_tool_v2-2.8.22/artifact_tool/generated/interface/models.py", line 32317, in hydrate_crdt_from_proto + File "/tmp/tmp.L2TH2Y5coc/artifact_tool_v2-2.8.22/artifact_tool/rpc/remote.py", line 749, in __call__ + File "/tmp/tmp.L2TH2Y5coc/artifact_tool_v2-2.8.22/artifact_tool/rpc/client.py", line 150, in call +artifact_tool.rpc.client.RemoteError: hydrateCrdtFromProto requires an empty collaborative document. diff --git a/Veyron-v1.2-full/examples/ground_truth.json b/Veyron-v1.2-full/examples/ground_truth.json new file mode 100644 index 0000000..1202f43 --- /dev/null +++ b/Veyron-v1.2-full/examples/ground_truth.json @@ -0,0 +1,16 @@ +{ + "note": "Example schema for future real-model blind evaluation.", + "model_id": "example-open-model", + "architecture": { + "family": "transformer", + "layers": 12, + "hidden_size": 768 + }, + "training": { + "objective": "causal_lm", + "stages": ["pretraining", "sft"] + }, + "data": { + "domains": ["text", "code"] + } +} diff --git a/Veyron-v1.2-full/pyproject.toml b/Veyron-v1.2-full/pyproject.toml new file mode 100644 index 0000000..dfe8e8a --- /dev/null +++ b/Veyron-v1.2-full/pyproject.toml @@ -0,0 +1,28 @@ +[build-system] +requires = ["setuptools>=68"] +build-backend = "setuptools.build_meta" + +[project] +name = "veyron" +version = "1.2.0" +description = "Evidence-first weight-only model checkpoint forensics" +requires-python = ">=3.10" +dependencies = [ + "safetensors>=0.5.3", + "numpy>=1.26", + "huggingface_hub>=0.24", + "torch>=2.2", +] + +[project.optional-dependencies] +dev = ["pytest>=8.0", "ruff>=0.8"] + +[project.scripts] +veyron = "veyron.cli:main" + +[tool.pytest.ini_options] +testpaths = ["tests"] + +[tool.ruff] +line-length = 100 +target-version = "py310" diff --git a/Veyron-v1.2-full/requirements.txt b/Veyron-v1.2-full/requirements.txt new file mode 100644 index 0000000..9617899 --- /dev/null +++ b/Veyron-v1.2-full/requirements.txt @@ -0,0 +1,7 @@ +safetensors>=0.5.3 +numpy>=1.26 +pytest>=8.0 +huggingface_hub>=0.24 +torch>=2.2 +hf_transfer>=0.1.6 +ruff>=0.8 diff --git a/Veyron-v1.2-full/run_veyron.bat b/Veyron-v1.2-full/run_veyron.bat new file mode 100644 index 0000000..234efab --- /dev/null +++ b/Veyron-v1.2-full/run_veyron.bat @@ -0,0 +1,48 @@ +@echo off +setlocal enabledelayedexpansion +title Veyron - Model Forensics Blind Test +cd /d "%~dp0" + +echo ============================================================ +echo VEYRON - Model Forensics Blind Test Launcher +echo ============================================================ +echo. + +REM --- Check Python is available --- +where python >nul 2>nul +if errorlevel 1 ( + echo [ERROR] Python was not found on PATH. + echo Install Python 3.10+ from https://python.org and re-run this file. + pause + exit /b 1 +) + +REM --- Create venv on first run only --- +if not exist ".venv\Scripts\python.exe" ( + echo [setup] First run detected. Creating virtual environment... + python -m venv .venv + if errorlevel 1 ( + echo [ERROR] Failed to create virtual environment. + pause + exit /b 1 + ) + echo [setup] Installing dependencies... this may take a few minutes. + ".venv\Scripts\python.exe" -m pip install --upgrade pip >nul + ".venv\Scripts\python.exe" -m pip install -r requirements.txt + if errorlevel 1 ( + echo [ERROR] Dependency installation failed. Check your internet connection. + pause + exit /b 1 + ) + echo [setup] Done. + echo. +) + +REM --- Enable fast parallel downloads --- +set HF_HUB_ENABLE_HF_TRANSFER=1 + +REM --- Run the interactive blind test --- +".venv\Scripts\python.exe" -m veyron.interactive + +echo. +pause diff --git a/Veyron-v1.2-full/tests/test_blind.py b/Veyron-v1.2-full/tests/test_blind.py new file mode 100644 index 0000000..3c422d3 --- /dev/null +++ b/Veyron-v1.2-full/tests/test_blind.py @@ -0,0 +1,61 @@ +""" +Regression test for the fused-QKV naming bug found via real-model blind +testing (GPT-2's c_attn convention was not recognized), plus a smoke +test of the full blind-scoring pipeline against a synthetic ground truth. +""" +from __future__ import annotations +from veyron.checkpoint import TensorRecord +from veyron.structure import infer_architecture +from veyron.hypotheses import generate_hypotheses +from veyron.blind_score import build_scorecard +from veyron.ground_truth import ground_truth_unavailable + + +def _fake_record(name, shape): + return TensorRecord( + name=name, shape=shape, dtype="float32", + numel=1, stats={"mean": 0.0, "std": 1.0, "sparsity": 0.0}, + fingerprint="x", + ) + + +def test_fused_qkv_naming_detected(): + """GPT-2-style fused QKV (c_attn) must be detected as attention, + not just split-QKV (q_proj/k_proj/v_proj) naming.""" + records = [ + _fake_record("transformer.h.0.attn.c_attn.weight", (768, 2304)), + _fake_record("transformer.h.0.mlp.c_fc.weight", (768, 3072)), + _fake_record("transformer.h.0.ln_1.weight", (768,)), + ] + findings = infer_architecture(records) + types = {f["type"] for f in findings} + assert "attention" in types + assert "feed_forward" in types + assert "architecture_family" in types + + +def test_split_qkv_naming_still_detected(): + """Don't regress the original Llama-style split QKV detection.""" + records = [ + _fake_record("model.layers.0.self_attn.q_proj.weight", (768, 768)), + _fake_record("model.layers.0.self_attn.k_proj.weight", (768, 768)), + _fake_record("model.layers.0.mlp.gate_proj.weight", (768, 3072)), + ] + findings = infer_architecture(records) + types = {f["type"] for f in findings} + assert "attention" in types + assert "feed_forward" in types + + +def test_training_objective_hypothesis_fires_on_fused_qkv(): + records = [_fake_record("transformer.h.0.attn.c_attn.weight", (768, 2304))] + hyps = generate_hypotheses(records) + assert any(h["type"] == "training_objective_clue" for h in hyps) + + +def test_insufficient_evidence_when_no_ground_truth(): + scorecard = build_scorecard([], [], ground_truth_unavailable("no config")) + assert scorecard["architecture"]["status"] == "INSUFFICIENT_EVIDENCE" + assert scorecard["data"]["status"] == "INSUFFICIENT_EVIDENCE" + assert scorecard["lineage"]["status"] == "INSUFFICIENT_EVIDENCE" + assert scorecard["overall_forensic_reliability"] is None diff --git a/Veyron-v1.2-full/tests/test_structure_v12.py b/Veyron-v1.2-full/tests/test_structure_v12.py new file mode 100644 index 0000000..c8d223a --- /dev/null +++ b/Veyron-v1.2-full/tests/test_structure_v12.py @@ -0,0 +1,18 @@ +from veyron.checkpoint import TensorRecord +from veyron.structure import infer_architecture + + +def record(name, shape): + return TensorRecord(name, shape, "float32", 1, {"mean": 0.0, "std": 1.0, "sparsity": 0.0}, "x") + + +def test_fused_qkv_geometry_and_gated_ffn_are_reported(): + records = [ + record("model.layers.0.self_attn.qkv.weight", (2304, 768)), + record("model.layers.0.mlp.gate_proj.weight", (3072, 768)), + record("model.layers.0.mlp.down_proj.weight", (768, 3072)), + ] + findings = {item["type"]: item for item in infer_architecture(records)} + assert findings["attention_layout"]["value"] == "fused-qkv-like" + assert findings["feed_forward_layout"]["value"]["gated"] is True + assert findings["hidden_size_clue"]["value"] == 768 diff --git a/Veyron-v1.2-full/tests/test_v1.py b/Veyron-v1.2-full/tests/test_v1.py new file mode 100644 index 0000000..fc01080 --- /dev/null +++ b/Veyron-v1.2-full/tests/test_v1.py @@ -0,0 +1,23 @@ +import json +from pathlib import Path + +def test_package_import(): + import veyron + assert veyron.__version__ == "1.2.0" + +def test_synthetic_benchmark(): + from veyron.benchmark import run_benchmark + result = run_benchmark() + assert result["accuracy"] == 1.0 + assert all(result["checks"].values()) + +def test_fingerprint_determinism(): + import torch, tempfile + from veyron.checkpoint import load_checkpoint + from veyron.fingerprint import model_fingerprint + with tempfile.TemporaryDirectory() as td: + p = Path(td) / "a.pt" + torch.save({"x": torch.arange(12).reshape(3,4).float()}, p) + r1 = load_checkpoint(p) + r2 = load_checkpoint(p) + assert model_fingerprint(r1) == model_fingerprint(r2) diff --git a/Veyron-v1.2-full/tests/test_v11.py b/Veyron-v1.2-full/tests/test_v11.py new file mode 100644 index 0000000..3006bae --- /dev/null +++ b/Veyron-v1.2-full/tests/test_v11.py @@ -0,0 +1,27 @@ +from pathlib import Path + +from veyron.blind_score import score_architecture +from veyron.blind_test import _is_url +from veyron.checkpoint import CheckpointError, load_checkpoint + + +def test_hf_url_detection(): + assert _is_url("https://huggingface.co/gpt2") + + +def test_missing_architecture_prediction_is_a_scored_miss(): + result = score_architecture([], {"available": True, "architecture_family": "transformer-like", + "layers": 2, "hidden_size": 64}) + assert result["status"] == "SCORED" + assert result["accuracy"] == 0.0 + + +def test_unknown_checkpoint_suffix_is_rejected(tmp_path): + path = tmp_path / "model.unknown" + path.write_bytes(b"not a checkpoint") + try: + load_checkpoint(path) + except CheckpointError as exc: + assert "Unsupported format" in str(exc) + else: + raise AssertionError("unsupported format was accepted") diff --git a/Veyron-v1.2-full/veyron/__init__.py b/Veyron-v1.2-full/veyron/__init__.py new file mode 100644 index 0000000..c68196d --- /dev/null +++ b/Veyron-v1.2-full/veyron/__init__.py @@ -0,0 +1 @@ +__version__ = "1.2.0" diff --git a/Veyron-v1.2-full/veyron/__main__.py b/Veyron-v1.2-full/veyron/__main__.py new file mode 100644 index 0000000..fed5d8e --- /dev/null +++ b/Veyron-v1.2-full/veyron/__main__.py @@ -0,0 +1,3 @@ +from .cli import main +if __name__ == "__main__": + main() diff --git a/Veyron-v1.2-full/veyron/benchmark.py b/Veyron-v1.2-full/veyron/benchmark.py new file mode 100644 index 0000000..8e93877 --- /dev/null +++ b/Veyron-v1.2-full/veyron/benchmark.py @@ -0,0 +1,60 @@ +from __future__ import annotations +from pathlib import Path +import json, tempfile + +def _make_state_dict(torch, kind="base"): + torch.manual_seed(7 if kind == "base" else 13) + d = 32 + v = 64 + layers = 3 + state = { + "model.embed_tokens.weight": torch.randn(v, d) * 0.02, + "model.norm.weight": torch.ones(d), + "lm_head.weight": torch.randn(v, d) * 0.02, + } + for i in range(layers): + prefix = f"model.layers.{i}" + state[f"{prefix}.self_attn.q_proj.weight"] = torch.randn(d, d) * 0.02 + state[f"{prefix}.self_attn.k_proj.weight"] = torch.randn(d, d) * 0.02 + state[f"{prefix}.self_attn.v_proj.weight"] = torch.randn(d, d) * 0.02 + state[f"{prefix}.self_attn.o_proj.weight"] = torch.randn(d, d) * 0.02 + state[f"{prefix}.mlp.gate_proj.weight"] = torch.randn(d*4, d) * 0.02 + state[f"{prefix}.mlp.up_proj.weight"] = torch.randn(d*4, d) * 0.02 + state[f"{prefix}.mlp.down_proj.weight"] = torch.randn(d, d*4) * 0.02 + state[f"{prefix}.input_layernorm.weight"] = torch.ones(d) + state[f"{prefix}.post_attention_layernorm.weight"] = torch.ones(d) + return state + +def run_benchmark(): + import torch + from .checkpoint import load_checkpoint + from .structure import infer_architecture + + with tempfile.TemporaryDirectory() as td: + path = Path(td) / "synthetic.pt" + truth = { + "architecture_family": "transformer-like", + "layers": 3, + "hidden_size": 32, + "parameter_count": sum(v.numel() for v in _make_state_dict(torch).values()) + } + torch.save(_make_state_dict(torch), path) + records = load_checkpoint(path) + findings = infer_architecture(records) + values = {f["type"]: f["value"] for f in findings} + checks = { + "architecture_family": values.get("architecture_family") == truth["architecture_family"], + "layers": values.get("layer_count_estimate") == truth["layers"], + "hidden_size": values.get("hidden_size_clue") == truth["hidden_size"], + } + score = sum(checks.values()) / len(checks) + result = { + "benchmark": "Veyron synthetic weight-only benchmark v1", + "ground_truth": truth, + "checks": checks, + "accuracy": score, + } + return result + +if __name__ == "__main__": + print(json.dumps(run_benchmark(), indent=2)) diff --git a/Veyron-v1.2-full/veyron/blind_score.py b/Veyron-v1.2-full/veyron/blind_score.py new file mode 100644 index 0000000..4fb97b1 --- /dev/null +++ b/Veyron-v1.2-full/veyron/blind_score.py @@ -0,0 +1,142 @@ +""" +Blind scoring: compares Veyron's weight-only findings against real +ground truth (from the model's own published config), and produces +an honest forensic reliability report. + +Design rule: a category only gets a numeric score if it is actually +checkable from what we have. Categories without verifiable ground +truth are marked INSUFFICIENT_EVIDENCE, not guessed. +""" +from __future__ import annotations + + +def _find(findings, ftype): + for f in findings: + if f["type"] == ftype: + return f + return None + + +def _find_hyp(hypotheses, htype): + for h in hypotheses: + if h["type"] == htype: + return h + return None + + +def score_architecture(findings, gt) -> dict: + """Architecture is checkable: family, layer count, hidden size are all + in the model's real config.json.""" + if not gt.get("available"): + return {"status": "INSUFFICIENT_EVIDENCE", "reason": gt.get("reason", "no ground truth")} + + checks = [] + detail = [] + + fam = _find(findings, "architecture_family") + fam_guess = fam["value"] if fam else None + if gt.get("architecture_family") is not None: + fam_correct = fam_guess == gt.get("architecture_family") + checks.append(fam_correct) + detail.append({"field": "architecture_family", "guess": fam_guess, + "truth": gt.get("architecture_family"), "correct": fam_correct}) + + layer_f = _find(findings, "layer_count_estimate") + layer_guess = layer_f["value"] if layer_f else None + layer_truth = gt.get("layers") + if layer_truth is not None: + layer_correct = layer_guess is not None and int(layer_guess) == int(layer_truth) + checks.append(layer_correct) + detail.append({"field": "layers", "guess": layer_guess, "truth": layer_truth, "correct": layer_correct}) + + hidden_f = _find(findings, "hidden_size_clue") + hidden_guess = hidden_f["value"] if hidden_f else None + hidden_truth = gt.get("hidden_size") + if hidden_truth is not None: + hidden_correct = hidden_guess is not None and int(hidden_guess) == int(hidden_truth) + checks.append(hidden_correct) + detail.append({"field": "hidden_size", "guess": hidden_guess, "truth": hidden_truth, "correct": hidden_correct}) + + if not checks: + return {"status": "INSUFFICIENT_EVIDENCE", "reason": "No comparable fields between findings and config"} + + accuracy = sum(1 for c in checks if c) / len(checks) + return {"status": "SCORED", "accuracy": accuracy, "checks": detail, "n_checks": len(checks)} + + +def score_training_objective(hypotheses, gt) -> dict: + """Training objective can ONLY be weakly checked: does the causal-LM + naming clue agree with what the config actually declares as its + architecture class? This is still an indirect, low-confidence check -- + Veyron cannot verify the actual training objective/loss used.""" + if not gt.get("available") or gt.get("is_causal_lm") is None: + return {"status": "INSUFFICIENT_EVIDENCE", + "reason": "Config does not declare a checkable training-objective class"} + + hyp = _find_hyp(hypotheses, "training_objective_clue") + guessed_causal = hyp is not None + truth_causal = bool(gt.get("is_causal_lm")) + correct = guessed_causal == truth_causal + return { + "status": "SCORED", + "accuracy": 1.0 if correct else 0.0, + "n_checks": 1, + "checks": [{"field": "is_causal_lm", "guess": guessed_causal, "truth": truth_causal, "correct": correct}], + "note": "Weak signal only: naming convention agreement, not a verified training objective.", + } + + +def score_data_domain(candidate_corpus_result) -> dict: + """Data-domain attribution requires a candidate corpus to compare + against. Without one supplied by the user, there is nothing to + verify against and Veyron correctly claims no data-domain knowledge.""" + if candidate_corpus_result is None: + return {"status": "INSUFFICIENT_EVIDENCE", + "reason": "No candidate corpus supplied -- data-domain attribution is unverifiable from weights alone"} + return candidate_corpus_result + + +def score_lineage(comparison_result) -> dict: + """Lineage requires a second checkpoint to compare against. Without + one, there is no lineage claim to make or score.""" + if comparison_result is None: + return {"status": "INSUFFICIENT_EVIDENCE", + "reason": "No second checkpoint supplied -- lineage/similarity is unverifiable from a single model"} + return comparison_result + + +def evidence_coverage(*category_results) -> float: + """Fraction of categories that had enough evidence to be scored at all.""" + scored = [c for c in category_results if c.get("status") == "SCORED"] + return len(scored) / len(category_results) if category_results else 0.0 + + +def high_confidence_claim_rate(findings, hypotheses, threshold=0.85) -> float: + """Fraction of ALL findings+hypotheses Veyron emitted that it labeled + with confidence >= threshold. This reflects Veyron's own self-reported + certainty distribution, not correctness.""" + all_items = list(findings) + list(hypotheses) + if not all_items: + return 0.0 + high = sum(1 for i in all_items if i.get("confidence", 0) >= threshold) + return high / len(all_items) + + +def build_scorecard(findings, hypotheses, gt, comparison_result=None, candidate_corpus_result=None) -> dict: + arch = score_architecture(findings, gt) + train = score_training_objective(hypotheses, gt) + data = score_data_domain(candidate_corpus_result) + lineage = score_lineage(comparison_result) + + overall_scored = [c["accuracy"] for c in (arch, train, data, lineage) if c.get("status") == "SCORED"] + overall = sum(overall_scored) / len(overall_scored) if overall_scored else None + + return { + "architecture": arch, + "training": train, + "data": data, + "lineage": lineage, + "overall_forensic_reliability": overall, + "evidence_coverage": evidence_coverage(arch, train, data, lineage), + "high_confidence_claim_rate": high_confidence_claim_rate(findings, hypotheses), + } diff --git a/Veyron-v1.2-full/veyron/blind_test.py b/Veyron-v1.2-full/veyron/blind_test.py new file mode 100644 index 0000000..77a8846 --- /dev/null +++ b/Veyron-v1.2-full/veyron/blind_test.py @@ -0,0 +1,303 @@ +""" +Veyron Blind Test -- real-model benchmark runner. + +Workflow: + 1. Take a HF repo URL / git link (or local folder) from the user. + 2. Download/copy the FULL repo into a "vault" folder (hidden from analyzer). + 3. Copy ONLY the weight file(s) into a bare "blind" folder -- strip + config.json, README, model card, tokenizer files, everything that + would leak the answer. + 4. Run Veyron's normal analysis pipeline against the blind folder only. + 5. AFTER Veyron has produced its findings, reveal the real config.json + from the vault and score Veyron's guesses against it. + 6. Print the forensic scorecard. + +This ordering matters: Veyron's code path never sees ground truth until +after it has committed to an answer, so the benchmark is a genuine blind +test, not a leak-and-check. +""" +from __future__ import annotations +import json +import os +import shutil +import sys +from urllib.parse import urlparse +from pathlib import Path + +from .checkpoint import load_checkpoint +from .structure import infer_architecture, structure_summary +from .hypotheses import generate_hypotheses +from .fingerprint import model_fingerprint +from .ground_truth import ground_truth_from_config, ground_truth_unavailable +from .blind_score import build_scorecard +from .scorecard_display import render_scorecard + +WEIGHT_SUFFIXES = {".safetensors", ".pt", ".pth", ".bin"} +CONFIG_NAME = "config.json" + +# Only fetch what forensics actually needs -- skip tokenizer/vocab/merges, +# redundant weight formats, and other repo bloat. Big speed win on its own. +ALLOW_PATTERNS = ["*.safetensors", "*.safetensors.index.json", "config.json"] + +# Enable HF's Rust-backed parallel chunked downloader if installed. +# 2-5x faster than the default downloader on most connections. +os.environ.setdefault("HF_HUB_ENABLE_HF_TRANSFER", "1") + + +def _is_url(s: str) -> bool: + return s.startswith("http://") or s.startswith("https://") + + +def _hf_transfer_available() -> bool: + try: + import hf_transfer # noqa: F401 + return True + except ImportError: + return False + + +def fetch_repo(source: str, vault_dir: Path, prefer_safetensors_only: bool = True) -> Path: + """Fetch a model repo into vault_dir. Supports: + - a local directory path (copied as-is) + - a Hugging Face repo URL (downloaded via huggingface_hub, if network allows) + Raises a clear error if network access to the source is unavailable -- + never fabricates a result. + + Speed optimizations applied: + - allow_patterns filters out tokenizer/vocab/redundant .bin weights + when a .safetensors file exists, so only what's needed is pulled. + - hf_transfer (if installed) is used automatically for parallel + chunked downloads. + """ + vault_dir.mkdir(parents=True, exist_ok=True) + + if not _is_url(source): + src = Path(source) + if not src.exists(): + raise FileNotFoundError(f"Local path does not exist: {src}") + if src.is_dir(): + shutil.copytree(src, vault_dir, dirs_exist_ok=True) + else: + shutil.copy2(src, vault_dir / src.name) + return vault_dir + + # HF-style URL: https://huggingface.co// + try: + from huggingface_hub import snapshot_download + except ImportError as e: + raise RuntimeError( + "huggingface_hub is required to fetch from a URL. " + "Install with: pip install huggingface_hub" + ) from e + + parsed = urlparse(source) + if parsed.netloc not in {"huggingface.co", "www.huggingface.co"}: + raise ValueError(f"Unrecognized model URL: {source}") + parts = [part for part in parsed.path.split("/") if part] + if len(parts) < 1: + raise ValueError(f"Unrecognized model URL: {source}") + # HF supports both /org/repo and /org/repo/tree/ URLs. + tree_index = parts.index("tree") if "tree" in parts else None + repo_parts = parts[:tree_index] if tree_index is not None else parts + if len(repo_parts) not in {1, 2}: + raise ValueError(f"Unrecognized model URL: {source}") + repo_id = "/".join(repo_parts) + revision = "/".join(parts[tree_index + 1:]) if tree_index is not None else None + if tree_index is not None and not revision: + raise ValueError(f"Missing revision in model URL: {source}") + + if not _hf_transfer_available(): + print(" [speed] hf_transfer not installed -- using default downloader.") + print(" [speed] For 2-5x faster downloads: pip install hf_transfer") + + patterns = ALLOW_PATTERNS if prefer_safetensors_only else None + + try: + downloaded = snapshot_download( + repo_id=repo_id, + local_dir=str(vault_dir), + allow_patterns=patterns, + max_workers=8, + revision=revision, + ) + except Exception as e: + # Some repos ship weights ONLY as .bin (no safetensors) -- retry + # without the safetensors-only filter instead of failing outright. + if patterns is not None: + print(" [fallback] No .safetensors found with filtered fetch, retrying full weight set...") + try: + downloaded = snapshot_download( + repo_id=repo_id, + local_dir=str(vault_dir), + allow_patterns=["*.bin", "*.pt", "*.pth", "config.json"], + max_workers=8, + revision=revision, + ) + except Exception as e2: + raise RuntimeError( + f"Could not download '{repo_id}'. Check your network connection " + f"and that the repo is public. Original error: {e2}" + ) from e2 + else: + raise RuntimeError( + f"Could not download '{repo_id}'. Check your network connection " + f"and that the repo is public. Original error: {e}" + ) from e + return Path(downloaded) + + +def make_blind_copy(vault_dir: Path, blind_dir: Path) -> Path: + """Copy ONLY weight files into blind_dir. No config, no README, no + tokenizer, no model card -- Veyron must guess from tensors alone.""" + blind_dir.mkdir(parents=True, exist_ok=True) + found = [] + for f in vault_dir.rglob("*"): + if f.is_file() and f.suffix.lower() in WEIGHT_SUFFIXES: + # Preserve relative layout: shards can share generic filenames. + dest = blind_dir / f.relative_to(vault_dir) + dest.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(f, dest) + found.append(dest) + for index in vault_dir.rglob("*.safetensors.index.json"): + dest = blind_dir / index.relative_to(vault_dir) + dest.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(index, dest) + if not found: + raise FileNotFoundError( + f"No weight files ({', '.join(WEIGHT_SUFFIXES)}) found in {vault_dir}" + ) + indexes = list(blind_dir.rglob("*.safetensors.index.json")) + return indexes[0] if indexes else found[0] + + +def _true_claims_only(findings, hypotheses, scorecard) -> dict: + """Extract only what Veyron actually got right, per the scorecard. + This is Veyron's own record of 'what I claimed and turned out true' -- + not a dump of every finding, and not the ground truth itself.""" + correct_fields = set() + for cat in ("architecture", "training"): + cat_result = scorecard.get(cat, {}) + if cat_result.get("status") == "SCORED": + for check in cat_result.get("checks", []): + if check.get("correct"): + correct_fields.add(check["field"]) + + # Map scored field names back to the finding/hypothesis types that produced them + field_to_types = { + "architecture_family": {"architecture_family"}, + "layers": {"layer_count_estimate"}, + "hidden_size": {"hidden_size_clue"}, + "is_causal_lm": {"training_objective_clue"}, + } + confirmed_types = set() + for field in correct_fields: + confirmed_types |= field_to_types.get(field, set()) + + true_findings = [f for f in findings if f["type"] in confirmed_types] + true_hypotheses = [h for h in hypotheses if h["type"] in confirmed_types] + + return { + "confirmed_findings": true_findings, + "confirmed_hypotheses": true_hypotheses, + "confirmed_field_count": len(correct_fields), + "note": "Only findings/hypotheses that were checked against real ground " + "truth and scored correct are included here. Unverified or " + "incorrect claims are excluded -- see the full report for those.", + } + + +def _save_run_folder(result: dict, runs_root: Path, model_name: str, source: str) -> Path: + import re + from datetime import datetime, timezone + + ts = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S") + label = re.sub(r"[^a-zA-Z0-9._-]+", "_", model_name or source).strip("_")[:60] or "run" + run_dir = runs_root / f"{ts}_{label}" + run_dir.mkdir(parents=True, exist_ok=True) + + (run_dir / "report.json").write_text( + json.dumps(result, indent=2, default=str), encoding="utf-8" + ) + + true_claims = _true_claims_only(result["findings"], result["hypotheses"], result["scorecard"]) + (run_dir / "true_findings.json").write_text( + json.dumps(true_claims, indent=2, default=str), encoding="utf-8" + ) + + (run_dir / "scorecard.txt").write_text(result["report_text"], encoding="utf-8") + + return run_dir + + +def run_blind_test(source: str, workdir: str = "./veyron_blind_test", model_name: str = "", + save_run: bool = True, runs_dir: str = "./veyron_runs") -> dict: + workdir = Path(workdir) + vault_dir = workdir / "vault" # hidden from analyzer -- has the real answer + blind_dir = workdir / "blind" # what Veyron actually sees + if workdir.exists(): + shutil.rmtree(workdir) + + print(f"[1/5] Fetching source into vault (hidden): {source}") + fetch_repo(source, vault_dir) + + print("[2/5] Building blind copy (weights only, config/README stripped)...") + weight_path = make_blind_copy(vault_dir, blind_dir) + print(f" Blind checkpoint: {weight_path.name}") + + print("[3/5] Running Veyron analysis -- NO ground truth visible at this stage...") + records = load_checkpoint(weight_path) + findings = infer_architecture(records) + hypotheses = generate_hypotheses(records) + fp = model_fingerprint(records) + summary = structure_summary(records) + + print("[4/5] Revealing ground truth from vault and scoring...") + config_path = vault_dir / CONFIG_NAME + if config_path.exists(): + gt = ground_truth_from_config(config_path) + else: + # search subdirs in case snapshot_download nested it + candidates = list(vault_dir.rglob(CONFIG_NAME)) + gt = ground_truth_from_config(candidates[0]) if candidates else ground_truth_unavailable( + f"No {CONFIG_NAME} found anywhere in fetched repo" + ) + + scorecard = build_scorecard(findings, hypotheses, gt) + + print("[5/5] Done.\n") + report_text = render_scorecard(scorecard, model_name=model_name or source) + print(report_text) + + result = { + "source": source, + "fingerprint": fp, + "structure_summary": summary, + "findings": findings, + "hypotheses": hypotheses, + "ground_truth": gt, + "scorecard": scorecard, + "report_text": report_text, + } + + if save_run: + run_dir = _save_run_folder(result, Path(runs_dir), model_name, source) + result["run_dir"] = str(run_dir) + print(f"\nSaved run: {run_dir}") + print(f" - report.json full results") + print(f" - true_findings.json only claims confirmed correct against ground truth") + print(f" - scorecard.txt the printed scorecard above") + + return result + + +def main(): + if len(sys.argv) < 2: + print("Usage: python -m veyron.blind_test [model_name]") + sys.exit(1) + source = sys.argv[1] + name = sys.argv[2] if len(sys.argv) > 2 else "" + run_blind_test(source, model_name=name) + + +if __name__ == "__main__": + main() diff --git a/Veyron-v1.2-full/veyron/checkpoint.py b/Veyron-v1.2-full/veyron/checkpoint.py new file mode 100644 index 0000000..16e678c --- /dev/null +++ b/Veyron-v1.2-full/veyron/checkpoint.py @@ -0,0 +1,234 @@ +from __future__ import annotations +from dataclasses import dataclass +from pathlib import Path +from typing import Any +import json +import hashlib + +_DEVICE = None + +def get_device(): + """Resolve compute device once per process. CUDA > MPS > CPU. + Fingerprinting always runs on CPU regardless -- SHA256 must be + reproducible byte-for-byte and GPU float reduction order can vary.""" + global _DEVICE + if _DEVICE is not None: + return _DEVICE + import torch + if torch.cuda.is_available(): + _DEVICE = torch.device("cuda") + elif getattr(torch.backends, "mps", None) and torch.backends.mps.is_available(): + _DEVICE = torch.device("mps") + else: + _DEVICE = torch.device("cpu") + return _DEVICE + +@dataclass +class TensorRecord: + name: str + shape: tuple[int, ...] + dtype: str + numel: int + stats: dict + fingerprint: str + spectrum: dict | None = None + +class CheckpointError(RuntimeError): + pass + +def _numel(shape): + n = 1 + for x in shape: + n *= int(x) + return n + +def _torch_stats(value, max_samples=1_000_000): + import torch + device = get_device() + t = value.detach().float().to(device, non_blocking=True).reshape(-1) + # Exact reductions on a 70B checkpoint are not a useful default. A + # deterministic stride keeps reports reproducible while bounding memory/time. + if t.numel() > max_samples: + step = (t.numel() + max_samples - 1) // max_samples + t = t[::step] + if t.numel() == 0: + return {"min": None, "max": None, "mean": None, "std": None, + "l1": None, "l2": None, "sparsity": None, + "q01": None, "q50": None, "q99": None} + finite = torch.isfinite(t) + if not bool(finite.all()): + t = t[finite] + if t.numel() == 0: + return {"min": None, "max": None, "mean": None, "std": None, + "l1": None, "l2": None, "sparsity": None, + "q01": None, "q50": None, "q99": None} + return { + "min": float(t.min()), + "max": float(t.max()), + "mean": float(t.mean()), + "std": float(t.std(unbiased=False)), + "l1": float(t.abs().sum()), + "l2": float(torch.linalg.vector_norm(t)), + "sparsity": float((t == 0).float().mean()), + "q01": float(torch.quantile(t, 0.01)), + "q50": float(torch.quantile(t, 0.50)), + "q99": float(torch.quantile(t, 0.99)), + } + +def _fingerprint(value): + # Deliberately CPU + float32 + contiguous: SHA256 must be reproducible + # byte-for-byte across machines/devices, and GPU op ordering can + # introduce nondeterministic rounding in reductions feeding into this. + t = value.detach().float().cpu().contiguous() + raw = t.numpy().tobytes() + return hashlib.sha256(raw).hexdigest() + +def _spectrum(value): + import torch + device = get_device() + t = value.detach().float().to(device, non_blocking=True) + if t.ndim != 2 or min(t.shape) < 2: + return None + # SVD grows rapidly; use a deterministic <=512x512 view. + if t.numel() > 262_144: + scale = max(1, int((t.numel() / 262_144) ** 0.5)) + t = t[::scale, ::scale] + try: + s = torch.linalg.svdvals(t) + if s.numel() == 0: + return None + s = s.cpu() + total = float((s ** 2).sum()) + top = min(16, s.numel()) + energy = float((s[:top] ** 2).sum() / total) if total else None + return { + "rank_estimate": int((s > (s.max() * 1e-6)).sum()), + "top_singular_values": [float(x) for x in s[:top]], + "top16_energy_ratio": energy, + "max_singular": float(s.max()), + "min_singular": float(s.min()), + } + except Exception: + return None + + +def _records_from_mapping(mapping, *, show_progress=True): + import sys, time + records = [] + items = list(mapping.items()) + total = len(items) + last_print = time.monotonic() + for i, (name, value) in enumerate(items): + if not hasattr(value, "shape"): + continue + try: + shape = tuple(int(x) for x in value.shape) + records.append(TensorRecord( + name=str(name), + shape=shape, + dtype=str(getattr(value, "dtype", "unknown")), + numel=_numel(shape), + stats=_torch_stats(value), + fingerprint=_fingerprint(value), + spectrum=_spectrum(value), + )) + except Exception: + continue + # Large tensors (e.g. wte/lm_head, SVD on big matrices) can each take + # real time -- print periodically so a long run doesn't look hung. + now = time.monotonic() + if show_progress and (now - last_print > 2.0 or i == total - 1): + print(f" analyzing tensors: {i + 1}/{total}", file=sys.stderr) + last_print = now + return records + +def checkpoint_sha256(path): + path = Path(path) + if path.is_dir(): + h = hashlib.sha256() + for child in sorted(p for p in path.rglob("*") if p.is_file()): + h.update(str(child.relative_to(path)).replace("\\", "/").encode()) + h.update(checkpoint_sha256(child).encode()) + return h.hexdigest() + h = hashlib.sha256() + with open(path, "rb") as f: + for chunk in iter(lambda: f.read(1024 * 1024), b""): + h.update(chunk) + return h.hexdigest() + +def _load_safetensors(path): + from safetensors import safe_open + import sys, time + records = [] + with safe_open(str(path), framework="pt", device="cpu") as f: + keys = list(f.keys()) + total = len(keys) + last_print = time.monotonic() + for i, name in enumerate(keys): + records.extend(_records_from_mapping({name: f.get_tensor(name)}, show_progress=False)) + now = time.monotonic() + if now - last_print > 2.0 or i == total - 1: + print(f" analyzing tensors: {i + 1}/{total}", file=sys.stderr) + last_print = now + return records + + +def _load_sharded_safetensors(index_path): + index = json.loads(Path(index_path).read_text(encoding="utf-8")) + weights = index.get("weight_map") + if not isinstance(weights, dict): + raise CheckpointError(f"Invalid safetensors index: {index_path}") + records = [] + for shard in sorted(set(weights.values())): + shard_path = Path(index_path).parent / shard + if not shard_path.exists(): + raise CheckpointError(f"Shard listed in index is missing: {shard_path}") + records.extend(_load_safetensors(shard_path)) + return records + + +def load_checkpoint(path, *, allow_unsafe_pickle=False): + path = Path(path) + if not path.exists(): + raise CheckpointError(f"Checkpoint does not exist: {path}") + if path.is_dir(): + indexes = sorted(path.glob("*.safetensors.index.json")) + if indexes: + return _load_sharded_safetensors(indexes[0]) + candidates = sorted(path.glob("*.safetensors")) + if len(candidates) == 1: + return _load_safetensors(candidates[0]) + raise CheckpointError("Directory must contain one .safetensors file or a safetensors index.") + if path.name.endswith(".safetensors.index.json"): + return _load_sharded_safetensors(path) + suffix = path.suffix.lower() + + if suffix == ".safetensors": + try: + from safetensors import safe_open + except ImportError as exc: + raise CheckpointError("Install safetensors first.") from exc + return _load_safetensors(path) + + if suffix in {".pt", ".pth", ".bin"}: + try: + import torch + except ImportError as exc: + raise CheckpointError("PyTorch is required for .pt/.pth/.bin.") from exc + try: + obj = torch.load(str(path), map_location="cpu", weights_only=True) + except TypeError as exc: + if not allow_unsafe_pickle: + raise CheckpointError( + "This PyTorch version cannot safely load legacy pickle checkpoints. " + "Use safetensors or explicitly opt in with --allow-unsafe-pickle for a trusted file." + ) from exc + obj = torch.load(str(path), map_location="cpu", weights_only=False) + if isinstance(obj, dict): + for key in ("state_dict", "model", "module"): + if isinstance(obj.get(key), dict): + return _records_from_mapping(obj[key]) + return _records_from_mapping(obj) + raise CheckpointError("Checkpoint does not contain a recognizable state dict.") + + raise CheckpointError(f"Unsupported format: {suffix}") diff --git a/Veyron-v1.2-full/veyron/cli.py b/Veyron-v1.2-full/veyron/cli.py new file mode 100644 index 0000000..1bd2755 --- /dev/null +++ b/Veyron-v1.2-full/veyron/cli.py @@ -0,0 +1,49 @@ +from __future__ import annotations +import argparse, json +from .checkpoint import load_checkpoint, CheckpointError +from .report import build_report, print_summary, save_json +from .similarity import compare +from .benchmark import run_benchmark +from .evaluation import run_evaluation + +def main(): + p = argparse.ArgumentParser(prog="veyron") + sub = p.add_subparsers(dest="command", required=True) + + a = sub.add_parser("analyze") + a.add_argument("checkpoint") + a.add_argument("--out") + a.add_argument("--allow-unsafe-pickle", action="store_true", + help="allow legacy pickle loading for a checkpoint you trust") + + c = sub.add_parser("compare") + c.add_argument("model_a") + c.add_argument("model_b") + + sub.add_parser("benchmark") + e = sub.add_parser("evaluate", help="score a local manifest of independently labelled checkpoints") + e.add_argument("manifest") + + args = p.parse_args() + try: + if args.command == "analyze": + records = load_checkpoint(args.checkpoint, allow_unsafe_pickle=args.allow_unsafe_pickle) + report = build_report(args.checkpoint, records) + print_summary(report) + if args.out: + save_json(report, args.out) + print(f"\nReport: {args.out}") + + elif args.command == "compare": + a_records = load_checkpoint(args.model_a) + b_records = load_checkpoint(args.model_b) + print(json.dumps(compare(a_records, b_records), indent=2)) + + elif args.command == "benchmark": + print(json.dumps(run_benchmark(), indent=2)) + + elif args.command == "evaluate": + print(json.dumps(run_evaluation(args.manifest), indent=2)) + + except CheckpointError as e: + p.error(str(e)) diff --git a/Veyron-v1.2-full/veyron/evaluation.py b/Veyron-v1.2-full/veyron/evaluation.py new file mode 100644 index 0000000..8f702d4 --- /dev/null +++ b/Veyron-v1.2-full/veyron/evaluation.py @@ -0,0 +1,35 @@ +"""Repeatable local evaluation from a declarative manifest. + +The manifest contains only checkpoint paths and independently supplied facts; +the analyzer never reads the expected fields while producing findings. +""" +from __future__ import annotations + +import json +from pathlib import Path + +from .checkpoint import load_checkpoint +from .structure import infer_architecture +from .hypotheses import generate_hypotheses +from .blind_score import build_scorecard + + +def run_evaluation(manifest_path): + manifest_path = Path(manifest_path) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + cases = manifest.get("cases") + if not isinstance(cases, list) or not cases: + raise ValueError("Evaluation manifest must contain a non-empty 'cases' array.") + results = [] + for case in cases: + checkpoint = Path(case["checkpoint"]) + if not checkpoint.is_absolute(): + checkpoint = manifest_path.parent / checkpoint + records = load_checkpoint(checkpoint) + scorecard = build_scorecard(infer_architecture(records), generate_hypotheses(records), case["ground_truth"]) + results.append({"id": case.get("id", str(checkpoint)), "scorecard": scorecard}) + scored = [r["scorecard"]["overall_forensic_reliability"] for r in results + if r["scorecard"]["overall_forensic_reliability"] is not None] + return {"suite": manifest.get("name", manifest_path.stem), "cases": results, + "mean_scored_reliability": sum(scored) / len(scored) if scored else None, + "scored_case_count": len(scored), "case_count": len(results)} diff --git a/Veyron-v1.2-full/veyron/fingerprint.py b/Veyron-v1.2-full/veyron/fingerprint.py new file mode 100644 index 0000000..e8e129b --- /dev/null +++ b/Veyron-v1.2-full/veyron/fingerprint.py @@ -0,0 +1,53 @@ +from __future__ import annotations +import hashlib, json, math +from collections import defaultdict + +def _round(x): + return None if x is None else round(float(x), 8) + +def tensor_signature(record): + s = record.stats + return { + "name": record.name, + "shape": list(record.shape), + "dtype": record.dtype, + "numel": record.numel, + "mean": _round(s.get("mean")), + "std": _round(s.get("std")), + "l1": _round(s.get("l1")), + "l2": _round(s.get("l2")), + "sparsity": _round(s.get("sparsity")), + "spectrum": record.spectrum, + } + +def model_fingerprint(records): + payload = [] + for r in sorted(records, key=lambda x: x.name): + payload.append(tensor_signature(r)) + raw = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode() + return hashlib.sha256(raw).hexdigest() + +def aggregate_profile(records): + means = [r.stats["mean"] for r in records if r.stats.get("mean") is not None] + stds = [r.stats["std"] for r in records if r.stats.get("std") is not None] + sparsity = [r.stats["sparsity"] for r in records if r.stats.get("sparsity") is not None] + return { + "tensor_mean_average": sum(means) / len(means) if means else None, + "tensor_std_average": sum(stds) / len(stds) if stds else None, + "tensor_sparsity_average": sum(sparsity) / len(sparsity) if sparsity else None, + "tensor_count": len(records), + } + +def layer_profiles(records): + buckets = defaultdict(list) + for r in records: + parts = r.name.replace("_", ".").split(".") + for i, part in enumerate(parts[:-1]): + if part.isdigit() and i > 0: + buckets[int(part)].append(r) + break + out = {} + for idx, rs in sorted(buckets.items()): + p = aggregate_profile(rs) + out[str(idx)] = p + return out diff --git a/Veyron-v1.2-full/veyron/ground_truth.py b/Veyron-v1.2-full/veyron/ground_truth.py new file mode 100644 index 0000000..8cfffdf --- /dev/null +++ b/Veyron-v1.2-full/veyron/ground_truth.py @@ -0,0 +1,69 @@ +""" +Ground truth resolution for real-model blind testing. + +This module NEVER touches the analyzer. It fetches the model's own +published config (config.json) so the benchmark has real facts to +score Veyron's blind guesses against. If no config is available, +fields are left as None ("unknown") rather than guessed. +""" +from __future__ import annotations +import json +import re +from pathlib import Path + + +# Common HF config key -> our normalized field name +_LAYER_KEYS = ["num_hidden_layers", "n_layer", "num_layers"] +_HIDDEN_KEYS = ["hidden_size", "n_embd", "d_model"] +_HEAD_KEYS = ["num_attention_heads", "n_head"] +_MODEL_TYPE_KEYS = ["model_type", "architectures"] + + +def ground_truth_from_config(config_path) -> dict: + """Parse a real config.json into a normalized ground-truth dict. + Only fields we can actually check against Veyron's findings are populated. + """ + config_path = Path(config_path) + if not config_path.exists(): + return {"available": False, "reason": f"No config found at {config_path}"} + + try: + cfg = json.loads(config_path.read_text(encoding="utf-8")) + except Exception as e: + return {"available": False, "reason": f"Could not parse config: {e}"} + + def first(keys): + for k in keys: + if k in cfg and cfg[k] is not None: + return cfg[k] + return None + + layers = first(_LAYER_KEYS) + hidden = first(_HIDDEN_KEYS) + heads = first(_HEAD_KEYS) + model_type = cfg.get("model_type") + architectures = cfg.get("architectures") + + is_causal_lm = False + if architectures: + is_causal_lm = any( + re.search(r"(CausalLM|LMHeadModel|ForCausalLM|GPT|Llama|Mistral|Qwen)", a, re.I) + for a in architectures + ) + + transformer_keys = {"hidden_size", "n_embd", "d_model", "num_hidden_layers", "n_layer", "num_layers"} + return { + "available": True, + "source": str(config_path), + "architecture_family": "transformer-like" if transformer_keys & set(cfg) else None, + "layers": layers, + "hidden_size": hidden, + "attention_heads": heads, + "model_type": model_type, + "architectures": architectures, + "is_causal_lm": is_causal_lm, + } + + +def ground_truth_unavailable(reason: str) -> dict: + return {"available": False, "reason": reason} diff --git a/Veyron-v1.2-full/veyron/hypotheses.py b/Veyron-v1.2-full/veyron/hypotheses.py new file mode 100644 index 0000000..4f57752 --- /dev/null +++ b/Veyron-v1.2-full/veyron/hypotheses.py @@ -0,0 +1,43 @@ +from __future__ import annotations +import re + +def generate_hypotheses(records): + names = " ".join(r.name.lower() for r in records) + findings = [] + + if any(x in names for x in ("q_proj", "k_proj", "v_proj", "c_attn", "in_proj", "w_qkv")): + findings.append({ + "type": "training_objective_clue", + "value": "causal-language-model-like architecture", + "confidence": 0.58, + "evidence": "Decoder-style attention projection naming is compatible with causal LM architectures.", + "warning": "Architecture alone cannot prove the training objective." + }) + + adapter = any(x in names for x in ("lora", "lora_a", "lora_b", "adapter")) + if adapter: + findings.append({ + "type": "adaptation_clue", + "value": "adapter/LoRA-related tensors present", + "confidence": 0.98, + "evidence": "Adapter-like tensor names were found." + }) + + merged = any(x in names for x in ("merged", "merge", "delta")) + if merged: + findings.append({ + "type": "merge_clue", + "value": "possible merged/delta checkpoint", + "confidence": 0.55, + "evidence": "Merge/delta naming clues were found.", + "warning": "Naming is not proof of a merge." + }) + + return findings + +def caution(): + return { + "exact_training_code_recovery": False, + "exact_dataset_recovery": False, + "reason": "Final weights are not a unique encoding of the original pipeline." + } diff --git a/Veyron-v1.2-full/veyron/interactive.py b/Veyron-v1.2-full/veyron/interactive.py new file mode 100644 index 0000000..da151cd --- /dev/null +++ b/Veyron-v1.2-full/veyron/interactive.py @@ -0,0 +1,46 @@ +""" +Interactive launcher for Veyron's real-model blind test. +Prompts for a Hugging Face repo URL (or local checkpoint folder), +runs the full fetch -> blind copy -> analyze -> reveal -> score +pipeline, prints the forensic scorecard, and optionally saves JSON. +""" +from __future__ import annotations +import json +import sys +from pathlib import Path + +from .blind_test import run_blind_test + + +def main(): + print("=" * 60) + print(" VEYRON -- Model Forensics Blind Test") + print("=" * 60) + print() + print("Paste a Hugging Face model URL (e.g. https://huggingface.co/gpt2)") + print("or a local path to a folder containing model weights + config.json.") + print() + source = input("Model link or path: ").strip() + if not source: + print("No input given, exiting.") + sys.exit(1) + + name = input("Optional label for this run (press Enter to skip): ").strip() + + try: + result = run_blind_test(source, model_name=name) + except Exception as e: + print() + print(f"FAILED: {e}") + print() + print("This is a real error, not a fake result -- Veyron does not") + print("fabricate a scorecard when it cannot actually fetch or read") + print("the checkpoint.") + sys.exit(1) + + print() + input("Press Enter to close...") + + +if __name__ == "__main__": + main() diff --git a/Veyron-v1.2-full/veyron/report.py b/Veyron-v1.2-full/veyron/report.py new file mode 100644 index 0000000..94a3548 --- /dev/null +++ b/Veyron-v1.2-full/veyron/report.py @@ -0,0 +1,54 @@ +from __future__ import annotations +import json +from pathlib import Path +from .checkpoint import checkpoint_sha256 +from .fingerprint import model_fingerprint, aggregate_profile, layer_profiles, tensor_signature +from .structure import infer_architecture, structure_summary +from .hypotheses import generate_hypotheses, caution + +def build_report(path, records): + return { + "veyron_version": "1.2.0", + "checkpoint": { + "path": str(Path(path).resolve()), + "sha256": checkpoint_sha256(path), + }, + "scope": { + "input": "model checkpoint weights", + "method": "weight-only forensic analysis", + "claims_are_probabilistic": True, + "analysis_limits": { + "statistics": "deterministic bounded sampling for very large tensors", + "spectra": "deterministic bounded matrix sampling", + "identity": "not inferred from weights alone", + }, + }, + "structure": structure_summary(records), + "fingerprint": model_fingerprint(records), + "aggregate_profile": aggregate_profile(records), + "layer_profiles": layer_profiles(records), + "findings": infer_architecture(records), + "hypotheses": generate_hypotheses(records), + "limitations": caution(), + "tensors": [tensor_signature(r) for r in records], + } + +def print_summary(report): + s = report["structure"] + print("VEYRON v1") + print("="*60) + print(f"Parameters : {s['parameter_count']:,} ({s['parameter_count_billions']:.4f}B)") + print(f"Tensors : {s['tensor_count']:,}") + print(f"Fingerprint: {report['fingerprint'][:32]}...") + print("\nFindings:") + for f in report["findings"]: + print(f" [{f['confidence']:.0%}] {f['type']}: {f['value']}") + print("\nHypotheses:") + for f in report["hypotheses"]: + print(f" [{f['confidence']:.0%}] {f['type']}: {f['value']}") + print("\nBoundary:") + print(" Exact training script: NOT CLAIMED") + print(" Exact dataset: NOT CLAIMED") + +def save_json(report, path): + Path(path).write_text(json.dumps(report, indent=2), encoding="utf-8") diff --git a/Veyron-v1.2-full/veyron/scorecard_display.py b/Veyron-v1.2-full/veyron/scorecard_display.py new file mode 100644 index 0000000..a3aad0a --- /dev/null +++ b/Veyron-v1.2-full/veyron/scorecard_display.py @@ -0,0 +1,56 @@ +from __future__ import annotations + +BAR_WIDTH = 20 + + +def _bar(fraction, width=BAR_WIDTH): + filled = int(round(fraction * width)) + return "\u2588" * filled + "\u2591" * (width - filled) + + +def _dashed_bar(width=BAR_WIDTH): + return "\u2591" * width # empty/hollow bar -- no evidence + + +def render_scorecard(scorecard: dict, model_name: str = "") -> str: + lines = [] + title = "MODEL FORENSICS BENCHMARK" + if model_name: + title += f" -- {model_name}" + lines.append(title) + lines.append("\u2500" * 60) + lines.append("") + + labels = [ + ("architecture", "Architecture"), + ("training", "Training "), + ("data", "Data "), + ("lineage", "Lineage "), + ] + + for key, label in labels: + cat = scorecard[key] + if cat.get("status") == "SCORED": + pct = cat["accuracy"] * 100 + lines.append(f" {label} {_bar(cat['accuracy'])} {pct:5.1f}% ({cat['n_checks']} checkable field(s))") + else: + lines.append(f" {label} {_dashed_bar()} INSUFFICIENT EVIDENCE") + lines.append(f" {' ' * len(label)} -> {cat.get('reason', 'not verifiable from this input')}") + + lines.append("") + overall = scorecard["overall_forensic_reliability"] + if overall is not None: + lines.append(f" Overall forensic reliability {_bar(overall)} {overall*100:5.1f}%") + else: + lines.append(" Overall forensic reliability " + _dashed_bar() + " NO SCORABLE CATEGORIES") + lines.append("") + lines.append(f" Evidence coverage : {scorecard['evidence_coverage']*100:.0f}%" + f" ({sum(1 for k,_ in labels if scorecard[k]['status']=='SCORED')}/4 categories had real ground truth)") + lines.append(f" High-confidence claims : {scorecard['high_confidence_claim_rate']*100:.0f}%" + f" (share of Veyron's own findings rated \u226585% confidence)") + lines.append("") + lines.append(" NOTE: 'High-confidence claims' reflects Veyron's self-reported") + lines.append(" certainty, not verified accuracy. Only categories marked SCORED") + lines.append(" above were checked against the model's real published config.") + + return "\n".join(lines) diff --git a/Veyron-v1.2-full/veyron/similarity.py b/Veyron-v1.2-full/veyron/similarity.py new file mode 100644 index 0000000..06a47cd --- /dev/null +++ b/Veyron-v1.2-full/veyron/similarity.py @@ -0,0 +1,41 @@ +from __future__ import annotations +import math + +def _vector(records): + vals = [] + for r in sorted(records, key=lambda x: x.name): + s = r.stats + vals.extend([ + float(s.get("mean") or 0), + float(s.get("std") or 0), + float(s.get("sparsity") or 0), + ]) + return vals + +def cosine(a, b): + n = min(len(a), len(b)) + if n == 0: + return 0.0 + a, b = a[:n], b[:n] + dot = sum(x*y for x,y in zip(a,b)) + na = math.sqrt(sum(x*x for x in a)) + nb = math.sqrt(sum(x*x for x in b)) + return dot/(na*nb) if na and nb else 0.0 + +def compare(records_a, records_b): + by_a = {r.name: r for r in records_a} + by_b = {r.name: r for r in records_b} + names_a, names_b = set(by_a), set(by_b) + overlap = len(names_a & names_b) / max(1, len(names_a | names_b)) + aligned = [name for name in sorted(names_a & names_b) if by_a[name].shape == by_b[name].shape] + deltas = [] + for name in aligned: + a, b = by_a[name].stats, by_b[name].stats + scale = max(abs(float(a.get("std") or 0)), abs(float(b.get("std") or 0)), 1e-12) + deltas.append(abs(float(a.get("mean") or 0) - float(b.get("mean") or 0)) / scale) + return { + "tensor_name_jaccard": overlap, + "shape_aligned_tensor_coverage": len(aligned) / max(1, len(names_a | names_b)), + "aligned_mean_normalized_delta": sum(deltas) / len(deltas) if deltas else None, + "interpretation": "similarity evidence, not proof of shared training lineage" + } diff --git a/Veyron-v1.2-full/veyron/structure.py b/Veyron-v1.2-full/veyron/structure.py new file mode 100644 index 0000000..f5939e1 --- /dev/null +++ b/Veyron-v1.2-full/veyron/structure.py @@ -0,0 +1,149 @@ +"""Conservative structural inference from checkpoint names and geometry. + +Every finding carries its evidence. This module intentionally describes +compatible structures, never an asserted model identity or training history. +""" +from __future__ import annotations + +import re +from collections import Counter + + +def has(records, pattern): + rx = re.compile(pattern, re.I) + return any(rx.search(r.name) for r in records) + + +def infer_layers(records): + indices = [] + patterns = [ + re.compile(r"(?:^|\.)(?:layers|h|blocks)\.(\d+)(?:\.|$)", re.I), + re.compile(r"(?:^|_)(?:layer|block)[._]?(\d+)(?:_|\.|$)", re.I), + ] + for record in records: + for rx in patterns: + match = rx.search(record.name) + if match: + indices.append(int(match.group(1))) + break + if not indices: + return None + unique = sorted(set(indices)) + contiguous = unique == list(range(unique[-1] + 1)) + return { + "value": unique[-1] + 1, + "confidence": 0.96 if contiguous else 0.72, + "evidence": f"Detected {len(unique)} layer indices from {unique[0]} to {unique[-1]}." + + (" Indices are contiguous." if contiguous else " Indices have gaps."), + } + + +def infer_hidden_size(records): + candidates = [] + for record in records: + if len(record.shape) != 2: + continue + a, b = record.shape + name = record.name.lower() + if any(k in name for k in ("c_attn", "w_qkv", "qkv", "in_proj")): + # Fused QKV has one dimension near 3×width; width is the smaller one. + candidates.append(min(a, b)) + elif any(k in name for k in ("q_proj", "k_proj", "v_proj", "o_proj")): + candidates.extend(x for x in (a, b) if 32 <= x <= 32768 and x <= min(a, b) * 3) + elif any(k in name for k in ("embed", "embedding", "wte", "word_embeddings")): + candidates.append(min(a, b)) + if not candidates: + candidates = [r.shape[0] for r in records if len(r.shape) == 2 and r.shape[0] == r.shape[1] + and 32 <= r.shape[0] <= 32768] + if not candidates: + return None + value, count = Counter(candidates).most_common(1)[0] + return {"value": value, "confidence": min(0.97, 0.60 + 0.06 * count), + "evidence": f"Model-width candidate {value} recurs in {count} projection/embedding tensors."} + + +def _attention_layout(records): + fused, split = [], [] + for record in records: + if len(record.shape) != 2: + continue + name, shape = record.name.lower(), record.shape + if any(token in name for token in ("c_attn", "w_qkv", "qkv", "in_proj")): + fused.append(shape) + elif any(token in name for token in ("q_proj", "k_proj", "v_proj", "query", "key", "value")): + split.append(shape) + if fused: + thirds = sum(1 for a, b in fused if a == 3 * b or b == 3 * a) + return {"type": "attention_layout", "value": "fused-qkv-like", "confidence": 0.93 if thirds else 0.74, + "evidence": f"Found {len(fused)} fused-QKV-named matrices; {thirds} have 3× geometry."} + if split: + return {"type": "attention_layout", "value": "split-qkv-like", "confidence": 0.90, + "evidence": f"Found {len(split)} split Q/K/V-named projection matrices."} + return None + + +def _mlp_layout(records, hidden): + ratios, gated = [], 0 + for record in records: + if len(record.shape) != 2: + continue + name = record.name.lower() + if not any(token in name for token in ("mlp", "ffn", "fc1", "c_fc", "up_proj", "gate_proj", "down_proj")): + continue + if "gate" in name: + gated += 1 + if hidden and hidden["value"] in record.shape: + other = max(record.shape) + ratios.append(round(other / hidden["value"], 3)) + if not ratios: + return None + ratio = Counter(ratios).most_common(1)[0][0] + return {"type": "feed_forward_layout", "value": {"expansion_ratio": ratio, "gated": gated > 0}, + "confidence": 0.88 if len(ratios) >= 2 else 0.68, + "evidence": f"{len(ratios)} FFN matrices support an approximate {ratio}× expansion" + + (" and gate projections are present." if gated else ".")} + + +def infer_architecture(records): + findings = [] + qkv = has(records, r"(q_proj|k_proj|v_proj|qkv|query|key|value|c_attn|w_qkv|in_proj)") + mlp = has(records, r"(mlp|ffn|gate_proj|up_proj|down_proj|fc1|fc2|c_fc|c_proj)") + norm = has(records, r"(norm|layernorm|ln_)") + emb = has(records, r"(embed|embedding|wte|word_embeddings)") + out = has(records, r"(lm_head|output|classifier)") + if qkv and mlp: + findings.append(("architecture_family", "transformer-like", 0.90, + "Attention and feed-forward projections co-occur across the tensor inventory.")) + if qkv: + findings.append(("attention", "attention projections detected", 0.93, + "Query/key/value-like tensor names detected.")) + if mlp: + findings.append(("feed_forward", "MLP/FFN detected", 0.92, "Feed-forward projection tensor names detected.")) + if norm: + findings.append(("normalization", "normalization detected", 0.88, "Normalization-like tensor names detected.")) + if emb: + findings.append(("embedding", "input embedding detected", 0.86, "Embedding-like tensor names detected.")) + if out: + findings.append(("output_head", "output head detected", 0.72, "Output-head-like tensor names detected.")) + layer, hidden = infer_layers(records), infer_hidden_size(records) + for extra in (_attention_layout(records), _mlp_layout(records, hidden)): + if extra: + findings.append(extra) + if layer: + findings.append(("layer_count_estimate", layer["value"], layer["confidence"], layer["evidence"])) + if hidden: + findings.append(("hidden_size_clue", hidden["value"], hidden["confidence"], hidden["evidence"])) + normalized = [] + for finding in findings: + if isinstance(finding, dict): + normalized.append(finding) + else: + kind, value, confidence, evidence = finding + normalized.append({"type": kind, "value": value, "confidence": confidence, "evidence": evidence}) + return normalized + + +def structure_summary(records): + total = sum(r.numel for r in records) + return {"parameter_count": total, "parameter_count_billions": total / 1e9, + "tensor_count": len(records), "dtype_counts": dict(Counter(r.dtype for r in records))}