Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,18 @@ Initial public foundation (v0.1).
native vision), AOI 53.2, grade D. Classified open-weights, not closed-frontier; grounded to
primary sources, with the offensive-cyber safety finding and the new Model-as-a-Service licence
gate as the load-bearing risks.
- Three new inference-provider entries, each drafted from automated research and then
independently re-verified by a Cowork browser session (see the "Corrected" entries below for
what changed on verification): Fireworks AI (AOI 64.8, grade C - a Zero Data Retention posture
and the Response API's 30-day-default retention exception are independently confirmed via
readable docs, but the Terms of Service and DPA turned out to be unreachable to a real browser,
so confidentiality is coded unknown rather than asserted), Baseten (AOI 73.6, grade B - Zero
Data Retention by default plus a genuinely MUTUAL confidentiality clause, confirmed on two
independent live fetches, and a real 99.9% SLA; no published EU region for the managed
service), and Nebius / Token Factory (AOI 60.8, grade C - its central finding, that the binding
Terms of Service contradict the marketing claim of no training on customer content, is
confirmed verbatim on independent re-read; EU-region and SLA claims were corrected downward on
the same pass).

### Changed

Expand All @@ -38,6 +50,63 @@ Initial public foundation (v0.1).
grant the aggregate entry had described. Safety was re-scored from 4 to 3 across all three, because
no published model-level safety evaluation was found for any release; Ultra's ownership is now
substantial on the OpenMDW "without restriction" grant.
- Refreshed the Together AI entry's model catalogue and pricing (last verified 2026-08-06, past the
30-day fast-moving SLA): the served catalogue moved forward a generation (DeepSeek V3/R1 to
V4-generation, Qwen3 to 3.5/3.7/3.8, plus new GLM-5.3 and Kimi K3 entries), Mistral/Mixtral was
removed from serverless, and the prior "embeddings" claim is corrected - Together's own docs now
state no embedding models are offered via serverless. Score unchanged (governance terms have not
materially changed since the last pass).
- Corrected all four inference-provider entries above against an independent Cowork browser
verification pass, replacing draft findings from automated web-fetch research with directly
read, human-supervised evidence. Fireworks AI: the Terms of Service and DPA turned out to be
unreachable to a real browser (page-level noindexed) - confidentiality moved from
`functional_only` to `unknown`, sub-processor disclosure from `true` to `false`, and
`trains_on_inputs` from `never` to `opt_in_only` (headline 71.6 -> 64.8, grade B -> C; ownership
substantial -> partial). Baseten: the mutual confidentiality clause was confirmed on two
independent live fetches, and a real 99.9% SLA was found where none had been located, but a
prior ISO 27001 claim was retracted for lack of any basis (headline 71.2 -> 73.6, grade
unchanged at B). Nebius: the entry's central finding (training by default under the binding
Terms of Service, contradicting marketing) was confirmed verbatim, but the EU-region count
(four -> two public self-serve), a claimed inference SLA (retracted - none exists), and a
"dedicated-only" catalogue claim (models were simply superseded, not gated) were all corrected
downward, while compliance strengthened with a named SOC 2 auditor (headline 65.2 -> 60.8,
grade unchanged at C; also corrected the ownership level from `limited` to `partial` per the
ownership rule, independent of this pass). Together AI: the confidentiality-disclaimed finding
and the catalogue corrections were both independently confirmed unchanged, but `zdr_available`
was corrected from `yes_default` to `yes_optional` (Zero Data Retention is account-level opt-in,
not the default) - a real field-level fix with no score impact, since data_governance was
already capped at 3 by the confidentiality disclaimer.
- Freshness re-verification sweep across the 21 model entries the daily freshness cron flagged as
49-58 days past the 30-day fast SLA (surfaced when PR #22's CI ran `validate.py` with the
freshness check active). A Cowork browser session re-checked each entry's licence, safety
posture, and availability against current primary sources - not a bulk date-bump. 20 of 21
re-verified clean with no score change; several flag newer-generation releases (DeepSeek V4
line and V3.1/V3.2, a licence-divergent GLM-5 generation, Nemotron 3.5 Lightning 30B-A3B, OLMo 3,
EuroLLM-22B-2512, Qwen3.5/3.6/3.8, Kimi K2.5/K2.6/K2.7-Code) as new entities for a future build
pass, deliberately not folded into the existing entries' scores. The one entry that needed real
judgment, not just a re-check, was `soofi`: the research surfaced that Soofi-S-Base's model card
reads "closed-beta" and the repo is gated, in tension with the project's own "open-source"
marketing - but the entry's 2026-07-25 correction already accounts for exactly this (openness
held at the open_weights ceiling, ownership already partial), so no further downgrade was
warranted. Full research pack archived under
`docs/freshness-sweeps/2026-09-21/`.
- Second freshness re-verification sweep, covering the remaining 8 stale entries (3 hosting
providers, 5 inference providers) - the same daily cron finding, out of scope for the model
sweep above. Six re-verified clean with no score change (Hugging Face, Berget, Groq, Infercom,
plus two hosting providers - llama.cpp/GGUF and Ollama - where a fresh in-window CVE each
(CVE-2026-52131, CVE-2026-85180) reinforced rather than moved their existing grade). Two needed
real re-grades: DeepInfra's Terms of Service was rewritten (effective 2026-08-17) to add a
genuine mutual confidentiality clause and make Zero Data Retention contractual and controlling
with a bounded exception, lifting it from 52.0 D to 56.8 C; Runware's current Terms no longer
contain the adverse, non-confidentiality clause a prior pass had found - the customer now owns
Outputs outright with no broad licence granted to Runware - lifting the adverse hard flag and
moving data_governance from 2 to 3 (45.6 D to 50.4 D; the grade itself did not cross into C
because compliance and residency, unrelated to this finding, were not re-examined). Runware's
correction carries a load-bearing caveat: the removal is inferred from the current text's
absence of the old clauses, since web archives were proxy-blocked and no verbatim historical
diff was possible. `python scripts/validate.py` now reports 0 errors registry-wide for the
first time this session. Research pack archived under
`docs/freshness-sweeps/2026-09-21-providers/`.

### The index

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# AOI freshness re-verification sweep - hosting/inference providers - 2026-09-21

The second half of the PR #22 freshness debt: **8 stale provider entries** (3 hosting/distribution, 5 inference), all 46-58 days past SLA. Real re-check against current primary sources, not a bulk date-bump. Every entry re-read this pass; `retrieved` flags honest per file. This lands `validate.py` at 0 errors registry-wide.

## Summary table

| Entity | Type | Result | What changed | Action |
|---|---|---|---|---|
| `hugging-face` (79.2 B) | hosting | clean | scanning policy verbatim unchanged; no HF-native signing; July intrusion predates window | bump, hold |
| `llama-cpp-gguf` (56.4 C) | hosting | delta | new CVE-2026-52131 (2026-09-01, CVSS 7.5, `gguf_reader::read` reachable assertion) | bump, hold C |
| `ollama` (55.2 C) | hosting | delta | new CVE-2026-85180 (2026-09-03, tensor-blob redirect SSRF); OMS signing PR #11573 in-progress, not merged | bump, hold C |
| `berget` (70.4 B) | inference | clean + positive | explicit confidentiality intact; refreshed DPA 2026-09-10 (EEA-only + customer region choice + 5 named sub-processors); ISO 27001 still a claim | bump, hold B |
| `groq` (70.0 B) | inference | clean | Sec 10 mutual confidentiality verbatim unchanged; training prohibited; "Eligible Customers" ZDR gap persists; Helsinki ~14-mo track record (no contractual region-pinning); Trust Center Vanta-gated | bump, hold B |
| `infercom` (76.0 B) | inference | clean | on-prem/air-gap still shipped (load-bearing strength); functional_only confirmed; no training (Sec 2.4); ISO 27001:2022 dated cert; DPA v1.3 PDF gated | bump, hold B |
| `deepinfra` (52.0 D) | inference | **DELTA - material positive** | ToS rewrite 2026-08-17: Section 17 mutual confidentiality (was functional_only); Section 7(b) Zero Data Retention contractual + controlling; training opt-in (hard flag lifted). Residual: disclosed Google/Anthropic model routing | RE-GRADED up: 52.0 D -> 56.8 C |
| `runware` (45.6 D) | inference | **DELTA - material** | current ToS has no adverse SUBMISSIONS clause; §2 user owns Outputs; §3 no-train commitment; private-model confidentiality - adverse cap no longer supported | RE-GRADED up: 45.6 D -> 50.4 D (hard flag lifted) |

## Code-session application (2026-09-21)

Both re-grade recommendations were applied, each with the flagged caveats carried into the entry:

- **deepinfra**: confidentiality functional_only -> mutual; data_governance 3 -> 4; headline 52.0 -> 56.8 (D -> C). Compliance (SOC 2/ISO dated status, DPA/SCCs) was NOT re-examined this pass (trust centre 403'd again) and stays unchanged pending a future check. Open caveat carried into the entry: the rewritten ToS reads as an enterprise/Service-Order MSA at the canonical /terms URL - credited to the base self-serve product as the standard reading of a canonical Terms page, flagged for explicit re-confirmation if a separate self-serve-only ToS surfaces.
- **runware**: confidentiality adverse -> functional_only (not mutual/explicit - no general confidentiality clause covers ordinary prompts/outputs, only the pre-existing Private Model Protection clause); adverse hard flag lifted; trains_on_inputs unclear -> never; customer_data_ownership provider_may_use -> customer_retains; data_governance 2 -> 3; headline 45.6 -> 50.4 (stays D - other dimensions, notably compliance at 1 and residency at 2, were not re-examined and remain the binding constraints). Load-bearing caveat carried into the entry: the confidentiality-clause removal is inferred from the current text's absence of the old clauses, not a verbatim historical diff - all web archives (web.archive.org, archive.ph, timetravel.mementoweb) were proxy-blocked this pass.

## New security findings (hosting) - no grade change

- **CVE-2026-52131** (llama-cpp-gguf, 2026-09-01, CVSS 7.5) - reachable assertion in `gguf_reader::read`. Reinforces the existing C rationale.
- **CVE-2026-85180** (ollama, 2026-09-03) - tensor-blob redirect SSRF to internal/metadata hosts, a separate code path from the earlier fix. Offset by the not-yet-merged OMS signing PR #11573.

## Items still requiring a manual eyeball / re-attempt (tool limits, gated sources)

- **runware Aug-2026 archive diff** - all web archives proxy-blocked; adverse removal inferred, not verbatim-diffed. Retry when the proxy allows if a verbatim diff is later required.
- **deepinfra trust center** (trust.deepinfra.com/compliance) - 403 Sprinto/JS-gated; SOC 2 / ISO 27001 dates not re-confirmed.
- **deepinfra MSA scope** - confirm the rewritten ToS genuinely governs standard self-serve API users, not only enterprise Service Orders.
- **groq Trust Center** (trust.groq.com) - Vanta JS-gated; SOC 2 Type II remains claim-grade this pass, dated certificate not read.
- **infercom DPA v1.3** - full text gated behind a redirect; substantive clauses not read this pass (not load-bearing, since functional_only already stands on the Terms).

## Net result

`python scripts/validate.py` now reports **0 errors registry-wide** for the first time this session. Two material re-grades (deepinfra, runware) and six clean re-verifications, all independently checked against current primary sources rather than bulk-dated.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Research return: ai2-olmo (freshness re-verification) - 2026-09-21

**Verdict: DELTA (supersession) -> bump `last_verified` to 2026-09-21, and add a new sibling.**

- **Licence:** unchanged for the tracked line - OLMo 2 remains **Apache-2.0, fully open** (weights + data + training code). No revision.
- **Supersession - FLAG (add sibling):** AI2 has shipped **OLMo 3**, also **Apache-2.0** and **fully_open**, trained on **Dolma 3**. This is a genuinely new generation continuing the fully-open posture - recommend adding **OLMo 3** as a new sibling. The existing **OLMo 2** entry remains valid (still a live, correctly-graded fully-open model), so this is add-sibling, not replace.
- **Safety / availability:** repos live, ungated; no adverse finding - the strong openness posture (the reason for the high B grade) is intact and extended.

`retrieved: true` - OLMo 2 + OLMo 3 model cards / LICENSE re-read; both Apache-2.0 fully-open. Clause file: `model_card__olmo-3__2026-09-21.md`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Research return: deepseek-r1-distill-llama (freshness re-verification) - 2026-09-21

**Verdict: re-verified CLEAN -> bump `last_verified` to 2026-09-21.**

- **Licence:** unchanged. The R1-Distill-Llama repos remain MIT for the DeepSeek distillation, with the underlying Llama base still governed by the Llama Community Licence (the dual-origin note in the entry still holds). No licence-text revision this pass.
- **Supersession:** no newer distill has replaced the tracked Llama-based R1 distills. **FLAG:** the V4/V3.2 generation exists but DeepSeek has not (as of this check) published an equivalent *R1-Distill-Llama* refresh under it - flagged only so the code session knows the distill line has not been re-cut.
- **Safety / availability:** repos live, ungated, no new safety finding.

`retrieved: true` - model card + LICENSE re-read. No new clause file.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Research return: deepseek-r1-distill-qwen (freshness re-verification) - 2026-09-21

**Verdict: re-verified CLEAN -> bump `last_verified` to 2026-09-21.**

- **Licence:** unchanged. The R1-Distill-Qwen repos remain MIT for the DeepSeek distillation, with the underlying Qwen base under its own Apache-2.0/Qwen licence per size (the entry's per-variant note still holds). No licence-text revision this pass.
- **Supersession:** no newer Qwen-based R1 distill has replaced the tracked set. **FLAG:** the DeepSeek V4/V3.2 generation exists but has not re-cut this distill line; separately the base Qwen family has moved to Qwen3.5/3.6/3.8 (see the `qwen` return) - not folded here.
- **Safety / availability:** repos live, ungated, no new safety finding.

`retrieved: true` - model card + LICENSE re-read. No new clause file.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Research return: deepseek-r1 (freshness re-verification) - 2026-09-21

**Verdict: re-verified CLEAN -> bump `last_verified` to 2026-09-21.**

- **Licence:** unchanged. `deepseek-ai/DeepSeek-R1` HF repo `LICENSE` still MIT (weights) with the code under MIT; model card licence tag reads `mit`. No revision to the licence text since the last check.
- **Supersession:** no newer *R1* point release has replaced the tracked `DeepSeek-R1` (the `-0528` update predates the last verification and is already reflected). **FLAG (do not fold):** DeepSeek has since shipped a materially newer generation - **DeepSeek-V4 line** (V4.1-Flash, V4-Flash-Vision-Exp, V4-Pro-0813, V4-Flash-0731) and **V3.1 / V3.2** - as separate reasoning/base families. These are new entities, not an update to R1; flagged for the code session to decide new-sibling vs new-family.
- **Safety / availability:** repo live, ungated, no new red-team finding, no takedown or gating change.

`retrieved: true` - model card + LICENSE re-read this pass. No new clause file (nothing moved on the tracked entry).
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Research return: deepseek-v3-mit (freshness re-verification) - 2026-09-21

**Verdict: re-verified CLEAN -> bump `last_verified` to 2026-09-21.**

- **Licence:** unchanged. The MIT-licensed DeepSeek-V3 variant remains MIT (this is the arm of the split that carries MIT for both code and weights). No licence-text revision this pass.
- **Supersession:** **FLAG (do not fold).** DeepSeek-V3 has been superseded by **V3.1 and V3.2**, which are also MIT-licensed, and by the **V4 line**. The tracked `deepseek-v3-mit` entry is not wrong, but it no longer represents the current head of the V3 line - flagged for the code session to decide whether to add V3.1/V3.2 as siblings.
- **Safety / availability:** repo live, ungated, no new safety finding.

`retrieved: true` - model card + LICENSE re-read. No new clause file (tracked entry unchanged; supersession flagged only).
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Research return: deepseek-v3-original (freshness re-verification) - 2026-09-21

**Verdict: re-verified CLEAN -> bump `last_verified` to 2026-09-21.**

- **Licence:** unchanged and the split still holds. The "original" DeepSeek-V3 repo remains **dual-licensed - code under MIT, model weights under the DeepSeek License Agreement (v1.0)**. This is exactly the divergence that justifies keeping `deepseek-v3-original` separate from `deepseek-v3-mit`, and it is still real this pass (re-confirmed against the repo `LICENSE` / `LICENSE-MODEL` split). No revision to the DeepSeek License Agreement text.
- **Supersession:** same V3.1/V3.2 -> V4 supersession as the MIT variant; flagged, not folded.
- **Safety / availability:** repo live, ungated, no new safety finding.

`retrieved: true` - model card + both LICENSE files re-read; the MIT-vs-DeepSeek-License split verified still present. No new clause file.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Research return: eurollm (freshness re-verification) - 2026-09-21

**Verdict: DELTA (supersession) -> bump `last_verified` to 2026-09-21, and add a new sibling.**

- **Licence:** unchanged for the tracked line - **EuroLLM-9B** remains **Apache-2.0** and **gated** (accept-terms access on HF). No revision.
- **Supersession - FLAG (add sibling):** a newer **EuroLLM-22B-2512** has shipped. Recommend adding it as a new sibling; the tracked 9B entry remains valid. Confirm the 22B's licence tag against its raw repo `LICENSE` before scoring (expected Apache-2.0, consistent with the family).
- **Safety / availability:** repos live (gated); no adverse finding.

`retrieved: true` - EuroLLM-9B card/LICENSE re-read (still apache-2.0, gated); 22B sibling flagged. Clause file: `model_card__eurollm-22b__2026-09-21.md`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Research return: glm-4-9b-chat-original (freshness re-verification) - 2026-09-21

**Verdict: re-verified CLEAN -> bump `last_verified` to 2026-09-21.**

- **Licence:** unchanged. `THUDM/glm-4-9b-chat` remains under the custom **`glm-4`** licence, and the **commercial-registration clause** (registration/notification required for commercial use) is **intact** - re-read verbatim this pass. No revision.
- **Supersession:** the GLM-5 generation exists (see the `glm` return) but does **not** supersede this specific legacy `glm-4-9b-chat` weight - the entry still points at a live, distinct repo.
- **Safety / availability:** repo **live**, accessible; no takedown, no new adverse finding.

`retrieved: true` - model card + glm-4 LICENSE re-read; commercial-registration clause re-confirmed. No new clause file.
Loading
Loading