Skip to content

task: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #348

Open

Description

@fake0fan

Current state: Existing Issue author @fake0fan has submitted PR #351, now 1694a0b, but this Issue still records Proposed, unassigned ownership and no accepted implementation delegation. The PR now contains the six content/coverage files plus 80 regenerated packaged knowledge files, and keeps Refs #348 rather than closure wording. At the prior 18c9a16 head, Ready had four software-check failures from stale packaged projections (including the reported 6 failed / 3538 passed full-check result). The 1694a0b5 incremental review clears the stale-index delta by static inspection; the subsequent current-head Ready result now has all six jobs passing, including both full-check and installed-package versions. This supersedes the earlier pending/full-check failure state, not the remaining review questions. The exact-head declaration now names claude-opus-5 and claude-opus-5-5 with effort unknown and extends its exception request to the generated records. The policy check still fails: explicit independent owner exception approval is absent, material AI author 1's model is unapproved, and author 2's effort is not allowed. The reviewer also retains the source/provenance-marker questions, blog-figure sourcing, and branch-only source-pin integration question; no merge strategy or new acceptance policy is chosen here. Source/index consistency and CI do not establish model/performance efficacy, acceptance or authority to close this Issue.
Decision needed: The requester/maintainers need to record the accepted scope/delegation and name the accountable and acceptance owners, which remain TBD below, including the existing gate-ownership/cross-link question. A submitted PR or reviewer's suggestions do not make those decisions.
Next action: Existing PR author @fake0fan retains the remaining source clarification and re-review work, accurate revision-specific documentation, and any exact-head policy-exception request to an authorized independent owner. The generated/packaged-consistency failure is no longer an outstanding repair at 1694a0b. Preserve the original proposed scope and acceptance text below; this status grants no implementation, policy-exception, merge or closure authority.

Goal and acceptance

methods/performance-ceiling-analysis L1 turns a model config plus a hardware
table entry into a runtime floor. It gates neither half of the demand side: the
skill has no procedure for checking that the weight and KV counts it divides by a
peak describe the model actually being deployed. safetensors appears nowhere
under this owner, while the mechanics already exist next door in
weight-loading-optimization
and a byte-level weight inventory already exists as an artifact pattern under
h3-c.

An L1 floor computed from a wrong parameter count is wrong in a way no roofline
check detects, because the roofline never looks at the model again. This issue
adds the missing prerequisite and records the external source it is derived from.

  • Problem / desired outcome: a reviewed-source prerequisite that anchors L1's
    weight demand against the checkpoint before any number derived from it is
    reported, with its blind spot stated rather than implied.
  • Completion criteria:
    • A reference document and a typed source-identity.json under
      methods/performance-ceiling-analysis/references/, recording the pinned
      commit, per-file git blob and sha256 digests, and the verbatim-vs-summary
      coverage of each inspected source.
    • A prerequisite step in SKILL.md covering two gates: the bidirectional
      per-tensor checkpoint join, and the per-rank weight ledger. The first needs no
      parallelism layout; passing it certifies nothing about per-rank bytes.
    • An explicit statement of what the weight anchor does not anchor (KV
      semantics: a windowed and a global attention with the same head dims have
      byte-identical weights).
    • A metadata.json source entry and a docs/reference-matrix.md row.
    • locally_verified stays false; empirical_validation stays not_run.

Scope and constraints

Ownership and authorization

  • Proposed by / actual requester: repository owner
  • Status: Proposed
  • Accountable parent (Issue-level owner, if assigned): TBD
  • Slice implementer(s) / acting contributor (if assigned): TBD
  • Goal-acceptance owner: TBD
  • Accepted decision or scoped delegation: Not yet granted

Inputs and deliverables

  • Relevant sources, revisions or linked issues:
    • github.com/shenh10/HuggingArch @ facbbbc96856e558b9084a94a5070b2e2c86cf0b
      (MIT; single init repo commit dated 2026-11-04T02:24:14Z), inspected
      2026-11-04. Readset: docs/validation.md, docs/measured_calibration.md,
      docs/evaluation_model.md, docs/known_limitations.md,
      backend/arch/validate/_matching.py, backend/arch/validate/_weight_ledger.py,
      backend/arch/validate/_ablation.py, backend/measure/scope.py,
      kernel_bench/emit.py.
    • A companion blog post dated 2026-10-08, read in part. The repository
      post-dates it by roughly four weeks and carries no history, so the post's
      figures are not attributable to this commit and are recorded as unverified
      where they are recorded at all.
    • Related: [RFC] Per-task SOL modeling and a KDA-flow gap metric for fused operators #303 (proposed; not a dependency of this issue).
  • Expected artifacts: one PR touching only the files named under completion
    criteria.

Completion and closure

  • Who accepts completion and may close: the named acceptance owner, not the PR merge.
  • Remaining uncertainty / next decision: whether the two gates belong to this
    owner alone or should be cross-declared as a reference from
    weight-loading-optimization; the PR proposes the former with a cross-link.

Activity

  1. amy-why-3459 commented on Nov 4, 2026

    @amy-why-3459
    Collaborator

    Thanks, the scoping here is clear, and separating "the weight anchor certifies X" from "it says nothing about KV semantics" is the right framing. A few points to settle before a PR:

    1. Where the gates live. weight-loading-optimization already owns the safetensors mechanics (references/fastsafetensors-transfer-readset.md). I agree with putting the correctness gate under performance-ceiling-analysis, since that's where the L1 number gets consumed, but please make the cross-link go both ways so the loading skill doesn't end up growing a second, slightly different checkpoint-inventory procedure.
    2. Reusing the existing inventory shape. repositories/h3-c/audit-h3-runtime-residency/references/minimax-h3-weight-inventory.json is already a byte-level inventory. Will the per-tensor join reuse that schema (or name the fields it shares with it), rather than adding a new one? One shape would make the two artifacts comparable.
    3. What the join compares. It would help to spell out the comparison key and tolerance: tensor name, dtype, shape, and byte size from the safetensors header against what the config implies, and how tied embeddings, quantization scales/zero-points, and MTP/draft heads are counted. Those are the usual places where config-derived parameter counts and checkpoint bytes disagree.
    4. Per-rank ledger inputs. Since the second gate needs a parallelism layout, please state which TP/EP/PP sharding rules it assumes (replicated vs. sharded tensors, expert placement) and that it stops if the layout isn't given, instead of falling back to a default.
    5. Source identity. Since HuggingArch has a single unversioned commit, recording git blob and sha256 per inspected file is good. I'd also keep the blog-post figures out of SKILL.md entirely, not just marked unverified, so they can't get quoted as numbers later.

    The out-of-scope list (no #303 bound model, no Class A/B vocabulary, locally_verified: false) looks right to me.

  2. added
    area:knowledgeKnowledge line: packs, skill library, knowledge admission
    kind:taskBounded delivery task with a delegated slice
    on Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:knowledgeKnowledge line: packs, skill library, knowledge admissionkind:taskBounded delivery task with a delegated slice

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions