Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions clean/converter/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,13 @@ the version still decides the map's features. A map without the field gets the p
not text, or whose preset does not use the map's version (for example `"preset": "K"` in a version-1 map) is "converter
data damaged". This is how a further preset that reuses an existing map version is added: one row in that table.

**Preset L** (Model L, issue #89) is such a row: `"preset": "L"` in a map of version 1 or 4. Its maps use the record
kinds of preset M's (no new element format, op or source kind); its parameters are a block of their own in the same
library file as preset M's, found by its own length and hashes like any source. The map file itself
(`data/preset-L.v1.map.json`) comes with the converter data; until it exists, `tests/run_tests.py` covers preset L with a
synthetic library file holding an L block directly before an M block, and maps for both: each converts its own block,
and a file without a preset's block fails with that preset's message.

## Preset F (map version 5, spec 57)

wronggpu-convert --map data\preset-F.v1.map.json --dll <nvngx_dlssd.dll> [--dll <other> ...] --out <dir> [--raw]
Expand Down Expand Up @@ -112,6 +119,7 @@ Preset K, for the spec team (48-T1, 48-T6):
| 11 | file smaller than W (or above 4 GiB) | This is not a DLSS model file. |
| 12 | M: W not found. K: T not found | This DLSS file has no preset-M model. Version 310.5 or newer is needed. / This DLSS file has no preset-K model. Version 310.1 or newer is needed. (+ "This file is version a.b.c.d." when the file has a version resource) |
| 12 | F: W not found (spec 57 section 8) | This DLSS file has no preset-F (Ray Reconstruction) model. Version 310.7 or newer is needed. (+ the file version when there is one) |
| 12 | L: W not found (a map with `"preset": "L"`) | This DLSS file has no preset-L model. Version 310.9 or newer is needed. (+ the file version when there is one; the version named is provisional) |
| 13 | M: W found, D or K not found. K: T found, O not found (spec 48 draft 4 section 8). F: W found, another source (or every variant of IMG-IN) not found. The log names the missing source | This DLSS build is not supported yet (model data found, but other parts differ). |
| 14 | F only (spec 57 section 8): a constant's sites disagree, a `const_f16x2` with unequal halves, a `const_f16lo` with a non-zero high half, a non-finite value where 57.6.1 or 57.6.2 forbids it, an RN16 overflow, the two reads of an `also` tensor differ, a `require` or check fails, the attention constants fail 57.6.4. The log names the tensor or check (never a value). Not an installer error: the next file is tried | This DLSS build is not supported yet (model data found, but other parts differ). |
| 20 | map invalid: unreadable, bad JSON, bad format or version, unknown op / locate / element / encoding / dtype (or one its version does not define), missing field, unknown perm, perm not dividing its axis, `axes` length ≠ `shape` length, a segment that is not a positive multiple of 4 or runs past the end of T, K sources other than T (and O), `capped_softmax` without 8 × 8 last axes, a source F that is not a keyed optional `lz4-block` block source with variants, a field outside the decoded text, variants with different numbers of fields, a `digest` list whose length is not the number of variants (or a list for a group without variants), a group that mixes tensors from F and from other sources, … Also (spec 08 section 4.3): a variant of F matched its hashes, but the block does not decompress to `decoded_length`, does not match `decoded_sha256`, or a field is not eight hexadecimal digits | Installer error: the converter data is damaged. … |
Expand Down
3 changes: 3 additions & 0 deletions clean/converter/src/api.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ const char* const kMsgNoPresetK = "This DLSS file has no preset-K model. Version
// Spec 57 section 8.
const char* const kMsgNoPresetF =
"This DLSS file has no preset-F (Ray Reconstruction) model. Version 310.7 or newer is needed.";
// Model L (#89). The DLSS version named is provisional: the first version known to carry a preset-L model; one line to
// change when the first shipping version is confirmed.
const char* const kMsgNoPresetL = "This DLSS file has no preset-L model. Version 310.9 or newer is needed.";
const char* const kMsgUnsupported = "This DLSS build is not supported yet (model data found, but other parts differ).";
const char* const kMsgMapDamaged = "Installer error: the converter data is damaged. Please reinstall WrongGPU.";
const char* const kMsgMapDoesNotFit =
Expand Down
1 change: 1 addition & 0 deletions clean/converter/src/api.h
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ extern const char* const kMsgNotModel;
extern const char* const kMsgNoPresetM;
extern const char* const kMsgNoPresetK;
extern const char* const kMsgNoPresetF;
extern const char* const kMsgNoPresetL;
extern const char* const kMsgUnsupported;
extern const char* const kMsgMapDamaged;
extern const char* const kMsgMapDoesNotFit;
Expand Down
3 changes: 3 additions & 0 deletions clean/converter/src/map.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1378,6 +1378,9 @@ const std::vector<PresetInfo>& known_presets() {
{"M", (1u << 1) | (1u << 4), kMsgNoPresetM}, // spec 08 (version 4 adds the source F)
{"K", (1u << 2) | (1u << 3), kMsgNoPresetK}, // spec 48
{"F", 1u << 5, kMsgNoPresetF}, // spec 57
// Model L (#89): the same record kinds as preset M's map (no new element format, op or source kind), so the
// same map versions; its parameters are a block of their own in the same library file, found by its own hashes
{"L", (1u << 1) | (1u << 4), kMsgNoPresetL},
};
return presets;
}
Expand Down
5 changes: 3 additions & 2 deletions clean/converter/src/map.h
Original file line number Diff line number Diff line change
Expand Up @@ -221,15 +221,16 @@ struct Perm {
// model of that preset. A map names its preset in the optional top-level field "preset"; a map without it gets the preset
// its version had before the field existed (versions 1 and 4: M; 2 and 3: K; 5: F), so the shipped maps load as before.
struct PresetInfo {
const char* name; // "M", "K", "F"
const char* name; // "M", "K", "F", "L"
unsigned versions; // bit v set: map version v is allowed for this preset
const char* no_model_message; // api.cpp kMsgNoPreset*: the DLL has no model of this preset
bool allows(int version) const { return version >= 0 && version < 32 && (versions >> version & 1u) != 0; }
};
// The known presets, in a fixed order; find_preset: by name (exact, upper case), nullptr when unknown.
const std::vector<PresetInfo>& known_presets();
const PresetInfo* find_preset(const std::string& name);
// The preset a map without a "preset" field has: the one its version always meant ("" for no known version).
// The preset a map without a "preset" field has: the one its version always meant ("" for no known version). The first
// row of known_presets that allows the version: M before L, so a map without the field is never preset L.
std::string preset_for_version(int version);

struct Map {
Expand Down
84 changes: 84 additions & 0 deletions clean/converter/tests/cases.py
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,90 @@ def mini_case(seed=3, version=None):
return case


# ---------------------------------------------------------------- two presets in one file (Model L, #89)

def _pair_map(rng, preset, label, w_halves, extra_stage):
"""One preset's synthetic map and sources: W (f16le), D (u16le), K (keyed pair), and for preset L one stage more
(a stand-in for its extra blocks: an fp8 matrix, a bias, a scaled norm scale)."""
w = bytes(random_finite_halves(rng, w_halves, max_exp_field=0x14))
d = bytes(rng.randbytes(2048))
k = struct.pack("<ff", rng.choice([0.5, 0.75, 1.25]), rng.choice([-1.5, -2.0, 3.0]))
key = sha256(label.encode() + w)
m = {
"format": "wronggpu-converter-map",
"version": 1,
"model": "synthetic preset-%s case" % preset,
"channel_order": "natural",
"sources": {
"W": source_entry("W", "fake preset-%s parameter block" % preset, "f16le", w),
"D": source_entry("D", "fake dither table", "u16le", d),
"K": source_entry("K", "fake constants", "u32le", k, locate="pair", gaps=[12], keyed=True, key=key),
},
"key": {"label": label, "from": "W"},
"perms": {},
"stages": [
{"stage": "enc0", "digest": "", "tensors": [
tensor("W", [8, 12], "float32", "W", {"name": "fp8"}, 0, [lin(12), lin(1)]),
tensor("b", [8], "float32", "W", {"name": "f16"}, 96, [lin(1)]),
tensor("T", [4, 16, 16], "uint8", "D", {"name": "u16_shr", "shift": 9}, 0, [lin(256), lin(16), lin(1)]),
tensor("sigma1", [1], "float32", "K", {"name": "u32_as_f32"}, 0, [lin(0)]),
tensor("sigma2", [1], "float32", "K", {"name": "u32_as_f32"}, 1, [lin(0)]),
]},
{"stage": "enc1", "digest": "", "tensors": [
tensor("gamma1", [16], "float32", "W", {"name": "f16_scaled", "mul": SQRT96, "div": SQRT96}, 200, [lin(1)]),
]},
],
}
if extra_stage:
m["stages"].insert(0, {"stage": "inenc_extra", "digest": "", "tensors": [
tensor("W1", [32, 20], "float32", "W", {"name": "fp8"}, 1200, [lin(20), lin(1)]),
tensor("b1", [32], "float32", "W", {"name": "f16"}, 1840, [lin(1)]),
tensor("gamma", [32], "float32", "W", {"name": "f16_scaled", "mul": SQRT8, "div": "0x3f800000"}, 1872, [lin(1)]),
]})
return m, {"W": w, "D": d, "K": k}


def preset_pair_case(seed=31):
"""A synthetic library file with two presets' parameter blocks: preset L's W block directly before preset M's, each
with its own D and K elsewhere, and maps for both (the L map names its preset; the M map has no "preset" field, as
the shipped one). Returns (case_l, case_m, file_l_only, file_m_only); each case's file holds both blocks. Our own
sizes and bytes, nothing taken from a real file."""
rng = random.Random(seed)
ml, sl = _pair_map(rng, "L", "pair label L", 2304, True)
ml["preset"] = "L"
mm, sm = _pair_map(rng, "M", "pair label M", 2048, False)

def layout(with_l, with_m):
buf = bytearray(rng_file.randbytes(96 * 1024))
at = {}
if with_l:
at["L"] = {"W": 1600, "D": 40_000, "K": 60_001}
if with_m:
at["M"] = {"W": 1600 + len(sl["W"]), "D": 50_000 + 6, "K": 70_003} # M's W directly after L's
for p, src in (("L", sl), ("M", sm)):
if p in at:
place(buf, at[p]["W"], src["W"])
place(buf, at[p]["D"], src["D"])
place(buf, at[p]["K"], src["K"][:4])
place(buf, at[p]["K"] + 12, src["K"][4:])
return bytes(buf), at

rng_file = random.Random(seed + 1)
both, at = layout(True, True)
l_only, _ = layout(True, False)
m_only, _ = layout(False, True)
out = []
for name, m, s in (("pair-L", ml, sl), ("pair-M", mm, sm)):
c = Case(name)
c.file = both
c.map = m
c.sources = s
c.expected, c.key = finalize_map(m, s)
c.placements = at[name[-1]]
out.append(c)
return out[0], out[1], l_only, m_only


# ---------------------------------------------------------------- source F (spec 08 sections 4.3, 4.4; map version 4)

def f_elements(rng, n=5):
Expand Down
46 changes: 46 additions & 0 deletions clean/converter/tests/run_tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,51 @@ def test_shipped_map_validates(r, shipped_map):
r.check("real map: nothing written", not os.path.exists(os.path.join(out, "model.wgcm")))


def test_preset_l(r):
"""Model L (#89): a map that names preset L ("preset": "L", map version 1) next to a preset-M map, both reading one
synthetic library file in which L's parameter block lies directly before M's. Each map converts its own block, and
only it; a file without a preset's block fails with that preset's message (exit 12)."""
case_l, case_m, l_only, m_only = cases.preset_pair_case()
lp = r.write("preset-l/map-L.json", case_l.map)
mp = r.write("preset-l/map-M.json", case_m.map)
both = r.write("preset-l/both.dll", case_l.file)
lo = r.write("preset-l/l-only.dll", l_only)
mo = r.write("preset-l/m-only.dll", m_only)
for label, c, p in (("preset L", case_l, lp), ("preset M next to L", case_m, mp)):
out = r.path("preset-l", "out-" + c.name, "x")[:-2]
code, so, se = r.run(p, [both], out)
r.check(label + ": exit 0", code == 0 and SUCCESS in so, so + se[-600:])
with open(p, "rb") as f:
check_output(r, label, c, out, f.read())
r.check(label + ": W found where it was placed", ("source W at offset %d " % c.placements["W"]) in se, se[-800:])
r.check("preset L: its extra stage is in the L output only",
"inenc_extra" in case_l.expected and "inenc_extra" not in case_m.expected and
os.path.isdir(os.path.join(r.work, "preset-l", "out-pair-L", "raw", "inenc_extra")) and
not os.path.exists(os.path.join(r.work, "preset-l", "out-pair-M", "raw", "inenc_extra")))
msg_l = "This DLSS file has no preset-L model. Version 310.9 or newer is needed."
for label, p, f, want in (("preset-L map, a file with only preset M's block", lp, mo, msg_l),
("preset-M map, a file with only preset L's block", mp, lo, MSG["no_preset_m"])):
code, so, se = r.run(p, [f], r.path("preset-l", "out-fail", "x")[:-2])
r.check(label + ": exit 12 with that preset's message", code == EXIT["no_preset_m"] and want in so, "code %d, out %r" % (code, so))
# the preset name decides only the message: the same L map without its field reads as preset M, and converts the same
plain = copy.deepcopy(case_l.map)
del plain["preset"]
pp = r.write("preset-l/map-L-no-field.json", plain)
code, so, se = r.run(pp, [mo], r.path("preset-l", "out-fail2", "x")[:-2])
r.check("the L map without its \"preset\" field: read as preset M (its message)", code == 12 and MSG["no_preset_m"] in so, so)
out = r.path("preset-l", "out-no-field", "x")[:-2]
code, so, se = r.run(pp, [both], out)
r.check("...and it converts the same tensors", code == 0, so + se[-400:])
with open(pp, "rb") as f:
check_output(r, "L map without the field", case_l, out, f.read())
# a preset name that does not fit the map version, or is not known: converter data damaged, before any file is read
for bad in ("K", "F", "X", "l"):
m2 = copy.deepcopy(case_l.map)
m2["preset"] = bad
code, so, se = r.run(r.write("preset-l/bad-%s.json" % bad, m2), [both], r.path("preset-l", "out-bad", "x")[:-2])
r.check("\"preset\": %r in a version-1 map: converter data damaged" % bad, code == EXIT["map_damaged"] and MSG["map_damaged"] in so, so)


def mutate(m, fn):
m2 = copy.deepcopy(m)
fn(m2)
Expand Down Expand Up @@ -1013,6 +1058,7 @@ def main():
r = Runner(os.path.abspath(args.exe), os.path.abspath(args.work))
t0 = time.time()
test_errors(r)
test_preset_l(r)
test_filter(r, args.t7)
test_k_errors(r)
if os.path.exists(args.map):
Expand Down
13 changes: 11 additions & 2 deletions clean/converter/tests/unit_tests.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -600,7 +600,11 @@ static void test_api_files(const fs::path& work) {
CHECK(preset_for_version(1) == "M" && preset_for_version(4) == "M" && preset_for_version(2) == "K" && preset_for_version(3) == "K" &&
preset_for_version(5) == "F" && preset_for_version(0).empty() && preset_for_version(6).empty());
CHECK(find_preset("M") && find_preset("K") && find_preset("F") && !find_preset("m") && !find_preset("") && !find_preset("X") &&
known_presets().size() == 3);
known_presets().size() == 4);
// preset L (#89): the map versions of preset M, its own message; never the preset of a map without the field
CHECK(find_preset("L") && find_preset("L")->allows(1) && find_preset("L")->allows(4) && !find_preset("L")->allows(2) &&
!find_preset("L")->allows(3) && !find_preset("L")->allows(5) && std::string(find_preset("L")->no_model_message) == kMsgNoPresetL &&
std::string(kMsgNoPresetL).find("preset-L") != std::string::npos);
CHECK(std::string(find_preset("M")->no_model_message) == kMsgNoPresetM && std::string(find_preset("K")->no_model_message) == kMsgNoPresetK &&
std::string(find_preset("F")->no_model_message) == kMsgNoPresetF);
{
Expand All @@ -617,7 +621,12 @@ static void test_api_files(const fs::path& work) {
CHECK(load_map_bytes(with_preset(u.map_v4, "4", "\"M\""), pm4, sink).code == kOk && pm4.map && pm4.map->preset_info() == find_preset("M"));
// the same map bytes with and without "preset":"M" describe the same tensors (only the file hash differs)
CHECK(pm4.map && pm4.map->stages.size() == m4.map->stages.size() && pm4.map->sources.size() == m4.map->sources.size());
for (const char* bad : {"\"K\"", "\"F\"", "\"X\"", "\"m\"", "\"\"", "7", "null"}) {
LoadedMap lm1, lm4;
CHECK(load_map_bytes(with_preset(u.map_v1, "1", "\"L\""), lm1, sink).code == kOk && lm1.map && lm1.map->preset_name == "L" &&
lm1.map->preset() == 'L' && lm1.map->preset_info() == find_preset("L"));
CHECK(load_map_bytes(with_preset(u.map_v4, "4", "\"L\""), lm4, sink).code == kOk && lm4.map && lm4.map->preset_name == "L" &&
lm4.map->stages.size() == m4.map->stages.size());
for (const char* bad : {"\"K\"", "\"F\"", "\"X\"", "\"m\"", "\"l\"", "\"\"", "7", "null"}) {
LoadedMap bm;
const Result r = load_map_bytes(with_preset(u.map_v1, "1", bad), bm, sink);
if (r.code != kMapDamaged) std::printf("preset field %s\n", bad);
Expand Down
Loading