Skip to content

Converter search: ignore lists for upscaler DLLs another installer placed - #46

Merged
phantoslayer21 merged 5 commits into
mainfrom
21-convert-ignore
Nov 14, 2026
Merged

phantoslayer21 merged 5 commits into
mainfrom
21-convert-ignore

Conversation

@phantoslayer21

@phantoslayer21 phantoslayer21 commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

When another program's installer has put its own copies of the upscaler's DLLs into a game or library folder, the
converter's automatic search could pick them up. This adds ignore lists to every search for a file to convert (model
data and Ray Reconstruction data; the window, install, model convert, rr-data convert).

  • Lists read: %LOCALAPPDATA%\WrongGPU\ignore-files.txt (the one an installer writes to), ignore-files.txt in
    the launcher's own data folder (portable / --data=), and on the two convert verbs --ignore-list=<file> and
    --ignore=<entry>.
  • Format (version 1): a full path, or a full path followed by sha256=<hex>; # comment lines. Documented in
    clean/installer/README.md, "Ignore lists".
    • A path matches the same file (case, slashes, . and .., links and junctions resolved, hard links by file identity).
    • A path + hash entry matches only when both match: that file, at that path, with that SHA-256. An identical copy
      at another path (the player's own game) is still converted, and so is the same path holding other bytes after an
      update.
    • A hash alone is not a list entry. In a list it is skipped with a warning, because it would skip every
      identical copy, the player's own too. Only --ignore=sha256=<hex> on the command line takes one, and the README
      says that it matches every identical copy.
    • Bad lines, and a path that is a folder, are skipped with a warning naming the list and line.
  • An empty list, or none, never changes which file is chosen (the candidates, their order and the winner).
    Every conversion now also writes a sha256<TAB><hex> line to model-source.txt, with or without a list. That is one
    more line in that file; the converted data is unchanged.
  • A skipped file is named with its reason in the log, on the console and in the JSON result. A chosen file
    (--model=, --dll=, the window) on a list is not converted either; the player is told which list and line.
  • Fails closed: a list that is there but can't be read (after short retries on a sharing violation), or can't be
    decoded (UTF-16 big-endian, UTF-16 without a BOM, not UTF-8), stops the conversion with exit 10. So does a
    well-known list whose attributes can't be read for a reason other than "not found". A candidate that can't be hashed
    while a hash entry applies to it is left out and named.
  • A conversion is reused only while its recorded source is not on a list (the same both-must-match rule for a path +
    hash entry), so data converted from a legitimate identical copy is reused. The recorded source is compared as it was
    recorded (path text and SHA-256); the recorded path is never opened again.
  • remove-data, the launcher's uninstall and the last game's uninstall keep ignore-files.txt and say so. A
    player's list is never deleted:
    if another program made the list again meanwhile, it is kept beside it as
    ignore-files.txt.kept-<n>; if it can't be moved or copied out at all, it stays in the moved-aside folder, which is
    then never deleted as a whole. The result the player sees says where it is.
  • --ignore / --ignore-list on a verb that does not apply them stop it with exit 10 and a message. A bare
    --ignore (no =) is an error and never takes the next argument.
  • CI builds and runs the new convert-ignore ctest.

Also in this pull request, beyond the ignore lists:

  • --library=<folder> on the two convert verbs: that folder's sub-folders are searched as games, for this run only.
  • rr-data prepare (WRONGGPU_TESTING builds only): the install's RR data step, so the tests can check RR reuse.
  • A re-conversion that replaces listed model data no longer requires the output-filter weights: the first allowed
    file that converts is used, with or without them.

Fixes #21

Review findings (independent review, 2026-11-13)

Blocking

  • Path + hash entries matched by path OR hash; RecordedSourceReason matched the recorded hash alone. Fixed in
    src/wg_core/convert_ignore.cpp / .h: one rule (EntryReason) for candidates and recorded sources. A path + hash
    entry matches only when both match. A hash alone is refused in a list (warned) and taken only from
    --ignore=sha256= (src/app/cli.cpp). The README warns that such a hash matches every identical copy and no longer
    shows a hash-only line in its example. Tests: two identical DLLs, only one listed by path + hash, the other is still
    the candidate tried (tests/convert_ignore_tests.cpp, through the real candidate search) and still converts
    (tests/run_convert_ignore_tests.py, for model data and RR data). Model and RR data converted from a legitimate
    identical copy are reused.

Should fix

  • The CI test for "an empty list changes nothing" could not fail. Fixed: the candidate step of FindModel is now
    AllowedModelCandidates (src/wg_core/model.cpp). tests/convert_ignore_tests.cpp drives it with no list, with
    empty lists in each place one is read (%LOCALAPPDATA%, the launcher's data folder, both, --ignore-list=), and
    with a list that does change the result, so the test can fail. The conversion itself needs the embedded map, which
    the CI build doesn't have. run_convert_ignore_tests.py covers that part and was run (results below).
  • Every conversion now writes a sha256 line to model-source.txt. Said above and in the README: "changes nothing"
    means the choice of file.
  • A folder entry parsed cleanly and ignored nothing. Fixed in convert_ignore.cpp (ParseIgnoreLine): it is
    warned about, with its list and line, like any bad line.
  • PutIgnoreListBack could lose the list silently. Fixed in convert_ignore.cpp: when both the move and the copy
    fail, the log says the list could not be put back, with both errors. (Superseded in round 3, item 1 below: the list
    is now never lost.)
  • StartCleanup change: split out (follow-up). The new folder naming, the move retries, their test seam and the
    200-iteration stress test are removed from this branch. StartCleanup (src/wg_core/record.cpp) is the code on
    main plus the ignore-list put-back. Not changed, because it is this pull request's own "uninstalls keep the list"
    feature: %LOCALAPPDATA%\WrongGPU is created again only when an ignore-files.txt was there, to hold it. With no
    list, the folder goes as before (unit test "no list: nothing kept, the folder goes").

Notes

  • Fail-open cases: fixed, they now fail closed (convert_ignore.cpp). A candidate that can't be hashed while a hash
    entry applies is left out. A list in UTF-16 big-endian, UTF-16 without a BOM or not UTF-8 is unreadable (exit 10). So
    is a well-known list whose attributes can't be read for a reason other than "not found".
  • --ignore / --ignore-list accepted on every verb: fixed in cli.cpp. On a verb that doesn't apply them they
    stop it with exit 10 and a message.
  • A bare --ignore ate the next argument: fixed in cli.cpp. A bare --ignore, and now also a bare
    --ignore-list, is a usage error (exit 10) and never takes the next argument. Testing found that a bare
    --ignore-list still took the next option as its file; that is fixed in the last commit, with two new checks.
  • README "UNC was tested through \\localhost": reworded to what the tests cover: a UNC entry through the PC's own
    administrative share, where reachable (the tests say so and skip otherwise). In the runs below these UNC checks ran;
    none were skipped. Shares on other machines are not covered.
  • "the spec team's synthetic preset-F file": reworded to "the synthetic preset-F file" in
    tests/run_convert_ignore_tests.py.
  • Extras not in the stated purpose: listed above: --library=, the test-only rr-data prepare, and output-filter
    weights no longer required on re-conversion.

Second independent review (2026-11-13), round 3

Must fix

  1. A failed list restore was followed by deletion of the only saved list. Fixed in
    src/wg_core/convert_ignore.cpp / .h (PutIgnoreListBack now returns ListKept), src/wg_core/launcher.cpp
    (RemoveSharedData, SelfUninstall), src/wg_core/record.cpp (StartCleanup) and src/app/ui_words.cpp (the
    Remove data window). The list is never deleted or scheduled for deletion:
    • put back in place when it can be (as before);
    • when another program made ignore-files.txt again meanwhile (or the move and copy fail for another reason), the
      player's list is moved, or copied, to ignore-files.txt.kept-<n> (first free n) beside it, never over it;
    • when it can't leave the moved-aside folder at all, it stays there: the rest of that folder is removed, nothing
      deletes that folder's tree (no rmdir /s in StartCleanup's hidden cleanup, no recursive DeleteWhenFree in
      the launcher; what is still held is deleted file by file, never the list);
    • a list that is a link (a reparse point) is no longer skipped: it is moved as the link (a copy copies the link,
      never its target);
    • each case is said in the result the player sees (kept of remove-data, the uninstall's output, a game
      uninstall's warnings, the window's message) and logged.
      Tests (both cases, "made again meanwhile" and "can't be put back", on each cleanup path): tests/convert_ignore_tests.cpp
      runs StartCleanup, RemoveSharedData and SelfUninstall through a test seam (SetPutBackHookForTest) that acts
      between the folder's move and the put-back; the last two against a private registry hive of the test's own
      (RegLoadAppKey), so no installed WrongGPU is seen or changed. Also PutIgnoreListBack alone (.kept-1, .kept-2,
      left in the moved folder, a list that is a link). tests/run_convert_ignore_tests.py [8] runs remove-data and
      self-uninstall through the command line with the same two cases (WRONGGPU_TEST_LIST_PUTBACK, test builds only).
  2. A recorded source was compared by its present-day file identity. Fixed in src/wg_core/convert_ignore.cpp
    (RecordedSourceReason, PathLook): the recorded path is never opened. Its text, normalised (case, / or \,
    ., .., a \\?\ prefix), is compared with the entry's path (as written and as it resolves now), and the stored
    SHA-256 with the entry's. model.cpp and rr.cpp call it unchanged. Old records without a SHA-256: the path text
    alone decides, so a path + hash entry naming the recorded path lists it (its bytes can't be checked); not listed,
    they stay "unknown" as before. Tests: the source path made a hard link and a symbolic link to a listed file with the
    same bytes after the conversion (path entry and path + hash entry): tests/convert_ignore_tests.cpp (not listed,
    while a candidate at that path today is skipped) and tests/run_convert_ignore_tests.py [4] and [5] (the model data
    and the RR data are still reused).

Can follow (done here)
3. --ignore-list read the whole file before the size check. Fixed in src/app/cli.cpp: each --ignore-list
file is read once, by the lists' own bounded reader (ReadIgnoreListBytes, now public: the size is checked on the
open handle before anything is read, 16 MB or more is refused, a non-disk file too); the search parses those bytes
(Context::ignoreListBytes) and does not read the file again. Test: a 16 MB --ignore-list file, exit 10 with the
size named (run_convert_ignore_tests.py [2]).
4. Automatic-search failure pointed the player to the log rather than the list line. Fixed in
src/wg_core/model.cpp, src/wg_core/rr.cpp and convert_ignore.cpp (IgnoredExample): when every file found is
on a list (and in the "none converted" message when some were skipped), the message itself names the first file
skipped with its list and line (or --ignore= argument), how many others were skipped and the log's path, and says
to remove that line if it names a file of the player's own game. An unreadable or undecodable list: the message names
the file and says what to do (close the program that holds it, or fix the file or delete it). Tests:
run_convert_ignore_tests.py [2] (model, RR) and [6] (a list held open, a list in UTF-16 without a BOM).

Tested

Round 3 (this head, 3fd8fc7), run locally on Windows, CPU only, Ninja Release /MT with MSVC:

  • CI "launcher" step (TESTING off: ctest -R "^(wg-core|launcher-core|coexist-core|convert-ignore|source-size|wording)$"):
    6/6 passed (wg-core and launcher-core are the launcher's CPU tests that cover the changed cleanup code).
  • convert-ignore-tests.exe run alone: 219 checks passed (192 at the previous head). The retarget checks ran through
    a hard link and a symbolic link; the RemoveSharedData / SelfUninstall cases ran against the private hive (none
    skipped).
  • WRONGGPU_TESTING launcher build with the installer suite's test map, full ctest: 11/11 passed.
  • run_convert_ignore_tests.py --synth-f ...: 155 passed, 0 failed, 0 skipped (128 at the previous head). That
    includes the new [8] (both put-back cases for remove-data and self-uninstall) and the hard- and symbolic-link
    retarget cases for the model and RR data. A first run had 2 failures in the new message checks (they expected the
    --log= file; the message correctly names the log the skip lines go to); the checks were corrected to read the log
    named in the message.
  • CI converter step (clean/converter, unchanged): 4/4 passed.
  • python tools/ci/check_line_endings.py: 1145 files checked, 0 problems.

Earlier rounds: "Before" is the earlier head of this pull request (62110c4); "after" is the round-2 head.

  • CI "launcher" step (TESTING off: ctest -R "^(wg-core|launcher-core|coexist-core|convert-ignore|source-size|wording)$"):
    before 6/6 passed; after 6/6 passed.
  • WRONGGPU_TESTING launcher build with the installer suite's test map, full ctest: before 11/11 passed; after
    11/11 passed.
  • convert-ignore-tests.exe run alone (same build): before 94 checks passed; after 192 checks passed.
  • python clean/installer/tests/run_convert_ignore_tests.py --launcher <that build>\WrongGPU.exe --work <dir> --synth-f <folder of synth-common-v0.bin>:
    before 104 passed, 0 failed, 0 skipped. On the intermediate head: 126 passed, 1 failed (the bare --ignore-list
    above). After: 128 passed, 0 failed, 0 skipped.
  • CI converter step (ctest of clean/converter, unchanged by this pull request): after 4/4 passed.
  • python tools/ci/check_line_endings.py: after, 1145 files checked, 0 problems.

Not tested

  • tests/run_launcher_tests.py (which runs tests/launcher_ignore_cases.py) and tests/run_installer_tests.py: they
    install into generated games. Not run in round 3 either. The game uninstall's cleanup (StartCleanup) is covered by the
    unit test in both new cases, and remove-data / self-uninstall by the unit test and run_convert_ignore_tests.py.
  • The Remove data window's message for a list not put back as it was (ui_words.cpp): built, not shown on screen.
  • A real race with another installer (the tests make the folder again through a seam at the exact point).
  • UNC matching against a share on another machine.
  • No game was run.

Rebased on main 234515a (conflict in clean/installer/src/wg_core/record.cpp and record.h, resolved by keeping main's StartCleanup from #53 and adding this pull request's ignore-list put-back to it, as on the earlier head; the rebase also restores main's #include <atomic> and cleanup test seam in record.cpp); tests re-run: CI converter ctest 4/4, CI launcher step 6/6 (with convert-ignore), WRONGGPU_TESTING build full ctest 11/11, run_convert_ignore_tests.py 107 passed, 0 failed, 2 skipped (the rr-data and RR reuse groups: run without --synth-f).

…aced

Every search for a file to convert (model data and Ray Reconstruction data; the
window, install, model convert, rr-data convert) now skips files named in an
ignore list: %LOCALAPPDATA%\WrongGPU\ignore-files.txt (the one an installer
writes to), ignore-files.txt in the launcher's own data folder, and the
--ignore-list=<file> / --ignore=<entry> options. An entry is a full path, a path
with its SHA-256, or a SHA-256 alone; paths match the same file (case, slashes,
. and .., links and junctions, hard links). An empty list or none never changes
which file wins.

- A skipped file is named with its reason in the log, the console and the JSON
  result; a chosen file on a list is not converted either, and the player is
  told which list and line.
- A list that exists but can't be read fails closed (exit 10) after short
  retries on a sharing violation.
- A conversion is reused only while its recorded source (path and, since this
  change, SHA-256 in model-source.txt) is not on a list.
- The uninstalls keep ignore-files.txt and say so.
- The shared-folder cleanup picks a target name no earlier cleanup uses and
  retries a sharing violation or access denied a bounded number of times.

The format is documented in clean/installer/README.md ("Ignore lists").
Tests: convert-ignore-tests (ctest convert-ignore, now in CI);
run_convert_ignore_tests.py and launcher_ignore_cases.py drive the launcher.

Fixes #21

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
…w fixes

A path + hash entry now matches a candidate only when it is the same file at
that path AND has that SHA-256. A copy with the same bytes under another path
(the player's own game, next to a vendor DLL another installer placed
unmodified) is no longer skipped, and a model converted from such a legitimate
copy is no longer listed (its repair reuses the data again). A hash alone is not
a list entry any more (warned and skipped); only --ignore=sha256=<hex> on the
command line takes one, and the README says it matches every copy with that
hash. The recorded-source check (RecordedSourceReason) follows the same rule.

Also:
- the CI test of "an empty list changes nothing" now drives the real candidate
  search (AllowedModelCandidates, split out of FindModel) with no list and with
  an empty list in each well-known place and --ignore-list=, and also checks a
  list that does change it; the README says "changes nothing" means the choice
  of file (model-source.txt gains its sha256 line with or without a list)
- a folder entry is warned about like any bad line
- PutIgnoreListBack logs when neither the move nor the copy could put the list back
- the converter-search change to StartCleanup (folder naming, move retries and
  their tests) is left out of this branch; only the ignore list put-back stays
- --ignore / --ignore-list on a verb that does not apply them exit 10 with a
  message; a bare --ignore (no "=") is an error and takes no next argument
- fail closed: a candidate whose SHA-256 can't be computed while a hash entry
  applies is left out; a list in UTF-16 big-endian, UTF-16 without a BOM or not
  UTF-8 is an unreadable list; GetFileAttributesW failing on a well-known list
  for a reason other than "not found" is an unreadable list (exit 10)
- wording: README UNC paragraph says what the tests cover; test header wording

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
The command-line usage comment in cli.cpp named only --ignore=<path> and
--ignore=sha256=<hex>. It now also names --ignore=<path> sha256=<hex>, which
matches only when both match, and says that a hash alone matches every
identical copy, as the README does.

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
…ument

--ignore-list without "=" still read the next argument as its file, also
when that argument was another option (--noscan became the list's name). It
now needs its "=" as --ignore does: a bare --ignore-list is the usage error
"--ignore-list needs its file" (exit 10), whatever follows it. The README
says so for both options. run_convert_ignore_tests.py checks a bare
--ignore-list followed by an option and followed by a file.

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
…e matched as recorded

Second review, round 3:
- PutIgnoreListBack returns ListKept. When another program made ignore-files.txt
  again meanwhile, the player's list is kept beside it as ignore-files.txt.kept-<n>;
  when it can't leave the moved-aside folder, it stays there and no cleanup deletes
  that folder's tree (StartCleanup, RemoveSharedData, SelfUninstall). A list that is
  a link is moved as the link. The player's result says where it is.
- RecordedSourceReason compares the recorded path text (normalised) and the stored
  SHA-256 with the entry; the recorded path is never opened again. An old record
  without a SHA-256: the path text alone decides.
- --ignore-list is read once by the lists' bounded reader (ReadIgnoreListBytes);
  the search parses those bytes.
- When every file found is ignored, the message names the first file with its list
  and line and points to the log; an unreadable list's message says fix or delete it.
Tests on each cleanup path (unit tests with a seam and a private registry hive;
command-line suite with WRONGGPU_TEST_LIST_PUTBACK), retargeted source cases for
model and RR data, the 16 MB --ignore-list case.

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
j0nsh1n pushed a commit to j0nsh1n/wronggpu that referenced this pull request Nov 13, 2026
…eanup; bounded retries

StartCleanup moves %LOCALAPPDATA%\WrongGPU aside and lets a hidden cmd.exe delete
it. The target name was <folder>.cleanup-<pid>-<GetTickCount64()>; that clock
moves in steps of about 16 ms, so a second cleanup of the same folder within one
step (an install's restore followed by the last game's uninstall, or Remove data
right after an uninstall) aimed at the first cleanup's folder, which is deleted
only seconds later. The move failed with error 183 and the folder was left in
place ("cleanup deferred").

- The target name adds a process-wide sequence number and is checked free first.
- A sharing violation (32) or an access denied (5: how a folder rename fails while
  another program, such as a scanner, has a just-written file in it open) is tried
  again, at most 5 times with waits from 50 ms doubling, each wait logged with the
  error. Any other error is final; the deferred line now names the error.
- Test seam SetCleanupMoveForTest (record.h).

Tests (wg-core, every build): two cleanups of one folder back to back, 20 runs;
error 5 for 3 tries then success; error 5 every time (5 tries, folder left,
deferred line names the error); error 32 twice then success; error 183 once
(deferred at once).

Split out of phantoslayer21#46 at review.

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
@phantoslayer21
phantoslayer21 merged commit 59f66b6 into main Nov 14, 2026
2 checks passed
@phantoslayer21
phantoslayer21 deleted the 21-convert-ignore branch November 14, 2026 00:41
phantoslayer21 added a commit that referenced this pull request Nov 14, 2026
Conflicts in record.h / record.cpp with the ignore-list change (#46):
StartCleanup now returns the line about a kept ignore-files.txt, and the
restore reports it. The restore keeps this branch's order: the record is
removed before the backup, and there is no kept-backup path (originals
that cannot go back are kept in WrongGPU-originals).

Found by the installer suite's direct-route cases ([1] to [11], [14]),
run with a direct-route build:
- An update or setup over an earlier install starts the new wronggpu.ini
  from the earlier one when that was WrongGPU's (its recorded bytes),
  read before the restore; the restore deals with the path by the rule,
  so the player's choices stay and an original from before WrongGPU is
  backed up again. The move of an earlier direct install from another
  folder carries that file to the exe folder the same way.
- The game root above the install folder (where the game's working
  folder is and Collect logs looks) gets a runtime line of its own
  ("<game-root>", resolved from the record's exe): WrongGPU's wronggpu-*
  logs there go with the uninstall; other files there stay.
- Suite expectations follow the rule: a hand-written wronggpu.ini stays
  after the uninstall; a game update's FSR DLL stays and the earlier one
  is kept in WrongGPU-originals (two generations both kept); a file held
  without delete sharing refuses the setup before anything changes
  (exit 2); OptiScaler's shared logs stay; an older version's record
  carries the hash of the wronggpu.ini it wrote.

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Converter: an ignore list for upscaler DLLs that another installer placed

1 participant