Repository navigation
Runtime build: built-in files as string literals (lower compile memory) - #235
Merged
Merged
Conversation
…lists write_embedded wrote every built-in file (all code objects, each also under its gfx1201 name, and every DXIL of dx12/) as a brace list of decimal bytes: tens of millions of integer elements in one generated embedded_files.cpp, one compiler node each. Compiling that one file was the memory peak of the runtime build. Each file is now one string literal in adjacent 64-byte pieces (3-digit octal escapes, printable bytes as themselves): the same bytes in the library, the same table line, a smaller file and far less compiler memory. A runtime built by MSVC would hit its 64 KB string-literal limit: the generated file says so with #error (clang++ and g++ build the runtime). clean/tests/test_embedded_table.py: the runtime's own lookup (wg_read_file, through clean/tests/embed_dump.cpp) walks the compiled table; every file's SHA-256 equals its source file's (synthetic files with every byte value, an empty file, quotes / backslashes / trigraph text; a mutant with one changed byte is caught), and a --build mode for the table of a real build folder. Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The runtime library's built-in files (
clean/runner/build.pywrite_embedded->embedded_files.cpp) were written as brace lists of decimal bytes: one compiler node per byte. Compiling that single generated file was the memory peak of every runtime build. They are now string literals in 64-byte pieces (3-digit octal escapes, printable bytes as themselves). The library holds the same bytes, and the table line keeps its exact format, whichtest_no_model_k.pyparses.Measured on one development build folder (Model L runtime, 133 built-in files):
embedded_files.cpp: 95 MB -> 78 MB.--runtime-only), peak system memory on a 32 GB machine with other work running:Byte proof:
clean/tests/test_embedded_table.pycompiles the runtime's own lookup (embedded.cpp) with the generated table and a small harness (clean/tests/embed_dump.cpp). It reads every entry back throughwg_read_fileand compares its SHA-256 with the source file.??sequences, plus a mutant with one changed byte, which the test catches. 2 tests, OK.--buildon the real folder above: 133 files, 26,122,717 bytes, 0 mismatches.MSVC would hit its 64 KB string-literal limit, so the generated file stops with
#errorwhen a file is larger than that. Only clang++ and g++ build the runtime (find_host_cxx).