Use UTF-8 for filesystem paths

AGENTS.md

@@ -12,14 +12,15 @@ Forge is a Lua-scriptable build tool. The C++ core builds a dependency graph of
 - `src/forge/lua/forge` contains the standard Lua API/rules; `src/forge/lua/forge/cc` contains C/C++ toolchain rules.
 - `src/forge/forge_test` contains Forge’s UnitTest++ test suite and Lua test fixtures.
 - `src/assert`, `src/cmdline`, `src/error`, `src/luaxx`, and `src/process` are small local support libraries.
-- `src/boost`, `src/lua`, and `src/unittest-cpp` are vendored/submodule code; avoid changing them unless explicitly needed.
+- `src/lua` and `src/unittest-cpp` are submodule code; avoid changing them unless explicitly needed.
 
 ## Docs map
 
 - `docs/getting-started/running-forge.md` explains CLI commands and variables.
 - `docs/getting-started/configuring-forge.md` explains root `forge.lua` build scripts and toolsets.
 - `docs/getting-started/writing-buildfiles.md` explains `*.forge` buildfiles.
 - `docs/more-details/rules.md`, `dependencies.md`, and `outdated-calculation.md` explain the core build model.
+- `docs/more-details/character-encoding.md` describes Forge's UTF-8 contract for strings, paths, and the graph cache.
 - `docs/reference/` documents Lua API functions, `Target`, and `Toolset`.
 - `docs/cc-module/cc.md` documents the C/C++ Lua rules.
 
@@ -32,8 +33,9 @@ Forge is a Lua-scriptable build tool. The C++ core builds a dependency graph of
 
 ## Conventions and guardrails
 
-- C++ is generally C++11; MSVC builds use C++14. Warnings are treated as errors.
-- Follow surrounding C++ style and existing `sweet::forge` namespaces. Use Boost.Filesystem rather than introducing `std::filesystem`.
+- C++ is generally C++17. Warnings are treated as errors.
+- Follow surrounding C++ style and existing `sweet::forge` namespaces.
 - Lua/buildfiles use 4-space indentation, semicolon-separated table entries, and portable `/` paths. Convert to native paths only when passing paths to external tools.
+- All strings inside Forge — target ids, paths, branches, filenames, env values — are UTF-8 byte sequences. Strings from Lua are trusted as UTF-8 (no validation). No Unicode normalization is performed; strings compare byte-for-byte. Windows uses a UTF-8 ACP manifest at the OS boundary. See `docs/more-details/character-encoding.md` for the full contract.
 - Do not edit generated or ignored outputs such as `bootstrap/`, `debug/`, `release/`, `shipping/`, `.forge`, `*.o`, or `*.obj`.
 - When changing behavior, check whether corresponding updates are needed in the C++ core, Lua bindings/rules, docs/reference, and `src/forge/forge_test`.

bootstrap-linux.bash

@@ -10,7 +10,7 @@ cc() {
     for file in $1; do
         echo $file...
         local DEFINES="-DBUILD_VARIANT_DEBUG -DBUILD_VERSION=\"bootstrap\" -DLUA_USE_POSIX -DLUA_USE_DLOPEN"
-        local INCLUDE_DIRS="-I $SRC -I $SRC/lua/src -I $SRC/boost"
+        local INCLUDE_DIRS="-I $SRC -I $SRC/lua/src"
         local FLAGS="-x c -g"
         $CC $DEFINES $INCLUDE_DIRS $FLAGS -o $file.o -c $file
     done
@@ -20,27 +20,25 @@ cxx() {
     for file in $1; do
         echo $file...
         local DEFINES="-DBUILD_VARIANT_DEBUG -DBUILD_VERSION=\"bootstrap\" -DLUA_USE_POSIX -DLUA_USE_DLOPEN"
-        local INCLUDE_DIRS="-I $SRC -I $SRC/lua/src -I $SRC/boost"
-        local FLAGS="-x c++ -std=c++11 -fexceptions -frtti -fPIC -march=native -g -Wno-deprecated-declarations"
+        local INCLUDE_DIRS="-I $SRC -I $SRC/lua/src"
+        local FLAGS="-x c++ -std=c++17 -fexceptions -frtti -fPIC -march=native -g -Wno-deprecated-declarations"
         $CXX $DEFINES $INCLUDE_DIRS $FLAGS -o $file.o -c $file
     done
 }
 
 archive() {
-    $AR -rcs $1 *.o    
+    $AR -rcs $1 *.o
 }
 
 link_forge() {
-    $CXX *.o -g -L $LIB -lforge -lforge_lua -lprocess -lcmdline -lluaxx -lerror -llua -lassert -lboost_filesystem -lboost_system -lpthread -ldl -o $BIN/forge
+    $CXX *.o -g -L $LIB -lforge -lforge_lua -lprocess -lcmdline -lluaxx -lerror -llua -lassert -lpthread -ldl -o $BIN/forge
 }
 
 link_forge_hooks() {
     $CXX *.o -shared -g -o $BIN/libforge_hooks.so
 }
 
 mkdir -p $LIB
-echo boost_system; pushd $SRC/boost/libs/system/src; cxx "*.cpp"; archive $LIB/libboost_system.a; popd
-echo boost_filesystem; pushd $SRC/boost/libs/filesystem/src; cxx "*.cpp"; archive $LIB/libboost_filesystem.a; popd
 echo lua; pushd $SRC/lua/src; cc '*.c'; archive $LIB/liblua.a; popd
 echo assert; pushd $SRC/assert; cxx '*.cpp'; archive $LIB/libassert.a; popd
 echo forge; pushd $SRC/forge; cxx '*.cpp'; archive $LIB/libforge.a; popd

bootstrap-macos.bash

@@ -10,7 +10,7 @@ cc() {
     for file in $1; do
         echo $file...
         local DEFINES="-DBUILD_VARIANT_DEBUG -DBUILD_VERSION=\"bootstrap\" -DLUA_USE_POSIX -DLUA_USE_DLOPEN"
-        local INCLUDE_DIRS="-I $SRC -I $SRC/lua/src -I $SRC/boost"
+        local INCLUDE_DIRS="-I $SRC -I $SRC/lua/src"
         local FLAGS="-x c -g"
         $CC $DEFINES $INCLUDE_DIRS $FLAGS -o $file.o -c $file
     done
@@ -20,27 +20,25 @@ cxx() {
     for file in $1; do
         echo $file...
         local DEFINES="-DBUILD_VARIANT_DEBUG -DBUILD_VERSION=\"bootstrap\" -DLUA_USE_POSIX -DLUA_USE_DLOPEN"
-        local INCLUDE_DIRS="-I $SRC -I $SRC/lua/src -I $SRC/boost"
-        local FLAGS="-x c++ -std=c++11 -stdlib=libc++ -fexceptions -frtti -g"
+        local INCLUDE_DIRS="-I $SRC -I $SRC/lua/src"
+        local FLAGS="-x c++ -std=c++17 -stdlib=libc++ -fexceptions -frtti -g"
         $CXX $DEFINES $INCLUDE_DIRS $FLAGS -o $file.o -c $file
     done
 }
 
 archive() {
-    $AR -rcs $1 *.o    
+    $AR -rcs $1 *.o
 }
 
 link_forge() {
-    $CXX *.o -g -L $LIB -lassert -lcmdline -lforge -lforge_lua -lprocess -lluaxx -lerror -llua -lboost_filesystem -lboost_system -o $BIN/forge
+    $CXX *.o -g -L $LIB -lassert -lcmdline -lforge -lforge_lua -lprocess -lluaxx -lerror -llua -o $BIN/forge
 }
 
 link_forge_hooks() {
     $CXX *.o -Xlinker -dylib -g -o $BIN/forge_hooks.dylib
 }
 
 mkdir -p $LIB
-echo boost_system; pushd $SRC/boost/libs/system/src; cxx "*.cpp"; archive $LIB/libboost_system.a; popd
-echo boost_filesystem; pushd $SRC/boost/libs/filesystem/src; cxx "*.cpp"; archive $LIB/libboost_filesystem.a; popd
 echo lua; pushd $SRC/lua/src; cc '*.c'; archive $LIB/liblua.a; popd
 echo assert; pushd $SRC/assert; cxx '*.cpp'; archive $LIB/libassert.a; popd
 echo forge; pushd $SRC/forge; cxx '*.cpp'; archive $LIB/libforge.a; popd

bootstrap-mingw.bash

@@ -9,8 +9,8 @@ CXX=${CXX:-g++}
 cc() {
     for file in $1; do
         echo $file...
-        local DEFINES="-DBUILD_VARIANT_DEBUG -DBUILD_VERSION=\"bootstrap\" -DBOOST_ALL_NO_LIB -D_WIN32_WINNT=0x0a00"
-        local INCLUDE_DIRS="-I $SRC -I $SRC/lua/src -I $SRC/boost"
+        local DEFINES="-DBUILD_VARIANT_DEBUG -DBUILD_VERSION=\"bootstrap\" -D_WIN32_WINNT=0x0a00"
+        local INCLUDE_DIRS="-I $SRC -I $SRC/lua/src"
         local FLAGS="-x c -g"
         $CC $DEFINES $INCLUDE_DIRS $FLAGS -o $file.o -c $file
     done
@@ -19,28 +19,26 @@ cc() {
 cxx() {
     for file in $1; do
         echo $file...
-        local DEFINES="-DBUILD_VARIANT_DEBUG -DBUILD_VERSION=\"bootstrap\" -DBOOST_ALL_NO_LIB -D_WIN32_WINNT=0x0a00"
-        local INCLUDE_DIRS="-I $SRC -I $SRC/lua/src -I $SRC/boost"
-        local FLAGS="-x c++ -std=c++11 -fexceptions -frtti -fPIC -march=native -g -Wno-deprecated-declarations"
+        local DEFINES="-DBUILD_VARIANT_DEBUG -DBUILD_VERSION=\"bootstrap\" -D_WIN32_WINNT=0x0a00"
+        local INCLUDE_DIRS="-I $SRC -I $SRC/lua/src"
+        local FLAGS="-x c++ -std=c++17 -fexceptions -frtti -fPIC -march=native -g -Wno-deprecated-declarations"
         $CXX $DEFINES $INCLUDE_DIRS $FLAGS -o $file.o -c $file
     done
 }
 
 archive() {
-    $AR -rcs $1 *.o    
+    $AR -rcs $1 *.o
 }
 
 link_forge() {
-    $CXX *.o -g -L $LIB -lforge -lforge_lua -lprocess -lcmdline -lluaxx -lerror -llua -lassert -lboost_filesystem -lboost_system -o $BIN/forge
+    $CXX *.o -g -L $LIB -lforge -lforge_lua -lprocess -lcmdline -lluaxx -lerror -llua -lassert -o $BIN/forge
 }
 
 link_forge_hooks() {
     $CXX *.o -shared -g -L $LIB -lassert -o $BIN/libforge_hooks.dll
 }
 
 mkdir -p $LIB
-echo boost_system; pushd $SRC/boost/libs/system/src; cxx "*.cpp"; archive $LIB/libboost_system.a; popd
-echo boost_filesystem; pushd $SRC/boost/libs/filesystem/src; cxx "*.cpp"; archive $LIB/libboost_filesystem.a; popd
 echo lua; pushd $SRC/lua/src; cc '*.c'; archive $LIB/liblua.a; popd
 echo assert; pushd $SRC/assert; cxx '*.cpp'; archive $LIB/libassert.a; popd
 echo forge; pushd $SRC/forge; cxx '*.cpp'; archive $LIB/libforge.a; popd

bootstrap-windows.bat

@@ -8,15 +8,13 @@ set OBJ=%~dp0bootstrap
 set LIBS=%~dp0bootstrap\lib
 set BIN=%~dp0bootstrap\bin
 
-set DEFINES=/DBUILD_VARIANT_DEBUG /DBUILD_VERSION=\"bootstrap\" /DBOOST_ALL_NO_LIB /D_WIN32_WINNT=0x0a00
-set INCLUDE_DIRS=/I %SRC% /I %SRC%\lua\src /I %SRC%\boost
+set DEFINES=/DBUILD_VARIANT_DEBUG /DBUILD_VERSION=\"bootstrap\" /D_WIN32_WINNT=0x0a00
+set INCLUDE_DIRS=/I %SRC% /I %SRC%\lua\src
 set CC_FLAGS=%DEFINES% %INCLUDE_DIRS% /c /MTd /TC /Zi
-set CXX_FLAGS=%DEFINES% %INCLUDE_DIRS% /c /MTd /TP /std:c++14 /EHsc /GR /Zi
+set CXX_FLAGS=%DEFINES% %INCLUDE_DIRS% /c /MTd /TP /std:c++17 /EHsc /GR /Zi
 
 if not exist "%OBJ%" mkdir %OBJ%
 if not exist "%LIBS%" mkdir %LIBS%
-echo boost_system & pushd %SRC%\boost\libs\system\src & del *.obj & cl.exe %CXX_FLAGS% *.cpp & lib.exe /out:%LIBS%\boost_system.lib *.obj & popd
-echo boost_filesystem & pushd %SRC%\boost\libs\filesystem\src & del *.obj & cl.exe %CXX_FLAGS% *.cpp & lib.exe /out:%LIBS%\boost_filesystem.lib *.obj & popd
 echo lua & pushd %SRC%\lua\src & del *.obj & cl.exe %CC_FLAGS% *.c & lib.exe /out:%LIBS%\lua.lib *.obj & popd
 echo assert & pushd %SRC%\assert & del *.obj & cl.exe %CXX_FLAGS% *.cpp & lib.exe /out:%LIBS%\assert.lib *.obj & popd
 echo forge & pushd %SRC%\forge & del *.obj & cl.exe %CXX_FLAGS% *.cpp & lib.exe /out:%LIBS%\forge.lib *.obj & popd
@@ -27,6 +25,6 @@ echo luaxx & pushd %SRC%\luaxx & del *.obj & cl.exe %CXX_FLAGS% *.cpp & lib.exe
 echo process & pushd %SRC%\process & del *.obj & cl.exe %CXX_FLAGS% *.cpp & lib.exe /out:%LIBS%\process.lib *.obj & popd
 
 if not exist "%BIN%" mkdir %BIN%
-set LIBRARIES=boost_system.lib boost_filesystem.lib lua.lib assert.lib forge.lib forge_lua.lib cmdline.lib error.lib luaxx.lib process.lib
+set LIBRARIES=lua.lib assert.lib forge.lib forge_lua.lib cmdline.lib error.lib luaxx.lib process.lib
 echo forge/forge & pushd %SRC%\forge\forge & del *.obj & cl.exe %CXX_FLAGS% *.cpp & link /out:%BIN%\forge.exe /libpath:%LIBS% /debug:full /pdb:%BIN%\forge.pdb *.obj %LIBRARIES% & popd
 echo forge/forge_hooks & pushd %SRC%\forge\forge_hooks & del *.obj & cl.exe %CXX_FLAGS% forge_hooks_windows.cpp ImportDescriptor.cpp & link /dll /out:%BIN%\forge_hooks.dll /libpath:%LIBS% /debug:full /pdb:%BIN%\forge_hooks.pdb *.obj assert.lib & popd

docs/cc-module/cc.md

@@ -62,14 +62,11 @@ forge:Executable '${bin}/forge' {
     '${lib}/error_${architecture}';
     '${lib}/assert_${architecture}';
     '${lib}/liblua_${architecture}';
-    '${lib}/boost_filesystem_${architecture}';
-    '${lib}/boost_system_${architecture}';
 
     libraries = libraries;
-    
+
     forge:Cxx '${obj}/%1' {
-        defines = {    
-            'BOOST_ALL_NO_LIB';
+        defines = {
             ('BUILD_VERSION="\\"%s\\""'):format( version );
         };
         'Application.cpp', 

docs/installation.md

@@ -52,6 +52,8 @@ mklink %USERPROFILE%\bin\forge.exe %USERPROFILE%\forge\bin\forge.exe
 
 NOTE: Forge uses [*vswhere*](https://github.com/Microsoft/vswhere/wiki) to find the Visual C++ compiler for Visual Studio 2015 and later and so requires Visual Studio 15.2 (26418.1 Preview) or later that install *vswhere* to a known location to work out of the box.  Visual Studio 2013, 2012, and 2010 may work but aren't tested.
 
+NOTE: Forge embeds a UTF-8 active code page manifest into `forge.exe` so that the narrow Win32 APIs round-trip UTF-8 paths and identifiers.  This requires Windows 10 version 1903 (build 18362) or later at runtime.  See [Character Encoding](more-details/character-encoding.md) for the encoding contract.
+
 ### Github Actions
 
 To install and use Forge from Github Actions:

docs/more-details/character-encoding.md

@@ -0,0 +1,25 @@
+---
+layout: page
+title: Character Encoding
+parent: More Details
+nav_order: 150
+---
+
+- TOC
+{:toc}
+
+Forge treats all internal strings as opaque UTF-8 byte sequences.  This applies to target identifiers, paths, branches, filenames, environment values, and anything else that flows between Lua, the C++ core, and the operating system.
+
+Paths from operating system APIs are treated as UTF-8.  On Linux and macOS the filesystem and process APIs are UTF-8 by default.  On Windows, Forge declares a UTF-8 active code page in its application manifest so that the narrow `*A` Win32 and C runtime APIs round-trip UTF-8 bytes.  Windows 10 version 1903 or later is required because of this.
+
+## Normalization
+
+Forge does not perform Unicode normalization.  Strings are compared as byte sequences.  Two strings that look identical on screen but use different Unicode normalization forms (for example NFC `café` versus NFD `café`) are not equal to Forge.
+
+In practice this means that paths originating from different sources must use the same byte representation to match.  If a target is authored in a buildfile and the same path is later reported by a filesystem hook, both must agree byte-for-byte.
+
+This is deliberate to keep Forge simple.  It's the authors responsibility to keep identifiers in build scripts and paths consistent.
+
+## Validation
+
+Forge does no validatation that strings are well-formed UTF-8.  Invalid byte sequences will pass through unchanged and may produce unreadable output when printed, but they will not crash the build or corrupt the build cache.

docs/more-details/index.md

@@ -28,3 +28,5 @@ This sections provides more detail about different parts of Forge.  Read as your
 - [String Interpolation](string-interpolation.md) describes the string interpolation used to expand `${...}` references in toolset tags and target identifiers.
 
 - [Automatic Dependencies](automatic-dependencies.md) describes how implicit dependencies can be automatically detected by tracing calls that open files for reading or writing when running a tool.
+
+- [Character Encoding](character-encoding.md) describes how Forge treats text as UTF-8 internally, at the operating system boundary, and in the graph cache.

docs/reference/path-and-string-functions.md → docs/reference/path-functions.md

@@ -1,6 +1,6 @@
 ---
 layout: page
-title: Path and String Functions
+title: Path Functions
 parent: Reference
 nav_order: 4
 ---
@@ -42,9 +42,7 @@ function basename( path )
 
 Get the basename of a path.
 
-Extract the basename from `path` (everything except for the extension of which
-the dot "." is considered part, i.e. the dot "." is not returned as part of
-the basename).
+Extract the basename from `path` (everything except for the extension, of which the dot "." is considered part).  Filenames that begin with a dot, such as `.gitignore`, are treated as having no extension — the entire filename is returned as the basename.
 
 **Parameters:**
 
@@ -80,7 +78,7 @@ function extension( path )
 
 Get the extension from a path.
 
-Extract the extension portion of `path` (including the period ".").  If there is no period in `path` then returns the empty string.  Multiple dots will return the right-most period and trailing characters.  A trailing period will return the string ".".
+Extract the extension portion of `path` (including the period ".").  If there is no period in `path` then returns the empty string.  Multiple dots will return the right-most period and trailing characters.  A trailing period will return the string ".".  Filenames that begin with a dot, such as `.gitignore`, are treated as having no extension — the empty string is returned.
 
 **Parameters:**
 
@@ -180,22 +178,6 @@ The leaf portion of a path is the last element.
 
 Returns the last element of `path`
 
-### lower
-
-~~~lua
-function lower( value )
-~~~
-
-Convert a string to lower case.
-
-**Parameters:**
-
-- `value` the string to convert to lower case
-
-**Returns:**
-
-The string `value` converted to lower case.
-
 ### native
 
 ~~~lua
@@ -272,19 +254,3 @@ If the optional `path` parameter is omitted then the calling script's directory
 **Returns:**
 
 The path `path` expressed as absolute relative to the calling Lua script's directory.
-
-### upper
-
-~~~lua
-function upper( value )
-~~~
-
-Convert a string to upper case.
-
-**Parameters:**
-
-- `value` the string to convert to upper case
-
-**Returns:**
-
-The string `value` converted to upper case.

docs/reference/rule-functions.md

@@ -2,7 +2,7 @@
 layout: page
 title: Rule Functions
 parent: Reference
-nav_order: 5
+nav_order: 6
 ---
 
 - TOC

docs/reference/string-functions.md

@@ -0,0 +1,47 @@
+---
+layout: page
+title: String Functions
+parent: Reference
+nav_order: 5
+---
+
+- TOC
+{:toc}
+
+## Overview
+
+String functions operate on, ahem, strings.
+
+## Functions
+
+### lower
+
+~~~lua
+function lower( value )
+~~~
+
+Convert a string to lower case.
+
+**Parameters:**
+
+- `value` the string to convert to lower case
+
+**Returns:**
+
+The string `value` converted to lower case.
+
+### upper
+
+~~~lua
+function upper( value )
+~~~
+
+Convert a string to upper case.
+
+**Parameters:**
+
+- `value` the string to convert to upper case
+
+**Returns:**
+
+The string `value` converted to upper case.

docs/reference/system-functions.md

@@ -2,7 +2,7 @@
 layout: page
 title: System Functions
 parent: Reference
-nav_order: 6
+nav_order: 7
 ---
 
 - TOC

docs/reference/target.md

@@ -2,7 +2,7 @@
 layout: page
 title: Target
 parent: Reference
-nav_order: 7
+nav_order: 8
 ---
 
 - TOC

docs/reference/toolset.md

@@ -2,7 +2,7 @@
 layout: page
 title: Toolset
 parent: Reference
-nav_order: 8
+nav_order: 9
 ---
 
 - TOC