Load MeTTa source as UTF-8 regardless of the ambient locale

[lafsar]

Problem

.metta source files are UTF-8 by convention — that is what editors produce — but PeTTa reads them without specifying an encoding, so SWI-Prolog falls back to the encoding flag, which is derived from the ambient locale:

% src/filereader.pl
load_metta_file(Filename, Results, Space) :- read_file_to_string(Filename, S, []),
                                             process_metta_string(S, Results, Space).

Where the locale is not UTF-8, multi-byte characters in string literals or symbol names are decoded in the locale codepage instead, and each one silently becomes several wrong characters. In the common case there is no error — the program simply runs with corrupted strings.

This is not Windows-specific. It reproduces under any non-UTF-8 locale, including the C/POSIX locale that minimal container images, CI runners and systemd units commonly have.

Reproduction

A .metta file saved as UTF-8 containing:

!(println! "café naïve — em dash")

The correct length of that string is 35 characters.

environment	current_prolog_flag(encoding, E)	length read
Linux, LANG=C.UTF-8	utf8	35	correct
Linux, LANG=C	text	24	Illegal multibyte Sequence
Windows 11, en-US	text (cp1252)	39	silent

Minimal repro on Linux, no Windows required:

$ printf '!(println! "caf\303\251 \342\200\224")\n' > /tmp/u.metta

$ LANG=C swipl -g 'read_file_to_string("/tmp/u.metta",S,[]), string_length(S,L), writeln(L), halt.'
Warning: '/tmp/u.metta':2:0: Illegal multibyte Sequence
24

$ LANG=C swipl -g 'set_prolog_flag(encoding,utf8), read_file_to_string("/tmp/u.metta",S,[]), string_length(S,L), writeln(L), halt.'
21

(21 is correct for that shorter one-line file; 35 is the longer example in the table.)

A note for anyone verifying this on Windows: the corruption can look correct when printed, because the console is itself cp1252 — mojibake renders "properly" there while true UTF-8 renders as ?. String length is the reliable signal, not what you see in the terminal. This cost me a wrong conclusion before I measured.

Why this is worth fixing rather than documenting

The same locale default applies to open/3, so files written by PeTTa and files written by anything else disagree about encoding. In a project that uses PeTTa's Python interop, this surfaces as a hard failure in one direction and silent corruption in the other:

• Prolog writes → Python reads: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x97
• Python writes → Prolog reads: no error, wrong characters

I hit the first of these in production. An append-only .metta log accumulated lone cp1252 bytes from one writer, and every subsequent load truncated at the first bad byte — the only visible symptom being that older entries had "disappeared".

Fix

One directive at the top of src/main.pl, before anything opens a stream:

:- set_prolog_flag(encoding, utf8).

Compatibility

This is a no-op where the locale is already UTF-8, which includes this repo's CI.

I specifically checked that it cannot disturb test.sh, which greps stdout for the ✅ / ❌ markers. In a swipl:10.0.2 container, reading a .metta file containing ❌ and writing it back out:

locale	without the change	with the change
C.UTF-8	grep matches	grep matches
C	grep does not match	grep does not match

Under C.UTF-8 the output bytes are identical with and without the patch, so the harness is unaffected. Under LANG=C the markers already fail to round-trip today; this change neither fixes nor worsens that.

One honest behaviour change: a pre-existing non-UTF-8 .metta file now warns (Illegal UTF-8 continuation) instead of silently mis-decoding. I would argue that is the better failure, but it is a change.

Alternative, if you prefer a narrower diff

Passing the encoding explicitly at the single read site:

read_file_to_string(Filename, S, [encoding(utf8)])

I verified this also yields the correct 35. It fixes source loading without touching a global flag, but leaves open/3 in lib/lib_import.pl and any Python interop still locale-dependent. Happy to switch to this form, or to set the flag in metta.pl instead of main.pl if PeTTa is expected to be loaded as a library rather than through its CLI entry point — just say which you prefer.

Testing

• Verified on Windows 11 (cp1252) and in a swipl:10.0.2 container under both C.UTF-8 and C locales.
• examples/roman_test.metta produces identical output with and without the change on my machine. It does report one pre-existing failure there (map-flat (partial + (1)) (1 2 3)) which reproduces on unpatched main as well — unrelated to this change, but mentioning it in case it is news.