Skip to content

Commit fbbb270

Browse files
committed
feat(lua): backport 5.1 hexadecimal number literals via source transpile
Lua 5.0's lexer reads only decimal digits + one '.' + 'e' exponent, so a `0x...` literal is a compile error (hence addons resort to `tonumber("0x...", 16)`). Lua 5.1 added hex literals. As with `#` / `%` / `...`, we rewrite the SOURCE at the luaL_loadbuffer chokepoint: each hex INTEGER token becomes the exact decimal it denotes (unsigned), which 5.0's lexer accepts and reads as the same double 5.1 would. RewriteHex runs first in the transpile pipeline, gated on a "0x"/"0X" presence scan; it reuses the tokenizer (so strings/comments are never touched) and inserts no newlines (line numbers preserved). Hex floats (0x1.8p3) and > 64-bit literals are left as-is — vanishingly rare, and they fail exactly as today (no regression). Also fixed SkipNumber to stop a number at a `..` so `0xFF.."x"` (hex then concat) tokenizes cleanly. Toggle: _classicapi_SetTranspileOption("HexLiterals", false). Docs: API.md "Lua 5.1 syntax" section + README behaviors row. Verified in-game (19/19 harness asserts) and cross-checked against retail: 0xFF->255, 0xFFFFFFFF->4294967295, bit.band(0xF0,0x0F)->0, "0xFF" untouched all match modern WoW exactly.
1 parent a06a064 commit fbbb270

3 files changed

Lines changed: 136 additions & 29 deletions

File tree

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -234,7 +234,7 @@ Transparent engine tweaks — no API to call, they just fix a vanilla limitation
234234
|-------|--------|
235235
| Tooltip line cap | Lifts `GameTooltip`'s hard 30-line limit to 60 for every `GameTooltipTemplate` frame (`GameTooltip`, `ShoppingTooltip1/2`, `ItemRefTooltip`, AtlasLoot, …). Stat-heavy tooltips and comparison blocks (e.g. pfUI's eqcompare) no longer have their extra lines silently dropped. Done in pure C++ by growing the engine's FontString pool at tooltip-creation time. |
236236
| Inline textures | Draws inline texture markup (`\|T…\|t`) as icons in FontStrings, chat, and tooltips, the way 4.3.4+ clients do. Vanilla 1.12 shows the raw escape as literal text instead. This covers item and spell icons, raid-target markers, and the coin icons in money strings. `GetStringWidth` and `GetStringHeight` count the icons, so measured width and text wrapping stay correct. Done in pure C++ by hooking the engine's text pipeline — no addon. |
237-
| Lua 5.1 syntax | Compiles the Lua 5.1 length (`#`), modulo (`%`), and `...`-expression syntax that vanilla's Lua 5.0 rejects, by rewriting addon source before it compiles. Each addon file also receives its `(name, table)` through `...` (`local name, tbl = ...`). See [Lua 5.1 syntax](docs/API.md#lua-51-syntax-length-modulo-and-vararg). |
237+
| Lua 5.1 syntax | Compiles the Lua 5.1 length (`#`), modulo (`%`), `...`-expression, and `0x` hex-literal syntax that vanilla's Lua 5.0 rejects, by rewriting addon source before it compiles. Each addon file also receives its `(name, table)` through `...` (`local name, tbl = ...`). See [Lua 5.1 syntax](docs/API.md#lua-51-syntax). |
238238
| Lua 5.1 environment protection | `getfenv` / `setfenv` honor a `__environment` metatable field — the Lua 5.1 sandbox form — in addition to vanilla's raw `__fenv`. See [getfenv / setfenv environment protection](docs/API.md#getfenv--setfenv-environment-protection). |
239239
| Multi-flavor & conditional TOC loading | Loads modern multi-flavor addons that ship one folder. Selects a version-specific TOC (`<Name>_ClassicAPI.toc` or `<Name>_Turtle.toc`) and the matching keybinding file (`Bindings_ClassicAPI.xml` / `Bindings_Turtle.xml`), accepts a comma-separated `## Interface:` version list (compatible when it includes the client version `11200`), and honors per-line `[AllowLoadGameType]` / `[AllowLoadTextLocale]` conditions and `[Family]` / `[Game]` / `[TextLocale]` path variables inside a TOC. See [Conditional and multi-flavor TOC loading](docs/API.md#conditional-and-multi-flavor-toc-loading). |
240240
| SavedVariables loaded first | Honors the modern `## LoadSavedVariablesFirst` TOC directive: a flagged addon's SavedVariables load before its Lua runs, so file-scope code sees restored config (instead of vanilla's `nil`). See [SavedVariables loaded first](docs/API.md#savedvariables-loaded-first). |

docs/API.md

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -343,7 +343,7 @@ build instructions.
343343
- [`C_LossOfControl.GetActiveLossOfControlData(index)`](#c_lossofcontrolgetactivelossofcontroldataindex)
344344

345345
- [Lua](#lua)
346-
- [Lua 5.1 syntax: length, modulo, and vararg](#lua-51-syntax-length-modulo-and-vararg)
346+
- [Lua 5.1 syntax](#lua-51-syntax)
347347
- [`getfenv` / `setfenv` environment protection](#getfenv--setfenv-environment-protection)
348348
- [`select(index, ...)`](#selectindex-)
349349
- [`table.wipe(t)`](#tablewipet)
@@ -8486,17 +8486,19 @@ backported addons find them. Most are single-function additions (several are
84868486
just the 5.0→5.1 renames — `string.gmatch`←`gfind`, `math.fmod`←`math.mod`);
84878487
`coroutine.*` restores the whole stripped coroutine library.
84888488

8489-
### Lua 5.1 syntax: length, modulo, and vararg
8489+
### Lua 5.1 syntax
84908490

8491-
1.12 runs Lua 5.0. It cannot compile three pieces of Lua 5.1 syntax that
8491+
1.12 runs Lua 5.0. It cannot compile four pieces of Lua 5.1 syntax that
84928492
modern addons use. These are the length operator `#`, the modulo operator
8493-
`%`, and `...` used as an expression. ClassicAPI rewrites addon source to
8494-
the 5.0 equivalent before it compiles, so all three work:
8493+
`%`, `...` used as an expression, and `0x` hexadecimal number literals.
8494+
ClassicAPI rewrites addon source to the 5.0 equivalent before it compiles,
8495+
so all four work:
84958496

84968497
```lua
84978498
local n = #myTable -- length operator
84988499
local r = a % b -- modulo operator
84998500
local args = { ... } -- ... as an expression, not only in a parameter list
8501+
local mask = 0xFF00 -- hex number literal
85008502
```
85018503

85028504
The rewrite is transparent. You do not call anything. It runs on every
@@ -8512,6 +8514,9 @@ What each form does:
85128514
`-1 % 3` is `2`, while `math.mod(-1, 3)` is `-1`.
85138515
- `...` as an expression yields all the varargs. The `...` in a function
85148516
parameter list stays as the vararg declaration.
8517+
- `0xFF00` becomes its decimal value (`65280`) — the same number Lua 5.1
8518+
produces. Vanilla's lexer rejects `0x` literals, so without this an addon
8519+
needs `tonumber("0xFF00", 16)`.
85158520

85168521
**Addon file arguments.** A modern addon reads its name and its private
85178522
table from the file arguments:
@@ -8534,14 +8539,17 @@ needs that addon's opt-in.
85348539
- A nested long string or comment (`[[ a [[ b ]] c ]]`) matches at the
85358540
first close, not by depth. This is a 5.0-only form that addons almost
85368541
never use.
8542+
- Only integer hex is converted. Hex *floats* (`0x1.8p3`) and literals
8543+
wider than 64 bits are left as-is. Both are almost nonexistent in addon
8544+
code.
85378545
- Error line numbers stay correct. The rewrite adds no new lines.
85388546
- The globals `__len` and `__mod` are the rewrite's helper functions.
85398547
Treat them as internal. Do not call them directly.
85408548
- To turn a rewrite off for diagnosis, call
85418549
`_classicapi_SetTranspileOption(name, false)`, where `name` is
8542-
`"Length"`, `"Modulo"`, or `"VarargExpansion"`. This reverts affected
8543-
chunks to the state that fails to compile, so use it only to answer
8544-
"is the rewrite breaking this addon?".
8550+
`"Length"`, `"Modulo"`, `"VarargExpansion"`, or `"HexLiterals"`. This
8551+
reverts affected chunks to the state that fails to compile, so use it
8552+
only to answer "is the rewrite breaking this addon?".
85458553

85468554
### `getfenv` / `setfenv` environment protection
85478555

src/luasyntax/Transpile.cpp

Lines changed: 119 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -14,17 +14,19 @@
1414
// Lua 5.1 syntax backport (source-level transpile).
1515
//
1616
// Vanilla's Lua is 5.0. It lacks the 5.1 operators `#` (length) and `%`
17-
// (modulo) AND `...` used as an expression, so modern addon ports that use
18-
// them fail to COMPILE. We don't touch the 5.0 parser/VM (there is no
19-
// OP_LEN / OP_MOD and no free opcode); instead we rewrite the SOURCE before it
20-
// reaches the parser: co-hook `luaL_loadbuffer` (the one function every compile
21-
// funnels through — file scripts, `loadstring`, XML `<OnLoad>`) and rewrite:
17+
// (modulo), `...` used as an expression, AND `0x` hexadecimal number
18+
// literals, so modern addon ports that use them fail to COMPILE. We don't
19+
// touch the 5.0 parser/VM (there is no OP_LEN / OP_MOD and no free opcode);
20+
// instead we rewrite the SOURCE before it reaches the parser: co-hook
21+
// `luaL_loadbuffer` (the one function every compile funnels through — file
22+
// scripts, `loadstring`, XML `<OnLoad>`) and rewrite:
2223
// #operand -> __len(operand)
2324
// a % b -> __mod(a, b)
2425
// ... -> unpack(arg) (the `...` EXPRESSION, not the decl)
26+
// 0xHH... -> <decimal> (5.0's lexer rejects hex literals)
2527
// `__len` / `__mod` are C globals we register (see below); `unpack` and the
26-
// 5.0 `arg` table are already present. RewriteChunk runs the vararg pass first,
27-
// then the # / % precedence parser.
28+
// 5.0 `arg` table are already present. RewriteChunk runs the hex pass first,
29+
// then the vararg pass, then the # / % precedence parser.
2830
//
2931
// Why a real parser (not regex / one-term-each-side): `#` is prefix but `%`
3032
// is binary infix, so its operands must be delimited by PRECEDENCE —
@@ -50,10 +52,15 @@
5052
// use first-close matching, not depth. Real addons ~never nest these.
5153
// * `__len` on a table returns a border (bisection) = 5.1 `#`; it ignores
5254
// any `table.setn` count (5.1 has no setn — correct `#` semantics).
55+
// * Hex literals: only INTEGER `0x…` (up to 16 hex digits) are converted,
56+
// to the exact decimal the value represents — the same double 5.1 would
57+
// produce. Hex FLOATS (`0x1.8p3`) and > 64-bit literals are left as-is
58+
// (vanishingly rare in addons; they fail to compile exactly as today, so
59+
// no regression). The value is unsigned (`0xFFFFFFFF` -> 4294967295).
5360
// * Diagnostic `_classicapi_TranspileLength(src)` returns the full rewrite.
5461
// * Toggles via `_classicapi_SetTranspileOption(name, bool)` /
5562
// `_classicapi_GetTranspileOption(name)` (name = "Length" / "Modulo" /
56-
// "VarargExpansion").
63+
// "VarargExpansion" / "HexLiterals").
5764
// * `...` expands to `unpack(arg)`, which is faithful in every position and
5865
// preserves embedded nils via `arg.n` (this build's `unpack` honors it).
5966
// Only the `...` in a function's parameter list is left intact.
@@ -88,11 +95,12 @@ namespace {
8895

8996
constexpr size_t NPOS = static_cast<size_t>(-1);
9097

91-
// Runtime switches, default ON — a `#`/`%`/`...`-bearing chunk does not compile
92-
// on 5.0 today, so enabling by default cannot regress working addons.
98+
// Runtime switches, default ON — a `#`/`%`/`...`/`0x`-bearing chunk does not
99+
// compile on 5.0 today, so enabling by default cannot regress working addons.
93100
bool g_lenEnabled = true;
94101
bool g_modEnabled = true;
95102
bool g_varargEnabled = true;
103+
bool g_hexEnabled = true;
96104

97105
// lua_rawgeti(L, idx, n) — push table_at_idx[n] without metamethods. Not
98106
// exposed via Game::Lua; used to probe table elements for the border search.
@@ -181,7 +189,9 @@ size_t SkipNumber(const char *src, size_t len, size_t pos) {
181189
while (i < len) {
182190
char c = src[i];
183191
if (c == '.') {
184-
if (seenDot)
192+
// A second dot, or the `..` concat operator, ends the number so
193+
// `0xFF.."x"` / `1..2` split cleanly (the concat is not consumed).
194+
if (seenDot || (i + 1 < len && src[i + 1] == '.'))
185195
break;
186196
seenDot = true;
187197
i++;
@@ -720,13 +730,87 @@ bool RewriteVararg(const char *src, size_t len, std::string &out) {
720730
return true;
721731
}
722732

723-
// Run every syntax rewrite over a chunk (vararg first, then # / %). Returns
724-
// true and fills `out` if anything changed.
733+
// ============================================================================
734+
// Hex-literal pass: rewrite 5.1 `0x…` integer literals to decimal.
735+
//
736+
// Lua 5.0's lexer reads only decimal digits + one `.` + `e` exponent, so a
737+
// `0x…` literal fails to compile (which is why addons resort to
738+
// `tonumber("0x…", 16)`). We convert each hex INTEGER token to the exact
739+
// decimal it denotes — Lua then reads it as the same double 5.1 would. The
740+
// value is unsigned. Hex floats and > 64-bit literals are left untouched (they
741+
// fail to compile as they do today — no regression). Runs before the other
742+
// passes; the tokenizer already skips strings/comments, so `"0xFF"` / `--0xFF`
743+
// are safe, and it never inserts a newline (line numbers preserved).
744+
// ============================================================================
745+
746+
inline bool IsHexDigit(unsigned char c) {
747+
return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F');
748+
}
749+
750+
bool ContainsHexPrefix(const char *src, size_t len) {
751+
for (size_t i = 0; i + 1 < len; i++)
752+
if (src[i] == '0' && (src[i + 1] == 'x' || src[i + 1] == 'X'))
753+
return true;
754+
return false;
755+
}
756+
757+
bool RewriteHex(const char *src, size_t len, std::string &out) {
758+
if (src == nullptr || len == 0 || !g_hexEnabled)
759+
return false;
760+
if (!ContainsHexPrefix(src, len))
761+
return false;
762+
763+
std::vector<Token> toks;
764+
Tokenize(src, len, toks);
765+
766+
out.clear();
767+
out.reserve(len);
768+
size_t p = 0;
769+
bool any = false;
770+
for (const Token &t : toks) {
771+
if (t.kind != TK_NUMBER || t.end - t.start < 3)
772+
continue; // need "0x" + >= 1 digit
773+
if (src[t.start] != '0')
774+
continue;
775+
char x = src[t.start + 1];
776+
if (x != 'x' && x != 'X')
777+
continue;
778+
const size_t nDigits = t.end - (t.start + 2);
779+
if (nDigits == 0 || nDigits > 16)
780+
continue; // no digits, or wider than uint64 — leave as-is
781+
782+
uint64_t val = 0;
783+
bool pure = true;
784+
for (size_t i = t.start + 2; i < t.end; i++) {
785+
unsigned char c = static_cast<unsigned char>(src[i]);
786+
if (!IsHexDigit(c)) { pure = false; break; } // hex float ('.'/'p') or junk
787+
val = val * 16 + ((c <= '9') ? (c - '0') : ((c | 0x20) - 'a' + 10));
788+
}
789+
if (!pure)
790+
continue;
791+
792+
out.append(src + p, t.start - p);
793+
out.append(std::to_string(val));
794+
p = t.end;
795+
any = true;
796+
}
797+
if (!any)
798+
return false;
799+
out.append(src + p, len - p);
800+
return true;
801+
}
802+
803+
// Run every syntax rewrite over a chunk (hex first, then vararg, then # / %).
804+
// Returns true and fills `out` if anything changed.
725805
bool RewriteChunk(const char *src, size_t len, std::string &out) {
806+
std::string h;
807+
bool didHex = RewriteHex(src, len, h);
808+
const char *cur = didHex ? h.data() : src;
809+
size_t curLen = didHex ? h.size() : len;
810+
726811
std::string a;
727-
bool didVararg = RewriteVararg(src, len, a);
728-
const char *cur = didVararg ? a.data() : src;
729-
size_t curLen = didVararg ? a.size() : len;
812+
bool didVararg = RewriteVararg(cur, curLen, a);
813+
if (didVararg) { cur = a.data(); curLen = a.size(); }
730814

731815
std::string b;
732816
if (RewriteAll(cur, curLen, b)) {
@@ -737,6 +821,10 @@ bool RewriteChunk(const char *src, size_t len, std::string &out) {
737821
out = std::move(a);
738822
return true;
739823
}
824+
if (didHex) {
825+
out = std::move(h);
826+
return true;
827+
}
740828
return false;
741829
}
742830

@@ -896,11 +984,21 @@ int __fastcall LoadBuffer_h(void *L, const char *buff, unsigned size, const char
896984
if (buff == nullptr || size == 0)
897985
return g_origLoadBuffer(L, buff, size, name);
898986

899-
// Syntax transpile: vararg first, then # / %.
987+
// Syntax transpile: hex literals first, then vararg, then # / %.
988+
std::string hx;
989+
const char *body = buff;
990+
size_t bodyLen = size;
991+
if (RewriteHex(body, bodyLen, hx)) {
992+
body = hx.data();
993+
bodyLen = hx.size();
994+
}
995+
900996
std::string va;
901-
bool didVararg = RewriteVararg(buff, size, va);
902-
const char *body = didVararg ? va.data() : buff;
903-
size_t bodyLen = didVararg ? va.size() : size;
997+
bool didVararg = RewriteVararg(body, bodyLen, va);
998+
if (didVararg) {
999+
body = va.data();
1000+
bodyLen = va.size();
1001+
}
9041002

9051003
std::string ops;
9061004
if (RewriteAll(body, bodyLen, ops)) {
@@ -982,6 +1080,7 @@ const Toggle kToggles[] = {
9821080
{"Length", &g_lenEnabled},
9831081
{"Modulo", &g_modEnabled},
9841082
{"VarargExpansion", &g_varargEnabled},
1083+
{"HexLiterals", &g_hexEnabled},
9851084
};
9861085
bool *FindToggle(const char *name) {
9871086
if (name)

0 commit comments

Comments
 (0)