Skip to content

Latest commit

 

History

History
406 lines (355 loc) · 17.9 KB

emit_python.md

File metadata and controls

406 lines (355 loc) · 17.9 KB

Python Emission

-emit-python lowers the shared MLIR pipeline into runnable Python source. The emitter aims for a near 1:1 translation of the MATLAB input — MATLAB for i = 1:N becomes Python for i in range(1, N+1):, matrix arithmetic uses inline numpy operators (A @ B, A.T, np.linalg.inv(A)), classdef code becomes proper Python class blocks with __init__ / @property / @staticmethod / dunder operator overloads, and string-literal disp calls collapse to bare print(...). The matlab_runtime import only appears when the body actually references the shim.

Scope

Driver flag:

Flag Output
-emit-python Self-contained .py source

When the emitted body references the shim it imports runtime/matlab_runtime.py, a NumPy-backed runtime helper module. Programs that only do numpy / arithmetic / class work skip the runtime import entirely. No LLVM toolchain or C compiler is needed at runtime:

build/matlabc -emit-python foo.m > foo.py
PYTHONPATH=runtime python3 foo.py

You can also use:

just emit-python examples/hello.m
just compile-python examples/hello.m
just test-emitpython

Status

What exists today:

Current positioning:

  • useful for inspection, portability, and rapid execution without a C toolchain
  • not as mature as the LLVM / C / C++ paths
  • some numerically sensitive programs are intentionally skipped in the Python test lane

Architecture reuse

Everything up to runLowerIO is shared with the other backends. The pipeline diverges at the same fork point as -emit-c and -emit-cpp:

AST ──► MLIR ──► [SlotPromotion, ..., LowerIO]
                       │
                       ├──► [scf→cf, arith→llvm, ...] ──► .ll
                       ├──► emitC()       ──► .c / .c++
                       └──► emitPython()  ──► .py

The emitter targets a small closed MLIR subset and remaps runtime calls to import matlab_runtime as rt.

Behavior

MLIR Python output
%c = arith.constant 1.0 : f64 1.0 (shortest round-tripping repr; 0.05, not 0.050000000000000003)
%c = arith.constant 1 : i64 1
%s = arith.addf %a, %b v1 = a + b
arith.mulf / divf / subf * / / / -
arith.cmpi / cmpf ==, !=, <, <=, >, >=
arith.andi / ori / xori &, |, ^ (on int); and / or / != for i1
arith.select %c, %a, %b v0 = a if c else b
casts (sitofp, fptosi, etc.) float(x) / int(x) / no-op
scf.if %c -> T { yield %a } else { yield %b } if c: v0 = a / else: v0 = b (no v0 = 0 predeclaration; the yields cover both paths)
scf.if with statically-known c only the live branch is emitted; e.g. polymorphic dispatch's if 2 == 2 collapses
scf.while matching for i = a:s:b for i in range(a, b+1, s): (or rt.frange(a, b, s) for non-integer bounds)
scf.while (general) while cond: <body> — or while True: ...; if not cond: break; ... when the before-region has work
func.func @foo(%a, %b) -> T def foo(a, b): (param names taken from matlab.name); pass for empty bodies
func.func @foo(...) -> (T0, T1) (multi-return) def foo(...): returning a Python tuple
func.call @foo(%a) foo(a) (inlined into the use site when single-use)
func.call with N>1 results a, b = foo(args) Python tuple unpacking
func.return %a, %b (multi-return) return a, b
matlab.{add,sub,emul,matmul,ediv,matdiv} (unregistered scalar) + - * / operators
matlab.{eq,ne,lt,le,gt,ge} (unregistered scalar) == != < <= > >=
matlab.short_or / matlab.short_and Python or / and (short-circuit semantics preserved)
matlab.call_builtin @matlab_global_set_f64 (persistent write) <fn>.<name> = <v> matching the module-level <fn>.<name> = <init> decl below the function
llvm.call @matlab_<helper>(...) rt.<helper>(...) (every matlab_* symbol drops its prefix). For matrix builtins the call is rewritten to inline numpy / Python operators — see the Numpy rewrite for matrix builtins note below.
llvm.call @matlab_disp_str(%p, %len) rt.disp_str(...) — the (ptr, length) C ABI tail is dropped, since Python strings carry their own length
llvm.call %fp(%a) (indirect) v0 = fp(a)
llvm.alloca (scalar slot) skipped — load/store emit as plain Python name = ...
llvm.alloca (matrix-literal array) slot = [v0, v1, ...] (the GEP+store pairs that filled it are folded in)
llvm.store %v, %p scalar slot: name = v; array element: name[idx] = v
llvm.load %p scalar slot: bare name; array element: name[idx]
llvm.getelementptr %base[%i] tracked as (base, i) pair; materialized only at the load/store consumer
llvm.mlir.global @s = "..." inlined as a Python string literal at every addressof use site (no top-of-file declaration)

Emitter behavior worth knowing:

  • Strings inline: rt.disp_str("Hello") (or print("Hello")) directly, instead of a top-of-file __matlab_str0 = "Hello" declaration referenced by rt.disp_str(__matlab_str0, 13). When the argument is a string literal, the rt.disp_str call collapses further to print(...) — byte-equivalent output, but reads as plain Python.

  • Scalar disp substitution: rt.disp_f64(x) collapses to print(f'{x:g}'). Output matches MATLAB's %g format for every finite value; the runtime path stays for matrices (disp_mat) where MATLAB's multi-column alignment rules don't have a clean f-string equivalent.

  • Conditional imports: the import matlab_runtime as rt and import numpy as np lines are added only when the emitted body actually references those modules. A pure-arithmetic program like examples/factorial.m emits zero import lines.

  • Matrix disp substitution: rt.disp_mat(M) collapses to plain print(M). Numpy's bracket / dotted-float matrix repr diverges from MATLAB's right-aligned %7g columns, so the Python lane uses per-test <name>.stdout-python overrides for tests that print matrices; the C / C++ goldens stay on the shared <name>.stdout. Programs that only do matrix display now drop the import matlab_runtime as rt line entirely (e.g. examples/matrix_mult.py).

  • Numpy rewrite for matrix builtins: the matrix subset of the runtime ABI is emitted as inline numpy / Python-operator expressions rather than rt.<helper> calls. Specifically:

    Runtime call Emitted as
    matlab_matmul_mm(A, B) A @ B
    matlab_add_mm/ms/sm A + B
    matlab_sub_mm/ms/sm A - B
    matlab_emul_mm/ms/sm A * B
    matlab_ediv_mm/ms/sm A / B
    matlab_transpose(A) A.T
    matlab_inv(A) np.linalg.inv(A)
    matlab_det(A) np.linalg.det(A)
    matlab_mldivide_mm(A, B) np.linalg.solve(A, B)
    matlab_norm(A) np.linalg.norm(A)
    matlab_trace(A) np.trace(A)
    matlab_zeros(m[, n]) np.zeros((m, n))
    matlab_ones(m[, n]) np.ones((m, n))
    matlab_eye(n[, m]) np.eye(n) / np.eye(n, m)
    matlab_mat_from_buf(buf, m, n) np.array(buf).reshape(m, n)
    matlab_sqrt_m / exp_m / sin_m / … np.sqrt(A) / np.exp(A) / np.sin(A) / …

    MATLAB-semantics helpers (slice1 / slice2, MATLAB-style column-major reductions like sum / mean on a 2-D matrix, disp_mat's right-aligned %7g formatting, eig / lu / qr / svd whose return shapes don't map 1:1 to numpy) stay on the runtime path — they don't have a clean one-line numpy equivalent.

  • For loops: scf.while ops produced by LowerSeqLoops::lowerForOp collapse to native for i in range(...):. When init/end/step are integer literals, Python's range is used; otherwise the rt.frange(start, end, step) generator preserves MATLAB's inclusive start:step:end semantics.

  • elif chaining: an scf.if whose else-region is a single nested scf.if (the shape MATLAB elseif lowers to) emits as elif <cond>:, walking the cascade down to a final else: block. Result-bearing scf.if chains have their result SSA values aliased so each branch writes to the shared local.

  • Static-condition folding: scf.if whose condition is a constant-vs-constant comparison (e.g. the if nargin == 2 baked in by polymorphic dispatch) emits only the live branch.

  • select(c, 1.0, 0.0) fold: emitted as float(c) rather than the literal ternary 1.0 if c else 0.0 (covers MATLAB's logical-to-double coercion).

  • Pure-read inlining: a single-use rt.obj_get_f64 (or another helper on the known-pure allowlist: size, numel, length, ndims, isempty, …) inlines past peer pure-read calls. Savings.interest collapses to a single return line: return rt.obj_get_f64(obj, "Balance") * rt.obj_get_f64(obj, "Rate").

  • Scalar slot collapse: alloca + store + load patterns become plain Python variables. The slot = 0 pre-declaration is dropped when the slot is unconditionally written before any read, including via an scf.if whose both branches store into the slot (covers MATLAB return-slot patterns).

  • Trailing return: dropped when it sits at the end of a void function; Python's implicit return covers it.

  • Multi-return functions: emit as return a, b Python tuples; call sites use a, b = f(args) tuple unpacking. No <typing> / typing.Tuple annotations — duck-typed by Python's runtime.

  • Persistent + if isempty(x); x = init; end: lowers to a module-level <fn>.<name> = <init> line just below the function definition. The if isempty(_) ... end chain is suppressed in the body so reads are bare <fn>.<name> and writes are <fn>.<name> = <v>. Persistents that have only datapath updates (no isempty guard) fall back to the legacy <fn>.<name> = 0.0 initializer.

  • String-length operands dropped: runtime helpers whose C ABI takes (ptr, i64-length) (disp_str, fprintf_str, fprintf_f64*, input_num) or (name_ptr, name_len) (obj_get_f64, obj_set_f64) lose the length operand at the call site. The Python runtime stubs make the length parameter optional so hand-written callers still work.

  • break / continue un-lowering: the frontend's did_break / did_continue flag slots are re-lowered to native Python break / continue, with the guarding & !did_break conjunct stripped from the loop condition.

  • Per-function name scope: each def gets a fresh identifier table so parameters keep their natural names (x, i) instead of accumulating _2, _3 suffixes from earlier functions.

  • User-source variable names: SlotPromotion propagates an alloc's name attribute onto the defining op of the value being stored, and LowerTensorOps carries it from concat_col onto the resulting mat_from_buf call. The Python emitter then prefers the propagated name over a fresh vN id, so a MATLAB A = [1 2; 3 4] emits as A = np.array(...).reshape(2, 2) rather than v0 = ....

  • MATLAB classdef → Python class: every function tagged with matlab.class_name is collected into a class block. The constructor becomes __init__ (with the synthesized obj_new allocation suppressed and the trailing return self dropped); regular methods bind the first parameter to self; static methods (matlab.method_kind = "static") emit @staticmethod and skip the self slot; Dependent properties (matlab.method_name = "get.X") emit @property; and operator overloads (eq, ne, lt, le, gt, ge, plus, minus, mtimes/times, mrdivide/rdivide, uminus) become Python dunders. Field reads / writes inside method bodies route through the same obj.X rewrite as elsewhere, so self.Balance = self.Balance + amt reads naturally.

  • Class-method call sites: <ClassName>__<method>(args) is rewritten according to the method kind:

    • constructor → ClassName(args)
    • regular method → args[0].method(args[1:])
    • static method → ClassName.method(args)
    • operator → args[0] OP args[1] (e.g. acc == other)
    • get-property → args[0].PropName (no parens)
  • Top-level script body is hoisted to module scope (mirrors the LowerIO rename of @script to @main).

  • Comments and blank lines from the MATLAB source propagate to the emitted Python, similar to the C/C++ emitter.

Runtime Model

The backend pairs the emitted source with a NumPy-backed runtime shim (runtime/matlab_runtime.py) that implements MATLAB-semantics helpers the emit can't express inline. The emitter's job is to route around the shim wherever a clean Python / numpy idiom exists, and to call into the shim only for behaviour that needs MATLAB-specific semantics (column-major reductions, MATLAB-style slicing, string-with-length helpers, parfor dispatch, handles, etc.). Many programs end up importing only numpy (or nothing at all — see the conditional-import note above).

Representative actual signatures:

def disp_f64(v):             ...      # matches MATLAB %g formatting
def disp_mat(A):             ...      # right-aligned %7g columns (C-lane only)
def fprintf_f64(fmt, *args): ...      # %d / %f / %g / %s expansion
def slice2(A, rows, cols):   ...      # 1-indexed row/col selection
def sum(A):                  ...      # column reduction for 2-D matrices
def frange(start, end, step):...      # MATLAB-style for-loop generator

def obj_new(*_):             ...      # allocates a `_MatObj` instance
def obj_set_f64(o, name, ...): ...    # routes through setattr() on _MatObj
def obj_get_f64(o, name, ...): ...    # routes through getattr() on _MatObj
def parfor_dispatch(start, step, end, body, state): ...

Coverage:

  • matrix creation, arithmetic, and decompositions through NumPy
  • MATLAB-formatted display and fprintf printf-spec expansion
  • structs, cells, classdef objects via Python containers / attributes
  • file I/O and string-handling helpers
  • parfor reductions, function handles, and runtime error plumbing

Testing

The Python backend has its own execution lane:

just test-emitpython

That runner:

  • emits Python from every test/Run/*.m
  • runs the result with python3
  • diffs stdout against the same golden files used by other backends
  • honors .sorted markers for nondeterministic parfor output
  • honors .skip-emit-python markers for known divergences

Sample output

% factorial.m
function y = fact(n)
  if n <= 1
    y = 1;
  else
    y = n * fact(n - 1);
  end
end
disp("fact(1..6):");
for i = 1:6
  disp(fact(i));
end

emits (note no import lines — this body needs neither the runtime shim nor numpy):

def fact(n):
    if n <= 1.0:
        y = 1.0
    else:
        y = n * fact(n - 1.0)
    return y

print("fact(1..6):")
for i in range(1, 7):
    print(f'{fact(i):g}')

A second example showing elseif chaining and select(c, 1.0, 0.0) folding, from examples/traffic_action.m:

def traffic_action(color, is_emergency):
    if is_emergency != 0.0:
        a = 9.0
    elif color < 1.0:
        a = 0.0
    elif color > 3.0:
        a = 0.0
    elif color == 1.0:
        a = 1.0
    elif color == 2.0:
        a = 2.0
    else:
        a = 3.0 if color == 3.0 else 0.0
    return a

A third example, examples/matrix_mult.m, showing the numpy rewrite — no matlab_runtime import is needed at all because matrix display also collapses to plain print(M). User-source variable names (A, B) propagate through the slot-promotion pipeline, so the emitted identifiers match the MATLAB:

import numpy as np

slot = [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 10.0]
A = np.array(slot).reshape(3, 3)

slot_2 = [1.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 3.0]
B = np.array(slot_2).reshape(3, 3)

print("A * B =")
print(A @ B)

print("A .* B =")
print(A * B)

print("A' =")
print(A.T)

A fourth example, examples/bank_account.m, showing the full classdef → Python class translation: constructor → __init__, Dependent get-properties → @property, operator overloading → dunder, inheritance → Python subclass, field reads / writes → attribute access:

class BankAccount:
    def __init__(self, id, bal):
        self.Id = id
        self.Balance = bal
    def deposit(self, amt):
        self.Balance = self.Balance + amt
    def withdraw(self, amt):
        self.Balance = self.Balance - amt
    @property
    def Overdrawn(self):
        return float(self.Balance < 0.0)
    def __eq__(self, other):
        return float(self.Id == other.Id)

class Savings(BankAccount):
    def __init__(self, id, bal, rate):
        self.Id = id
        self.Balance = bal
        self.Rate = rate
    def interest(self):
        return self.Balance * self.Rate

acc = BankAccount(1001.0, 500.0)
acc.deposit(250.0)
print(f'{acc.Balance:g}')               # 750
print(f'{acc.Overdrawn:g}')              # 0
print(f'{acc == BankAccount(1001.0, 0.0):g}')   # 1
print(f'{Savings(2000.0, 1000.0, 0.05).interest():g}')  # 100

Limitations

This backend should be treated as experimental.

Known constraints:

  • some numerically sensitive cases are skipped in the Python lane
  • parity is strongest on common scalar, matrix, control-flow, and runtime-library programs, not on every edge case
  • the generated Python aims to read like the natural translation of the MATLAB source — not perfectly idiomatic hand-written Python, but close enough to inspect and modify
  • if you need the most mature code generation path, prefer -emit-c, -emit-cpp, or -emit-llvm