Skip to content

Draw formulas, coloured code, underline and scripts in markdown notes - #11

Open
sanjaydotpro wants to merge 8 commits into
mainfrom
capy/markdown-rendering
Open

sanjaydotpro wants to merge 8 commits into
mainfrom
capy/markdown-rendering

Conversation

@sanjaydotpro

Copy link
Copy Markdown
Contributor

Markdown notes now draw what their markdown says instead of showing its source. That covers TeX formulas, fenced code coloured in its own language, underline, superscript and subscript, and line breaks in table cells. This is the first of three stacked PRs for markdown-first notes (HIS-4): the paste and copy-out PR stacks on it, and the default flip stacks on that.

Formulas, code and underline in a dark note

The same note in light

With the caret in the display formula, its TeX shows as written

Formulas

  • TeX between dollar signs is read as math by pandoc's rules. $x^2$ is a formula, while "lunch was $5 or $10" stays as words.
  • A formula is drawn in place of its source with flutter_math_fork. Inline formulas sit on the baseline at 1.1× the size of the words around them. Display formulas ($$…$$) are centred on a line of their own, and shrink rather than overflow. When the caret is inside a formula, its TeX shows as written.
  • Some common TeX that flutter_math_fork lacks is adapted:
    • align, alignat and split are read as aligned and alignedat.
    • equation is unwrapped.
    • \sgn is defined.
    • A phantom's bare sized delimiter gets braces.
  • TeX that doesn't parse stays as written, in code style.
  • Each formula is laid out once offscreen, and only its size and baseline reach the text layout. The calculator never reads a formula, and screen readers read its TeX from the note's text.

Code, underline, scripts and table cells

  • A fenced code block whose info string names a language is coloured in that language (re_highlight), using the palette's colours.
  • <u> and <ins> underline their words, and <sup> and <sub> raise and lower them. Where the face has no raised glyph for a character, it falls back to smaller text on the baseline, like the n in 2ⁿ in the golden above.
  • <br> in a table cell breaks the cell's line.

Opening long notes

The macOS app keeps the engine's accessibility tree switched on for every user (AccessibilityTree.swift, so dictation tools can write into the editor). RenderEditable re-reads the semantics of every span in the whole note each time it adopts a widget, so 350 formulas meant 350 walks of a 5,000-line note. The editor's root span now works its semantics out once and keeps them.

Separately, dart_markdown stopped reading plain words in runs once any inline syntax of ours was registered, and fell back to reading a word at a time. A small word-run syntax restores main's behaviour.

Profile builds on an M3 Pro, Flutter 3.47.4, semantics on as in the app, main and this branch run alternately. The note has 5,000 lines with 350 formulas and 30 code blocks:

main this branch
Opening, ms to first frame (three opens) 2330 / 1783 / 1710 2722 / 1971 / 2017
Typing in prose, p90 frame build 2024 ms 2105 ms
Typing in a code block, p90 1987 ms 2061 ms
Typing after a formula, p90 2007 ms 2062 ms

What's left, about 0.2–0.4 s on this note, is mostly mounting the formula widgets and laying out the extra styled runs. Typing at about 2 s a key in a 5,000-line note is already the case on main. The per-line editor (HIS-14) is the fix for that. benchmark/markdown_rendering.dart reproduces these runs.

Size and licences

The release app grows by 4.30 MB (+3.4%): 3.65 MB of AOT code, plus KaTeX's fonts at 0.64 MB. The new packages are flutter_math_fork (Apache-2.0), re_highlight (MIT), provider and nested (MIT), and tuple (BSD-2-Clause). KaTeX's fonts are MIT. Their notice ships as an asset, because the package's own LICENSE doesn't carry it.

Testing

  • Built and tested with Flutter 3.47.4 (~/dev/flutter) after flutter pub get --enforce-lockfile. The pinned 3.47.2 at ~/development/flutter is damaged on this Mac.
  • flutter analyze is clean. Every commit passes analyze, dart format and the markdown and editor suites.
  • In the full suite, the only failures are the 45 goldens that also fail on main under 3.47.4 (pixel drift). The three new goldens are above.
  • The profile runs used the bundle id com.kapybara.kapynotes.renderperf, set on the command line only.

Not fixed here: on main, a table row with no pipes that ends in a backslash escape makes dart_markdown's TableSyntax._parseRow throw, and that chunk of the note loses its styling.

Open in Capy

The field works out its semantics information, an entry for every run
of the note's text, each time its semantics are marked dirty, and it
marks them for every widget it takes in. Opening a note walked the whole
of it once for each picture in it, whenever the semantics tree is on,
and in the macOS app it always is: AccessibilityTree.swift switches it
on at launch so that dictation tools can write into the editor. The span
the controller builds now works the information out once and keeps it.

Formulas, drawn as widgets in the text by the commits that follow, are
where it shows: in a debug build, a 5,000-line note with 350 of them
opened in 16.1 s without this and in 4.8 s with it.
A note pasted from ChatGPT, Gemini or Wikipedia writes its maths as $…$
among words and $$…$$ set apart. Both are now read as formulas, by the
rules pandoc reads them by, so that a note about money keeps its prices:
the opening $ of $…$ has no space after it, and the closing one no space
before it and no digit after it, which keeps "$5 and $10" two prices
rather than one formula. A $$ starting a line opens a block that runs to
the $$ ending one, with no blank line between, so the lines of an
aligned equation stay TeX rather than becoming a list.

A formula is literal, the way code is: its TeX is shown in code's face
with its dollar signs quiet, the calculator reads none of it, and
spelling leaves it alone. In a table's cell it stays its TeX.

dart_markdown reads plain words in runs until it is given an inline
syntax of its own, and then a word at a time, in case that syntax starts
at a space or a letter. The dollar sign's never does, so a syntax for
runs of words keeps the reading of a long note as quick as it was:
otherwise every word becomes a text node of its own, and the parse of a
5,000-line note takes more than twice as long.
A formula outside a table is now drawn by flutter_math_fork where its
TeX was, a tenth bigger than the words around it and in their colour:
among words on their baseline, and set apart in the middle of a line of
its own, centred across the writing column. With the caret in it, it
shows its TeX again. TeX it cannot read stays written out in code's
face, never an error box, and a formula wider than the column is drawn
smaller rather than cut off.

A text engine has to be told how big every placeholder is before it
breaks lines, and everything that measures the note apart from the field
has only the numbers, so each formula is laid out once in a widget tree
of its own that is never shown, and its size kept. Its widget is kept
too, one for every formula in the note: flutter_math_fork keys what it
builds with a GlobalKey held in the parsed TeX, so one widget in two
places would take its lines from the first.

What pandoc and Wikipedia write that flutter_math_fork does not read is
read as what it has: align, alignat and split become aligned and
alignedat, an equation's environment goes, \sgn is the sign function,
and a phantom of a sized delimiter written without braces gets them. A
formula's glyphs stay out of the semantics tree, where the note's text
already gives its TeX. A note with a formula gives up the forced
one-line strut, as one with a picture or a table does, so that moving
the caret onto a formula never moves every line.

flutter_math_fork is Apache-2.0. The KaTeX fonts it bundles, about
660 KB, are MIT, and their notice now ships with the app.
A fenced code block whose info string names one of 35 languages, or one
of highlight.js's names for them (py, ts, sh, yml and the like), is
coloured by re_highlight, highlight.js's grammars ported to Dart. Its
keywords, strings, numbers, comments, titles, types and punctuation take
their colours from the note's own palette, so code reads the same in
light and dark rather than in some editor's theme. A block is coloured
once for as long as its language and its code stay the same, so typing
in one block leaves the rest alone. A block with no language, or one it
does not know, stays plain code.
A <u> or <ins> closed among the same words underlines them, and <sup>
and <sub> raise and lower them. The tags hide like any inline marker,
showing again with the caret against them, and the calculator reads the
words without them. A tag that is never closed, or that crosses another,
stays the HTML it is written as, and other inline HTML looks as it did.

A text engine cannot move words off their baseline, so a superscript
uses the face's own raised glyphs where it has one for every character:
digits in Shantell Sans, and digits, small letters and +=() in the
system face on Apple's platforms. Anything else is drawn smaller on the
baseline. Either way the line keeps its height. A table's grid draws
them too, so its run style is now told the words it styles.
<br>, <br/> and <br /> in a GFM table's cell start a new line of it,
whether the grid fits its columns to the page or keeps their natural
widths, and the room the note keeps for each row is measured from the
same lines, so the two agree. At natural widths, where a cell is
otherwise cut to one line, each of its lines is cut short with an
ellipsis of its own, since a painter only shortens the last line it may
draw. Outside a table a <br> is shown as written, as before.

A random test now checks that a note mixing formulas, HTML, tables and
fenced code is read the same in pieces as whole.
One note with formulas among words and set apart, TeX that does not
parse, prices, <u>, <sup> and <sub>, a Python block, and a table with a
<br> and a formula in its cells: drawn in dark, in light, and in dark
with the caret in the display formula, which then shows its TeX.
@sanjaydotpro
sanjaydotpro force-pushed the capy/markdown-rendering branch from e4609b0 to 23bd2f0 Compare November 17, 2026 05:30
benchmark/markdown_rendering.dart opens a 5,000-line note with 350
formulas and 30 fenced code blocks three times, then types into it a key
every 33 ms in prose, in a code block and after a formula, and prints
how long each opening took to its first frame and how long the typing
frames took to build and to draw. It is an app rather than a test,
because only profile mode says anything about frame times, and its
header says how to run it under a bundle id of its own.

On an M3 Pro, with the semantics tree on as it always is in the app,
main opened the note in 1.6 to 1.8 s after its first opening and this
branch in 2.0 to 2.2 s, and a key in it took about 2 s to draw on both.
@sanjaydotpro
sanjaydotpro force-pushed the capy/markdown-rendering branch from 23bd2f0 to eadf170 Compare November 17, 2026 05:31

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant