From 25621c3a06d908440e36f9d89040cd82d1be8ac5 Mon Sep 17 00:00:00 2001 From: Lucas Vogel Date: Sun, 27 Sep 2026 21:45:21 -0400 Subject: [PATCH] docs: the note now carries what the posts promise Every launch post ends by pointing here, and the note was written six days before half of what they describe existed. Three gaps. Markdig was never named. The note said the words were "recovered with a Markdown parser", which is the vaguest possible phrasing of the strongest claim in the project. A .NET reader wants the library, and wants to know why it has to be one we did not write: our own reader would share our escaping assumptions, forget to interpret what the serialiser forgot to escape, and agree the document was fine. Shown now with the welded-emphasis case, where Markdig reads two words back as one token. The serialiser had one paragraph. It now has a section, because that is where the difficult problems are: two escaping passes, why code spans are deliberately not escaped, and the rule that drops formatting from a span with no letters or digits in it. Rendered faithfully, one document in the corpus would have lost 3,440 of its 4,664 words to an italic full stop. --report did not exist when this was written. It does now, and it is the only channel there is for documents we will never be allowed to see. Also corrects the performance section. It described the chained-predicate bug as the finding; the larger one is that any predicate in a match pattern is expensive and its content is irrelevant, which is a different defect, filed separately as phoenixmldb-xslt#95. The old text also promised the engine fix was probably done by now. It is not. Attribution fixed throughout: "one real document lost 3,440 words" had no subject and read as though docmd lost them. A naive serialiser would have; the coverage check is why it never shipped. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_018wZEgtuzaZPswykiGEBxbz --- docs/notes/introducing-docmd.md | 144 ++++++++++++++++++++++++++++++-- 1 file changed, 137 insertions(+), 7 deletions(-) diff --git a/docs/notes/introducing-docmd.md b/docs/notes/introducing-docmd.md index 2e3fece..0c3cf37 100644 --- a/docs/notes/introducing-docmd.md +++ b/docs/notes/introducing-docmd.md @@ -114,13 +114,75 @@ Torque to **40 Nm** before sealing. Notice `slug="vent-inspection"` and `ordered="false"` — the stylesheet has already decided the hard things, and the serialiser only has to write them down. +## The serialiser is where the hard problems live + +The stylesheet has the interesting ideas. The serialiser has the difficult ones, and they are +difficult because Markdown is context-sensitive: almost every decision depends on something +outside the thing being written. + +**Escaping runs in two passes.** Characters that can start markup anywhere get escaped as each +text node is written: + +```csharp +SearchValues.Create(['\\', '`', '*', '_', '[', ']', '<', '>', '|', '~']) +``` + +That set is deliberately narrower than CommonMark's full punctuation list. Escaping everything +is legal and produces backslash-strewn output nobody wants to read. But `#`, `-`, `>` and `1.` +only mean anything at the *start of a line*, and a single text node has no idea where it will +land — so line-leading escaping runs afterwards, over the assembled line. + +**Code spans are deliberately not escaped.** That is the entire point of a code span. The fence +widens instead when the content contains a backtick. + +**Adjacent markup is merged before anything is written**, which is the `**Safety****Review**` +case above. + +And then there is the rule that is the best argument for the whole split: + +```csharp +if (!trimmed.Any(char.IsLetterOrDigit)) { builder.Append(text); return; } +``` + +Word italicises a trailing full stop constantly, because of how a sentence gets selected rather +than anything the author meant. The md-XML records that faithfully. But `*.*` is valid in +isolation and breaks the instant it touches another character, leaving the asterisks visible as +text. Rendered faithfully, one document in our corpus would have lost 3,440 of its 4,664 words +to the pattern — and we know that number because the coverage check reported it, which is the +only reason it never shipped. + +So a span containing no letters or digits gets no delimiters at all. The formatting loses and +the words survive, which is the right way round. + +That one line is why the serialiser is C# and not more XSLT. It requires looking at the +*content* of a span, judging that an author's formatting cannot be expressed, and dropping it. +XSLT could express it. You would be writing character-class predicates in a language built for +tree dispatch, inside the file where you are trying to say what a document means — and Markdown +has a dozen rules of this shape. + ## Every conversion is measured `docmd` compares the words a reader can see in the `.docx` against the words recovered from its -own finished Markdown — recovered with a Markdown parser. That detail is the -design - if our escaping is wrong, an asterisk we failed to escape reads back as markup and the -word beside it goes missing. Our own code does not get a vote on whether our own output is -correct. +own finished Markdown — recovered with **Markdig**, which we did not write. That detail is the +design, and it is worth being precise about why. + +After writing the Markdown we need to know which words are in it, and that is itself a parsing +problem: `**bold**` contains the word *bold*, not the asterisks. If we answered that question +with our own reader, it would share our assumptions. Where the serialiser forgot to escape +something, the reader would forget to interpret it, and both halves would agree the document was +fine. Markdig has no idea what we meant. It knows CommonMark, which is what a reader's renderer +knows too. + +Here is that difference on a real bug. Word splits one bold phrase across several runs, and a +serialiser that joins them carelessly emits the second line: + +``` +**Safety** **Review** Markdig reads: [Safety | Review] correct +**Safety****Review** Markdig reads: [Safety****Review] one token, two words lost +``` + +The source had two words. The output has one. Coverage reports both as missing — and it reports +them because the checker is not us. When words go missing, `docmd` prints this to stderr and carries on converting: @@ -155,6 +217,52 @@ comes with it. The full account lives in [docmd's limitations page](https://github.com/phoenixmldb/docmd/blob/main/docs/limitations.md), and it is a dated measurement against a fixed corpus rather than a claim. +## Telling us, without sending us the document + +Measuring every conversion makes `docmd` the only instrument we have for documents we will never +be allowed to see. For a while none of that could reach us, and the reason was our own doing: +the warning printed on stderr welds the diagnostic to the content. + +``` +! 2 of 3754 words did not survive conversion (99.9 % kept). +! missing 'Here' near "the Discount List Click Here Click 'Add' to use" +! one of them sits inside . +``` + +The middle line is the user's document. Anyone willing to help had to redact it by hand or say +nothing, and documents that convert badly are disproportionately contracts, invoices and +internal procedures. So people said nothing. + +The half we actually need was always the safe half. `` is the entire actionable fact; +that the word was "Here" tells us nothing. `--report` prints that half: + +```console +$ docmd report.docx --report +docmd coverage report + docmd 0.2.3 · PhoenixmlDb.Xslt 2.0.0 · .NET 10.0.12 · linux-x64 + + 1 document(s), 3,754 words + 0 intact · 1 with losses · 2 lost (0.053 %) + + causes, by occurrences + drawing 1 + txbxContent 1 +``` + +No filename, no path, no document property, no word of the text. It aggregates across a folder, +so an organisation can audit its whole estate and send a single page. + +The renderer is never handed the text in the first place — it takes a type with no field capable +of holding it. "We strip the sensitive parts" is a claim that decays with every future edit; +"the sensitive parts are not in scope" is a property of the code's shape, and one a test can +check. + +One place it deliberately says less than it could. An element name from a published schema is +printed as itself, but a wrapper from a custom XML part reports only as `foreign` — because a +template author names those elements, and `AcmeCorpContractValue` is an ordinary thing to find +in a real document. You still learn that an unrecognised construct cost someone a word, which is +enough to start the conversation. + ## What a stylesheet gets wrong, and why we chose it anyway The single most useful thing we learned is this: **XSLT's built-in rules lose nothing.** Feed a @@ -204,15 +312,37 @@ something nobody has heard of. The point is that it now says so. `docmd` runs on our own [PhoenixmlDb.Xslt](https://github.com/phoenixmldb/phoenixmldb-xslt) engine, bringing the most recent standards to a long-neglected parsing engine, which gives us some interesting tradeoffs. -For example, one Word document took 124 seconds to parse, where comparable ones took merely two. The cause was a single match -pattern with two chained predicates: +For example, one Word document took 124 seconds to convert where comparable ones took two. The +cause was a single match pattern carrying two chained predicates: ```xml ``` -This may require some performance tweaking on our part, so we've filed a bug with ourselves to pore over for an upcoming release - hopefully, by the time you read this, we've already handled it. As we stretch into the long-tail of bugfixes in the parser, however, you can expect more solutions like this to come into the .NET Core ecosystem, though hopefully not just from us. +Identical output, 169 times apart. Rewriting that one pattern took the document to 19 seconds, +and no correctness test could have caught it, because the answer never changed. + +Chasing it further turned up something larger, and it is the more useful finding. **Any** +predicate in a match pattern is expensive on this engine, and the predicate's *content* turns +out to be irrelevant: + +``` +no predicate 51 ms on a 3,000-node document +w:p[@zzz] 2836 ms tests one attribute, matches nothing +w:p[A and B] 3065 ms walks every descendant twice +``` + +A predicate that touches a single attribute and matches nothing costs as much as one that walks +every descendant twice. That says the expense is in dispatch rather than evaluation, which is a +different bug from the one we started with and is +[filed accordingly](https://github.com/phoenixmldb/phoenixmldb-xslt/issues/95). The conjunctive +form is still the right thing to write — it buys a much smaller constant — it simply does not +buy linearity, which the original report implied and we have since corrected. + +Both are open against the engine. Building a product on your own parser is how you find the +defects a conformance suite cannot: conformance measures whether the answer is right, never what +it cost. Take a look at what we've put together, and feel free to reach out. If you're new to XSLT, we've built MCP servers for [XSLT](https://www.nuget.org/packages/xslt-mcp) and [XQuery](https://www.nuget.org/packages/xquery-mcp) on NuGet that can help you familiarize yourself (and your agent) with what we're putting together over here.