Sanitizer strips inner MathML/AsciiMath from <stem>, losing notation content on YAML round-trip

[opoudjis]

Summary

Relaton::Bib::Sanitizer.sanitize strips MathML (and any other non-basicdoc inline markup) from inside <stem> elements, even though <stem> itself is preserved. The result is that bibliographic titles containing <stem> lose the entire mathematical or technical-notation content of the stem on YAML round-trip.

Reproducer

require "relaton/bib"

input  = "Prefix <stem><math><mi>d</mi><mn>6</mn></math><asciimath>d_6</asciimath></stem> Suffix"
output = Relaton::Bib::Sanitizer.sanitize(input)

puts input
puts output

Output:

Prefix <stem><math><mi>d</mi><mn>6</mn></math><asciimath>d_6</asciimath></stem> Suffix
Prefix <stem>d6d_6</stem> Suffix

The <stem> wrapper survives (it's in ALLOWED), but <math>, <mstyle>, <msub>, <mi>, <mn>, and <asciimath> are all unwrapped. Only their text content remains: d, 6, d_6 — concatenated, with no element markup to disambiguate.

End-to-end symptom

Downstream, metanorma-generic's spec Metanorma::Generic::BibdataConfig preserves embedded MathML when deserialising a bibdata title (spec) fails. Expected YAML:

content: |-
  Internal Standard Reference Data for qNMR: 4,4-Dimethyl-4-silapentane-1-sulfonic acid-<stem block="false" type="MathML">
    <math>
      <mstyle displaystyle="false">
        <msub>
          <mi>d</mi>
          <mn>6</mn>
        </msub>
      </mstyle>
    </math>
    <asciimath>d_6</asciimath>
  </stem> [ISRD-07]

Actual YAML (post-Sanitizer):

content: "Internal Standard Reference Data for qNMR: 4,4-Dimethyl-4-silapentane-1-sulfonic
  acid-<stem block=\"false\" type=\"MathML\">\n  \n    \n      \n        d\n        6\n
  \     \n    \n  \n  d_6\n</stem> [ISRD-07]"

The MathML structure is gone; only whitespace skeleton and the bare text nodes (d, 6, d_6) survive.

Root cause

Sanitizer.sanitize_children recurses into every child element before deciding whether to unwrap. It checks the ALLOWED set only at the parent level, so the recursion strips disallowed descendants of <stem> (which is itself allowed) before the parent-level allow-check can spare them.

Current logic (lib/relaton/bib/sanitizer.rb, lines 31–39):

def self.sanitize_children(node)
  node.children.to_a.each do |child|
    next unless child.element?

    child.name = RENAME[child.name] if RENAME.key?(child.name)
    sanitize_children(child)                                  # ← always recurses
    child.replace(child.children) unless ALLOWED.include?(child.name)
  end
end

For <stem>, the recursion is wrong: <stem> is an opaque container holding non-basicdoc inline notation (MathML / AsciiMath / LaTeX), not basicdoc inline markup that needs sanitising. Its contents should be preserved verbatim.

Proposed fix

Add an OPAQUE set for elements whose contents are out-of-band notation, and skip recursion into them:

# Elements whose contents are non-basicdoc inline notation (MathML,
# AsciiMath, LaTeX) and must be preserved verbatim rather than sanitised
# against ALLOWED.
OPAQUE = %w[stem].freeze

def self.sanitize_children(node)
  node.children.to_a.each do |child|
    next unless child.element?

    child.name = RENAME[child.name] if RENAME.key?(child.name)
    next if OPAQUE.include?(child.name)   # opaque: leave inner XML untouched
    sanitize_children(child)
    child.replace(child.children) unless ALLOWED.include?(child.name)
  end
end

This preserves the existing semantics for every ALLOWED element except <stem>, and converts <stem> from "strip everything inside that isn't in ALLOWED" to "leave inner content untouched".

Test plan

A new spec case covering the MathML-inside-<stem> reproducer above, plus the existing AsciiMath-inside-<stem> shape if not already covered. The downstream metanorma-generic spec then passes once a new relaton-bib release lands.

🤖

[ronaldtse]

It does not seem that the <stem> element is part of our grammar here.

Can you explain why Metanorma needs Relaton to accept <stem>?

To be transparent, Relaton will no longer accept random XML hacks from now on because it fully lutaml-model based.

[opoudjis]

Bibliographic entries routinely include mathematical expressions as plain text in both titles and abstracts. In fact, the test case that catches this is there because a BIPM document with MathML in its title and abstract triggered a crash on Relaton processing of XML by Metanorma.

A Relaton using Lutaml-Model that ignores mathematical content is not fit for purpose.

Relaton uses basic-models for its grammar for text, and stem is absolutely part of the grammar: https://github.com/metanorma/basicdoc-models/blob/7206afee7233ee425f4344d12d9d082283a5ab97/grammars/basicdoc.rnc#L1055 .

MathML is currently encoded in the grammar as any-element. This can be addressed by forcing ingest of the entire MathML grammar as a separate RNC, included in the basic-models grammar.

The issue of extraneous namespace models has come up several times in the past with Lutaml-Model, and I have not changed my view that the inflexibility of Lutaml-Model towards arbitrary XML is a liability that will compromise it repeatedly in the future. But the need of Lutaml-Model to deal with MathML is not an abstraction, it is a current failing, masked by the fact that it is infrequent.

The test case this has crashed against is Relaton XML being read back into the Relaton lutaml-model, in order for its content to be accessible for Liquid templating. Requests from @petervwyatt mean this functionality should be generalised from metanorma-generic to all of Metanorma, as was always intended.

This initiative is harnessing the strategic move of Relaton towards a modelling language. What it models needs to include MathML.

In order to expedite this issue, I am going to change the encoding in basicdoc-models of stem, to use all-of-MathML (including Semantic MathML) explicitly, by including the MathML grammar, and namespace support: metanorma/basicdoc-models#35.

Lutaml-Model has to deal with extraneous namespaces and MathML. They are text it processes, and they will keep being text it processes. If "Relaton will no longer accept random XML hacks", then it will accept namespaced MathML, because the requirement is not going to go away.