Skip to content

fix!: complete and faithful lossless output - #217

Merged
bc-m merged 1 commit into
bc-m:mainfrom
mislavkos:fix/lossless-output
Oct 30, 2026
Merged

bc-m merged 1 commit into
bc-m:mainfrom
mislavkos:fix/lossless-output

Conversation

@mislavkos

Copy link
Copy Markdown
Contributor

Summary

--lossless is meant to keep everything in the export. Rebuilding exports from lossless output (see the follow-up PR) showed where it didn't:

  • Persistent stores (PersistentStoreCatalog, FileMaker 2026, SaXML 2.3.0.0) weren't exploded, so their values were lost. They now go to persistent_stores/.
  • Carriage returns in calculation text were dropped where they were part of a CRLF line break, because prepare_xml_content split with str::lines().
  • Indentation of calculation text: removing an item's indentation also removed tabs from continuation lines of CDATA and multi-line text values. Only markup lines are unindented now.
  • The XML declaration is kept as the first line of the skeleton, since it differs between FileMaker versions.
  • Modified fields (FileMaker 20–22): the ModifyAction section of these exports has one FieldCatalog per field. All of them were written to tables__modify_action/<table> - ID <id>.xml, so each overwrote the previous one and only the last field of each table survived. They're now written to tables__modify_action/<table> - ID <id>/<field> - ID <id>.xml, and the skeleton names the field as well, so the order is kept.
  • File names derived from the content (accounts, relationships, libraries, modified layouts) are escaped like all other file names. A / in such a name made the write fail, so the file was missing.

Breaking change

Marked fix! because the output layout changes for FileMaker 20–22 exports: modified fields move from one file per table to one file per field. Anything that reads tables__modify_action/<table> - ID <id>.xml by path needs to follow. The first run after upgrading produces a large one-time diff, mostly fields that were missing until now: in one real-world export, about 1,500 new field files. Releasing this as 0.10.0 rather than a patch might be the clearer signal.

Other one-time changes on upgrade:

  • persistent_stores/ appears for FileMaker 2026 exports that have persistent stores.
  • Every skeleton.xml gains the declaration line (lossless only).
  • Calculations with CRLF line breaks gain their \r.
  • Files named after content with characters like | or ? are renamed, e.g. | becomes _.

Testing

  • Unit tests for each fix; snapshots updated (declaration line, per-field files for fmSyntaxColorizer).
  • Exploded 82 real exports (FileMaker 20 to 2026, UTF-8 and UTF-16, single files and split exports) with --lossless and rebuilt each from the output: all identical to the originals.
  • About 3–5% slower on a 330 MB export.

- Explode PersistentStoreCatalog (FileMaker 2026) into persistent_stores
- Keep carriage returns and the indentation of calculation text
- Keep the XML declaration in the skeleton
- Write each modified field of a FileMaker 20-22 export to its own file;
  they all overwrote one file per table, so most of them were missing
- Escape file names derived from the content, like all other file names

BREAKING CHANGE: modified fields of FileMaker 20-22 exports move from
tables__modify_action/<table> - ID <id>.xml to
tables__modify_action/<table> - ID <id>/<field> - ID <id>.xml. The first run
after upgrading changes many files once, mostly by adding the fields that
were missing until now.
@bc-m
bc-m merged commit 4a94b5a into bc-m:main Oct 30, 2026
1 check passed
@mislavkos
mislavkos deleted the fix/lossless-output branch November 3, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants