Skip to content

Microsoft Access data recovery support. - #116

Open
pplupo wants to merge 31 commits into
Euro-Office:mainfrom
pplupo:db-support
Open

Microsoft Access data recovery support.#116
pplupo wants to merge 31 commits into
Euro-Office:mainfrom
pplupo:db-support

Conversation

@pplupo

@pplupo pplupo commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Description

This PR introduces native support for importing and reading database and data-warehouse files directly into the spreadsheet editor. It adds unified data-extraction pipelines for:

  • Microsoft Access Databases (.mdb)
  • SQLite Databases (.sqlite, .db)
  • DuckDB Databases (.duckdb)
  • Parquet Files (.parquet)

Motivation

For a sovereign and independent office suite, true data sovereignty means ensuring users are never locked out of their historical data. Legacy Microsoft Access (.mdb) databases hold decades of critical public, enterprise, and personal records. Relying on proprietary or Windows-exclusive software to open them poses a severe risk to long-term data accessibility. By natively supporting .mdb file reading, Euro-Office guarantees that users can independently recover and migrate their historical data at any time.

Furthermore, once the unified database-to-spreadsheet conversion pipeline was built to support .mdb recovery, it was a natural and low-overhead extension to include support for modern embedded databases like SQLite, as well as analytics-heavy formats like DuckDB and Parquet. This empowers data analysts and researchers to quickly inspect massive datasets directly from the comfort of their office suite, bridging the gap between legacy data recovery and modern data science.

Technical Details

  • Vendored libmdb (from mdbtools) directly into the source tree to ensure consistent cross-platform builds without relying on fragmented system packages (resolves iconv and locale struct mismatches).
  • Integrated libduckdb for reading DuckDB files and efficiently querying .parquet files via DuckDB's view functionality.
  • Added standard SQLite file parsing.
  • Built a unified translation layer that iterates through database tables and seamlessly converts them into individual worksheet tabs within the spreadsheet engine.

@pplupo
pplupo requested a review from a team as a code owner June 30, 2026 12:42
@pplupo
pplupo requested review from DmySyz and chrip and removed request for a team June 30, 2026 12:42
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
pplupo added 9 commits July 7, 2026 18:21
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Our vendored libmdb sources (libmdb/*.c) use a newer MdbHandle struct
that includes the 'locale' field. The system mdbtools-dev on Ubuntu Jammy
ships an older version without this field, causing 'MdbHandle has no member
named locale' errors.

Since all libmdb .c files are compiled directly into BinDocument, there
is no need to link against the system libmdb (-lmdb). Remove the linkage
to eliminate the struct mismatch.

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
iconv.c has two code paths:
- #ifdef HAVE_ICONV: uses iconv_t (standard POSIX, no locale field needed)
- #else: uses mdb->locale (requires newer MdbHandle with locale field)

The system mdbtools-dev on Ubuntu Jammy has an older MdbHandle without
the 'locale' field. Defining HAVE_ICONV selects the iconv_t code path
which is the correct choice on Linux (iconv is always available via glibc)
and avoids the missing struct member error entirely.

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Two fixes for iconv.c build errors with HAVE_ICONV defined:
1. Add 'ICONV_CONST=' compile definition - on Linux glibc, iconv()
   takes non-const char**, so ICONV_CONST must be empty
2. Add libmdb/mdbtools.h shim that redirects to our vendored
   mdbtools/mdbtools.h - ensures libmdb sources always use our
   version of MdbHandle (with iconv_in/iconv_out fields) regardless
   of what system mdbtools-dev provides

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
pplupo and others added 14 commits July 7, 2026 22:01
BinDocument's CMakeLists.txt has always listed these 20 libmdb/*.c
files as sources, and MdbEngine was switched to link against them
instead of the system mdbtools-dev package, but the .c files
themselves were never committed -- only the accompanying headers.
Vendored verbatim from the official mdbtools v1.0.1 release, matching
the version string already present in the committed mdbver.h.

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Two separate build-hygiene/licensing fixes for the database engines:

- DuckDB was vendored as raw prebuilt binaries (libduckdb.so 51MB,
  libduckdb_static.a 63MB) committed directly to the repo. Switch to
  vcpkg's duckdb port instead, matching how sqlite3 is already pulled
  in. Removes ~114MB from history; DuckDbEngine.cpp only ever used the
  public C API (duckdb.h), so no vendored C++ header was needed either.

- mdbtools (LGPLv2+) was compiled directly into BinDocument's static
  library. Static-linking an LGPL library into an AGPL application
  without satisfying LGPL section 6 (relinking) is a licensing risk.
  Build it as its own shared library (mdbtools.so) and link BinDocument
  against it dynamically instead -- the standard way to consume an LGPL
  dependency from a differently-licensed application.

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
The duckdb port exports find_package(DuckDB CONFIG REQUIRED) (capitalized,
per its own usage file) with a plain, unnamespaced 'duckdb'/'duckdb_static'
target -- not 'duckdb::duckdb'. Verified against the port's usage file on
the build server rather than guessed.

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
…b move

x2tlib links BinDocument (a static library) and therefore needs its own
direct links to duckdb and mdbtools -- PRIVATE dependencies of a static
library don't propagate to its consumers. It still had an IMPORTED
duckdb pointing at the now-deleted vendored .so, plus a reference to a
target named mdb that was never actually defined anywhere. Point both
at the real targets and drop the now-invalid manual libduckdb.so copy step.

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
The project sets -fvisibility=hidden globally via set_default_options(),
which silently hid every mdb_* C API symbol from mdbtools.so's exported
symbol table -- the functions were compiled in but not linkable from
outside the .so, causing 'undefined reference to mdb_open' etc. at the
final executable link step despite mdbtools being correctly listed as
a dependency. Override back to default visibility for this target,
since its entire purpose is external linkage.

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Hendrik Leidinger <hendrik.leidinger@nextcloud.com>
Assisted-by: Claude Code:Opus 4.8
Add x64-linux-v2 (default) and x64-linux-baseline overlay triplets and
thread ARCH_TRIPLET/ARCH_MARCH_FLAGS build args through the core cmake
invocation, so downstream builders can compile against plain x86-64
without the team maintaining a second officially-supported toolchain.

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
…d-migration

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>

# Conflicts:
#	.docker/core.bake.Dockerfile
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
…arch-baseline

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>

# Conflicts:
#	.docker/core.bake.Dockerfile
…d-migration

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
@chrip

chrip commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Thanks for putting this together, @pplupo and for the care that clearly went into it. The mdbtools LGPL handling and the engine abstraction are solid pieces of work.

But my personal read is that this isn't a fit for core. Pulling database recovery into core means a bigger shipped build, a new parser of untrusted binary input in our default attack surface, and ongoing maintenance of the DB engines for something we don't currently have a use case for. That's a real cost, and I don't think it's one we should take on right now.

I'll raise this PR at our next team meeting and put the scope question to everyone. I'll follow up here once we've talked it through.

@pplupo

pplupo commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

@chrip Thanks for taking the time to review this.

Regarding the attack surface, the risk is heavily mitigated by how the abstraction works. The parser strictly extracts raw data from these files; it does not execute any embedded logic or binaries. The extracted information is purely treated as a data source and fed directly into the spreadsheet engine.

On the question of the use case, this feature directly addresses the core philosophy of Euro-Office: true data sovereignty. For users with decades of critical records locked in MS Office Access mdb files, we need to provide an independent recovery path to break that vendor lock-in. There is a demonstrated market need for this, which is why alternatives like LibreOffice already support it. It is a powerful enabler for users migrating to our ecosystem.

IMHO, this is a must-have.

I appreciate you bringing this to the team meeting, and I look forward to hearing the team's thoughts.

@chrip

chrip commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

@pplupo This PR prompted a discussion about what Euro-Office stands for and the direction we want to take it in. As the outcome, I was convinced that we shouldn't reject a feature this valuable. Freeing data from formats like Access is exactly what it is about, and we build with the community, so a useful feature that someone is willing to implement belongs here.

The one issue is timing. We're mid-stabilization on the project right now and don't want to land a change this size on an unstable trunk, so we're putting it on hold rather than merging now. We'll keep the PR open and pick it back up once things are stable.

@chrip
chrip removed their request for review July 24, 2026 10:20
pplupo added 7 commits July 27, 2026 11:32
Adds a BerkeleyDbEngine implementing the existing IDatabaseEngine
interface, alongside SqliteEngine/DuckDbEngine/MdbEngine. Berkeley DB
is a schema-less key/value store, so each table is exposed with a
synthetic {Key, Value} schema and no primary/foreign keys.

- New extension .bdb, plus a content sniff (SQLite's 16-byte magic
  header) to disambiguate the already-ambiguous .db extension between
  SQLite and Berkeley DB.
- Registers AVS_OFFICESTUDIO_FILE_SPREADSHEET_BDB and wires it through
  OfficeFileFormatChecker2's detection/extension tables.
- Links unofficial::berkeleydb::libdb (vcpkg, pinned to 18.1.40, which
  is licensed AGPLv3 same as this application) into BinDocument and
  x2tlib, mirroring the existing sqlite3/duckdb integration.

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
vcpkg's berkeleydb port only carries the old 4.8.30 (Sleepycat
License) release; there is no port revision for the 6.0.20+ line
that Oracle relicensed under AGPLv3 (the same license as this
application), which is what the BerkeleyDB support was built
against. Vendor the upstream source verbatim under
DatabaseEngines/berkeleydb/ (matching the existing mdbtools
precedent for vendoring a dependency vcpkg can't supply) and drive
its own autotools build via ExternalProject_Add, producing an
imported 'berkeleydb' target that BinDocument and x2tlib link
against in place of the vcpkg package.

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
…I utils, GCC strictness)

Three issues surfaced by an actual build attempt:

- The repo-wide '**/Makefile.*' gitignore rule silently dropped
  dist/Makefile.in from the previous commit despite 'git add' reporting
  success, breaking configure's config.status. Force-add it, and vendor
  the one other file config.status needs that lives outside src/dist
  (test/tcl/include.tcl), rather than the whole test/ tree.
- BDB's default 'all'/'install' targets also build/install the CLI
  utilities (db_archive, db_dump, ...), whose sources under util/ were
  deliberately not vendored (unneeded for a read-only import engine).
  Build/install the library-only targets instead.
- BDB 18.1.40's K&R-style C trips -Wincompatible-pointer-types, which
  this GCC treats as a hard error by default; pass
  CFLAGS=-Wno-error=incompatible-pointer-types, the standard workaround
  for building this codebase with a modern compiler.

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
Qt 6.10.1 (fetched prebuilt via aqtinstall) requires libicuuc.so.73;
core's own vendored ICU was pinned to 74, so any process descending
from the Qt-linked desktop app (which loads both versions into the
same process tree) ends up with both ICU 73 and 74 loaded side by
side. This corrupts state in a way that's hard to pin down exactly,
but is 100% reproducible: x2t, invoked via fork/exec as a direct
child of DesktopEditors, silently fails to parse its own task XML
despite reading the correct bytes (confirmed via strace) -- while the
identical operation succeeds when x2t is invoked from any other
parent, or when an extra process generation is inserted between
DesktopEditors and x2t (e.g. via gdbserver).

Pinning core's vendored ICU to 73 instead of 74 keeps a single
consistent ICU version across the whole process tree.

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
ICU only gained a top-level LICENSE file starting around release 74;
release-73-1's LICENSE lives at icu4c/LICENSE (which exists in both
old and new releases), so the version bump to 73 broke the vendoring
script's license copy step.

Signed-off-by: Peter P. Lupo <pplupo@gmail.com>
RetGal pushed a commit to RetGal/core that referenced this pull request Aug 4, 2026
nc-build.py cloned depot_tools then ran 'git pull origin main', tracking HEAD.
gclient_paths.patch (drops four @functools.lru_cache decorators from gclient_paths.py)
was authored against an older depot_tools; upstream f065bb3b0 (2026-07-13, 'Add gclient
getconfig subcommand') reworked gclient_paths.py, so the patch stopped applying and core
failed to build on all arches (blocking #143 and Euro-Office#116).

Pin depot_tools to 6e5a13d2598ee48c9c7afc750401533f30dde16e (newest revision the patch
applies against; verified with git apply --check) and set DEPOT_TOOLS_UPDATE=0 so it
can't self-update back to HEAD during gclient sync. Verified: core builds and x2t links
on linux/arm64.

Assisted-by: ClaudeCode:claude-opus-4-8
Signed-off-by: Christoph Schaefer <christoph.schaefer@nextcloud.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants