diff --git a/servers/hades/Dockerfile b/servers/hades/Dockerfile index c0251e0..32ed628 100644 --- a/servers/hades/Dockerfile +++ b/servers/hades/Dockerfile @@ -10,7 +10,7 @@ RUN mkdir -p /opt/hades /opt/scripts /var/hades # Hades 2.x: single uberjar covering import / index / compact / serve for # SNOMED, LOINC and FHIR packages. -ADD https://github.com/wardle/hades/releases/download/v2.0.206/hades.jar /opt/hades/hades.jar +ADD https://github.com/wardle/hades/releases/download/v2.0.282/hades.jar /opt/hades/hades.jar COPY build-databases.sh /opt/scripts/build-databases.sh RUN chmod +x /opt/scripts/build-databases.sh diff --git a/servers/hades/README.md b/servers/hades/README.md index 2f8a5b9..7250e9d 100644 --- a/servers/hades/README.md +++ b/servers/hades/README.md @@ -26,7 +26,7 @@ The builder service consumes everything under `../../.tx-content/`: | ------------------------------------- | ------------------------------------ | | `SnomedCT_*.zip` | `/var/hades/snomed.db` (Hermes) | | `Loinc_*.zip` / `loinc-*.zip` | `/var/hades/loinc.db` (FTRM SQLite) | -| `*.tgz` (FHIR NPM packages) | `/var/hades/packages/-/` | +| `*.tgz` (FHIR NPM packages) | `/var/hades/fhir.db` (FTRM SQLite) | Multiple SNOMED zips (intl, US, UK) are imported into the same Hermes DB — the composite serves each module/version distinctly. @@ -40,20 +40,21 @@ path passed to `serve`: java -Xmx8g -jar hades.jar serve --port 8080 \ /var/hades/snomed.db \ /var/hades/loinc.db \ - /var/hades/packages/hl7.fhir.r4.core-4.0.1/package \ - /var/hades/packages/hl7.terminology.r4-7.0.1/package \ - ... + /var/hades/fhir.db ``` -FHIR packages are loaded **in-memory** rather than into SQLite — boot -takes ~15 s longer, but every CodeSystem / ValueSet / ConceptMap -lookup becomes a hashmap hit. With six standard HL7 packages this is -~600 MB of resident heap; the `-Xmx8g` ceiling gives comfortable -headroom for the in-memory data + Hermes' Lucene mmap caches + -transient `$expand` working sets. On RAM-constrained hosts, `hades -import fhir.db ` and `serve … fhir.db` (instead of the -package directories) keeps the resident footprint to ~80 MB at the -cost of small per-request JDBC overhead — see the +FHIR packages are built into a single **FTRM SQLite container** +(`fhir.db`) rather than served in-memory. SQLite is mmap'd, so the +*corpus at rest* lives off-heap, and the indexed/FTS query paths are +faster than scanning an in-memory corpus on search and intensional +`$expand`. Heap is still sized at `-Xmx8g`, though: `$expand` builds +each result set in heap to serialize it, so concurrent large +expansions (e.g. 200k-member VSAC/PHINVADS value sets) need multiple +GB regardless of where the corpus is stored — a measured ~6 GB working +set for 30 concurrent large expansions. The in-memory alternative — +serving the unpacked package directories instead of `fhir.db` — trades +that for hashmap-hit lookups and is the right choice only on large-RAM +hosts or for request-scoped overlays; see the [in-memory vs SQLite section](https://github.com/wardle/hades#in-memory-vs-sqlite-container) in hades' README. @@ -67,9 +68,9 @@ hades compact ## Loading Terminologies -Place each artefact in `tx-benchmark/.tx-content/`. The builder is -idempotent — it skips packages already extracted and rebuilds the -SNOMED/LOINC DBs only when `REBUILD_DB=1` or the destination is empty. +Place each artefact in `tx-benchmark/.tx-content/`. Set `REBUILD_DB=1` +to drop and rebuild `snomed.db` / `loinc.db` / `fhir.db` from scratch; +otherwise the builder imports into whatever containers already exist. ## Known limitations diff --git a/servers/hades/build-databases.sh b/servers/hades/build-databases.sh index 63a2fba..299a4db 100755 --- a/servers/hades/build-databases.sh +++ b/servers/hades/build-databases.sh @@ -4,16 +4,16 @@ set -eu DB_ROOT="${DB_ROOT:-/var/hades}" SNOMED_DB="${DB_ROOT}/snomed.db" LOINC_DB="${DB_ROOT}/loinc.db" -PKG_DIR="${DB_ROOT}/packages" +FHIR_DB="${DB_ROOT}/fhir.db" SOURCE="${SOURCE_DIR:-/source-data}" HADES="${HADES:-java -jar /opt/hades/hades.jar}" if [ "${REBUILD_DB:-0}" = "1" ]; then - rm -rf "${SNOMED_DB}" "${LOINC_DB}" "${PKG_DIR}" + rm -rf "${SNOMED_DB}" "${LOINC_DB}" "${FHIR_DB}" fi -mkdir -p "${DB_ROOT}" "${PKG_DIR}" +mkdir -p "${DB_ROOT}" EXTRACT_DIR="$(mktemp -d)" trap 'rm -rf "${EXTRACT_DIR}"' EXIT @@ -61,16 +61,28 @@ if [ "${found_loinc}" = "1" ]; then ${HADES} compact "${LOINC_DB}" fi -# FHIR packages are loaded in-memory at serve time, not pre-built into a DB. +# FHIR packages are built into a single FTRM SQLite container (fhir.db) +# rather than served in-memory: lower memory footprint and faster on the +# search / intensional-expand paths. `import` auto-indexes after each +# call, so pass `--no-index` while importing each package sequentially, +# then `index` + `compact` once at the end (compact vacuums only — it +# does not index). +found_fhir=0 for tgz in "${SOURCE}"/*.tgz; do [ -f "${tgz}" ] || continue name="$(basename "${tgz}" .tgz)" - dest="${PKG_DIR}/${name}" - if [ ! -d "${dest}" ]; then - mkdir -p "${dest}" - echo "Extracting FHIR package ${tgz} -> ${dest}" - tar xzf "${tgz}" -C "${dest}" - fi + dest="${EXTRACT_DIR}/pkg-${name}" + mkdir -p "${dest}" + echo "Extracting FHIR package ${tgz} -> ${dest}" + tar xzf "${tgz}" -C "${dest}" + echo "Importing FHIR package ${name} -> ${FHIR_DB}" + ${HADES} import --no-index "${FHIR_DB}" "${dest}" + found_fhir=1 done +if [ "${found_fhir}" = "1" ]; then + ${HADES} index "${FHIR_DB}" + ${HADES} compact "${FHIR_DB}" +fi + ls -la "${DB_ROOT}" diff --git a/servers/hades/docker-compose.yaml b/servers/hades/docker-compose.yaml index 495767e..a58a7ba 100644 --- a/servers/hades/docker-compose.yaml +++ b/servers/hades/docker-compose.yaml @@ -21,12 +21,9 @@ services: - > test -d /var/hades/snomed.db || { echo 'snomed.db missing — run builder first'; exit 1; }; set --; - for d in /var/hades/snomed.db /var/hades/loinc.db; do + for d in /var/hades/snomed.db /var/hades/loinc.db /var/hades/fhir.db; do [ -e "$$d" ] && set -- "$$@" "$$d"; done; - for d in /var/hades/packages/*; do - [ -d "$$d" ] && set -- "$$@" "$$d/package"; - done; exec java -Xmx8g -jar /opt/hades/hades.jar serve --port 8080 "$$@" ports: - "7006:8080"