LogMapLLM Experimental

This repository contains experiment data and on-going work related to LogMapLLM, a project that extends the LogMap ontology matching system with an LLM oracle.

LogMapLLM

Expert knowledge is often required for effective ontology matching. LogMap has an interactive matching mode for this purpose, where uncertain mappings, denoted $M_{ask}$, are escalated for review by a human oracle. It is plausible, however, to replace the human expert by a large language model (LLM); a paradigm known as LLMs-as-Oracles. By assembling prompts for each correspondence in $M_{ask}$ and consulting an LLM via an OpenAI-compatible API (e.g., via OpenRouter, to vLLM, etc.), we fold the response back into the proposed alignment before refinement (i.e., the repair step).

Experimental features (this repository):

• extended prompt construction (class, property, instance prompt templates),
• few-shot ontology-driven prompting with RAG-based features,
• collective anchor-based few-shot ontology-driven prompting,
• and self-hosted, open-weight-based evaluation.

Usage

python -m logmap_llm --config path/to/config.toml

with a config like:

[alignmentTask]
task_name = "mouse-human"
onto_source_filepath = "data/mouse.owl"
onto_target_filepath = "data/human.owl"

[oracle]
model_name = "qwen/qwen3-32b"
api_key = "ENV:OPENROUTER_API_KEY"    # or "EMPTY" for a local vLLM/SGLang server

[outputs]
logmapllm_output_dirpath = "output/logmapllm"
logmap_initial_alignment_output_dirpath = "output/initial-alignment"
logmap_refined_alignment_output_dirpath = "output/refined-alignment"

# optional: score the refined alignment against a reference
[evaluation]
evaluate = true
reference_alignment_path = "data/reference.rdf"

The config is a TOML file validated by a Pydantic schema; at minimum it needs [alignmentTask] (task name and the two ontology paths), [oracle] (model name), and [outputs] (three output directories). You also need a LogMap bundle on disk (logmap-matcher-4.0.jar, java-dependencies/, parameters.txt), found under ./logmap by default or wherever alignmentTask.logmap_parameters_dirpath points.

The CLI flags:

• --config PATH (or -c) — the TOML configuration file; defaults to configs/default_config.toml.
• --reuse-align — reuse an existing LogMap alignment instead of re-running the matcher.
• --reuse-prompts — reuse previously built oracle prompts; implies --reuse-align.
• --no-cache — disable owlready2 quadstore caching and parse the ontologies from scratch.
• --run-root DIR — root all outputs under DIR, creating logmapllm-outputs, logmap-initial-alignment and logmap-refined-alignment subdirectories.

The run executes five phases in order: (1) align, (2) prompt build, (3) oracle consultation, (4) refinement, and (5) evaluation; followed by reporting. Prompt building and evaluation each run in their own subprocess, since owlready2 and JPype contained to a single process has be known to cause problems; we also separate the DeepOnto evaluator into its own JVM spawned by a subprocess when used.

Project Structure

logmap_llm/
├── __main__.py       python -m logmap_llm -> calls -> pipeline.runner.main()
├── config/           contains the TOML schema (Pydantic) and loader
├── pipeline/         the five-phase driver: runner, orchestration, paths, reporting
├── oracle/           LLM client (manager.py) and threaded consultation campaign
│   ├── prompts/      user/developer prompt templates per entity lane (class/property/instance)
│   └── rag/          query-specific few-shot retrieval from LogMap anchors
├── evaluation/       alignment and oracle scoring, run as a subprocess harness
├── experiments/      experimental batch harness: batch.toml -> sealed batch directory
├── ontology/         owlready2 access layer, annotation indices, sibling retrieval
├── bridging.py       LogMap's Java MappingObjectStr <-> pandas DataFrames
├── interface.py      JPype wrapper around LogMap's LogMapLLM_Interface
├── constants.py      shared enums, column names, separators, defaults
└── utils/            logging, atomic writes, subprocess bootstrap

Roadmap

• Refined implementation (beyond experimental, on-going work).
• Remove owlready2; opt to use LogMap itself for ontology access.
• Easily sharable embedding stores for community reuse.
• Improved documentation.

License

Apache License 2.0