metaforge

Forensic metadata extraction. Any file in, structured records out — with provenance on every single value.

Most metadata tools tell you what they found. metaforge also tells you where it came from, how confident that is, and what it could not collect — because a timestamp without a source is not evidence, it is a rumour.

---

Table of contents

• Why
• Requirements
• Installation
• Usage
• Extractors
• Output format
• Design principles
• Findings it surfaces
• Platform notes
• Evidence handling
• Extending
• Limitations
• Roadmap
• License

---

Why

Three problems this tool exists to solve honestly:

Provenance. Every normalized value carries the extractor, its version, the library used, and the original raw key it came from. When someone asks "where did this timestamp come from?", the answer is in the record, not in your memory.

Timezones. EXIF DateTimeOriginal has no timezone. None. Converting it to UTC means inventing an offset, and inventing precision is indefensible in a report. metaforge stores the original string, the parsed UTC value only when the offset is genuinely known, and a flag saying which.

Blind spots, stated out loud. Every record says which platform produced it and what that platform could not collect — NTFS $MFT entry-modified time, POSIX ownership on Windows, alternate data streams under WSL.

---

Requirements

Requirement	Version	Notes
Python	3.11 or newer	Uses StrEnum and modern typing syntax
pydantic	2.x	The only hard dependency
python-magic	0.4.27+	Optional. Improves type detection. Awkward on Windows — skip it there, the built-in signature table covers the common formats

Every extractor is pure standard library. No exiftool, no Perl, no native binaries.

---

Installation

Windows / PowerShell

git clone https://github.com/MafiaBV1/metaforge.git
cd metaforge
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -e .

Linux / macOS

git clone https://github.com/MafiaBV1/metaforge.git
cd metaforge
python -m venv .venv && source .venv/bin/activate
pip install -e .
pip install -e ".[magic]"      # optional, better type detection

Installing with -e . gives you a metaforge command. Without installing, python -m metaforge works the same from the project root.

---

Usage

metaforge <path> [--text] [--pretty] [--raw] [--kind KIND] [--list-extractors]

Flag	Effect
--text	Human-readable report with a NOTABLE section per file
--pretty	Indented JSON instead of one dense line
--raw	Include the raw extraction blob alongside normalized fields
--kind KIND	Only emit records of this kind. Repeatable
--list-extractors	Print registered extractors and their versions, then exit

Examples

# Readable report for one file
metaforge suspicious.pdf --text

# Full JSON case file for a whole directory
metaforge .\evidence --pretty --raw > case001.json

# Only image records
metaforge .\photos --kind file.image.jpeg --pretty

# What can this build actually parse?
metaforge --list-extractors

Directory scans walk in sorted order, so two runs over the same tree produce records in the same sequence.

---

Extractors

Extractor	Applies to	Notable output
filesystem	everything	MACB timestamps, ownership, entropy, content-vs-extension mismatch
indicators	everything	ASCII + UTF-16 strings, URLs, routable IPs, registry keys, GUIDs, AWS keys, JWTs, private-key markers
ntfs-ads	everything (Windows)	Alternate data streams, Zone.Identifier download URL and referrer
png	PNG	Chunk sequence, tEXt/zTXt/iTXt, eXIf, CRC errors, data appended after IEND
jpeg	JPEG	EXIF, GPS with altitude and fix time, camera and lens serials, XMP edit history, data after EOI
pdf	PDF	/Info and XMP including inside compressed object streams, revisions via /Prev, active content with byte context, embedded URLs
ooxml	docx / xlsx / pptx / odf	Authorship trail, editing minutes, macros, external template references
zip	zip / jar / apk	One record per entry, path traversal, encryption, compression ratio
gzip	gzip	Original filename and mtime of the compressed file
pe	exe / dll / sys	Linker timestamp, PDB path, section entropy, ASLR/DEP/CFG mitigations
elf	ELF	Class, machine, interpreter, toolchain comment
email	.eml	Received chain, X-Originating-IP, per-attachment records with hashes
mp4 / id3	mp4 / mov / mp3	mvhd times using the 1904 epoch, device tags, ID3v2 frames

---

Output format

One source yields 1..N records. A JPEG yields one. A zip yields one per entry. An email yields one per attachment. A pcap would yield one per request.

{
  "record_id": "b78e536a2bc5851a09de3c0b93197ab9",
  "kind": "file.document.pdf",
  "index": 0,
  "source": {
    "path": "...",
    "sha256": "...",
    "md5": "...",
    "size": 232716,
    "mime": "application/pdf",
    "mime_method": "signature",
    "acquired_at": "2026-07-24T22:02:29Z"
  },
  "fields": {
    "created": {
      "value": {
        "raw": "D:20230722142936+02'00'",
        "utc": "2023-07-22T12:29:36Z",
        "offset_known": true,
        "clock": "embedded"
      },
      "provenance": {
        "extractor": "pdf",
        "extractor_version": "0.1.0",
        "library": "stdlib",
        "raw_key": "PDF./Info/CreationDate"
      }
    }
  }
}

record_id is deterministic: the same evidence produces the same id on any machine, forever.

Timestamps

Field	Meaning
raw	The value exactly as stored in the file
utc	Parsed UTC — null when the offset was not knowable
offset_known	Whether the file actually stated a timezone
clock	filesystem, embedded, transmitted, or unknown

Filesystem and embedded clocks disagree constantly, and that disagreement is often the evidence.

---

Design principles

1. No value without provenance. FieldValue requires a Provenance, so there is no convenient path that skips it.
2. Raw is kept. Normalization is lossy; the raw extraction sits alongside it as ground truth.
3. Deterministic output. Same input, same bytes out. Sorted keys, sorted directory walks, stable ids.
4. Extractors stream. extract() is a generator, so a 2 GB pcap does not have to fit in memory before the first record lands.
5. Failure is a record, not a crash. A malformed file produces an error.extraction record with the traceback. One bad parser never costs you the other 4,000 files.
6. Absence over invention. A field that cannot be determined is omitted, not guessed. Windows reports no uid rather than the zero Python invents.

---

Findings it surfaces

These are the fields worth grepping for:

Field	Why it matters
extension_matches_content	false means a renamed file — a PNG called invoice.txt
entropy_bits_per_byte	Above ~7.5 means compressed, encrypted, or packed
trailing_bytes_after_iend / _eoi	Data past the end of the image. Nothing renders it; plenty hides in it
download_url / referrer_url	Where a file came from, straight out of NTFS
active_content_context	The bytes around /JavaScript, so you can judge the hit instead of trusting it
contains_macros	VBA project inside an Office document
external_references	Remote template loads from a document
path_traversal_entries	Zip Slip
high_entropy_sections	Packed or encrypted PE section
pdb_path	Developer's build path — routinely leaks a username
original_filename	What a .gz used to be called
access_time_set_by_this_examination	The atime you are looking at was set by this tool, not the suspect

---

Platform notes

Run metaforge on the same platform as the evidence.

os.stat uses identical field names across platforms and means different things by them:

Field	POSIX	Windows
st_ctime	inode metadata change	file creation
st_birthtime	macOS/BSD only	3.12+
st_uid / st_gid	real owner	always 0, meaningless
st_ino	inode	NTFS file index

metaforge labels each by what it actually is on the running platform and records that platform on every record.

Native Windows additionally reads NTFS alternate data streams. These are invisible through WSL's drvfs layer, and examining NTFS evidence from WSL also yields synthetic uid/gid/mode values that are artifacts of the translation layer rather than facts about the evidence.

---

Evidence handling

Acquisition opens files read-only and never writes, renames, or changes permissions. But reading a file updates its access time on most filesystems — unavoidable from userspace. When metaforge detects that it has done this, it says so in access_time_set_by_this_examination rather than leaving a misleading timestamp in your output.

For real evidence: work from a write-blocked copy or a forensic image, and mount read-only (-o ro,noatime).

---

Extending

One file in extractors/, one import line in extractors/__init__.py. Nothing else changes.

from ..models import FieldValue, Record, Source
from ..registry import ExtractorBase, register

@register
class MyExtractor(ExtractorBase):
    name = "my-format"
    version = "0.1.0"
    library = "stdlib"
    mimes = ("application/x-my-format",)   # "*/*" runs on everything
    kind = "file.myformat"

    def extract(self, source: Source):
        fields = {
            "some_field": FieldValue(
                value=parsed_value,
                provenance=self.provenance("Spec.FieldName"),
            )
        }
        yield Record.build(source=source, kind=self.kind, fields=fields)

Import order in extractors/__init__.py is record order.

---

Limitations

Stated plainly, because a forensic tool that hides its gaps is worse than one that has none:

• No test suite. The parsers have been exercised against generated files with known metadata, but nothing pins that behaviour against regressions.
• NTFS $MFT entry-modified time is not collected. os.stat does not expose it; reaching it requires parsing $MFT directly.
• PDF encryption is detected, not defeated. Encrypted content is reported as encrypted and left alone.
• pe, mp4 and id3 are unverified against real-world samples at time of writing.
• Indicators are candidates, not conclusions. A URL in a file means the bytes are there. It does not mean the file contacted it.
• String scanning caps at 8 MB per file, flagged as scan_truncated.

---

Roadmap

• SQLite case store with cross-artifact correlation (same camera serial, same author, same document ID across files)
• Timeline reconstruction across a whole directory
• GPS to GeoJSON export
• Pixel-plane analysis for images (LSB entropy, bit-plane uniformity)
• pcap / HTTP request records
• SigMF signal capture metadata
• Chain-of-custody audit log

---

License

MIT — see LICENSE.