Change to datamodel-code-generator and pydantic 2

[jimisola]

Summary

Our current solution of custom classes, parsing and validation of reqstool yaml files is not core business logic.
It would be nice if we can use a library/tool for that in order to just focus on the business logic.

Blocked by:

• Code not generated for patternProperties in JSON Schemas koxudaxi/datamodel-code-generator#1851

Proposal

Change to datamodel-code-generator and pydantic 2

We can then generate pydantic 2 datamodels from JSON Schemas using datamodel-code-generator.
Using pydantic we can then populate are datamodels by having pydantic parse and validate the data.

It could be helpful to use dynaconf as a preprocessor to get support environment variables and dotenv, integration with HashiCorp Vault and Redis etc.

Pydantic has support for environment variables but both options have to be investigated.

[jimisola]

Analysis

What it would replace

Currently the project uses:

• Hand-written @dataclass models (~15 model classes across models/, common/dataclasses/, filters/, imports.py, implementations.py)
• ruamel.yaml for YAML parsing with manual field-by-field mapping in model generators
• jsonschema (Draft 2020-12) for syntax validation against 6 JSON schema files
• Manual enum conversion, UrnId construction, and cross-reference wiring in each generator

Why the blocker matters

The schemas in common.schema.json are highly complex — nested allOf with if-then-not conditions, patternProperties two levels deep, oneOf for mutual exclusivity, and cross-file $ref to common.schema.json#/$defs/.... The datamodel-code-generator issue koxudaxi/datamodel-code-generator#1851 likely blocks clean generation from these patterns.

Considerations

• Pydantic 2 migration alone (without code generation) would already be valuable — it gives built-in YAML/dict parsing, validation, and serialization, replacing most of the manual mapping code in RequirementsModelGenerator, SVCsModelGenerator, etc.
• Code generation from schemas is the harder part due to schema complexity. Could be done incrementally — generate simple models, hand-tune complex ones.
• Impact is large: ~42 test files, ~15 model classes, 5+ model generators, 3 validators, and all commands depend on the current dataclass shapes. Would need careful migration.
• dynaconf seems orthogonal — the project doesn't need env var injection or vault integration for its YAML config files today.

Recommendation

Consider splitting into phases:

1. Migrate models to Pydantic 2 manually (tracked in Add support for environment variables #158)
2. Adopt code generation later when the blocker is resolved

Phase 1 alone eliminates most of the manual parsing/validation boilerplate.

[jimisola]

Benefits of Pydantic 2 Migration — Concrete Example

Using SVCsModelGenerator + SVCData as a before/after example since it touches all the key pain points.

Before (current): Manual parsing + separate validation

Model (models/svcs.py) — plain dataclass, no validation:

@dataclass(kw_only=True, frozen=True)
class SVCData:
    id: UrnId
    title: str
    description: str
    verification: VERIFICATIONTYPES
    instructions: str
    revision: Version
    lifecycle: LifecycleData = field(default_factory=lambda: LifecycleData(state=LIFECYCLESTATE.EFFECTIVE, reason=None))
    requirement_ids: List[UrnId] = field(default_factory=list)

Generator (model_generators/svcs_model_generator.py) — 60+ lines of manual dict-to-object mapping:

def __parse_svcs(self, data) -> dict[UrnId, SVCData]:
    r_result = {}
    self.semantic_validator._validate_no_duplicate_svc_ids(data=data)

    if "cases" not in data:
        return r_result

    for case in data["cases"]:
        urn_id = UrnId(urn=self.urn, id=case["id"])

        # Manual lifecycle extraction with nested if/in checks
        if "lifecycle" in case:
            lifecycle_state = LIFECYCLESTATE(case["lifecycle"]["state"])
            lifecycle_reason = case["lifecycle"]["reason"] if "reason" in case["lifecycle"] else None
        else:
            lifecycle_state = LIFECYCLESTATE.EFFECTIVE
            lifecycle_reason = None

        svc = SVCData(
            id=urn_id,
            requirement_ids=Utils.convert_ids_to_urn_id(ids=case["requirement_ids"], urn=self.urn),
            title=case["title"],
            description=case["description"] if "description" in case else None,
            verification=VERIFICATIONTYPES(case["verification"]),
            instructions=case["instructions"] if "instructions" in case else None,
            revision=self.__parse_svc_version(version=case["revision"], urn_id=urn_id),
            lifecycle=LifecycleData(state=lifecycle_state, reason=lifecycle_reason),
        )
        if svc.id not in r_result:
            r_result[svc.id] = svc

    return r_result

Validation — separate JSON Schema step before parsing:

if not SyntaxValidator.is_valid_data(
    json_schema_type=JsonSchemaTypes.SOFTWARE_VERIFICATION_CASES, data=data, urn=self.urn
):
    sys.exit(EXIT_CODE_SYNTAX_VALIDATION_ERROR)

Three separate layers (schema → manual mapping → dataclass) that must all stay in sync.

---

After (Pydantic 2): Declarative model with built-in validation

Model — validation, defaults, and parsing in one place:

class SVCData(BaseModel):
    model_config = ConfigDict(frozen=True)

    id: UrnId
    title: str
    description: str | None = None
    verification: VERIFICATIONTYPES
    instructions: str | None = None
    revision: Version
    lifecycle: LifecycleData = LifecycleData()
    requirement_ids: list[UrnId] = []

    @field_validator("revision", mode="before")
    @classmethod
    def parse_revision(cls, v: str) -> Version:
        return Version(v)

    @field_validator("verification", mode="before")
    @classmethod
    def parse_verification(cls, v: str) -> VERIFICATIONTYPES:
        return VERIFICATIONTYPES(v)

Generator — parsing collapses to:

def __parse_svcs(self, data) -> dict[UrnId, SVCData]:
    if "cases" not in data:
        return {}

    result = {}
    for case in data["cases"]:
        case["id"] = UrnId(urn=self.urn, id=case["id"])
        case["requirement_ids"] = Utils.convert_ids_to_urn_id(ids=case["requirement_ids"], urn=self.urn)
        svc = SVCData.model_validate(case)
        result.setdefault(svc.id, svc)
    return result

---

What this eliminates

Boilerplate	Before	After
if "key" in dict checks for optional fields	~10 per generator	0 — Pydantic handles defaults
Nested if/else for lifecycle extraction	6 lines per model	0 — nested model auto-parsed
Enum string → Enum conversion	Manual per field	Declarative via field_validator or use_enum_values
Separate JSON Schema validation step	Required before parsing	Can be replaced by Pydantic validation (or kept as defense-in-depth)
Type coercion (e.g. Version())	Manual in generator	field_validator on the model

Additional Pydantic 2 benefits

• Serialization for free — model.model_dump() / model.model_dump_json() replaces the manual JSON export code and jsonpickle dependency
• Better error messages — Pydantic validation errors include field paths, expected types, and received values out of the box
• Schema generation — SVCData.model_json_schema() can generate JSON schemas from models, potentially inverting the current flow (models become the source of truth instead of schemas)
• Performance — Pydantic 2's Rust core is significantly faster than manual dict traversal + jsonschema validation

[jimisola]

Additional Pydantic 2 opportunities found during codebase review

A simplification review of the codebase identified two more areas where Pydantic 2 would eliminate significant duplication. These have been noted on #158:

1. Filter parsing — RequirementFilter and SVCFilter parsing logic is duplicated across two generators (~40 lines each). Pydantic model_validate() would replace both.

2. Location parsing — Four near-identical methods parse git/local/maven/pypi locations. Pydantic discriminated unions could replace all four with a single model.

These reinforce the recommendation to pursue the Pydantic 2 migration (#158) as the primary vehicle for structural simplification — smaller targeted cleanups (lifecycle dedup, version parsing dedup, dict.get() usage, etc.) are being handled separately.

[jimisola]

Additional refactoring candidates for the Pydantic 2 migration

From a codebase simplification review, these two areas have significant duplication that would be naturally eliminated by the Pydantic 2 migration:

1. Duplicated filter parsing (requirements vs SVCs)

RequirementsModelGenerator.__parse_requirement_filters() (requirements_model_generator.py:268-317) and SVCsModelGenerator.__parse_svc_filters() (svcs_model_generator.py:79-122) are near-identical ~40-line methods that:

• Iterate over filter URNs
• Parse urn_ids includes/excludes → convert to Set[UrnId]
• Parse custom includes/excludes
• Construct filter objects (RequirementFilter / SVCFilter)

With Pydantic 2, this becomes declarative model validation — the filter classes would parse themselves from dicts via model_validate().

2. Duplicated location parsing (4 near-identical methods)

RequirementsModelGenerator has four methods with the same structure:

• __parse_location_local() (line 244)
• __parse_location_git() (line 253)
• __parse_location_maven() (line 208)
• __parse_location_pypi() (line 227)

Each checks if a location type key exists in the data dict, iterates the list, and constructs location instances. With Pydantic 2 discriminated unions, this could be a single LocationData model with type-based dispatch.

[jimisola]

Blocker resolved + feasibility testing results

datamodel-code-generator blocker is resolved

The original blocker (koxudaxi/datamodel-code-generator#1851 — broken patternProperties handling) is fully resolved in v0.54.1 (released Dec 2025):

• PR #2766: crash fix for boolean patternProperties
• Issue #2769 / PR #2770: multiple patternProperties entries now merge into a single regex — filed and merged same day by the maintainer

All 6 reqstool schemas generate cleanly with datamodel-codegen 0.54.1 with exit 0.

---

Schema design issues discovered during testing

Running generated models against all YAML fixtures in the test suite revealed two cases where patternProperties is used for fixed, known field names — which is semantically wrong and confuses the generator.

1. common.schema.json — filter field names

requirement_ids, svc_ids, and mvr_ids are defined via:

"patternProperties": { "^requirement_ids|svc_ids|mvr_ids$": { "": "#//filterids" } }

These are fixed field names, not dynamic keys. The regex also has an operator-precedence bug — | in regex makes it match (^requirement_ids) OR (svc_ids) OR (mvr_ids$), which accidentally accepts keys like my_svc_ids_here.

Fix: move them to properties with three explicit named entries.

2. annotations.schema.json — section names

implementations and tests are defined via:

"patternProperties": { "^implementations": { ... }, "^tests": { ... } }

Same issue — fixed names, not dynamic. ^implementations also matches implementations_extra, etc.

Fix: move them to properties.

Both fixes are non-breaking: every valid YAML that any real user would write is unaffected. The only change is that previously-accepted malformed keys (e.g. implementations_extra) are now correctly rejected.

A third minor bug was also found: in reqstool_config.schema.json, additionalProperties: false is placed inside properties (treated as a field named additionalProperties rather than as a schema keyword). Fix: move it to the correct level on the resources sub-object.

---

Test results

After applying the two patternProperties fixes (tested in-memory without modifying files):

• All YAML fixtures from the test suite: ✅ 100% pass with model_validate()
• All YAML fixtures from reqstool-demo: ✅ 100% pass with model_validate()

The schema fixes are therefore a prerequisite before generating Pydantic models, but once applied the path is clear.

[jimisola]

Implementation Plan — Pydantic v2 via datamodel-code-generator

Architecture: JSON Schema = single source of truth → datamodel-code-generator (dev-only) generates Pydantic v2 models → model_validate() replaces manual parsing. jsonschema runtime validation kept for user-facing errors.

Phases

Phase	Description	Scope
1. Regression baseline	Capture all CLI output (status, report, export) against all 12 in-repo test datasets + reqstool-demo. Store in baselines/ on the branch. Run and save full pytest output.	No code changes
2. JSON Schema fixes	Fix 3 schema bugs that block the code generator: patternProperties → properties in common.schema.json and annotations.schema.json, misplaced additionalProperties in reqstool_config.schema.json. Non-breaking. Regression diff.	3 schema files
3. Build tooling	Add pydantic>=2.0 to dependencies, datamodel-code-generator>=0.54.1 as dev-only. Set up codegen script. Generate models from schemas, review output, rename classes.	pyproject.toml, new generated/ dir
4. Migrate parsers	Replace manual dict→dataclass in all 5 model generators with model_validate(). ~530 lines of manual parsing → ~50 lines. Update all generator tests.	5 model generators, test files
5. Serialization	Replace jsonpickle + 3 custom handlers with Pydantic model_dump() / model_dump_json(). Verify JSON export output identical.	generate_json.py, pyproject.toml
6. Cleanup	Convert ALL remaining @dataclass data models to Pydantic (raw_datasets, combined_indexed_dataset, implementations, imports, urn_id, lifecycle, filters, config, etc.). Location classes stay as regular classes (infrastructure, not data). Full regression.	~20 files
7. (Future)	pydantic-settings for env variable support (dynaconf alternative). Out of scope for this PR.	—

Key decisions

• Code generation runs only when schemas change — generated files are committed (not generated on every build)
• jsonschema + referencing kept at runtime (user-facing errors match IDE)
• Goal: zero @dataclass for data models — location classes remain regular classes
• Each phase = separate PR

Branch: feat/140-pydantic-v2-datamodel-code-generator

Full plan (click to expand)

Plan: Issue #140 — Pydantic v2 via datamodel-code-generator (NEW)

Problem Statement

The current reqstool-client uses hand-rolled @dataclass models with manual YAML→dict→dataclass
conversion and jsonschema + referencing for validation. The goal is to adopt Pydantic v2,
eliminating manual parsing boilerplate, and to use datamodel-code-generator as a build-time
dev tool to generate those Pydantic models directly from the existing JSON Schemas.

---

Architecture Decision

JSON Schema is the single source of truth.

The 6 JSON Schema files in src/reqstool/resources/schemas/v1/ serve two audiences:

1. End-users — IDE validation, autocompletion when editing .yml files
2. Runtime — jsonschema currently validates user YAML before parsing

Direction: Schema → generate Pydantic (not Pydantic → generate Schema).

datamodel-code-generator is a dev/CI-only tool, not a runtime dependency.
jsonschema runtime validation is kept so user-facing error messages remain consistent with
what the IDE shows.

Why this is better than the old plan (PLAN_PYDANTIC_OLD.md)

The old plan proposed hand-writing Pydantic models. That plan was written when
koxudaxi/datamodel-code-generator#1851
(broken patternProperties handling) was unresolved.

The blocker is fully resolved in datamodel-code-generator 0.54.1 (Dec 2025):

• PR #2766: crash fix for boolean patternProperties
• Issue #2769 / PR #2770: multiple patternProperties merging into one regex — filed and merged same day

Verified: All 6 reqstool schemas generate cleanly with v0.54.1. All YAML fixtures pass after
the Phase 0 schema fixes described below.

---

Stack Changes

Layer	Current	Target
YAML loading	ruamel.yaml (safe mode)	Keep — Pydantic doesn't parse YAML natively
Schema validation	jsonschema Draft 2020-12 + referencing	Keep for user-facing runtime errors
Data models	@dataclass (frozen/plain)	Pydantic v2 BaseModel — generated from JSON Schemas
Dict→Model conversion	Manual per-field code in model generators	Model.model_validate(yaml_dict)
JSON serialization	jsonpickle + 3 custom handlers	model_dump() / model_dump_json()
Cross-ref validation	SemanticValidator (manual)	Keep as-is (business logic)
Lifecycle validation	LifecycleValidator (manual)	Keep as-is (business logic)
Expression language	Lark-based DSL	Keep as-is (accesses .id — interface-compatible)
Model source of truth	JSON Schema (doc) + @dataclass (code)	JSON Schema (generates Pydantic)

---

Phase 1 — Pre-Migration Regression Baseline (run BEFORE any code changes)

Goal: Capture the exact current behaviour of every CLI command against every test dataset.
After each phase of implementation, re-run and diff — zero unexpected changes allowed.

Baselines are stored in baselines/ on the feature branch (committed). This directory must be
removed before merging to main.

Datasets

In-repo — all under tests/resources/test_data/data/local/:

Dataset path	Notes
test_standard/baseline/ms-001	Primary MS — most complete fixture
test_standard/baseline/sys-001	System with multi-level imports
test_standard/empty_ms/ms-001	MS with empty requirements
test_standard/empty_ms/sys-001	System with empty MS imports
test_basic/baseline/ms-101	Basic MS
test_basic/lifecycle/ms-101	MS with lifecycle states (DEPRECATED/OBSOLETE)
test_basic/lifecycle/validation_error	Intentionally invalid YAML — must fail gracefully
test_basic/no_impls/basic/ms-101	MS with no annotations/implementations
test_basic/no_impls/with_error/ms-101	MS with error conditions
test_delete_mvr/ms-001	MS with MVRs
test_delete_mvr/sys-001	System with MVRs
test_errors/ms-101	Error-condition dataset

ext-001, ext-002 are sub-imports referenced by sys-001 — not standalone entry points.
with_requirements_config/ms-101 only has a config file — not a standalone entry point.

External — sibling project ../reqstool-demo/docs/reqstool:

Dataset path	Notes
../reqstool-demo/docs/reqstool	Real-world fixture with imports, SVCs, MVRs, annotations, JUnit results

Prerequisites for reqstool-demo: run ./mvnw verify in reqstool-demo first to generate target/ artifacts.

Baseline capture script

Run once on main (before switching to the feature branch), save all output to baselines/:

OUT="baselines"
mkdir -p "$OUT"

BASE="tests/resources/test_data/data/local"
DATASETS=(
  "test_standard/baseline/ms-001"
  "test_standard/baseline/sys-001"
  "test_standard/empty_ms/ms-001"
  "test_standard/empty_ms/sys-001"
  "test_basic/baseline/ms-101"
  "test_basic/lifecycle/ms-101"
  "test_basic/lifecycle/validation_error"
  "test_basic/no_impls/basic/ms-101"
  "test_basic/no_impls/with_error/ms-101"
  "test_delete_mvr/ms-001"
  "test_delete_mvr/sys-001"
  "test_errors/ms-101"
)

STRIP_ANSI='sed s/\x1b\[[0-9;]*m//g'

for DS in "${DATASETS[@]}"; do
  SLUG="${DS//\//_}"
  P="$BASE/$DS"

  # status
  hatch run python src/reqstool/command.py status local -p "$P" \
    2>&1 | eval $STRIP_ANSI > "$OUT/${SLUG}__status.txt"
  echo $? > "$OUT/${SLUG}__status.exitcode"

  # report asciidoc
  hatch run python src/reqstool/command.py report --format asciidoc local -p "$P" \
    > "$OUT/${SLUG}__report_adoc.txt" 2>&1
  echo $? > "$OUT/${SLUG}__report_adoc.exitcode"

  # report markdown
  hatch run python src/reqstool/command.py report --format markdown local -p "$P" \
    > "$OUT/${SLUG}__report_md.txt" 2>&1
  echo $? > "$OUT/${SLUG}__report_md.exitcode"

  # export (full)
  hatch run python src/reqstool/command.py export local -p "$P" \
    > "$OUT/${SLUG}__export.txt" 2>&1
  echo $? > "$OUT/${SLUG}__export.exitcode"
done

# Extra: export with filters on the primary dataset
DS="test_standard/baseline/ms-001"
SLUG="${DS//\//_}"
P="$BASE/$DS"

hatch run python src/reqstool/command.py export local -p "$P" --req-ids REQ_010 \
  > "$OUT/${SLUG}__export_req010.txt" 2>&1
hatch run python src/reqstool/command.py export local -p "$P" --svc-ids SVC_010 \
  > "$OUT/${SLUG}__export_svc010.txt" 2>&1
hatch run python src/reqstool/command.py export local -p "$P" --req-ids REQ_010 --svc-ids SVC_022 \
  > "$OUT/${SLUG}__export_req010_svc022.txt" 2>&1

# Deprecated alias smoke check
hatch run python src/reqstool/command.py generate-json local -p "$P" \
  > "$OUT/${SLUG}__generate_json.txt" 2>&1

# reqstool-demo (external sibling project)
DEMO="../reqstool-demo/docs/reqstool"
if [ -d "$DEMO" ]; then
  hatch run python src/reqstool/command.py status local -p "$DEMO" \
    2>&1 | eval $STRIP_ANSI > "$OUT/reqstool_demo__status.txt"
  echo $? > "$OUT/reqstool_demo__status.exitcode"
  hatch run python src/reqstool/command.py report --format asciidoc local -p "$DEMO" \
    > "$OUT/reqstool_demo__report_adoc.txt" 2>&1
  hatch run python src/reqstool/command.py report --format markdown local -p "$DEMO" \
    > "$OUT/reqstool_demo__report_md.txt" 2>&1
  hatch run python src/reqstool/command.py export local -p "$DEMO" \
    > "$OUT/reqstool_demo__export.txt" 2>&1
fi

echo "Baseline captured in $OUT"
ls "$OUT" | wc -l

Regression comparison (after each phase)

Re-run the same script into baselines-check/, diff against baselines/:

# (re-run same script with OUT="baselines-check")
diff -rq baselines/ baselines-check/
rm -rf baselines-check/

Any diff must be explained and accepted before continuing.

Unit test baseline

hatch run dev:pytest --cov=reqstool -v 2>&1 > baselines/pytest_output.txt

---

Phase 2 — JSON Schema Prerequisites (must land before any Pydantic work)

Three bugs in the current JSON Schemas must be fixed first. Without these, datamodel-code-generator
produces incorrect models (fields treated as extra/unknown inputs instead of named fields).

Fix 1: common.schema.json — patternProperties for known field names

File: src/reqstool/resources/schemas/v1/common.schema.json

Inside $defs.filters.patternProperties["^.+$"], the per-URN filter object currently defines
requirement_ids, svc_ids, and mvr_ids via:

"patternProperties": {
  "^requirement_ids|svc_ids|mvr_ids$": { "$ref": "#/$defs/filterids" }
}

Problem: These are fixed, known field names — patternProperties is semantically wrong here.
The regex ^requirement_ids|svc_ids|mvr_ids$ also has an operator-precedence bug:
due to how | works in regex, it matches (^requirement_ids) OR (svc_ids) OR (mvr_ids$),
which accidentally accepts keys like requirement_ids_extra, my_svc_ids_here, prefix_mvr_ids.

Fix: Move them to properties:

"properties": {
  "requirement_ids": { "$ref": "#/$defs/filterids" },
  "svc_ids":         { "$ref": "#/$defs/filterids" },
  "mvr_ids":         { "$ref": "#/$defs/filterids" }
}

Breaking change: No. Every valid YAML written by any real user is unaffected.
The only "breakage" is rejecting keys like my_svc_ids_here — which were never intended to be valid.

Fix 2: annotations.schema.json — patternProperties for known section names

File: src/reqstool/resources/schemas/v1/annotations.schema.json

In $defs.requirement_annotations, the two top-level section names implementations and tests
are defined via:

"patternProperties": {
  "^implementations": { ... },
  "^tests":           { ... }
}

Problem: Same as Fix 1 — these are fixed, known field names.
^implementations accidentally matches implementations_extra, implementations_v2, etc.

Fix: Move them to properties:

"properties": {
  "implementations": { ... },
  "tests":           { ... }
}

Breaking change: No. Same reasoning as Fix 1.
Note: This fix was explicitly mentioned by the datamodel-code-generator maintainer in the
original issue #1851 as a schema design issue.

Fix 3: reqstool_config.schema.json — additionalProperties misplaced

File: src/reqstool/resources/schemas/v1/reqstool_config.schema.json

additionalProperties: false appears inside properties (as if it were a field named
"additionalProperties"), not as a sibling to properties on the resources object:

// CURRENT (broken)
"properties": {
  "language": { ... },
  "resources": { ... },
  "additionalProperties": false   ← treated as a property field, not schema keyword
},
"additionalProperties": false     ← correct at top level

Fix: Move additionalProperties: false out of properties and add it correctly to the
resources sub-object:

"resources": {
  "type": "object",
  "additionalProperties": false,   ← correct: keyword on the resources object
  "properties": { ... }
}

Breaking change: No. The misplaced entry currently has no effect; adding it correctly just
closes a gap where unknown keys in resources were silently accepted.

---

Phase 3 — Setup build tooling

3.1 Add dependencies to pyproject.toml

• Add pydantic>=2.0,<3.0 to [project.dependencies]
• Add datamodel-code-generator>=0.54.1 to [tool.hatch.envs.dev.dependencies] (dev only)
• Keep jsonschema[format-nongpl] and referencing (runtime validation stays)

3.2 Set up code generation as a reproducible dev step

Add a hatch script (or Makefile target) to regenerate Pydantic models from schemas:

datamodel-codegen \
  --input src/reqstool/resources/schemas/v1/ \
  --input-file-type jsonschema \
  --output src/reqstool/models/generated/ \
  --output-model-type pydantic_v2.BaseModel \
  --use-schema-description \
  --use-field-description \
  --use-annotated \
  --field-constraints \
  --target-python-version 3.13

When to regenerate: Only when a JSON Schema changes. The generated files are committed to the
repo (same pattern as gRPC/protobuf generated code). This means:

• datamodel-code-generator is a dev-only dependency, not needed for CI builds or end-users
• Normal builds do NOT run the generator — they just use the committed Python files
• When a schema is updated, the developer runs the generation script and commits the result
• CI can optionally verify the generated files are up-to-date (run generator, diff, fail if stale)

3.3 Run generation and review output

• Generate models from all 6 schemas (after Phase 0 fixes)
• Review generated class names and rename where needed (e.g. Model → RequirementsData)
• Review if/then, oneOf, anyOf handling: common.schema.json and requirements.schema.json
  use advanced conditional schemas. Verify that the generated Pydantic models enforce these rules
  correctly (e.g., oneOf for includes/excludes in filterids, if/then for variant-dependent fields).
  Our in-memory testing passed all fixtures, but specifically test edge cases.
• Verify all YAML fixtures pass model_validate() (already tested in-memory: 100% pass rate)

---

Phase 4 — Migrate model generators

Goal: Replace manual dict→dataclass conversion with Model.model_validate(yaml_dict).

4.1 Migrate RequirementsModelGenerator

Largest file (~316 lines). Current flow:

1. Load YAML → dict
2. SyntaxValidator.is_valid_data() (JSON Schema)
3. Manual parsing: __parse_metadata(), __parse_implementations(), __parse_imports(),
   __parse_requirement_filters(), __parse_requirements()

New flow:

1. Load YAML → dict
2. SyntaxValidator.is_valid_data() (keep — user-facing errors)
3. RequirementsData.model_validate(yaml_dict) — Pydantic handles structure + type coercion

Note on LocationResolver: GitImplData, LocalImportData, etc. need a parent reference
that is context-dependent (not in YAML). Strategy: use Pydantic to validate the raw YAML
structure (URLs, paths, required fields), then wrap in LocationResolver objects as post-validate
business logic inside the generator.

4.2 Migrate SVCsModelGenerator (~100 lines)

Simpler. Replace manual field extraction with SVCsData.model_validate(yaml_dict).

4.3 Migrate AnnotationsModelGenerator (~60 lines)

Replace manual iteration over patternProperties keys with AnnotationsData.model_validate(yaml_dict).
After Phase 2 Fix 2, the generated model has named implementations and tests fields instead
of Dict[str, T].

4.4 Migrate MVRsModelGenerator (~50 lines)

Replace manual MVR parsing. Handle pass → passed field alias
(YAML uses pass; Python reserved word needs Field(alias="pass")).

4.5 Migrate TestdataModelGenerator (hand-written Pydantic)

JUnit XML test results have no JSON schema — they're parsed from XML, not YAML.
TestData and TestsData need hand-written Pydantic models (not generated).
Replace the dataclass definitions in src/reqstool/models/test_data.py with Pydantic equivalents.

4.6 Update tests for all migrated generators

• Update tests/unit/reqstool/model_generators/
• Pydantic ValidationError replaces jsonschema.ValidationError
• Error message format changes — update expectations

---

Phase 5 — Replace jsonpickle with Pydantic serialization

5.1 Add serialization config to Pydantic models

• UrnId → serialize as "urn:id" string via @field_serializer
• Version → serialize as {"major": N, "minor": N, "patch": N} via @model_serializer
• Enums → .value (Pydantic default)

5.2 Migrate GenerateJsonCommand

Replace:

jsonpickle.encode(cids, make_refs=False, keys=True, unpicklable=False)

With:

cids.model_dump_json()        # if CombinedIndexedDataset is Pydantic
# OR
json.dumps(serialize_cid(cids))  # if it stays as @dataclass

CombinedIndexedDataset is an internal aggregate — converting it to Pydantic is optional.
If kept as @dataclass, write a serialize() function that calls model_dump() on Pydantic
sub-models.

5.3 Remove jsonpickle

• Remove jsonpickle==4.1.1 from pyproject.toml
• Remove UrnIdHandler, RevisionHandler, JsonEnumHandler
• Remove the post-processing regex cleanup in generate_json.py

5.4 Verify JSON output is identical

Run regression smoke tests (see CLAUDE.md) before and after. Output must be byte-for-byte
identical unless a change is intentional.

---

Phase 6 — Cleanup

6.1 Remove jsonpickle dependency (done in 5.3)

6.2 Decide on jsonschema / referencing

Keep them — runtime validation produces user-facing errors that match what the IDE shows.
If a future decision is made to drop them, Pydantic error messages can be wrapped for
user-friendliness, but that is out of scope here.

6.3 Remove old @dataclass definitions — convert ALL data models to Pydantic

Delete ALL @dataclass data model definitions and convert to Pydantic BaseModel:

YAML-parsed models (replaced by generated models in Phase 1):

• src/reqstool/models/requirements.py
• src/reqstool/models/svcs.py
• src/reqstool/models/mvrs.py
• src/reqstool/models/annotations.py
• src/reqstool/models/test_data.py — hand-written Pydantic (no JSON schema; parsed from JUnit XML)

Internal aggregation models (convert to Pydantic for consistency — no mixing):

• src/reqstool/models/raw_datasets.py — RawDataset, CombinedRawDataset
• src/reqstool/models/combined_indexed_dataset.py — CombinedIndexedDataset
• src/reqstool/models/implementations.py — ImplementationDataInterface + Git/Local/Maven/Pypi variants.
  Replace __post_init__ with @model_validator(mode='after').
• src/reqstool/models/imports.py — same pattern as implementations
• src/reqstool/common/dataclasses/urn_id.py — UrnId with ConfigDict(frozen=True) for hashability
• src/reqstool/common/dataclasses/lifecycle.py — LifecycleData

Filter and config models:

• src/reqstool/filters/id_filters.py — IDFilters
• src/reqstool/filters/requirements_filters.py — RequirementFilter
• src/reqstool/filters/svcs_filters.py — SVCFilter
• src/reqstool/reqstool_config/reqstool_config.py — ReqstoolConfig, Resources
• src/reqstool/requirements_indata/requirements_indata.py — RequirementsIndata
  (has __post_init__ → @model_validator(mode='after'))
• src/reqstool/requirements_indata/requirements_indata_paths.py

Command/report internal data classes:

• src/reqstool/common/validator_error_holder.py
• src/reqstool/commands/status/statistics_container.py
• src/reqstool/commands/report/criterias/group_by.py
  (has __post_init__ → @model_validator(mode='after'))

Intentionally NOT converted (behavioral infrastructure, not data models):

• src/reqstool/locations/*.py — GitLocation, LocalLocation, MavenLocation, PypiLocation.
  These have methods like _make_available_on_localdisk() and represent infrastructure, not data.
  Pydantic BaseModel is designed for data validation, not general OOP with behavior.

Goal: zero @dataclass for data models — all data models use Pydantic BaseModel.
Location classes remain as regular classes (not dataclasses, not Pydantic).

6.4 Full regression test

hatch run dev:pytest --cov=reqstool

Plus regression smoke tests from CLAUDE.md against:

• tests/resources/test_data/data/local/test_standard/baseline/ms-001
• tests/resources/test_data/data/local/test_basic/baseline/ms-101
• ../reqstool-demo/docs/reqstool

All diffs must be empty unless intentional.

---

Phase 7 (Future / Optional) — pydantic-settings for config

Investigate replacing reqstool_config.yml parsing with pydantic-settings:

from pydantic_settings import BaseSettings
class ReqstoolSettings(BaseSettings):
    model_config = SettingsConfigDict(env_prefix='REQSTOOL_')

This would also handle env variable overrides (mentioned in issue #140 / dynaconf reference).
Out of scope for this PR.

---

Implementation Order

Phase 1 (Regression baseline)
  ├── 1.1  Capture all CLI output for all datasets + reqstool-demo
  └── 1.2  Run and save pytest baseline
           │
Phase 2 (Schema fixes — prerequisite)
  ├── 2.1  Fix common.schema.json: patternProperties → properties (req/svc/mvr_ids)
  ├── 2.2  Fix annotations.schema.json: patternProperties → properties (implementations/tests)
  └── 2.3  Fix reqstool_config.schema.json: move additionalProperties to resources object
           │
Phase 3 (Build tooling)
  ├── 3.1  Add pydantic + datamodel-code-generator to pyproject.toml
  ├── 3.2  Set up codegen script / hatch task
  └── 3.3  Generate + review models; rename where needed
           │
Phase 4 (Migrate parsers)
  ├── 4.1  RequirementsModelGenerator (largest)
  ├── 4.2  SVCsModelGenerator
  ├── 4.3  AnnotationsModelGenerator
  ├── 4.4  MVRsModelGenerator
  ├── 4.5  TestdataModelGenerator (hand-written Pydantic, no schema)
  └── 4.6  Update model generator tests
           │
Phase 5 (Serialization)
  ├── 5.1  Add serialization config to Pydantic models
  ├── 5.2  Migrate GenerateJsonCommand
  ├── 5.3  Remove jsonpickle
  └── 5.4  Verify JSON output identical
           │
Phase 6 (Cleanup)
  ├── 6.1  Convert ALL remaining data model @dataclass to Pydantic
  └── 6.2  Full regression test suite

Each phase should be a separate PR.

---

Risk Assessment

Risk	Impact	Mitigation
Generated class names differ from current	Import errors in model generators	Review + rename generated classes in 1.3 before migrating generators
UrnId as frozen Pydantic model — dict key hashability	{UrnId: value} dicts break	ConfigDict(frozen=True) auto-generates __hash__; verified in Pydantic v2
Expression language accesses .id attribute	Could break if generated field names differ	Attribute names controlled by schema field names (snake_case); id stays id
JSON export output format changes	Downstream consumers break	Regression smoke test in 3.4 catches this before merge
Pydantic ValidationError format differs from jsonschema	User-facing error messages change	jsonschema runtime validation is kept; Pydantic errors are a secondary layer
LocationResolver context dependency	Cannot fully model in Pydantic	Intentional: Pydantic validates raw YAML structure; LocationResolver wrapping stays as post-validate business logic