Skip to content

test(models): parametrize phantom-kwarg behavioral test across all Request models - #235

Merged
TexasCoding merged 1 commit into
mainfrom
audit/w3a-phantom-kwarg
May 21, 2026
Merged

TexasCoding merged 1 commit into
mainfrom
audit/w3a-phantom-kwarg

Conversation

@TexasCoding

Copy link
Copy Markdown
Owner

Wave 3-A of audit umbrella.

Pairs with the existing config-level test_request_bodies_use_extra_forbid to verify behavior, not just configuration. For each of the 23 Request/RequestOrder models in kalshi.models.__all__, builds minimal-valid kwargs via a new auto-builder (tests/_request_model_fixtures.py) and asserts pydantic.ValidationError with an extra_forbidden entry when a phantom field is passed.

23 new parametrized cases + 1 meta-test. Catches the failure mode where a contributor flips extra='allow' and adds a typo'd field — the config test catches the first miss; this test catches the second.

Closes #219
Refs #224

…quest models

Pairs with the existing config-level test_request_bodies_use_extra_forbid to verify behavior, not just configuration. Catches models where a contributor accidentally sets extra='allow' AND adds a typo'd field — the config test catches the first miss; this test catches the second.

Closes #219

Refs #224
@claude

claude Bot commented May 21, 2026

Copy link
Copy Markdown

Code Review — PR #235

Overall: Solid. The behavioral test is the right complement to the config-level gate. The auto-builder is clean and minimal. Two items worth addressing before merge, a few smaller nits.


What the PR does

Adds a parametrized behavioral test (test_request_model_rejects_phantom_field) that constructs each of the 23 Request/RequestOrder models with a minimal-valid kwargs set, then asserts that adding a phantom ghost_field raises ValidationError with an extra_forbidden error. Pairs with the existing config test (test_request_bodies_use_extra_forbid) to form a two-layer guard: config gate catches flipping extra=, behavioral gate catches the downstream consequence. Also adds a meta-test (test_minimal_kwargs_helper_handles_all_request_models) to give a clear failure path when the auto-builder encounters an annotation it can't resolve.


Issues

1. Meta-test ordering is inverted (minor correctness issue)

The meta-test docstring says:

"fails before the parametrized phantom test surfaces a confusing per-case error"

But in the file the meta-test is declared after test_request_model_rejects_phantom_field. With pytest's default declaration-order execution, the parametrized test would run and fail with a confusing RuntimeError before the meta-test runs to diagnose it. The two options:

  • Move test_minimal_kwargs_helper_handles_all_request_models to just before the parametrized test (fixes ordering as the docstring implies)
  • Or soften the docstring to say it's a diagnostic companion, not a pre-flight gate

2. _generate_value silently misses Enum fields

The walker handles primitives, Literal, Union, list, tuple, set, dict, and nested BaseModel. It does not handle Enum subclasses. Because Enum is a type and not a BaseModel, the isinstance(annotation, type) and issubclass(annotation, BaseModel) branch won't catch it, and _generate_value will raise RuntimeError. That's the intended signal — add an override — but it isn't documented. A one-liner in the module docstring alongside the existing list of supported types would make this expectation explicit:

# The walker handles primitives, ``Literal``, ``Union``/``Optional``,
# ``list[X]``, nested ``BaseModel`` subclasses, and ``UUID``. Enum,
# datetime, and other types must be supplied via ``_OVERRIDES``.

Smaller nits

  • _OVERRIDES by class name vs. class object: The comment explains the reason (cross-module portability). Fine. Worth noting that if two Request models in different submodules ever share a name (unlikely but not impossible), the wrong override would silently win. A short # no two Request models should share a name comment would be a cheap safety note.

  • Meta-test catches RuntimeError and TypeError but not ValidationError: If minimal_kwargs produces a value that passes the auto-builder but fails a model validator, the meta-test won't catch it — the parametrized test would blow up instead with a ValidationError. This is probably fine for now (the overrides system exists for exactly this case), but the except clause could say so:

    except (RuntimeError, TypeError) as e:  # ValidationError → override needed, caught by parametrized test
  • CLAUDE.md says no multi-paragraph docstrings — test_request_model_rejects_phantom_field has a 3-sentence docstring. Test files arguably warrant more prose, but worth being consistent.


What's good

  • The _OVERRIDES pattern is exactly the right design for edge cases — explicit, lazy (no import-time side effects), and self-documenting.
  • RuntimeError as the signal for unsupported annotations is the right choice: loud failure beats silent wrong value.
  • The ghost_field name is unambiguous; won't collide with real fields accidentally.
  • Both types.UnionType (Python 3.10 X | Y syntax) and typing.Union are handled — covers both annotation styles Pydantic models may use.
  • 150 lines of additions for 23 new test cases and a safety net is genuinely lean.

Fix the ordering issue and add the Enum/datetime note to the module docstring and this is good to go.

@TexasCoding
TexasCoding merged commit a3d4c52 into main May 21, 2026
5 checks passed
@TexasCoding
TexasCoding deleted the audit/w3a-phantom-kwarg branch May 21, 2026 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Testing: parametrize phantom-kwarg behavioral test across all Request models

1 participant