TLV file crash

[SoorajDe]

Summary

Running sigflow validate on a malformed TLV file crashes instead of returning warnings/diagnostics.

Reproduction

Run: python -m sigflow validate samples\malformed_frame.sgf
Smallest failing input (fixture): samples\malformed_frame.sgf

Expected Behavior

validate should not crash.
It should return diagnostics (warnings) like it does for other bad fixtures.
For ex: it should print frames=... diagnostics=... and exit cleanly.

Actual Behavior

The program crashes with:

sigflow.core.exceptions.ParseError: truncated input

Root cause

-TLVParser tries to read: value bytes for each TLV.
-If the file is cut short (truncated), it asks for bytes that don’t exist.
-The helper require() then raises ParseError("truncated input").
-That ParseError is not converted into a “warning/diagnostic”, so the whole command crashes.

Environment

• Python version: Python 3.10.11
• sigflow version or commit:
• Operating system: windows 11

[jmestwa-coder]

The truncation itself and resulting ParseError seem plausible, but before treating this as a bug I’d like to clarify the intended validation semantics for malformed TLV input.

Right now, it’s not yet clear whether structurally truncated binary data is expected to:

• fail fast with a hard parse error, or
• be downgraded into recoverable diagnostics during validation.

Can you point to existing cases in the repository where transport-level parse failures are intentionally converted into diagnostics rather than terminating validation? I’d like to confirm the expected contract before deciding whether this behavior is actually incorrect.

[SoorajDe]

I checked the current parser/validation behavior across the codebase to better understand the intended contract for malformed transport input.

From what I found, truncation currently appears to be treated as recoverable during validation:

• binary.py: truncated-header / truncated-payload paths call context.warn(...) and stop parsing gracefully
• tlv.py: tlv-truncated emits a warning and breaks parsing instead of raising
• protocol.py: parser failures are converted into parser-fallback diagnostics before trying the next parser

I also tested:

Engine().process(samples/truncated_stream.sgf)

which returned frames plus diagnostics without terminating execution.

Based on those cases, my understanding is that validation paths are generally designed to surface malformed/truncated input as diagnostics rather than hard parser failures.

That’s why the uncaught ParseError("truncated input") observed for validate samples/malformed_frame.sgf looked inconsistent with the existing handling behavior elsewhere in the parsing pipeline.

I’m still a final-year student and relatively new to some of the lower-level parser and validation design semantics, so it’s possible I may be missing part of the intended contract. I appreciate the clarification and the opportunity to better understand the expected behavior here.

[jmestwa-coder]

The additional examples do make the inconsistency concern more grounded in the current codebase behavior. However, I still want to better understand whether this specific ParseError("truncated input") path is intended to be recoverable or whether it represents a stricter structural invariant violation compared to the other warning-based truncation cases you referenced.

Some parser paths may intentionally tolerate partial/truncated input while others fail fast on corruption that breaks parsing guarantees, so I’d like to avoid assuming all truncation cases should automatically downgrade into diagnostics.

Could you narrow down the exact parser boundary involved here and explain why this particular failure path should follow the same recovery semantics as the existing warning-based cases?

[SoorajDe]

I checked the code again and I think the difference is happening at BaseParser.require().

From what I understood, require(data, offset, size) raises ParseError("truncated input") when the parser tries to read bytes that are not fully available.

In the other truncation cases I checked, the parser seems to detect the truncation earlier using checks like:

len(data) - offset < HEADER.size

and then it does context.warn(...) and stops parsing gracefully instead of raising an exception.

So maybe the inconsistency is not that “all truncation should become diagnostics”, but more like:

• truncation detected before require() → warning/diagnostic
• truncation detected inside require() → hard ParseError

In my reproduction, it looks like the parser reaches require() after partially decoding the structure, and then the exception propagates directly.

I’m still a final-year student and still learning some of these parser recovery semantics, so I might be misunderstanding the intended distinction between recoverable truncation and fail-fast parser states.

Could you confirm whether the stack trace for this case is expected to come from sigflow/parsers/base.py::require() and which parser path reaches it (binary.py or tlv.py)?

That would help me understand whether this specific boundary is intentionally fail-fast or whether it should behave similarly to the existing warning-based truncation paths.

[jmestwa-coder]

this is a much clearer breakdown of the distinction between early truncation detection and failures propagating through require().

Your current analysis sounds more grounded in the actual parser flow, but I don’t want to confirm the exact parser boundary or intended recovery semantics yet without validating the specific execution path in the codebase.

At this stage, I agree the behavior may represent an inconsistency in how truncation is handled across different parser paths, but it’s still important to determine whether the require() boundary is intentionally treated as fail-fast for certain structural violations.

Please continue narrowing down the exact parser path involved before we decide whether this should be treated as a recoverable validation diagnostic or an intentional hard parse failure.

[SoorajDe]

Makes sense. I agree we should verify the exact execution path before deciding whether this should be recoverable or intentionally fail-fast.

From what I understood so far:

• the “early truncation” cases are handled directly in the parsers using remaining-byte checks and context.warn(...)
• the hard failure path seems to happen when execution reaches BaseParser.require(), which raises ParseError("truncated input") if the requested bytes are not available

I’ll trace the failing path more carefully to confirm:

• whether the exception is actually coming from sigflow/parsers/base.py::require()
• which parser reaches it (binary.py or tlv.py)
• and whether the failure happens while reading a declared payload/value after partial decoding has already succeeded

I’m still learning these lower-level parsing/recovery flows, so I may be misunderstanding some of the intended parser boundaries here, but I’ll narrow down the exact path first before making conclusions about the expected behavior.

[jmestwa-coder]

That sounds like the right approach.

The distinction between parser-level recoverable conditions and invariant-enforcing require() failures is important here, so I’d prefer to understand the exact transition point before deciding whether the current behavior is actually inconsistent or intentionally strict.

In particular, confirming whether the failure occurs after a parser has already committed to a declared structure length/payload boundary will help clarify whether ParseError is serving as a hard integrity check rather than a recoverable truncation condition.

Please continue tracing the exact execution path and parser boundary involved before drawing conclusions about the intended recovery semantics. Once that path is confirmed, we can evaluate whether the appropriate fix belongs in parser behavior itself or only in the CLI-facing error presentation layer.

[SoorajDe]

Understood. I’ll trace the exact execution path in the code first before concluding whether this is an actual inconsistency or an intentionally strict parser check.

From what I understand right now, I need to verify:

which parser path is actually failing (BinaryFrameParser, TLVParser, or fallback flow)
whether the ParseError("truncated input") is coming from BaseParser.require()
and whether that happens only after the parser has already accepted a declared size/length from the input

I also want to compare that path with the existing warning-based truncation handling to see whether those cases are only handled before the parser reaches the strict require() boundary.

Once I trace the exact flow for the failing sample, it should be clearer whether:

the parser behavior itself is inconsistent or whether the parser is intentionally fail-fast and only the CLI/error presentation layer should handle it differently

[jmestwa-coder]

Yes, that’s exactly the level of validation I’d want before deciding whether this should change parser behavior or remain a strict parse failure.

Comparing the warning-based truncation paths against the require() path should help clarify whether they are intentionally handling different stages of parsing/invariant enforcement rather than representing inconsistent recovery behavior.

The key detail will be identifying whether the parser has already committed to a declared structural boundary before require() triggers. If so, the current fail-fast behavior may be intentional and the actionable issue may only be the uncaught CLI traceback presentation.

Please continue tracing the exact flow first, especially the transition point between soft validation/warning paths and hard invariant enforcement.

[SoorajDe]

I ran the truncation sample from the repo directly to trace the actual behavior.

Command:

Engine().process(samples/truncated_stream.sgf)

Result:

• no exception raised
• parser returned frames + diagnostics

Output:

frames 1
diagnostics: ('truncated-payload', 73, 'payload ended before declared length')

So in this truncation example, the parser handles the issue through the warning/diagnostic path instead of raising ParseError("truncated input").

From tracing the flow, this seems to happen because the parser detects the truncation before reaching BaseParser.require().

For the ParseError("truncated input") case, I think I need to inspect the exact traceback and parser path that reaches require() during the failing read.

If you can share the relevant traceback lines for the TLV failure path (especially which parser/function reaches require()), I should be able to narrow down whether:

• the parser is intentionally fail-fast at that boundary
• or whether the behavior is inconsistent with the existing truncation handling paths.

[jmestwa-coder]

The distinction you identified between early truncation handling and failures propagating through BaseParser.require() is useful context, and the truncated_stream.sgf behavior does suggest that at least some truncation paths are intentionally recoverable.

That said, I still don’t want to assume the require() path is unintended without confirming the exact failing parser flow and boundary involved in the TLV case.

Please continue tracing the precise call path and traceback for the failing sample so we can determine whether this represents:

• an intentional fail-fast structural boundary, or
• an actual inconsistency in validation recovery behavior.

Once the exact parser boundary and intended semantics are confirmed, feel free to open a minimal scoped PR for further review.

[SoorajDe]

I traced the failing execution path with a real run.

Repro:

python -m sigflow validate samples/malformed_frame.sgf

The traceback reaches:

sigflow/parsers/tlv.py:25
value = self.require(data, offset, length)

sigflow/parsers/base.py:20
ParseError("truncated input")

and then protocol.py re-raises after the fallback parsers fail.

From tracing the flow, this seems to happen after TLVParser has already parsed the TLV tag + declared length from the header. The failure occurs when reading the value payload crosses the available buffer, which causes require() to hard-fail.

That seems different from the existing binary truncation paths, where truncation is detected earlier and handled through context.warn(...)+break before reaching require().

So this looks like a “post-length-commit” truncation case rather than the earlier soft-truncation paths.

If the preferred behavior is to treat this TLV truncation similarly to the existing warning-based recovery paths, I can prepare a minimal parser-side fix:

• add an early bounds check in sigflow/parsers/tlv.py
• emit context.warn("tlv-truncated", ...)
• stop parsing gracefully before require()
• add a focused regression test

Otherwise, if the current parser behavior is intentionally strict, the alternative may be handling the ParseError more gracefully at the CLI validation layer instead of exposing the full traceback.

[jmestwa-coder]

LGTM,
raise a PR with the proposed fix

[SoorajDe]

I traced the failing TLV path, confirmed the require() boundary involved, and implemented the minimal parser-side recovery fix discussed earlier.

PR includes:

early bounds check for truncated TLV value payloads
tlv-truncated diagnostic emission instead of hard ParseError
focused regression test covering recoverable TLV truncation behavior

Verified with:

python -m pytest -q
all tests passing