Skip to content

fix: report the diagnostics pyval was swallowing (evaluation errors, disjunctions, metrics, --track) - #4

Open
ghostfix-pm wants to merge 9 commits into
SPL-BGU:mainfrom
ghostfix-pm:fix/honest-diagnostics
Open

ghostfix-pm wants to merge 9 commits into
SPL-BGU:mainfrom
ghostfix-pm:fix/honest-diagnostics

Conversation

@ghostfix-pm

Copy link
Copy Markdown

Summary

Four defects in the diagnostic paths, all of the same shape: pyval knew something was wrong and told the caller nothing. Each one is reproduced on the parent commit below, and the fixes come with regression tests. No packaging, no CI, no dependency changes.

Bug fixes

1. An unevaluable step crashed validation with a traceback. Numeric fluents have no default in PDDL, and the simulator raises UPUsageError when a condition or effect reads an instance :init never assigned. simulate let that escape, so the CLI died mid-report:

$ pyval init_domain.pddl init_problem.pddl init_plan.txt
unified_planning.exceptions.UPUsageError: The state {at(t1, l2): true, fuel(t1): 40, ...}
does not have a value for the value fuel(t2)

The same input now stops at the offending step, names the fluent and the repair, and refuses to give a verdict it did not earn:

Step 2: (drive t2 l1 l2) ✗ EVALUATION ERROR
  No value in the current state for fuel(t2): this problem's :init never assigns it,
  and PDDL defines no default. The step cannot be evaluated until it is initialised.
  Add (= (fuel t2) <value>) to :init.

=== Goal Check ===
Plan could not be validated: step 2 of 2 ((drive t2 l1 l2)) could not be evaluated,
so no verdict is given for the plan as a whole.

2. ✗ PRECONDITION FAILURE with nothing under it. decompose_preconditions decided or by counting collected sub-failures against the number of disjuncts, so (or (and (p) (q) (s)) (r)) produced four failures for two disjuncts and reported none; and any node it did not handle (forall, implication, object equality) fell straight through. Both produced a failed step with an empty unsatisfied_preconditions list. Disjuncts are now evaluated with StateEvaluator and an unhandled node gets a final branch, so:

Step 1: (go a) ✗ PRECONDITION FAILURE
  Unsatisfied: ((p(a) and q(a) and s(a)) or r(a))
    None of the 2 disjuncts holds: (p(a) and q(a) and s(a)); r(a)

Object equality (:equality) also no longer goes through the numeric path, where it answered "Could not evaluate numeric comparison" with type: "numeric".

3. Metrics silently disappeared. _evaluate_metric used state.get_value, which asserts on arithmetic sub-expressions — the same root cause 0.1.4 fixed for goals, left behind on the metric path. (:metric minimize (+ (* 2 (fuel l1)) 5)) returned no metric at all (expected 17.0), and :action-costs domains were dropped entirely because MinimizeActionCosts is not an expression over the final state. Both are reported now, and a metric that genuinely cannot be evaluated shows as n/a with the reason.

4. --track never matched anything. The README's own command pyval domain.pddl problem.pddl plan.txt --trajectory --track fuel printed "No numeric fluents to display.": the tracker compared a display name (fuel truck1) with the snapshot key (fuel(truck1)), and format_trajectory compared the raw reference with that same key. Both now accept the bare function name (every instance), the PDDL spelling and UPF's rendering; an unmatched reference warns with the fluents that do exist instead of printing an empty table.

Also fixed

The Phase-1 "numeric function has no initial value" warning was decided per declared name, so initialising (fuel t1) silenced it for fuel(t2), and it claimed the missing value "defaults to 0" — which is not what the simulator does (it raises). It now enumerates ground instances and reports the ones an action writes, with a valid PDDL repair hint. Static tables such as (distance ?a ?b) are left alone when they are partially specified, since that is normal.

Additive output changes

  • JSON gains is_valid, phases.goal_check (every goal and its verdict) and warnings; a failed step may carry evaluation_error. phases.goals is unchanged, so existing consumers keep working — goal_check is what distinguishes "checked and green" from "not checked", which goals: null could not.
  • ValidationResult gains goal_results (all goals) alongside the existing unsatisfied_goals.

Tests

107 passed, 8 skipped in 17.2s

40 new tests (67 → 107), each defect pinned by a test that fails on the parent commit. Text, verbose and trajectory output for every existing fixture is byte-identical to the parent commit; the JSON diff is purely additive (verified by diffing full reports for six fixtures before and after).

Files Changed

File Δ What
pyval/plan_simulator.py +152/-30 Catch evaluation failures into StepResult.evaluation_error; StateEvaluator metrics, :action-costs sum, unevaluable metric reason
pyval/diagnostics.py +177/-52 Disjuncts evaluated instead of counted; final branch for quantifiers/equality; :equality off the numeric path
pyval/syntax_checker.py +104/-11 Per-instance numeric :init coverage, capped enumeration, PDDL repair hints
pyval/report_formatter.py +68/-5 EVALUATION ERROR step lines, "could not be validated", n/a metrics, is_valid/goal_check/warnings, unified --track filter
pyval/numeric_tracker.py +76/-7 fluent_reference_matches/parse_fluent_reference, unknown-reference tracking
pyval/validator.py +38 Warning for --track values that select nothing
pyval/models.py +8 StepResult.evaluation_error, ValidationResult.goal_results
pyval/pddl_format.py +24 as_pddl_call: fuel(t2) → (fuel t2) for repair hints
tests/test_numeric_init.py +226 8 tests
tests/test_precondition_diagnostics.py +192 7 tests
tests/test_evaluation_errors.py +147 6 tests
tests/test_metrics.py +131 5 tests
tests/test_tracking.py +119 9 tests
tests/test_json_output.py +71 4 tests
README.md +12 Numeric initialisation, evaluation errors, accepted --track spellings
CHANGELOG.md +88 [Unreleased] entries

The Phase-1 warning was decided per declared function name, so a problem
that initialised (fuel t1) got no warning at all for fuel(t2) — and the
message claimed the missing value "defaults to 0", which PDDL does not do.
Reading such an instance raises UPUsageError from the simulator, so the
warning was both silent where it mattered and wrong where it fired.

Enumerate the ground instances of each numeric function and name the ones
:init leaves out, with a PDDL-shaped repair hint. Cap the enumeration at
250 instances per function (a two-argument function over a big domain
would explode) and summarise long missing lists. Still a warning: a plan
that never reads the instance remains valid.

Adds tests/test_numeric_init.py (7 tests).
A plan that reads a numeric fluent instance :init never assigned (the
common case: (fuel t1) is set, (fuel t2) is not) made UPF raise
UPUsageError from is_applicable, and validate() let it escape — the CLI
died with a traceback and the library raised at the caller. The same
happens for effects that write such an instance (sim.apply).

Catch the evaluation failure, stop the plan at that step and record why
on StepResult.evaluation_error: the fluent name plus the PDDL repair.
The report says "could not be validated" instead of "Plan is INVALID" —
an unevaluable step proves nothing about the plan as a whole — and the
JSON step carries the message.

Adds tests/test_evaluation_errors.py (6 tests) and a shared
pyval/pddl_format.py for the fuel(t2) -> (fuel t2) rendering.
Two shapes of precondition produced "PRECONDITION FAILURE" with nothing
under it. The OR branch compared the number of collected sub-failures
with the number of disjuncts, so (or (and p q s) r) yielded four
failures for two disjuncts and reported none of them. And the walker had
no branch for quantifiers, implication or object equality, so
(forall (?y) (p ?y)) fell through silently.

Evaluate each disjunct with UPF's StateEvaluator instead of counting
failures, and add a final branch that evaluates and reports anything the
walker does not special-case. Route EQUALS through the boolean path when
its operands are not numeric: an object equality is a boolean condition,
not a numeric deficit, and it used to read "Could not evaluate numeric
comparison". Render expressions with " = " so the text is PDDL-shaped.

Adds tests/test_precondition_diagnostics.py (7 tests), including an
invariant that a FAILED step always carries diagnostics.
(:metric minimize (+ (* 2 (fuel)) 5)) produced no metric at all: the
evaluation used state.get_value, which UPF asserts on for arithmetic
sub-expressions, and the exception was swallowed into a bare None —
indistinguishable from a problem that declares no metric. 0.1.4 fixed the
same bug for goals by moving to StateEvaluator; the metric path kept it.
It now reports 17.0 on the probe that used to return nothing.

Domains declaring :action-costs carry MinimizeActionCosts, which is not
an expression over the final state, so (:metric minimize (total-cost))
was dropped too. Sum each executed action's declared cost, evaluated in
the state the action was applied in.

When a metric still cannot be evaluated, say so (n/a plus the reason)
instead of dropping it.
phases.goals held only the unsatisfied goals and was null both when no
plan had been executed and when every goal passed, so a consumer could
not tell "checked and green" from "not checked" — the ambiguity 0.1.5
removed from the plain-text report, still present in the JSON. is_valid
was missing from the payload entirely.

Add both: is_valid at the top level, and phases.goal_check with every
goal and its verdict whenever the execution phase ran. ValidationResult
keeps the full goal list alongside the unsatisfied subset, and
phases.goals is untouched so existing consumers keep working.

Adds tests/test_json_output.py (4 tests).
pyval domain.pddl problem.pddl plan.txt --trajectory --track fuel printed
"No numeric fluents to display." for any fluent with arguments. Two
comparisons were wrong at once: NumericTracker filtered on a display name
("fuel truck1") while snapshots are keyed by UPF's rendering
("fuel(truck1)"), and format_trajectory compared the raw reference with
the same key.

Route both through fluent_reference_matches, which accepts the bare
function name (all instances), "fuel truck1" and "fuel(truck1)". A
reference that matches nothing now warns with the fluents that do exist
(new "Warnings" section in the text report, `warnings` in the JSON)
instead of handing back an empty table.

Adds tests/test_tracking.py (9 tests) and documents the accepted
spellings in the README.
Reporting every unassigned instance of every numeric function was noisy:
a static table like (distance ?a ?b) is routinely given only for the
pairs that exist, so ordinary logistics benchmarks grew a warning per
run. The instances that actually hurt are the ones an effect writes — a
read of a missing static pair now gets the precise step-level error from
the previous commit instead.

Report a function when an action assigns or updates it, or when it has no
initial value at all; stay quiet about partial coverage of read-only
tables.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant