Repository navigation
fix: report the diagnostics pyval was swallowing (evaluation errors, disjunctions, metrics, --track) - #4
Open
ghostfix-pm wants to merge 9 commits into
Open
fix: report the diagnostics pyval was swallowing (evaluation errors, disjunctions, metrics, --track)#4ghostfix-pm wants to merge 9 commits into
ghostfix-pm wants to merge 9 commits into
Conversation
The Phase-1 warning was decided per declared function name, so a problem that initialised (fuel t1) got no warning at all for fuel(t2) — and the message claimed the missing value "defaults to 0", which PDDL does not do. Reading such an instance raises UPUsageError from the simulator, so the warning was both silent where it mattered and wrong where it fired. Enumerate the ground instances of each numeric function and name the ones :init leaves out, with a PDDL-shaped repair hint. Cap the enumeration at 250 instances per function (a two-argument function over a big domain would explode) and summarise long missing lists. Still a warning: a plan that never reads the instance remains valid. Adds tests/test_numeric_init.py (7 tests).
A plan that reads a numeric fluent instance :init never assigned (the common case: (fuel t1) is set, (fuel t2) is not) made UPF raise UPUsageError from is_applicable, and validate() let it escape — the CLI died with a traceback and the library raised at the caller. The same happens for effects that write such an instance (sim.apply). Catch the evaluation failure, stop the plan at that step and record why on StepResult.evaluation_error: the fluent name plus the PDDL repair. The report says "could not be validated" instead of "Plan is INVALID" — an unevaluable step proves nothing about the plan as a whole — and the JSON step carries the message. Adds tests/test_evaluation_errors.py (6 tests) and a shared pyval/pddl_format.py for the fuel(t2) -> (fuel t2) rendering.
Two shapes of precondition produced "PRECONDITION FAILURE" with nothing under it. The OR branch compared the number of collected sub-failures with the number of disjuncts, so (or (and p q s) r) yielded four failures for two disjuncts and reported none of them. And the walker had no branch for quantifiers, implication or object equality, so (forall (?y) (p ?y)) fell through silently. Evaluate each disjunct with UPF's StateEvaluator instead of counting failures, and add a final branch that evaluates and reports anything the walker does not special-case. Route EQUALS through the boolean path when its operands are not numeric: an object equality is a boolean condition, not a numeric deficit, and it used to read "Could not evaluate numeric comparison". Render expressions with " = " so the text is PDDL-shaped. Adds tests/test_precondition_diagnostics.py (7 tests), including an invariant that a FAILED step always carries diagnostics.
(:metric minimize (+ (* 2 (fuel)) 5)) produced no metric at all: the evaluation used state.get_value, which UPF asserts on for arithmetic sub-expressions, and the exception was swallowed into a bare None — indistinguishable from a problem that declares no metric. 0.1.4 fixed the same bug for goals by moving to StateEvaluator; the metric path kept it. It now reports 17.0 on the probe that used to return nothing. Domains declaring :action-costs carry MinimizeActionCosts, which is not an expression over the final state, so (:metric minimize (total-cost)) was dropped too. Sum each executed action's declared cost, evaluated in the state the action was applied in. When a metric still cannot be evaluated, say so (n/a plus the reason) instead of dropping it.
phases.goals held only the unsatisfied goals and was null both when no plan had been executed and when every goal passed, so a consumer could not tell "checked and green" from "not checked" — the ambiguity 0.1.5 removed from the plain-text report, still present in the JSON. is_valid was missing from the payload entirely. Add both: is_valid at the top level, and phases.goal_check with every goal and its verdict whenever the execution phase ran. ValidationResult keeps the full goal list alongside the unsatisfied subset, and phases.goals is untouched so existing consumers keep working. Adds tests/test_json_output.py (4 tests).
pyval domain.pddl problem.pddl plan.txt --trajectory --track fuel printed
"No numeric fluents to display." for any fluent with arguments. Two
comparisons were wrong at once: NumericTracker filtered on a display name
("fuel truck1") while snapshots are keyed by UPF's rendering
("fuel(truck1)"), and format_trajectory compared the raw reference with
the same key.
Route both through fluent_reference_matches, which accepts the bare
function name (all instances), "fuel truck1" and "fuel(truck1)". A
reference that matches nothing now warns with the fluents that do exist
(new "Warnings" section in the text report, `warnings` in the JSON)
instead of handing back an empty table.
Adds tests/test_tracking.py (9 tests) and documents the accepted
spellings in the README.
Reporting every unassigned instance of every numeric function was noisy: a static table like (distance ?a ?b) is routinely given only for the pairs that exist, so ordinary logistics benchmarks grew a warning per run. The instances that actually hurt are the ones an effect writes — a read of a missing static pair now gets the precise step-level error from the previous commit instead. Report a function when an action assigns or updates it, or when it has no initial value at all; stay quiet about partial coverage of read-only tables.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four defects in the diagnostic paths, all of the same shape: pyval knew something was wrong and told the caller nothing. Each one is reproduced on the parent commit below, and the fixes come with regression tests. No packaging, no CI, no dependency changes.
Bug fixes
1. An unevaluable step crashed validation with a traceback. Numeric fluents have no default in PDDL, and the simulator raises
UPUsageErrorwhen a condition or effect reads an instance:initnever assigned.simulatelet that escape, so the CLI died mid-report:The same input now stops at the offending step, names the fluent and the repair, and refuses to give a verdict it did not earn:
2.
✗ PRECONDITION FAILUREwith nothing under it.decompose_preconditionsdecidedorby counting collected sub-failures against the number of disjuncts, so(or (and (p) (q) (s)) (r))produced four failures for two disjuncts and reported none; and any node it did not handle (forall, implication, object equality) fell straight through. Both produced a failed step with an emptyunsatisfied_preconditionslist. Disjuncts are now evaluated withStateEvaluatorand an unhandled node gets a final branch, so:Object equality (
:equality) also no longer goes through the numeric path, where it answered "Could not evaluate numeric comparison" withtype: "numeric".3. Metrics silently disappeared.
_evaluate_metricusedstate.get_value, which asserts on arithmetic sub-expressions — the same root cause 0.1.4 fixed for goals, left behind on the metric path.(:metric minimize (+ (* 2 (fuel l1)) 5))returned no metric at all (expected17.0), and:action-costsdomains were dropped entirely becauseMinimizeActionCostsis not an expression over the final state. Both are reported now, and a metric that genuinely cannot be evaluated shows asn/awith the reason.4.
--tracknever matched anything. The README's own commandpyval domain.pddl problem.pddl plan.txt --trajectory --track fuelprinted "No numeric fluents to display.": the tracker compared a display name (fuel truck1) with the snapshot key (fuel(truck1)), andformat_trajectorycompared the raw reference with that same key. Both now accept the bare function name (every instance), the PDDL spelling and UPF's rendering; an unmatched reference warns with the fluents that do exist instead of printing an empty table.Also fixed
The Phase-1 "numeric function has no initial value" warning was decided per declared name, so initialising
(fuel t1)silenced it forfuel(t2), and it claimed the missing value "defaults to 0" — which is not what the simulator does (it raises). It now enumerates ground instances and reports the ones an action writes, with a valid PDDL repair hint. Static tables such as(distance ?a ?b)are left alone when they are partially specified, since that is normal.Additive output changes
is_valid,phases.goal_check(every goal and its verdict) andwarnings; a failed step may carryevaluation_error.phases.goalsis unchanged, so existing consumers keep working —goal_checkis what distinguishes "checked and green" from "not checked", whichgoals: nullcould not.ValidationResultgainsgoal_results(all goals) alongside the existingunsatisfied_goals.Tests
40 new tests (67 → 107), each defect pinned by a test that fails on the parent commit. Text, verbose and trajectory output for every existing fixture is byte-identical to the parent commit; the JSON diff is purely additive (verified by diffing full reports for six fixtures before and after).
Files Changed
pyval/plan_simulator.pyStepResult.evaluation_error;StateEvaluatormetrics,:action-costssum, unevaluable metric reasonpyval/diagnostics.py:equalityoff the numeric pathpyval/syntax_checker.py:initcoverage, capped enumeration, PDDL repair hintspyval/report_formatter.pyEVALUATION ERRORstep lines, "could not be validated",n/ametrics,is_valid/goal_check/warnings, unified--trackfilterpyval/numeric_tracker.pyfluent_reference_matches/parse_fluent_reference, unknown-reference trackingpyval/validator.py--trackvalues that select nothingpyval/models.pyStepResult.evaluation_error,ValidationResult.goal_resultspyval/pddl_format.pyas_pddl_call:fuel(t2)→(fuel t2)for repair hintstests/test_numeric_init.pytests/test_precondition_diagnostics.pytests/test_evaluation_errors.pytests/test_metrics.pytests/test_tracking.pytests/test_json_output.pyREADME.md--trackspellingsCHANGELOG.md[Unreleased]entries