From 9eb130dcc21a1b7ca888c8ece87009bf4b223ab5 Mon Sep 17 00:00:00 2001 From: Bill Sommers <14631342+bsommers@users.noreply.github.com> Date: Wed, 11 Nov 2026 23:01:31 -0500 Subject: [PATCH 1/2] chore(release): 1.8.3 Date the 1.8.2 changelog entry (released 2026-11-08) and move the emitter, REPL and prover fixes that landed after that tag into a new 1.8.3 entry. Bump the package version to 1.8.3. Co-Authored-By: Claude Fable 5.2 --- CHANGELOG.md | 10 ++++++++-- ontologic.cabal | 2 +- package.yaml | 2 +- 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b1e5a0..d096755 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,9 +7,9 @@ All notable changes to OntoLogic are recorded here. The format follows Entries before 1.5.1 were back-filled from git history. v1.4.0 and v1.5.0 were tagged after the fact, on their version-bump commits. -## [1.8.2] - Unreleased +## [1.8.3] - 2026-11-11 -Fixes to the v1.8.1 solver resource controls, plus emitter, REPL and prover fixes. +Emitter, REPL and prover fixes found while writing the user guide. ### Fixed - **Repeated bounds in generated Python and Java.** A subclass that restated an inherited @@ -28,6 +28,12 @@ Fixes to the v1.8.1 solver resource controls, plus emitter, REPL and prover fixe Z3 or CVC5. - **VS Code snippets expanded to obsolete syntax.** They now use the current grammar, and the extension README and `build_extension.sh` report the real package version. + +## [1.8.2] - 2026-11-08 + +Fixes to the v1.8.1 solver resource controls. + +### Fixed - **`--solve-timeout` was ignored by CVC5.** `--tlimit` was passed after the `-` stdin marker, where cvc5 ignores options. It now comes before `-`. - **Timeouts behave the same on every backend.** External solvers now also run under a diff --git a/ontologic.cabal b/ontologic.cabal index 0dcad90..4fa9b10 100644 --- a/ontologic.cabal +++ b/ontologic.cabal @@ -5,7 +5,7 @@ cabal-version: 1.12 -- see: https://github.com/sol/hpack name: ontologic -version: 1.8.2 +version: 1.8.3 homepage: https://github.com/bsommers/OntoLogic#readme bug-reports: https://github.com/bsommers/OntoLogic/issues author: OntoLogic Team diff --git a/package.yaml b/package.yaml index 670328a..a0fc09f 100644 --- a/package.yaml +++ b/package.yaml @@ -1,5 +1,5 @@ name: ontologic -version: 1.8.2 +version: 1.8.3 github: "bsommers/OntoLogic" license: Apache-2.0 author: "OntoLogic Team" From eff108b0d90bc7ccc8ab65273a9f919d1b820abf Mon Sep 17 00:00:00 2001 From: Bill Sommers <14631342+bsommers@users.noreply.github.com> Date: Wed, 11 Nov 2026 23:13:32 -0500 Subject: [PATCH 2/2] feat(solver): warn when a prover without arithmetic gives an all-clear The native prover and Vampire (on TPTP) read gt/gte/lt/lte as uninterpreted predicates, so their SAT can miss a conflict that needs arithmetic, such as an integer gap (gt 5 with lt 6 on an Integer). When the ontology uses those comparisons and one of them gives an all-clear: - --solve prints a warning on stderr; - the REPL appends it to :solve, :model, :explain, :unsat and a refuted :prove; - the language server appends it to the ontologic.solve details. auto warns only when it falls back to the native prover. Z3 and CVC5 answers, and every UNSAT, are unchanged. Co-Authored-By: Claude Fable 5.2 --- CHANGELOG.md | 13 ++++- docs/user-guide/axioms-and-reasoning.md | 2 +- docs/user-guide/editor-and-lsp.md | 2 +- docs/user-guide/repl.md | 2 + docs/user-guide/troubleshooting.md | 3 ++ src/LSP.hs | 5 +- src/Main.hs | 9 ++++ src/OntoLogic.hs | 19 +++++++ src/REPL.hs | 30 ++++++++--- src/Solver.hs | 32 +++++++++++ test/MainSpec.hs | 6 +++ test/RegressionBugsSpec.hs | 71 ++++++++++++++++++++++++- test/fixtures/integer_gap.logic | 7 +++ 13 files changed, 187 insertions(+), 14 deletions(-) create mode 100644 test/fixtures/integer_gap.logic diff --git a/CHANGELOG.md b/CHANGELOG.md index d096755..e5f5ea9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,18 @@ tagged after the fact, on their version-bump commits. ## [1.8.3] - 2026-11-11 -Emitter, REPL and prover fixes found while writing the user guide. +Emitter, REPL and prover fixes found while writing the user guide, and a warning for answers +reached without arithmetic. + +### Added +- **Warning when a prover without arithmetic gives an all-clear.** The native prover and + Vampire read `gt`, `gte`, `lt` and `lte` without arithmetic, so their SAT can miss a conflict + that needs it (an integer gap such as `gt 5` with `lt 6`). When the ontology uses those + comparisons and one of them reports SAT or every concept satisfiable, `--solve` prints a + warning on stderr. The REPL adds the same line to `:solve`, `:model`, `:explain`, `:unsat` + and a refuted `:prove`, and the language server adds it to the `ontologic.solve` result. + `auto` warns only when it falls back to the native prover. Z3 and CVC5 answers, and every + UNSAT, are unchanged. ### Fixed - **Repeated bounds in generated Python and Java.** A subclass that restated an inherited diff --git a/docs/user-guide/axioms-and-reasoning.md b/docs/user-guide/axioms-and-reasoning.md index 3ea74da..01ef690 100644 --- a/docs/user-guide/axioms-and-reasoning.md +++ b/docs/user-guide/axioms-and-reasoning.md @@ -157,7 +157,7 @@ A model is one possible world, usually a tiny one. Concepts that are empty in th | `native` | Built into the binary (tableau and resolution). No install needed; honours `-T`; reports unknown on step or clause limits | | `auto` | Tries `z3`, `cvc5`, `vampire` in order, skipping any that is not installed; if none is installed, falls back to `native`. A solver that is installed but fails is reported, not skipped | -Practical guidance: use `z3` or `cvc5` for ontologies with numeric property bounds. `native` and `vampire` have no arithmetic. OntoLogic gives them one background axiom for each pair of numeric bounds in the file that no number can meet together (such as `gte 100` and `lt 50`), so they catch contradictions like the balance example above. They still miss contradictions that need real arithmetic, such as integer gaps (`gt 5` and `lt 6` on an `Integer`) or bounds derived from sums. The domain fixtures in `test/fixtures/domains/` are checked with Z3 (then CVC5) for this reason. For purely taxonomic and relational reasoning, all backends agree. If two provers disagree, trust UNSAT from an SMT solver and treat the other's SAT as incomplete. +Practical guidance: use `z3` or `cvc5` for ontologies with numeric property bounds. `native` and `vampire` have no arithmetic. OntoLogic gives them one background axiom for each pair of numeric bounds in the file that no number can meet together (such as `gte 100` and `lt 50`), so they catch contradictions like the balance example above. They still miss contradictions that need real arithmetic, such as integer gaps (`gt 5` and `lt 6` on an `Integer`) or bounds derived from sums. When one of them gives an all-clear on an ontology that uses `gt`, `gte`, `lt` or `lte`, OntoLogic prints a warning that the answer was reached without arithmetic and should be confirmed with `z3` or `cvc5`. The domain fixtures in `test/fixtures/domains/` are checked with Z3 (then CVC5) for this reason. For purely taxonomic and relational reasoning, all backends agree. If two provers disagree, trust UNSAT from an SMT solver and treat the other's SAT as incomplete. ## Time and memory limits diff --git a/docs/user-guide/editor-and-lsp.md b/docs/user-guide/editor-and-lsp.md index 3628d6e..8416bfb 100644 --- a/docs/user-guide/editor-and-lsp.md +++ b/docs/user-guide/editor-and-lsp.md @@ -16,7 +16,7 @@ Capabilities reported by `initialize`: | Formatting | Re-indents the whole document (honours the editor's tab size) | | Semantic tokens | Keyword, type, class, property, function, variable, string, number, comment, operator | | Code actions | Quick-fix entries "Verify SMT Consistency", "Explain SMT Unsatisfiable Core", "Inspect SMT Satisfying Model" | -| Execute command | `ontologic.solve`, `ontologic.explainUnsat`, `ontologic.inspectModel`. They run asynchronously with progress notifications and can be cancelled; they always use the `auto` prover | +| Execute command | `ontologic.solve`, `ontologic.explainUnsat`, `ontologic.inspectModel`. They run asynchronously with progress notifications and can be cancelled; they always use the `auto` prover. When `auto` falls back to the native prover on an ontology with numeric comparisons, the `ontologic.solve` SAT `details` end with a warning that the theory was checked without arithmetic | Imports use the same search path as the CLI, including `$ONTOLOGIC_PATH` (set it in the environment the editor is launched from). The server has no `--lib-path` option. diff --git a/docs/user-guide/repl.md b/docs/user-guide/repl.md index 46c4dd6..a55cda5 100644 --- a/docs/user-guide/repl.md +++ b/docs/user-guide/repl.md @@ -36,6 +36,8 @@ Input is read one line at a time, so each form or command must fit on one line. Anything that is not a command is parsed as DSL. A line may contain one or several top-level forms; they are merged into the session. The linter runs after each addition and, if it finds a problem, prints `⚠️ Lint warning: ...` for the first one; the forms stay in the session regardless. +The `native` and `vampire` provers have no arithmetic. When one of them is active (or `auto` has no external solver to use) and the session uses `gt`, `gte`, `lt` or `lte`, an all-clear from `:solve`, `:model`, `:explain`, `:unsat` or `:prove` ends with `⚠️ ... checks this theory without arithmetic ...`. Confirm such an answer with `:prover z3` or `:prover cvc5`. + ## Example session ```text diff --git a/docs/user-guide/troubleshooting.md b/docs/user-guide/troubleshooting.md index 082092a..f8e8e2b 100644 --- a/docs/user-guide/troubleshooting.md +++ b/docs/user-guide/troubleshooting.md @@ -48,6 +48,7 @@ Contradictions can be reported twice (once per operand order); fix the first and | `SMT Solver Error: Failed to execute Z3 solver: ... Ensure 'z3' is installed and available in PATH.` | You asked for a specific prover (`-p z3`) that is not installed. Install it, or use `-p auto`/`-p native` | | `Unknown prover 'x'. Valid options: z3, cvc5, vampire, native, auto` | Bad `-p` value | | `⚠️ --solve-memory is not supported by CVC5 and is ignored.` | Memory limits apply to Z3 only | +| `⚠️ The native prover checks this theory without arithmetic: ...` (or `Vampire checks ...`) | The ontology uses `gt`, `gte`, `lt` or `lte`, and the all-clear came from `native` or `vampire`. It can miss a conflict that needs arithmetic. Re-run with `-p z3` or `-p cvc5`. See [below](#sat-from-one-prover-unsat-from-another) | | `Concept coherence check could not run: ...` | Exit 2; the second solver step failed. Usually the same cause as an adjacent solver error | | `Native resolution prover timed out / step limit / clause limit exceeded` | The built-in prover gave up (reported as unknown). Use `z3` or `cvc5` | @@ -55,6 +56,8 @@ Contradictions can be reported twice (once per operand order); fix the first and The native prover and Vampire have no arithmetic. They are told which pairs of numeric bounds in the file exclude each other, so a plain conflict (a universal `gte 100` and an existential witness with `lt 50`) is UNSAT on every backend. A contradiction that needs real arithmetic, such as an integer gap (`gt 5` and `lt 6` on an `Integer`), is UNSAT under `-p z3` or `-p cvc5` and can look SAT under `-p native` or `-p vampire`. Prefer Z3 or CVC5 whenever properties have numeric bounds. +OntoLogic flags the answers this affects. When the ontology uses `gt`, `gte`, `lt` or `lte` and `native` or `vampire` gives an all-clear (SAT, or every concept satisfiable), `--solve` prints a warning on stderr that the theory was checked without arithmetic. The REPL adds the same line to its answer. `-p auto` warns only when it has fallen back to the native prover. An UNSAT from these provers needs no confirmation. + ### Auto picked the native prover `--prover auto` falls back to the native prover only when none of `z3`, `cvc5`, `vampire` is on `PATH`. Check with `which z3 cvc5 vampire`. Solvers installed through sandboxed packages (snap, flatpak) may not be visible or may not read your files. diff --git a/src/LSP.hs b/src/LSP.hs index aaa67bd..162a0d9 100644 --- a/src/LSP.hs +++ b/src/LSP.hs @@ -38,7 +38,7 @@ import OntoLogic import Parser (parseRegistry, parseRegistryWithSource) import Linter (lintRegistryFindings, lintRegistryFindingsWithSource, LintFinding(..)) import IR (compileRegistry, irClasses, compClassName, compParentClass, compProperties, compPropName, compPropType, compConstraints, CompiledProperty, compCardinality, compDefaultValue) -import Solver (checkConsistencyWith, explainUnsatWith, extractModel, Prover(AutoProver), UnsatExplanation(..)) +import Solver (checkConsistencyWith, explainUnsatWith, extractModel, arithmeticCaveat, Prover(AutoProver), UnsatExplanation(..)) data LSPState = LSPState { lspDocs :: Map.Map String String -- URI -> Content @@ -440,8 +440,9 @@ handleLSPRequestIO state raw case cmd of "ontologic.solve" -> do res <- checkConsistencyWith AutoProver reg + caveat <- arithmeticCaveat AutoProver reg let (status, details) = case res of - Sat _ -> ("sat", "Satisfiable (SAT): Consistent Model Found") + Sat _ -> ("sat", "Satisfiable (SAT): Consistent Model Found" ++ maybe "" (". " ++) caveat) Unsat -> ("unsat", "Unsatisfiable (UNSAT): Logical Contradiction Detected") Unknown u -> ("unknown", u) SolverError e-> ("error", e) diff --git a/src/Main.hs b/src/Main.hs index 46a1a95..4e1b8a5 100644 --- a/src/Main.hs +++ b/src/Main.hs @@ -20,6 +20,7 @@ import Solver , findUnsatConceptsWithOptions , explainUnsatWithOptions , untranslatedAxioms + , arithmeticCaveat , resolveProver , solverExitCode , worst @@ -177,6 +178,14 @@ runSolverCheck isFreeConsts showExplain showModel maybeP maybeTimeout maybeMem r Right empties -> do putStrLn ("❌ Unsatisfiable / empty concept(s): " ++ unwords empties) return (ExitFailure 1) + -- An all-clear from a backend without arithmetic can miss numeric conflicts: say so. + let allClear = case (res, coh) of + (Sat _, _) -> True + (_, Right []) -> True + _ -> False + if allClear + then arithmeticCaveat prover reg >>= mapM_ (\w -> hPutStrLn stderr ("⚠️ " ++ w)) + else return () exitWith (worst [solverExitCode res, coherenceCode]) processTarget :: Bool -> SourceTable -> String -> String -> Maybe String -> Maybe String -> Maybe String -> Bool -> Registry -> IO () diff --git a/src/OntoLogic.hs b/src/OntoLogic.hs index 11e96c1..8925848 100644 --- a/src/OntoLogic.hs +++ b/src/OntoLogic.hs @@ -503,3 +503,22 @@ disjointBoundPairs axs = | l@(lop, a) <- lowers, u@(uop, b) <- uppers , num a > num b || (num a == num b && (lop == "gt" || uop == "lt")) ] + +-- | Whether any axiom compares values with an ordering operator (gt, gte, lt, lte). A declared +-- relation that shares one of those names is a relation, not a comparison. +usesOrderingComparison :: Registry -> Bool +usesOrderingComparison reg = any go (axioms reg) + where + ordOps = filter (`notElem` map relName (relations reg)) ["gt", "gte", "lt", "lte"] + go f = case f of + Predicate op _ -> op `elem` ordOps + And a b -> go a || go b + Or a b -> go a || go b + Implies a b -> go a || go b + Iff a b -> go a || go b + Not a -> go a + ForAll _ b -> go b + Exists _ b -> go b + TypedForAll _ _ b -> go b + TypedExists _ _ b -> go b + _ -> False diff --git a/src/REPL.hs b/src/REPL.hs index c5af599..1ca6857 100644 --- a/src/REPL.hs +++ b/src/REPL.hs @@ -19,6 +19,7 @@ import Solver , proveSubsumptionWith , findUnsatConceptsWith , untranslatedAxioms + , arithmeticCaveat , Prover(..) , parseProver , proverName @@ -142,7 +143,11 @@ evalREPLCommand state input Left err -> "⚠️ Concept coherence check could not run: " ++ err Right [] -> "✅ All concepts are satisfiable (0 empty concepts)." Right empties -> "❌ Unsatisfiable / empty concept(s): " ++ unwords empties - return (state, warning ++ consistency ++ "\n" ++ coherence) + note <- case (res, coh) of + (Sat _, _) -> caveatLine state + (_, Right []) -> caveatLine state + _ -> return "" + return (state, warning ++ consistency ++ "\n" ++ coherence ++ note) | input `elem` [":explain", ":unsat-core", ":explain-unsat"] = do res <- explainUnsatWith (replProver state) (replRegistry state) @@ -150,7 +155,9 @@ evalREPLCommand state input Left err -> return (state, "❌ SMT Explanation Error:\n" ++ err) Right explanation -> if null (unsatCoreNames explanation) - then return (state, "✅ SMT Satisfiable (SAT) — No contradictory axioms or unsat cores detected.") + then do + note <- caveatLine state + return (state, "✅ SMT Satisfiable (SAT) — No contradictory axioms or unsat cores detected." ++ note) else return (state, unlines [ "❌ SMT Unsatisfiable (UNSAT) — Minimal Unsat Core Identified:" , " Core Assertions: " ++ unwords (unsatCoreNames explanation) @@ -162,10 +169,11 @@ evalREPLCommand state input | input `elem` [":model", ":inspect-model"] = do res <- checkConsistencyWith (replProver state) (replRegistry state) case res of - Sat _ -> + Sat _ -> do + note <- caveatLine state case extractModel res of - Just m -> return (state, "✅ SMT Satisfiable (SAT) — Extracted Valuation Model:\n" ++ m) - Nothing -> return (state, "✅ SMT Satisfiable (SAT) — Valuation model is empty or implicit.") + Just m -> return (state, "✅ SMT Satisfiable (SAT) — Extracted Valuation Model:\n" ++ m ++ note) + Nothing -> return (state, "✅ SMT Satisfiable (SAT) — Valuation model is empty or implicit." ++ note) Unsat -> return (state, "❌ SMT Unsatisfiable (UNSAT) — No satisfying model exists. Run ':explain' to view the unsat core.") Unknown out -> return (state, "⚠️ SMT Solver Result Inconclusive: " ++ out) SolverError err -> return (state, "❌ SMT Solver Error:\n" ++ err) @@ -177,7 +185,9 @@ evalREPLCommand state input res <- proveSubsumptionWith (replProver state) (replRegistry state) child parent case res of Subsumed -> return (state, "✅ Theorem Proved via " ++ proverName (replProver state) ++ ": '" ++ child ++ "' ⊑ '" ++ parent ++ "' (Every instance of " ++ child ++ " is necessarily a " ++ parent ++ ").") - NotSubsumed model -> return (state, "❌ Theorem Disproved via " ++ proverName (replProver state) ++ ": '" ++ child ++ "' is NOT subsumed by '" ++ parent ++ "'. Counterexample model:\n" ++ model) + NotSubsumed model -> do + note <- caveatLine state + return (state, "❌ Theorem Disproved via " ++ proverName (replProver state) ++ ": '" ++ child ++ "' is NOT subsumed by '" ++ parent ++ "'. Counterexample model:\n" ++ model ++ note) ProofError err -> return (state, "❌ Solver Error in Subsumption Proof:\n" ++ err) _ -> return (state, "⚠️ Usage: :prove ") @@ -185,7 +195,9 @@ evalREPLCommand state input res <- findUnsatConceptsWith (replProver state) (replRegistry state) case res of Left err -> return (state, "❌ SMT Solver Error:\n" ++ err) - Right [] -> return (state, "✅ All concepts are satisfiable (0 empty concepts) via " ++ proverName (replProver state) ++ ".") + Right [] -> do + note <- caveatLine state + return (state, "✅ All concepts are satisfiable (0 empty concepts) via " ++ proverName (replProver state) ++ "." ++ note) Right unsatList -> return (state, "❌ Unsatisfiable / Contradictory Concepts Found via " ++ proverName (replProver state) ++ ":\n" ++ unlines (map (" - " ++) unsatList)) | input == ":clear" = @@ -229,6 +241,10 @@ evalREPLCommand state input lintMsg = if null errs then "" else "\n⚠️ Lint warning: " ++ head errs in return (state { replRegistry = merged }, "Added to registry." ++ lintMsg) +-- | Extra line for an all-clear answer from a prover without arithmetic; empty when none applies. +caveatLine :: REPLState -> IO String +caveatLine state = maybe "" ("\n⚠️ " ++) <$> arithmeticCaveat (replProver state) (replRegistry state) + trim :: String -> String trim = f . f where f = reverse . dropWhile (`elem` (" \t\r\n" :: String)) diff --git a/src/Solver.hs b/src/Solver.hs index 647520c..cc65481 100644 --- a/src/Solver.hs +++ b/src/Solver.hs @@ -32,6 +32,8 @@ module Solver , emitSMTForConsistency , emitSMTBase , untranslatedAxioms + , backendWithoutArithmetic + , arithmeticCaveat , worst , extractModel , UnsatExplanation(..) @@ -52,11 +54,13 @@ import Emitters.TPTP (emitTPTP, emitTPTPSubsumption, emitTPTPCoherence) import FOLTheory (buildTheory, buildNamedTheory, theoryText, theorySkipped, conceptPred) import System.Process (readProcessWithExitCode) import System.Exit (ExitCode(..)) +import System.Directory (findExecutable) import System.IO.Error (isDoesNotExistError) import Control.Exception (try, IOException) import System.Timeout (timeout) import Data.List (isPrefixOf, isInfixOf) import Data.Char (toLower) +import Data.Maybe (isNothing) import qualified Data.Map as Map -- | Supported automated theorem prover backends @@ -223,6 +227,34 @@ worst codes untranslatedAxioms :: Registry -> [(FOL, String)] untranslatedAxioms = theorySkipped . buildTheory +-- | The backend that answers for this prover, when that backend has no arithmetic and so reads +-- ordering comparisons as uninterpreted predicates: the native prover, and Vampire on the +-- TPTP encoding. 'AutoProver' reaches the native prover only when no external solver is on +-- PATH; otherwise it sends SMT-LIB, where comparisons keep their arithmetic meaning. +backendWithoutArithmetic :: Prover -> IO (Maybe Prover) +backendWithoutArithmetic NativeProver = return (Just NativeProver) +backendWithoutArithmetic Vampire = return (Just Vampire) +backendWithoutArithmetic AutoProver = do + found <- mapM findExecutable ["z3", "cvc5", "vampire"] + return (if all isNothing found then Just NativeProver else Nothing) +backendWithoutArithmetic _ = return Nothing + +-- | Caveat for an all-clear answer (SAT, every concept satisfiable, not subsumed) from a backend +-- without arithmetic, on a theory that uses ordering comparisons. Such an answer can be wrong +-- where the conflict needs arithmetic, so callers show this next to it. UNSAT needs no caveat: +-- the background axioms given to these backends hold for every number. +arithmeticCaveat :: Prover -> Registry -> IO (Maybe String) +arithmeticCaveat p reg + | usesOrderingComparison reg = fmap caveat <$> backendWithoutArithmetic p + | otherwise = return Nothing + where + caveat backend = + who backend ++ " checks this theory without arithmetic: it catches numeric bounds that " ++ + "directly exclude each other, but can miss conflicts that need arithmetic. " ++ + "Confirm this answer with z3 or cvc5." + who NativeProver = "The native prover" + who other = proverName other + -- | Run raw SMT-LIB2 code through the Z3 solver via standard input runZ3 :: String -> IO (Either String String) runZ3 = runProver Z3 diff --git a/test/MainSpec.hs b/test/MainSpec.hs index ab89399..e882572 100644 --- a/test/MainSpec.hs +++ b/test/MainSpec.hs @@ -34,6 +34,12 @@ spec = do (_, _, err) <- readProcessWithExitCode "stack" ["exec", "ontologic-exe", "--", "-i", "logic/fleet.logic", "--solve", "-p", "native", "--solve-memory", "512"] "" err `shouldContain` "--solve-memory is not supported by Native" + it "warns when a prover without arithmetic gives an all-clear on numeric comparisons" $ do + (code, out, err) <- readProcessWithExitCode "stack" ["exec", "ontologic-exe", "--", "-i", "test/fixtures/integer_gap.logic", "--solve", "-p", "native"] "" + code `shouldBe` ExitSuccess + out `shouldContain` "SMT Satisfiable (SAT)" + err `shouldContain` "The native prover checks this theory without arithmetic" + it "runs -t lint successfully without -o flag" $ do (exitCode, stdout, _) <- readProcessWithExitCode "stack" ["exec", "ontologic-exe", "--", "-i", "logic/fleet.logic", "-t", "lint"] "" exitCode `shouldBe` ExitSuccess diff --git a/test/RegressionBugsSpec.hs b/test/RegressionBugsSpec.hs index a2e795a..45c83ff 100644 --- a/test/RegressionBugsSpec.hs +++ b/test/RegressionBugsSpec.hs @@ -1,17 +1,21 @@ -- | Regression tests for defects found while writing the 1.8.2 user guide: -- emitter output for subclasses, the REPL/CLI unsat-core explanation, and numeric-bound --- contradictions on provers without arithmetic. +-- contradictions on provers without arithmetic, including the caveat those provers attach +-- to an all-clear answer. module RegressionBugsSpec (spec) where import Test.Hspec +import Control.Monad (forM_) import Data.List (isInfixOf, isPrefixOf) +import Data.Maybe (isJust, isNothing) import Data.Text (pack) import System.Directory (findExecutable) import Text.Megaparsec (parse, errorBundlePretty) import OntoLogic import Parser (parseRegistry) import IR -import Solver (checkConsistencyWith, explainUnsatWith, Prover(..), UnsatExplanation(..)) +import REPL (evalREPLCommand, REPLState(..), initialREPLState) +import Solver (checkConsistencyWith, explainUnsatWith, arithmeticCaveat, Prover(..), UnsatExplanation(..)) import Emitters.Java (emitJavaWithPackage) import Emitters.OpenApi (emitOpenApi) import Emitters.Python (emitPython) @@ -62,6 +66,28 @@ sensorReg = load $ unlines , "(forall (s) (implies (SmartSensor s) (gt batteryLevel 100)))" ] +-- | An integer gap: no Integer is above 5 and below 6. Only arithmetic finds the conflict. +-- Same content as test/fixtures/integer_gap.logic. +gapReg :: Registry +gapReg = load $ unlines + [ "(def-concept Tank)" + , "(def-property level Tank Integer)" + , "(forall (t) (implies (Tank t) (gt level 5)))" + , "(exists (t) (and (Tank t) (lt level 6)))" + ] + +-- | A declared relation named like an ordering operator: a relation, not a comparison. +relationLtReg :: Registry +relationLtReg = load $ unlines + [ "(def-concept Version)" + , "(def-relation lt Version Version)" + , "(forall (a b) (implies (lt a b) (not (lt b a))))" + ] + +-- | A REPL session on the native prover, which has no arithmetic. +nativeSession :: Registry -> REPLState +nativeSession reg = initialREPLState { replProver = NativeProver, replRegistry = reg } + constraintsOf :: Name -> Name -> Registry -> [BoundConstraint] constraintsOf cls prop reg = concat [ compConstraints p @@ -123,6 +149,47 @@ spec = do unsatCoreNames e `shouldSatisfy` (not . null) unwords (unsatCoreDetails e) `shouldSatisfy` ("SmartSensor.batteryLevel" `isInfixOf`) + describe "all-clear answers from provers without arithmetic" $ do + it "recognises ordering comparisons, but not a relation that shares an operator name" $ do + usesOrderingComparison gapReg `shouldBe` True + usesOrderingComparison relationLtReg `shouldBe` False + usesOrderingComparison (load "(def-concept A)\n(def-concept B A)") `shouldBe` False + it "gives a caveat for the native prover and Vampire, not for Z3 or CVC5" $ do + arithmeticCaveat NativeProver gapReg >>= (`shouldSatisfy` maybe False ("The native prover" `isPrefixOf`)) + arithmeticCaveat Vampire gapReg >>= (`shouldSatisfy` maybe False ("Vampire" `isPrefixOf`)) + arithmeticCaveat Z3 gapReg `shouldReturn` Nothing + arithmeticCaveat CVC5 gapReg `shouldReturn` Nothing + it "gives no caveat when the theory has no ordering comparison" $ + arithmeticCaveat NativeProver relationLtReg `shouldReturn` Nothing + it "auto gives a caveat only when it falls back to the native prover" $ do + external <- mapM findExecutable ["z3", "cvc5", "vampire"] + caveat <- arithmeticCaveat AutoProver gapReg + isJust caveat `shouldBe` all isNothing external + it "the native prover answers SAT on an integer gap (the case the caveat is for)" $ + checkConsistencyWith NativeProver gapReg >>= (`shouldSatisfy` isSat) + it "Z3 answers UNSAT on the same integer gap" $ withProver "z3" $ + checkConsistencyWith Z3 gapReg >>= (`shouldSatisfy` isUnsat) + it "REPL :solve adds the caveat to a native SAT answer" $ do + (_, out) <- evalREPLCommand (nativeSession gapReg) ":solve" + out `shouldContain` "SMT Satisfiable (SAT)" + out `shouldContain` "The native prover checks this theory without arithmetic" + it "REPL :model, :unsat and :explain carry the same caveat" $ + forM_ [":model", ":unsat", ":explain"] $ \cmd -> do + (_, out) <- evalREPLCommand (nativeSession gapReg) cmd + out `shouldContain` "without arithmetic" + it "REPL :prove adds the caveat to a failed subsumption proof" $ do + (_, out) <- evalREPLCommand (nativeSession truckReg) ":prove Vehicle Truck" + out `shouldContain` "NOT subsumed" + out `shouldContain` "without arithmetic" + it "REPL adds no caveat to an UNSAT answer" $ do + (_, out) <- evalREPLCommand (nativeSession pumpConflictReg) ":solve" + out `shouldContain` "UNSAT" + out `shouldNotContain` "without arithmetic" + it "REPL adds no caveat when Z3 answers" $ withProver "z3" $ do + (_, out) <- evalREPLCommand ((nativeSession gapReg) { replProver = Z3 }) ":solve" + out `shouldContain` "UNSAT" + out `shouldNotContain` "without arithmetic" + describe "numeric-bound contradictions without arithmetic" $ do it "native prover finds the forall/exists bound conflict" $ checkConsistencyWith NativeProver pumpConflictReg >>= (`shouldSatisfy` isUnsat) diff --git a/test/fixtures/integer_gap.logic b/test/fixtures/integer_gap.logic new file mode 100644 index 0000000..4127cff --- /dev/null +++ b/test/fixtures/integer_gap.logic @@ -0,0 +1,7 @@ +; An integer gap: no Integer is above 5 and below 6, so this ontology is contradictory. +; Finding that needs arithmetic. Z3 and CVC5 answer UNSAT; the native prover and Vampire +; answer SAT and print a warning that they have no arithmetic. +(def-concept Tank) +(def-property level Tank Integer) +(forall (t) (implies (Tank t) (gt level 5))) +(exists (t) (and (Tank t) (lt level 6)))