Repository navigation
Multiset egraph - #42
Merged
Merged
Conversation
…b-pareto Wire srtree-db into reggression: - new commands (persist, db-load, db-top, db-distribution, db-count, db-pareto) with parsers, handlers, help and CLI/Python dispatch - Python API: persist, loadDB, dbTop, dbDistribution, dbCount, dbPareto - db-pareto runs the (fitness, size) pareto front in SQL - db-load recomputes cost-minimal bests (recalculateBestAll) since best/cost are not persisted; the db otherwise holds arbitrary nodes that blow up when expanded for printing/pattern matching - build deps: srtree-db, direct-sqlite, text; local packages wired into cabal.project; version bump 1.0.10 -> 1.0.11
…DB reuse) Builds a graph from test/equations.bingo (no stale egraph fixture), persists it, runs db-top/db-distribution/db-count/db-pareto, reloads via db-load and checks in-memory top/distribution still work (regression for the OOM from arbitrary _best nodes on load).
…ures Expose srtree-db's pushFit/refreshFitness as db-push-fit/db-refresh-fitness CLI commands and dbPushFit/dbRefreshFitness Python methods. Rebuild test/eqs.egraph, test/final.egraph, test/nasa_battery.egraph against the current ENode binary encoding so test_pyreggression*, test_modularity no longer crash on decode.
…utorial 10 - add db-import command + Reggression.importDB(): build an e-graph directly in SQLite by streaming expressions (structural, content-addressed), no in-memory graph. - re-enable db-eqsat dispatch and make dbEqSat recompute best/cost before saturating (imported DBs carry defaults). - dbTop: render the top-N out-of-core by installing the lazily loaded paged graph instead of relying on an in-memory graph. - rewrite tutorials/10_scale_dbEqSat.py to demonstrate importDB + dbEqSat end-to-end out-of-core.
…cale ceiling The eqsat rewrite engine currently saturates reliably up to ~50-60k e-classes; beyond that the n-ary matcher/merge step slows down sharply (a pre-existing engine limitation, independent of the out-of-core path). Default the demo to 20k equations (~40k classes, ~30s) and document the ceiling + SCALE_N/SCALE_ITERS tuning knobs, so the tutorial finishes quickly while still showing a fully out-of-core import->saturate->query pipeline.
… bounded The earlier '~50-60k class ceiling' note is obsolete; the matcher is now work-bounded and dbEqSat scales into the hundreds of thousands of classes. Default demo (~40k classes) finishes in ~11s.
…ount in tutorial 10 Util: route createDB/createDBBest through the ClassStore (allClasses) so they rebuild the pattern DB from the paged store instead of the resident _eClass map directly, correct for out-of-core graphs. tutorial 10: report the e-class count after dbEqSat saturation (seed -> after), so the growth from rewrites is visible.
Reggression object derives a dataset name from the dataset CSV basename; the db-* commands carry it and the query layer resolves it to a dataset id. importDB writes dataset_fit for the dataset; dbTop/dbDistribution/dbPareto rank that dataset's fitness.
…h-fitness All db-* commands that read/write fitness now carry the dataset name (derived from the dataset CSV basename on the Python object); handlers resolve it to a dataset id and the dataset-aware load/save/query layer uses it. importDB writes dataset_fit + expression_index for the dataset.
canonical/canonize now require ClassStore (bounded-cache store fallback on paged graphs), so the EGraphST helpers that call them are bumped from Monad m to ClassStore m.
The out-of-core eqsat matcher does not use cost/best (_consts, from import, is correct; _best/_cost are only for extraction, and the graph is discarded after dbEqSat). Removing the full O(n) read+rewrite-all-classes pass cuts dbEqSat runtime ~40% (200k: 125s -> 71s) and slightly lowers incremental RSS, with identical results (best fitness / top expressions unchanged).
The Python-loaded _binding previously used plain hs_init, which rejects RTS
options ('Most RTS options are disabled'). Switch the SWIG wrapper to
hs_init_with_rtsopts (declared explicitly) so the runtime accepts +RTS args,
and have the Python package set them:
- binding.i: hs_init_with_rtsopts + extern declaration.
- __init__.py: default GHCRTS to -F1.2 and pass [reggression,+RTS,-F1.2,
-RTS] to hs_rts_init in reggression_run.
Out-of-core eqsat churns through every class body; with the default GHC heap
growth (F=2) this leaves a large reserved-heap high-water mark. -F1.2 keeps the
embedded heap tight: ~24% lower process RSS at 200k (1505 -> 1140 MB) and 400k
(2613 -> 2030 MB) with ~16% runtime cost, identical results.
…ses) Adds a db-eqsat-frontier command (+ Python Reggression.dbEqSatFrontier): loads the paged graph, restricts the matcher to the persisted frontier via cpsBeginFrontier, runs eqsat, then cpsEndFrontier (clears the frontier). A full dbEqSat now also clears the frontier at the end (it re-saturates everything). The pure in-memory eggp loop is unchanged: the frontier is only active on a paged (DB-backed) graph and only when a frontier pass begins.
Add a db-insert command (+ Reggression.dbInsert): parses a single expression and inserts it into the lazily loaded (paged) e-graph via fromTree, so its subgraph is written through content-addressed (existing subexpressions dedup against the live tables) and every genuinely-new class is marked as part of the re-saturation frontier. A later dbEqSatFrontier then re-saturates only what changed. O(subgraph) work, O(1) memory. The pure in-memory insert is unchanged.
- dbInsert now returns the root e-class id (symmetric with the in-memory insert)
so the eggp loop can track and evaluate a DB-inserted expression, and records
the expression in expression_index via recordExpressionIndex.
- Add db-set-fit / Reggression.dbSetFit: write a single e-class's fitness into
dataset_fit, so an evaluated DB-inserted expression can be ranked by dbTop.
Together these make the DB-backed path a drop-in for the eggp loop: dbInsert ->
{dbEqSatFrontier | dbEqSat} -> dbSetFit -> dbTop, while the in-memory insert
path is unchanged (both options).
Shows the DB-backed eggp loop (authoritative SQLite e-graph): seed with importDB, then insert candidates via dbInsert (content-addressed, marks the frontier), record their fitness with dbSetFit, re-saturate with dbEqSat / dbEqSatFrontier, and rank with dbTop -- then 'resume' the search from the same database file with a fresh Reggression object. Uses a lightweight stand-in fitness (the loop owns model evaluation) and shallow expressions so dbTop's out-of-core extraction is robust; peaks at ~265 MB, far below the previous run's spike.
…utorial dbTop used printSimpleMultiExprs, which renders each top class through the full LaTeX/python/modular path (showModular / showPython / theta / dl). On a supersaturated or cyclic graph this could hang or explode memory (seen as ~13 GB RSS and a hang). Replace it with a simple, cycle-safe renderer that outputs just Id, Expression (via getBestExpr, which is now budget/cycle-safe) and Fitness -- fast and bounded regardless of graph shape. Update tutorials/11_eggp_db.py to use deep, realistic expressions (the earlier shallow version was only to dodge the extraction bug, which is now fixed).
- putEOL crashed with 'Data.ByteString.last: empty ByteString' when the command string parsed to empty (an empty _dbSpec). Guard B.null. - Reggression now defaults self.db to a temp SQLite file when no db is given, so db-native methods (insert/dbInsert/top/...) always have a valid target instead of emitting a malformed command. Temp db is removed on __del__. - import tempfile (removed by the DB-mode refactor).
DBInsert now takes an algorithm name (e.g. TIR, hl, operon, ...) resolved via algToAlgs (defaults to TIR on unknown). The command/parser, Py.hs insertCmd, and Python insert/dbInsert(alg=...) all thread it through, so a single expression can be parsed with any supported SRAlgs instead of hardcoded TIR.
Previously inserting into an empty DB failed with 'no e-graph stored'. On that case, create an empty paged graph and insert into it (subsequent inserts load the now-existing graph). Makes the eggp flow's first insert work without a separate seed step.
…sename DBOptimize called loadDataset on the dataset *name* (basename), which isn't a file, so it crashed with 'openBinaryFile: does not exist'. The Python optimize now sends the dataset file path; DBOptimize loads data from that path but uses the path's basename as the dataset identity for the dataset_fit lookup. Added basenameOf helper.
… in the right dataset DBOptimize used one 'ds' arg for both the dataset identity (dataset_fit lookup) and the data file to load for fitting. When we switched it to the file path, it created a spurious second dataset named by the path (e.g. opt2.csv) and wrote fitness there, which the query layer (reading the real dataset 'opt2') missed. Now the optimize command carries the dataset name (ds) and the dataset file path (dataPath) separately; DBOptimize writes dataset_fit under the name and loads data from the path.
…arams/Fitness/MSE/R^2/nll/DL) + CI rows DBReport previously returned plain text and loaded the dataset by its *name* (not a path), so CI also failed. It now takes the dataset file path separately (like optimize), loads the data, computes MSE/R^2/nll/DL with the fitted theta, and outputs a CSV DataFrame. When ci is requested it appends ci_param_lower/upper rows from the profile-likelihood computation (when profiles are produced). Note: the underlying getAllProfiles/profile computation can segfault on some multi-parameter models -- a separate, pre-existing CI-backend issue.
The report command now accepts an optional test-data path (Python sends self.testData, or '-' when absent). The handler loads the training set for the Training column and the test set for the Test column, computing MSE/R^2/nll/DL with the same fitted theta on both. Without a test set, Test stays NaN.
Fitness is the negative mean nll (-nll/n). For the test set we now compute it from the test nll and sample count so the Test column isn't NaN; training fitness still comes from the stored dataset_fit.fitness.
…nIndex after saveGraph The egraph DB schema (enode, eclass_node, ...) must exist before the first insert, especially with a separate fit DB. SaveGraph runs before the expression_index record so the node lookup against eclass_node succeeds.
…ize loss consistency importDB in split-DB mode wrote dataset_fit/expression_index into the egraph DB (importEqs uses a single connection), so report/top/optimize -- which read the fit DB -- saw no fitness: report Training fitness came back NaN and top was empty. Now the ImportDB handler copies those rows into the fit DB after the import. Report Test fitness now uses the same loss objective as optimize (threaded as a lossName arg like DBOptimize), instead of hardcoding -mean(nll) which only matched MSE. Training fitness comes from the stored dataset_fit.fitness; Test fitness is -loss on the test data, matching the training convention.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.