Implement Cluster 2 spark.write fidelity - #23
Merged
Conversation
saveAsTable now raises AnalysisException when the CSV already exists for error/errorIfExists, and skip the write without touching the file or temp view for ignore. Unknown modes raise ValueError. Co-authored-by: gkaran184 <gkaran184@gmail.com>
insertInto requires the table CSV to exist and raises a Spark-like
AnalysisException otherwise. Default behavior appends; overwrite=True
or mode('overwrite') replaces rows.
Co-authored-by: gkaran184 <gkaran184@gmail.com>
Unknown partition columns raise at write time. replaceWhere with overwrite deletes matching CSV rows via a pandas query, then appends the new frame atomically. Co-authored-by: gkaran184 <gkaran184@gmail.com>
saveAsTable stores delimiter, quote, escape, nullValue, and date/timestamp formats next to the CSV so read.table round-trips without repeating options. File csv() writes continue to pass options through to native Spark. Co-authored-by: gkaran184 <gkaran184@gmail.com>
File APIs write under base_path via native Spark. format('delta').save()
stores parquet on disk; unknown file formats raise instead of failing
with AttributeError.
Co-authored-by: gkaran184 <gkaran184@gmail.com>
Overwrite with a changed schema requires overwriteSchema=true. Append can union extra columns when mergeSchema=true; overlapping type conflicts still raise SchemaMismatchError. README documents the save-mode and schema matrix. Co-authored-by: gkaran184 <gkaran184@gmail.com>
Production writer chains that bucket and sort no longer raise AttributeError; bucketing is logged and not simulated. Co-authored-by: gkaran184 <gkaran184@gmail.com>
create/replace/append map to the CSV catalog writer. createOrReplace and overwritePartitions raise NotImplementedError with a saveAsTable migration hint. Co-authored-by: gkaran184 <gkaran184@gmail.com>
engineeringmadness
marked this pull request as ready for review
September 4, 2026 03:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements Cluster 2 from
docs/superpowers/specs/2026-09-03-spark-write-lakeflow-roadmap-design.md: SparkDataFrameWriter/ catalog save-mode fidelity backed by CSV + temp views.Done (W1–W8)
error/errorIfExists/ignoresave modesinsertInto(table must exist;overwrite=Trueormode("overwrite")replaces)partitionBycolumn validation +replaceWhereoverwrite-wheredelimiter,quote,escape,nullValue, date/timestamp formats) persisted forread.tableround-tripsparquet/json/save/format().save();deltafile saves are parquet-on-diskoverwriteSchema/mergeSchematruth table (documented in README)bucketBy/sortByaccepted no-opswriteTo(...).create()/replace()/append()façade;createOrReplace/overwritePartitionsraiseNotImplementedErrorTesting
python3.14 -m coverage run -m pytest tests/ -v→ 172 passed. Coverage report ~98%.