Skip to content

AGENTS.md

Latest commit

 

History

History
41 lines (24 loc) · 1.34 KB

AGENTS.md

File metadata and controls

41 lines (24 loc) · 1.34 KB
Raw

The ultimate aim of the project is to build a Proxy mock for the Spark object

Goals

A common Databricks pipeline uses the following commands to interact with Delta tables -

  1. spark.read.table("<schema_name>.<table_name>")

  2. spark.write.table("<schema_name>.<table_name>")

  3. spark.sql("SELECT * FROM <schema_name>.<table_name>")

The proxy replaces these functionalities with CSV files and Spark temp views.

Directory structure

All code is placed inside src directory and all modules are placed inside their own folders with __init__.py files. The main mock Spark object / proxy object is inside the testbricks package.

Recent Changes

  • 001-spark-mock-catalog: Added Python 3.14 + pyspark, pandas, pyarrow, numpy, py4j
  • 001-spark-mock-catalog: Added [if applicable, e.g., PostgreSQL, CoreData, files or N/A]

Active Technologies

  • Python 3.14 + pyspark, pandas, pyarrow, numpy, py4j (001-spark-mock-catalog)
  • Local files (CSV-compatible data + catalog registry) (001-spark-mock-catalog)

Development commands

PySpark requires Java (OpenJDK 21 is installed in the Cloud Agent environment).

# Install dependencies (also run automatically by the Cloud Agent install phase)
./.cursor/install.sh

# Run the full test suite with coverage (matches CI)
python3.14 -m coverage run -m pytest tests/ -v
python3.14 -m coverage report