Pipeflush (#3)

* adding info upto now

* adding info upto now

* update again

* adding scikit-learn context

* small changes

* upgrade pkgs

* saving progress till now; implemented an agent for filtering commits + start storing patch info

* validation working

* partial auto-installer working

* auto-installer working pipe-flush single

* partially working autobuilder

* added portkey

* Better backup sync

* update benchmarking

* use dotenv

* remove unneccessary type hints

* remove support for 3.13

* updating build agent to be reactive

* small errors

* lot of updates. Dockerfile revamp, pipeflush previous layers (#4)

* lot of updates. Dockerfile revamp, pipeflush previous layers

* Adding PORTKEY_MODEL_NAME function for easier control.

* Increase timeout for container conflict. This caused a crash.

* config_agent_backend should be on top level.

* Add additional logging for containers.

* Correcting small bugs in dockerfile building.

* Stop bypassing cache.

* Help avoid race conditions in caching.

* Update commit_collection code to remove caching + log tagged and perf improving commits.

* Update various scripts

* example notebook

* Update README

Author: atharvas

.github/workflows/main.yml

@@ -29,7 +29,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.10", "3.11", "3.12", "3.13"]
+        python-version: ["3.10", "3.11", "3.12"]
       fail-fast: false
     defaults:
       run:

.gitignore

@@ -195,4 +195,8 @@ cython_debug/
 
 tokens.env
 benchmark_results/
-artifacts/
+scratch/artifacts/
+output
+
+# Database files
+*.db

CONTRIBUTING.md

@@ -1,54 +1,62 @@
 .PHONY: install
 install: ## Install the virtual environment and install the pre-commit hooks
-	@echo "🚀 Creating virtual environment using uv"
+	@echo "Creating virtual environment using uv"
 	@uv sync
 	@uv run pre-commit install
 
 .PHONY: backup
 backup: ## Create a backup of the datasets, results, and analysis directories
-	@echo "🚀 Creating backup archive"
+	@echo "Syncing backup mirror with rsync"
 	@/usr/bin/env bash -euo pipefail -c '\
 		if [ ! -f tokens.env ]; then \
 			echo "❌ Error: tokens.env file not found"; exit 1; \
 		fi; \
-		BACKUP_DIR=$$(grep -E "^BACKUP_DIR=" tokens.env | head -n1 | cut -d "=" -f2-); \
+		BACKUP_DIR=$$(awk -F= '"'"'/^BACKUP_DIR=/{print $$2; exit}'"'"' tokens.env); \
 		if [ -z "$$BACKUP_DIR" ]; then \
 			echo "❌ Error: BACKUP_DIR not defined in tokens.env"; exit 1; \
 		fi; \
-		mkdir -p "$$BACKUP_DIR"; \
-		zip -qr "$$BACKUP_DIR/datasmith.bckp" artifacts/benchmark_results artifacts/raw; \
-		cp -f artifacts/cache.db "$$BACKUP_DIR/datasmith.cache.bckp"; \
+		DEST="$$BACKUP_DIR/datasmith.mirror"; \
+		mkdir -p "$$DEST"; \
+		rsync -a --delete --human-readable --info=stats1 \
+			scratch/ "$$DEST/scratch/"; \
 	'
-
 .PHONY: check
 check: ## Run code quality tools.
-	@echo "🚀 Checking lock file consistency with 'pyproject.toml'"
+	@echo "Checking lock file consistency with 'pyproject.toml'"
 	@uv lock --locked
-	@echo "🚀 Linting code: Running pre-commit"
+	@echo "Linting code: Running pre-commit"
 	@uv run pre-commit run -a
-	@echo "🚀 Static type checking: Running mypy"
+	@echo "Static type checking: Running mypy"
 	@uv run mypy
-	@echo "🚀 Checking for obsolete dependencies: Running deptry"
+	@echo "Checking for obsolete dependencies: Running deptry"
 	@uv run deptry src
 
 .PHONY: test
 test: ## Test the code with pytest
-	@echo "🚀 Testing code: Running pytest"
+	@echo "Testing code: Running pytest"
 	@uv run python -m pytest --cov --cov-config=pyproject.toml --cov-report=xml
 
 .PHONY: build
 build: clean-build ## Build wheel file
-	@echo "🚀 Creating wheel file"
+	@echo "Creating wheel file"
 	@uvx --from build pyproject-build --installer uv
 
 .PHONY: clean-build
 clean-build: ## Clean build artifacts
-	@echo "🚀 Removing build artifacts"
+	@echo "Removing build artifacts"
 	@uv run python -c "import shutil; import os; shutil.rmtree('dist') if os.path.exists('dist') else None"
 
+.PHONY: docker-clean
+docker-clean: ## Clean up dangling Docker images and containers
+	@echo "Cleaning up dangling Docker images and containers"
+	@docker system prune -f
+
+
 .PHONY: help
 help:
 	@uv run python -c "import re; \
 	[[print(f'\033[36m{m[0]:<20}\033[0m {m[1]}') for m in re.findall(r'^([a-zA-Z_-]+):.*?## (.*)$$', open(makefile).read(), re.M)] for makefile in ('$(MAKEFILE_LIST)').strip().split()]"
 
+
+
 .DEFAULT_GOAL := help