Fix/s3 reset artifact persistence - #34
Open
NSingh1227 wants to merge 547 commits into
Open
Conversation
…rg/model) for ingested models
…lly for exact matches
…autograder failures
…shooting autograder ID retrieval failures
- Update database.py to use /tmp directory in Lambda (only writable directory) - Enable USE_SQLITE=1 in CD workflow for production - Set SQLALCHEMY_DATABASE_URL environment variable for Lambda - Update reset endpoint to work with both SQLite and in-memory storage - Add comprehensive logging for SQLite initialization - Update README with SQLite production configuration details - Ensures artifacts persist across Lambda invocations for autograder compatibility
Testing local
…tibility
- Autograder calls /artifact/{type}/{id} (singular) but spec says /artifacts/{type}/{id} (plural)
- Add duplicate route handler that delegates to existing artifact_retrieve function
- Ensures compatibility with both autograder and OpenAPI spec patterns
- Fixes all ID retrieval failures (was 0/22, should now pass)
- Add count_artifacts_by_type() helper function in crud.py - Update models_upload, models_ingest, and artifact_create to query SQLite for accurate artifact count when USE_SQLITE=True - Prevents ID collisions and incorrect sequential numbering after Lambda cold starts - Falls back to len(artifacts_db) if SQLite query fails or SQLite is disabled - Fixes issue where all IDs would start with model-1- after cold start
- Update models_upload to use SQLite count for ID generation - Completes the ID generation fix across all artifact creation endpoints
- Create S3 storage adapter module (src/storage/s3_storage.py) - Add USE_S3 environment variable support - Update artifact creation/retrieval to use S3 when enabled - Priority: S3 > SQLite > in-memory storage - Add comprehensive S3 setup guide (S3_SETUP_GUIDE.md) - Supports artifact metadata and file storage in S3 - Falls back gracefully if S3 unavailable
- Remove unused datetime import from s3_storage.py - Fix f-string without placeholders in app.py - Update models_ingest endpoint to use S3 storage when enabled - Update models_upload endpoint to use S3 storage when enabled
- Add S3 storage support to models_upload endpoint - Update ID generation to prioritize S3 > SQLite > in-memory - Store artifact metadata in S3 when enabled
- Disable SQLite automatically in production/Lambda environments - Enable S3 automatically in production/Lambda environments - Update storage priority: S3 (production) > SQLite (local) > in-memory - Update CI/CD to set USE_S3=1 and remove USE_SQLITE from production - Update README to reflect production uses S3, local uses SQLite - Clean separation: production = S3, local = SQLite
- Add *.db, *.sqlite, *.sqlite3 patterns to ignore all database files - Specifically ignore test_id_gen.db - Add frontend/.env files - Add *.pyo and *.pyd to Python cache patterns
- Added S3 helper methods: list_artifacts_by_queries, list_artifacts_by_name, list_artifacts_by_regex, clear_all_artifacts - Updated all endpoints to use S3 (production) > SQLite (local) > in-memory priority - Updated endpoints: artifacts_list, models_enumerate, artifact_by_name, artifact_by_regex, artifact_retrieve, artifact_update, artifact_delete, artifact_audit, artifact_lineage, artifact_license_check, reset - Removed debug print statements and sys.stdout.flush() calls - Fixed storage layer selection logic (S3 for production, SQLite for local) - All endpoints now properly work with database storage - Code formatted with black and passes linting
- Updated models_download endpoint to use S3/SQLite priority - Updated model_artifact_rate endpoint to use S3/SQLite priority - Updated artifact_cost endpoint to use S3/SQLite priority - All endpoints now properly use S3 in production - Frontend API service fully synced with backend endpoints - Frontend components support all Milestone 3 features - Updated S3_SETUP_GUIDE.md with complete setup instructions - Added troubleshooting section and verification steps - All critical endpoints tested and verified - Ready for autograder testing
- Updated model_artifact_rate endpoint to use S3/SQLite priority - Updated artifact_cost endpoint to use S3/SQLite priority - All endpoints now properly use S3 in production - Frontend fully synced with backend (all methods and components) - Updated S3_SETUP_GUIDE.md with complete step-by-step instructions - Added troubleshooting section and verification steps - All critical endpoints tested and verified - Ready for autograder testing
- Updated model_artifact_rate endpoint to use S3/SQLite priority - Updated artifact_cost endpoint to use S3/SQLite priority - Fixed flake8 warnings by removing unused variables - All endpoints now properly use S3 in production - Complete frontend/backend sync for Milestone 3
- Removed s3:ListObjectsV2 from IAM policy (not a valid IAM action) - s3:ListBucket covers the ListObjectsV2 API operation - Updated both AWS CLI and Console examples - Fixed troubleshooting section to clarify the action mapping
- AWS_REGION is a reserved Lambda environment variable (cannot be modified) - Removed AWS_REGION from all environment variable examples - Added notes explaining AWS_REGION is automatically set by Lambda - Updated troubleshooting and Quick Reference sections - Code already handles AWS_REGION correctly via os.environ.get fallback
- Updated Lambda environment variables to use S3_BUCKET_NAME from GitHub secrets - Removed AWS_REGION (reserved Lambda variable, cannot be modified) - Added USE_SQLITE=0 to disable SQLite in production - Added comment explaining AWS_REGION is automatically set by Lambda - All S3 configuration now happens automatically on deployment
- Changed os.environ.get('AWS_LAMBDA_FUNCTION_NAME') to explicit None check
- This ensures mypy correctly infers bool type instead of Literal[True] | str | None
- Fixes CI/CD mypy typecheck failure
- Fix f-string placeholders (remove f-strings with no placeholders) - Fix blank line whitespace issues - Fix line length issues (split long lines across multiple lines) - Fix type ignore comment placement for mypy compatibility - All code quality checks now pass: black, flake8, mypy
Improve HF artifact matching and auth
- Convert stored_type string to ArtifactType enum before creating ArtifactMetadata - Add comprehensive error handling and logging for type conversion - Fix byID and byName endpoints to handle both in-memory and S3 storage - Add test_type_conversion.py to verify S3/memory type consistency - Enhanced debug logging to identify S3/memory mismatches This fixes autograder failures where artifacts stored in S3 (as JSON strings) could not be properly converted to ArtifactType enums when retrieved.
- Add blank line before middleware function (E302) - Remove whitespace from blank lines (W293) - Fix f-string without placeholders (F541) - Fix continuation line indentation (E128)
Fix ArtifactType enum conversion for S3/memory consistency
…revent overwrites during rapid creation
…rtifact retrieval tracing
…ion removal, and debug logging
…elds, add comprehensive response logging
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.