feat: add upload deduplication, README rewrite, and backend improvements#98
Merged
feat: add upload deduplication, README rewrite, and backend improvements#98
Conversation
Formatting, async Supabase migration, improved error handling, and logging across routes and services.
Add frontend Dockerfiles, ESLint, Prettier, Vercel config, and nginx. Update docker-compose, env example, and lint workflow.
Remove broken route-level tests from test_ingest (referenced removed functions). Update test_storage and test_dataset_name_validation for current service signatures.
23 tests exercising full HTTP request → route → service → response chain. Covers upload, search, graph, document CRUD, file-url, and health check endpoints. External services mocked at SDK boundary.
Replace standalone script with pytest-discoverable e2e test. Creates temp fixture data (no external mock_data needed), uses Cognee embedded defaults (LanceDB/KuzuDB), auto-skips when LLM_API_KEY is missing.
Run pytest on every PR touching backend/. Excludes broken test_storage and e2e test_cognee. Adds pip caching, pytest-asyncio dependency, and registers the e2e marker in pyproject.toml.
Architecture overview, key files, environment variables, run/test commands, branch naming conventions, and code review checklist.
…, and cross-page navigation Fixes the graph flipping/bouncing bug by stabilizing the force simulation (cooldownTicks, d3AlphaDecay, d3VelocityDecay, warmupTicks) and memoizing graph data to prevent unnecessary re-renders. Adds: - Click-to-inspect node detail panel with connected entities, related content (Cognee CHUNKS search), and source documents - Connected node highlighting: selected node glows, neighbors stay visible, unrelated nodes dim to 20% opacity - Graph node search (client-side filter with dropdown, zoom-to-node) - Search-to-graph bridge: "View in Graph" button on search result source cards navigates to /graph?dataset=X - URL param support: ?dataset= auto-selects filter, ?node= auto-selects and zooms to a node - Improved UI: overlaid controls, polished hover tooltip, degree-based node sizing, UUID label filtering Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Compute a SHA-256 hash of file contents at upload time and check for an existing completed document with the same hash before running the pipeline. Duplicates return the existing document immediately, skipping R2 upload, LLM classification, and Cognee ingestion. Frontend shows a distinct amber "Duplicate" card with a link to the existing document. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace outdated ETL-era README with practical setup instructions covering Docker and manual workflows, project structure, API endpoints, testing, linting, CI/CD, and branch/PR conventions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Delete legacy route and service modules that were superseded by the Cognee-based pipeline. Update api.py, CLAUDE.md, and related services to drop references to the removed modules. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…arch results Summary/Insights/Entities tabs were all rendering raw document chunks because the pipeline used SearchType.CHUNKS for every query. Switch to GRAPH_SUMMARY_COMPLETION for the summary and GRAPH_COMPLETION for insights and entities, and add _split_bulleted() to break the resulting narrative answers into discrete list items. Also swap the dataset-slug pill on search results for the underlying document filename (falling back to the dataset name when no source is attached) so users see the specific document rather than a sanitized client slug. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kaydencel
previously approved these changes
Apr 17, 2026
Collaborator
kaydencel
left a comment
kaydencel
left a comment
There was a problem hiding this comment.
merge per Gordon and Jeff's request
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
supabase/migrations/019_add_content_hash.sql— newcontent_hashcolumn + indexbackend/app/routes/documents.py— SHA-256 dedup check before pipelinebackend/app/services/document_metadata_service.py—find_document_by_hash(), hash increate_document()backend/tests/test_integration.py— 28 integration tests including 5 dedup testsfrontend/src/pages/UploadPage.tsx— duplicate detection UIfrontend/src/services/api.ts— updatedUploadedFiletypeREADME.md— full rewrite for developer onboardingTest plan
pytest tests/test_integration.py -v)tsc --noEmit)🤖 Generated with Claude Code