IndexedCP Python library by adithya1012 · Pull Request #26 · mieweb/indexedcp

adithya1012 · 2025-10-30T05:54:59Z

Python Implementation of IndexedCP

Short file upload with Encryption: https://youtube.com/shorts/00Lj3eQnLVs
Old One wo Encryption Short file upload: https://www.youtube.com/shorts/RSIwcoRnTu0

Summary

Complete Python port of IndexedCP with end-to-end encryption support. Implements RSA-4096 + AES-256-GCM encryption compatible with Node.js server and encrypted storage.

What's New

Encryption Support

RSA-4096 key wrapping + AES-256-GCM data encryption
Encrypted storage at rest (JSON-based, matching JavaScript implementation)
Keystore abstraction (filesystem backend with file permissions)
Per-stream ephemeral keys for secure file uploads
Public key caching for offline encryption
Backward compatible - encryption is optional

Core Features

Client: Chunked uploads with SQLite/encrypted storage backends
Crypto Utils: Low-level cryptographic primitives
Test Suite: 82 passing tests (15 integration tests require Node.js server)
Interactive Demos: Server and client demos with encryption
Documentation: Encryption guides and API references

Key Features

Encryption Architecture

Envelope encryption: RSA wraps AES session keys
Unique session keys: Per-file encryption isolation

Client Features

Chunked uploads with encryption
Offline encryption capability
Retry logic with exponential backoff
Progress callbacks
Background upload mode

Testing

82/82 tests passing
Full encryption workflow coverage
Backward compatibility verified
Integration tests

Interactive Demos

Server Demo (examples/server_demo.py):
- Displays configuration
- Real-time upload logging
- Clean shutdown handling
Client Demo (examples/client_demo.py):
- Simple file upload
- Batch uploads
- Large file chunking with DB inspection
- Background upload mode
- Automatic cleanup

Issues

adithya1012#12 -> 3 Sub issues
adithya1012#16 -> 4 Sub issues
adithya1012#21 -> 4 Sub issues

Copilot

Pull Request Overview

This PR introduces a Python port of the IndexedCP project, setting up the basic structure and tooling for a Python implementation alongside the existing Node.js version.

Adds Python package structure with src/indexedcp/ and tests/ directories
Configures Python development tooling (pytest, black, ruff, mypy) with appropriate settings
Creates basic smoke tests to validate pytest configuration and package import

Reviewed Changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`python/src/indexedcp/__init__.py`	Defines package metadata and version
`python/tests/__init__.py`	Initializes test package
`python/tests/test_basic.py`	Basic smoke tests for pytest configuration and package import
`python/requirements.txt`	Runtime dependencies (cryptography, aiohttp)
`python/requirements-dev.txt`	Development dependencies for testing and code quality
`python/pyproject.toml`	Project configuration and build metadata
`python/README.md`	Documentation for Python implementation
`.gitignore`	Adds Python-specific ignore patterns

horner

This commit is really not worthy of anything yet. Call it WIP still. It is the structure and the foundation on which to build. It’s always good to do a first commit like this and build on it. I’ll merge when we meet the goals.

- Initialize Python implementation of IndexedCP with minimal project setup. - Add colored logger module with comprehensive tests

Implement zero-dependency storage layer for Python IndexedCP following JavaScript keystore pattern. Uses Python's built-in sqlite3 module for persistence without external database dependencies.

…n layer Implemented IndexedCPClient with file chunking, SQLite storage backend, and HTTP upload queue. Created storage abstraction (BaseStorage, SQLiteStorage) with async context manager support. Added comprehensive test suite (44 tests) covering client operations, storage layer, and logging utilities. Includes examples and documentation following JS implementation patterns.

…tracking - Implement upload_buffered_files() with chunked uploads - Add exponential backoff retry logic (configurable) - Add progress callbacks (on_upload_progress, on_upload_error, on_upload_complete) - Add background upload process (non-blocking) - Add session tracking and automatic cleanup

- Add IndexedCPServer class with FastAPI backend for file uploads - Implement three path handling modes: ignore (default), sanitize, allow-paths - Add chunked upload reassembly with session tracking - Include API key authentication middleware - Add integration tests and interactive demos

horner · 2025-11-11T03:04:27Z

There is A lot in here that is too noisy. I would just do the TLDR and the details on how to use it. Not how it works.

I also think it should work like the node layer works. The files paths for be the same.

Reduced ReadMe content to 153 lines

Both client implementations use same path:
Client DB: ~/.indexcp/db/client.db (user's home directory)
Node Python

Correcting the Server path(CWD)
Server Node: this.outputDir = options.outputDir || process.cwd();
Server Python: upload_dir: str = "./uploads",

Fix: self.upload_dir = Path(upload_dir) if upload_dir else Path.cwd() LINK

horner · 2025-11-11T03:05:28Z

As a demonstration. This thing is way too complex. We want to show three or four lines of Python code for a client to work.

Previously I have included Multiple ways we can upload the file through client_demo.py
Short : https://www.youtube.com/shorts/RSIwcoRnTu0

I have updated the client_demo.py to 50 lines which includes the core 4 lines.

await client.initialize() await client.add_file(demo_file) results = await client.upload_buffered_files() await client.close()

horner · 2025-11-11T03:06:00Z

This is good.

Thank you !

Actually tho:
NOTE This is a beta package and encryption is not supported in the client or in the server.

Server code should removed

encryption on the client should be supported.

horner · 2025-11-11T03:08:45Z

+
+class IndexedCPClient:
+
+    def __init__(


I don't see any client side encryption

I have not included the Encryption in this PR (to keep it less code). I am working on it. adithya1012#22

- Server upload_dir now defaults to cwd() (matches Node.js) - Simplify README: focus on usage, remove implementation details - Simplify client_demo.py: 50 lines vs 380, shows 4-line usage

horner · 2025-11-13T01:37:01Z

edited the first comment to add the video https://www.youtube.com/shorts/RSIwcoRnTu0

horner · 2025-11-13T02:07:21Z

Actually tho:
NOTE This is a beta package and encryption is not supported in the client or in the server.

Server code should removed

encryption on the client should be supported.

- Implement RSA-4096 and AES-256-GCM encryption (crypto_utils.py) - Add pluggable keystore abstraction with filesystem implementation - Include 44 comprehensive tests (21 crypto + 23 keystore) - Set secure file permissions (0600/0700) with thread-safe locking

feat(python): Add crypto utilities and keystore system

- Updated examples to support Node server - Updated integration testing - Readme updated

- Introduced EncryptedStorage class for encrypted storage. - Updated create_storage function to support encrypted storage. - Modified FileSystemKeyStore and SQLiteStorage initialization logs for consistency. - Updated tests to support encryption features in IndexedCPClient.

adithya1012 mentioned this pull request Oct 30, 2025

Issue 1: Project Setup and Structure adithya1012/IndexedCP#13

Closed

8 tasks

adithya1012 force-pushed the python_indexedcp branch from 161c0c3 to c86b7f5 Compare October 30, 2025 06:10

horner requested a review from Copilot October 30, 2025 11:29

Copilot AI reviewed Oct 30, 2025

View reviewed changes

Comment thread python/README.md Outdated

Comment thread python/pyproject.toml

Comment thread python/tests/test_basic.py Outdated

Comment thread python/tests/test_basic.py Outdated

horner requested changes Oct 30, 2025

View reviewed changes

Comment thread python/tests/test_basic.py Outdated

Comment thread python/README.md Outdated

Comment thread python/README.md Outdated

adithya1012 changed the title ~~Python Project Setup and Structure~~ [WIP] Python Project Setup and Structure Oct 30, 2025

This was referenced Oct 31, 2025

Issue 2: Logger Module Implementation adithya1012/IndexedCP#14

Closed

Issue 3: Base Storage Interface (SQLite Abstraction) adithya1012/IndexedCP#15

Closed

[Python IndexedCP] Phase 1: Foundation & Core Infrastructure adithya1012/IndexedCP#12

Closed

adithya1012 force-pushed the python_indexedcp branch from 5f02d28 to c0cc3b1 Compare November 7, 2025 04:22

adithya1012 added 4 commits November 6, 2025 20:31

feat(python): add Python project structure and build configuration.

139f64e

- Initialize Python implementation of IndexedCP with minimal project setup. - Add colored logger module with comprehensive tests

feat: add pluggable storage abstraction with built-in SQLite backend

3e4e92c

Implement zero-dependency storage layer for Python IndexedCP following JavaScript keystore pattern. Uses Python's built-in sqlite3 module for persistence without external database dependencies.

adithya1012 force-pushed the python_indexedcp branch from c0cc3b1 to d89e366 Compare November 7, 2025 04:33

adithya1012 force-pushed the python_indexedcp branch from d89e366 to a549b5b Compare November 7, 2025 04:54

adithya1012 changed the title ~~[WIP] Python Project Setup and Structure~~ Python Project Setup and Structure Nov 7, 2025

adithya1012 requested a review from horner November 7, 2025 05:26

adithya1012 marked this pull request as ready for review November 7, 2025 05:26

horner requested changes Nov 11, 2025

View reviewed changes

PR Comments fix: align with Node.js defaults and simplify docs

05c0d28

- Server upload_dir now defaults to cwd() (matches Node.js) - Simplify README: focus on usage, remove implementation details - Simplify client_demo.py: 50 lines vs 380, shows 4-line usage

Added python info to the main readme

e33916c

horner requested changes Nov 13, 2025

View reviewed changes

adithya1012 and others added 3 commits November 13, 2025 15:24

Merge pull request #44 from adithya1012/python_indexcp-encryption

a97329c

feat(python): Add crypto utilities and keystore system

fix(python): Remove python server related.

746299f

- Updated examples to support Node server - Updated integration testing - Readme updated

adithya1012 changed the title ~~Python Project Setup and Structure~~ IndexedCP Python library Nov 21, 2025

adithya1012 force-pushed the python_indexedcp branch from 3665798 to 707543b Compare November 21, 2025 06:50

horner merged commit c836b35 into mieweb:main Nov 21, 2025
6 checks passed

Conversation

adithya1012 commented Oct 30, 2025 • edited by horner Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Python Implementation of IndexedCP

Summary

What's New

Key Features

Interactive Demos

Issues

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

horner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

horner commented Nov 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adithya1012 commented Oct 30, 2025 •

edited by horner

Loading