SwordJack: Feature Database Service

[SwordJack]

Hi all,

This PR introduces the isobase.database module, establishing a general-purpose, unified persistence boundary for IsoBase applications. By extracting and formalizing database operations, it replaces the ad-hoc SQLite implementation previously embedded in the workflow module, providing a solid foundation for both document/state storage and structured application data.

Description

This PR accomplishes the following:

• New Database Module (isobase.database)

  • MongoDB Support (mongo.py): Adds MongoDbService and MongoDbModel providing document CRUD, querying (find_many, count, etc.), schema management, and transaction wrapping.
  • SQL Support (sql.py): Adds SqlDbService and SqlDbModelMixin wrapping SQLAlchemy, compatible with SQLite and PostgreSQL. Provides standard repository-pattern operations matching the MongoDB API naming style.
  • Shared Abstractions: Unifies naming conventions (e.g., execute_transaction, insert, delete_many, mongo_db, sql_db) to keep API surfaces predictable across backends.

• Workflow Module Refactoring

  • Removes the dedicated isobase/workflow/database.py that handled SQLite state sync.
  • Migrates workflow execution state (like ExecutionEntity) to use the new isobase.database.sql module (isobase/workflow/models.py).
  • Updates the workflow module to cleanly depend on the shared database layer rather than managing its own connections.
  • Updates isobase/workflow/README.md to reflect the new state management architecture.

• Testing & Tooling

  • Adds integration tests against real MongoDB (test_integration_mongo.py) and PostgreSQL (test_integration_postgres.py) instances.
  • Adds general tests for the SQL layer (test_sql.py).
  • Updates pyproject.toml, environment.yml, and conda recipe.yaml to include pymongo, sqlalchemy, and psycopg2-binary.

Testing

• All unit and integration tests under test/database pass.
• Github Actions pre-compilation successful.
• Tested with python -m pytest test/database/.

Future Work

This module establishes the data storage base. Future PRs will build upon this to add vector storage abstractions (e.g., pgvector) and knowledge-base document chunking for Retrieval-Augmented Generation (RAG) capabilities.

---

Best,
SwordJack.