Skip to content

feat(storage): add DiskManager for raw page I/O - #336

Merged
VihaanReddyM merged 18 commits into
danieldeer:db-enginefrom
IndicKangaroo:db-engine
Oct 18, 2026
Merged

VihaanReddyM merged 18 commits into
danieldeer:db-enginefrom
IndicKangaroo:db-engine

Conversation

@IndicKangaroo

@IndicKangaroo IndicKangaroo commented Oct 5, 2026 •

Copy link
Copy Markdown

Description

Adds DiskManager, the lowest layer of the new storage engine. It does raw
fixed-size page I/O on the .sdb file and nothing else: read a page, write a
page, allocate an empty one. It doesn't know about keys, records or page
contents, and it doesn't call the serializer. Locking stays with the
concurrency manager.

Part of #290. Closes #332 .

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update

Key Changes

  • New src/seriousdb/disk_manager.py: DiskManager with read_page,
    write_page, allocate_page, page_count, sync and close ). Page n is at byte offset n * PAGE_SIZE.
  • page_count is read from the file size, so pages added by another handle
    are visible.
  • allocate_page always adds a zero-filled page at the end of the file.
    Reusing freed pages is left to the layer that owns the free list.
  • Opening a file whose size isn't a multiple of PAGE_SIZE raises. With
    repair_torn_page=True it truncates the partial last page instead, and
    never touches whole pages.
  • write_page and allocate_page don't fsync. sync() does, so the WAL
    ordering can be decided later.
  • Uses os.pread/os.pwrite where available, with a lock around seek and
    read/write otherwise (Windows).
  • New DiskManagerError(ApplicationError) in exceptions.py; OS failures
    still propagate as OSError.
  • No changes to existing behavior.

Validation

  • uv run pytest tests/test_disk_manager.py (Windows, Python 3.13): 17 passed, 3 skipped.
    The skips are the short read/write and EOF tests, which patch
    os.pread/os.pwrite and so only run on platforms that have them.
  • uv run pytest: 2 failed, 227 passed, 3 skipped in 4.80s,
    the failed ones were test_cache.py, which were failing due to implemention of caching before my commits.They are unrelated to this change.
  • uv run ruff check . and uv run ruff format --check .: All checks passed!
  • uv run ty check .: All checks passed!
  • Tests cover round trips and reopen, bounds checking, misaligned files and
    torn-tail repair, short reads and writes, use after close(), and
    concurrent allocate_page() calls on one instance.
  • Not tested: the os.pread/os.pwrite path and the short read/write tests
    on the real repo setup, since I ran them on Windows only.

Checklist

  • I reviewed the diff and kept it focused on the described change.
  • Python changes pass uv run ruff check . and uv run ruff format --check . , or this PR does not change Python.
  • Changed behavior is covered by appropriate tests, including a regression test for bug fixes; any limitations are explained under Validation.
  • Validation lists the checks or manual flows I ran and their results.
  • Relevant documentation is updated, or no documentation change is needed.

@IndicKangaroo IndicKangaroo changed the title Db engine feat(storage): add DiskManager for raw page I/O Oct 5, 2026
Comment thread src/seriousdb/disk_manager.py Outdated
Comment thread src/seriousdb/disk_manager.py Outdated
Comment thread src/seriousdb/disk_manager.py Outdated
Comment thread src/seriousdb/disk_manager.py Outdated
Comment thread src/seriousdb/disk_manager.py Outdated
Comment thread src/seriousdb/exceptions.py
Comment thread .gitignore
@mohammedFawzy0111

Copy link
Copy Markdown
Contributor

I will have to run the pread and pwrite tests. I will test it on my linux to confirm

Comment thread src/seriousdb/disk_manager.py
Comment thread src/seriousdb/disk_manager.py
@IndicKangaroo

Copy link
Copy Markdown
Author

I will have to run the pread and pwrite tests. I will test it on my linux to confirm

sure, would wait for the results

@VihaanReddyM

Copy link
Copy Markdown
Collaborator

Will check this out soon it's been a busy week.

@mohammedFawzy0111

Copy link
Copy Markdown
Contributor

I will have to run the pread and pwrite tests. I will test it on my linux to confirm

Sorry for being late I was very busy .
I have ran the tests and they are working as intended.
Platform: Arch linux.

2026-10-08_10-10-1791446246

@IndicKangaroo

Copy link
Copy Markdown
Author

I will have to run the pread and pwrite tests. I will test it on my linux to confirm

Sorry for being late I was very busy . I have ran the tests and they are working as intended. Platform: Arch linux.

2026-10-08_10-10-1791446246

Awesome, thanks for verifying this on Linux for me! Much appreciated

@IndicKangaroo

Copy link
Copy Markdown
Author

@mattsva and @VihaanReddyM can you please review this pr? i think this is complete..

@VihaanReddyM VihaanReddyM left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for such a late review, I have midterms, family issues and a lot of stuff went on in life. I really apologize 🙏

Comment thread flake.lock
Comment thread src/seriousdb/disk_manager.py
@IndicKangaroo

Copy link
Copy Markdown
Author

Sorry for such a late review, I have midterms, family issues and a lot of stuff went on in life. I really apologize 🙏

No worries at all, take your time, and thanks for the review.

@VihaanReddyM
VihaanReddyM merged commit 2c4553e into danieldeer:db-engine Oct 18, 2026
5 checks passed
@mattsva

mattsva commented Oct 18, 2026

Copy link
Copy Markdown
Collaborator

@IndicKangaroo srry for not review, I did not had the time.

It looks pretty good tho, thanks @VihaanReddyM for merging it 😃

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants