Skip to content

DM-54457: Avoid unnecessary file size check on remote storage#1348

Draft
dhirving wants to merge 2 commits into
mainfrom
tickets/DM-54457
Draft

DM-54457: Avoid unnecessary file size check on remote storage#1348
dhirving wants to merge 2 commits into
mainfrom
tickets/DM-54457

Conversation

@dhirving
Copy link
Copy Markdown
Contributor

When using FormatterV2 to write a file, we now keep track of the file size that it wrote to use when recording validation information in the datastore. This lets us avoid a call to remote storage to retrieve the size of the file after writing it. In particular, this extra call was contributing to performance problems during heavy processing workloads with the FrDF WebDAV implementation.

Checklist

  • ran Jenkins
  • added a release note for user-visible changes to doc/changes
  • (if changing dimensions.yaml) make a copy of dimensions.yaml in configs/old_dimensions

@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 23, 2026

Codecov Report

❌ Patch coverage is 91.42857% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.52%. Comparing base (0bdf4f7) to head (9b31333).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
python/lsst/daf/butler/datastores/fileDatastore.py 70.00% 0 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1348      +/-   ##
==========================================
- Coverage   89.53%   89.52%   -0.01%     
==========================================
  Files         370      370              
  Lines       50250    50267      +17     
  Branches     5876     5879       +3     
==========================================
+ Hits        44989    45003      +14     
- Misses       3873     3874       +1     
- Partials     1388     1390       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Make `FormatterV2.write_direct` and `FormatterV2.write_locally_then_move` private.  Public methods intended for override `to_bytes` and `write_local_file` already exist that replace these.
When using FormatterV2 to write a file, we now keep track of the file size that it wrote to use when recording validation information in the datastore.  This lets us avoid a call to remote storage to retrieve the size of the file after writing it.  In particular, this extra call was contributing to performance problems during heavy processing workloads with the FrDF WebDAV implementation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant