Skip to content

Skip writing unchanged generated files to preserve mtimes#4396

Open
iainmcgin wants to merge 1 commit intobufbuild:mainfrom
iainmcgin:generate-skip-unchanged-files
Open

Skip writing unchanged generated files to preserve mtimes#4396
iainmcgin wants to merge 1 commit intobufbuild:mainfrom
iainmcgin:generate-skip-unchanged-files

Conversation

@iainmcgin
Copy link

@iainmcgin iainmcgin commented Mar 16, 2026

Contributes towards #3126.

Summary

buf generate previously wrote output files unconditionally, touching mtimes even when content was identical to what was already on disk. For mtime-based build systems (cargo, make, ninja, file-watching dev servers), a no-op buf generate would cascade into unnecessary downstream rebuilds.

The protoc plugin protocol gives plugins no visibility into the output directory—they emit File{name, content} on stdout and the invoking tool handles filesystem writes—so this fix lives in buf's write path rather than in individual plugins.

Approach

Replaces the unconditional storage.Copy in the directory output path (bufprotopluginos.writeDirectory) with a compare-then-write loop. For each generated file:

  • Read existing content from the OS bucket
  • If it matches the new content, skip the write (mtime preserved)
  • Otherwise write as before

Read errors during comparison (file doesn't exist, permissions, etc.) fall through to write, so worst-case behavior is identical to before.

Scope

Testing

  • 4 unit tests in response_writer_test.go covering unchanged/changed/new/mixed file scenarios
  • 1 integration test in generate_test.go that runs buf generate twice and verifies the output mtime is preserved

buf generate previously wrote output files unconditionally, touching
mtimes even when content was identical to what was already on disk.
For mtime-based build systems (cargo, make, ninja), a no-op generate
would cascade into unnecessary downstream rebuilds.

This change replaces the unconditional storage.Copy in the directory
output path with a compare-then-write loop. Files whose content matches
what is already on disk are skipped, preserving their mtimes. Read
errors during comparison (e.g. file does not exist, permissions) fall
through to write, preserving existing behavior.

Scope is limited to directory output; zip/jar output continues to write
unconditionally. Interaction with --clean is already correct: clean
deletes first, so comparison finds nothing and writes everything.

Closes bufbuild#3126
@iainmcgin iainmcgin marked this pull request as ready for review March 17, 2026 00:09
Copy link
Contributor

@emcfarlane emcfarlane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this contribution! The skip-unchanged behavior is a good improvement.

However, this doesn't fully close #3126. That issue is about making --clean a per-plugin "smart" clean, including stale file deletion. This PR is a useful step toward that, but the core of the issue remains open.

Could you update the PR description to say "Contributes to #3126" instead of "Closes #3126" so the issue stays open?


- Add support for `--rbs_out` as a `protoc_builtin` plugin (requires protoc v34.0+).
- Add relevant links from CEL LSP hover documentation to either <celbyexample.com> or <protovalidate.com>
- `buf generate` now skips writing output files when the content matches what's already on disk, preserving modification times for mtime-based build systems like cargo and make.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- `buf generate` now skips writing output files when the content matches what's already on disk, preserving modification times for mtime-based build systems like cargo and make.
- Skip writing unchanged output files in `buf generate` to preserve modification times

Copy link
Member

@doriable doriable left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thanks for putting this up! +1 to Ed's comments on fixing the PR description.

@iainmcgin
Copy link
Author

Changed to "contributes towards" as requested

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants