Batch the reporting-data purge so it survives large tables#197
Open
kennygutierrez wants to merge 1 commit into
Open
Batch the reporting-data purge so it survives large tables#197kennygutierrez wants to merge 1 commit into
kennygutierrez wants to merge 1 commit into
Conversation
On sites that have been collecting CSP reports for a while, SecurityReportTo can hold millions of rows. The purge job's single unbounded DELETE exceeds the SqlCommand timeout and rolls back, so the table never shrinks. Changes: - Add ReportingOptions.PurgeBatchSize (default 5000; <= 0 restores the original single-DELETE behavior). - IReportingRepository.PurgeReporingData gains an optional int? batchSize. Source-compatible with existing callers. - SQL provider switches to DELETE TOP (@batchsize) when a batch size is supplied; ElasticSearch ignores it since DeleteByQuery already handles bulk deletes. - PurgeReporintgDataJob loops batch-by-batch, reports progress via OnStatusChanged, and is now IsStoppable.
Owner
|
Thanks for creating this, I really appreciate it. Can I ask that you change the target branch to release/v3.2.0. I can then update the version numbers and documentation. |
Author
|
Andrew, I changed the target. Hope I did it right! |
Owner
|
@kennygutierrez I will take a look at getting this merged and released. It may take a few days as I am just about to go on holiday. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On sites that have been collecting CSP reports for a while, SecurityReportTo can hold millions of rows. The purge job's single unbounded DELETE exceeds the SqlCommand timeout and rolls back, so the table never shrinks.
Changes: