perf(core): optimize nest_for_ard group partitioning, process_nested_list, and batch apply_fmt_fun - #610
Open
kpagacz wants to merge 2 commits into
Open
perf(core): optimize nest_for_ard group partitioning, process_nested_list, and batch apply_fmt_fun#610kpagacz wants to merge 2 commits into
kpagacz wants to merge 2 commits into
Conversation
…list, and batch apply_fmt_fun
Flamegraph profiling of cards::ard_summary() and cards::ard_missing() revealed
critical allocation and CPU bottlenecks during data partitioning, parameter updates,
and statistic formatting:
1. cards::nest_for_ard() dynamically generated quosures and executed dplyr::filter()
and dplyr::select() G times across all N observations (O(G * N) complexity), repeatedly
scanning the dataset and evaluating tidyselect column specifications.
2. .process_nested_list_as_df() executed per-variable dplyr::filter() followed by
dplyr::rows_update(), forcing hash key table validation and full dataframe copies.
3. cards::apply_fmt_fun() performed row-by-row purrr::pmap() iteration, resolving
function aliases, creating closures via label_round(), and executing scalar format()
calls independently for each cell.
Changes:
1. cards/R/nest_for_ard.R:
- Replaced dynamic quosure filtering loop with O(N) integer index matching via
vctrs::vec_match() against the unique group combination grid.
- Sliced row subsets directly using base R matrix/dataframe indexing on pre-split
integer indices, bypassing dplyr::filter() and tidyselect evaluations per stratum.
- Replaced mutate(across(..., as.list)) with direct in-place list column wrapping.
2. cards/R/ard_summary.R:
- Refactored .process_nested_list_as_df():
* Extracted unique (variable, stat_name) pairs upfront and grouped into a lookup list.
* Calculated formula selectors per variable and computed 1-to-1 integer indices via
vctrs::vec_match().
* Replaced dplyr::rows_update() with direct positional assignment x[[new_column]][idx] <- vals.
3. cards/R/apply_fmt_fun.R:
- Grouped rows sharing identical fmt_fun definitions and resolved function aliases
once per group rather than per row.
- Vectorized formatting across statistic vectors for supported primitive and
standard rounding functions, falling back to scalar evaluations only when required.
- Maintained full condition capture parity (stat_fmt, warning, error) matching
existing ARD S3 structure.
Verification and Impact:
- Unit tests: 865 passed, 0 failed, 0 warnings (test-ard_summary, test-ard_missing,
test-nest_for_ard, test-apply_fmt_fun, test-ard_tabulate, etc.).
- cards::nest_for_ard() microbenchmark: 181 ms -> 28 ms (6.5x speedup, -87% memory allocation).
- .process_nested_list_as_df() microbenchmark: 4.71 ms -> 0.73 ms (6.5x speedup, -70% memory allocation).
- tbl_summary (50 vars): 797.6 ms -> 516.1 ms (1.55x speedup, -19% memory allocation).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flamegraph profiling of cards::ard_summary() and cards::ard_missing() revealed following possible performance gains:
Changes:
cards/R/nest_for_ard.R:
cards/R/ard_summary.R:
cards/R/apply_fmt_fun.R:
What changes are proposed in this pull request?
nest_for_ardgroup partitioning,process_nested_list, and batchapply_fmt_fun, @kpagaczReference GitHub issue associated with pull request. e.g., 'closes #'
Pre-review Checklist (if item does not apply, mark is as complete)
usethis::pr_merge_main()devtools::test_coverage()Reviewer Checklist (if item does not apply, mark is as complete)
pkgdown::build_site(). Check the R console for errors, and review the rendered website.devtools::test_coverage()When the branch is ready to be merged:
NEWS.mdwith the changes from this pull request under the heading "# cards (development version)". If there is an issue associated with the pull request, reference it in parentheses at the end update (seeNEWS.mdfor examples).Optional Reverse Dependency Checks:
Install
checkedwithpak::pak("Genentech/checked")orpak::pak("checked")