diff --git a/CLAUDE.md b/CLAUDE.md index ce4246d..aae6a14 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -58,7 +58,7 @@ The table registry is the central data structure that defines all ACS tables the - `raw_variables` - named vector of ACS variable codes (for manual sources) - `definitions` - list of DSL objects (`define_percent()`, `define_sum()`, `define_complement()`, `define_metadata()`) describing derived variables. Codebook entries and MOE-propagation strategy are derived from each object's `type`. -There are 33 registered internal tables. +There are 34 registered internal tables. ### Table selection API @@ -82,7 +82,7 @@ Both construct names and internal names are accepted by `compile_acs_data(tables `tables` can contain three kinds of elements (mix freely inside a `list()`): - **Registered table names** (e.g., `"race"`, `"snap"`) — use `list_tables()` to see them all. -- **Raw ACS table codes** (e.g., `"B25070"`, `"C15002B"`) — any valid Detailed/Collapsed code is auto-processed at runtime: raw variables are fetched, the label hierarchy is parsed, and percentages are computed automatically. The `denominator` parameter controls the percentage denominator (`"parent"` for nearest subtotal, `"total"` for `_001`, or a specific code like `"B25070_001"`). If a raw code is already covered by a registered table, the registered version is used. +- **Raw ACS table codes** (e.g., `"B25070"`, `"C15002B"`) — any valid Detailed/Collapsed code is auto-processed at runtime: raw variables are fetched, the label hierarchy is parsed, and percentages are computed automatically. The `denominator` parameter controls the percentage denominator (`"parent"` for nearest subtotal, `"total"` for `_001`, or a specific code like `"B25070_001"`). Raw codes are always auto-processed, even when a registered table covers the same code (`"B22003"` returns the auto-processed table; `"snap"` returns the registered one). If a requested code overlaps a registered table included in the same call, the registered version wins and the auto entry is dropped with a warning. - **DSL definition objects** from `define_percent()`/`define_sum()`/`define_complement()`/`define_metadata()` — let users layer custom derived variables on top of the requested tables; results land in the returned data frame and the codebook with MOEs computed automatically. When `tables` are specified: @@ -110,6 +110,7 @@ Note: don't hard-code a list of block-group tables — derive it from the codebo 1. **`R/table_registry.R`** - Central registry, DSL constructors (`define_percent()`, `define_sum()`, `define_complement()`, `define_metadata()`), `validate_definition()`, `execute_definitions()`, `resolve_tables()`, `collect_raw_variables()`, `expand_codebook_entry()`, `list_tables()`, and all `register_table()` calls. 2. **`R/compile_acs_data.R`** - `compile_acs_data()` (entry point), `fetch_acs()` (per-year/per-state tidycensus calls + ZCTA-style "super-state" geographies), `safe_divide()`. + **`R/cache.R`** - Opt-in disk cache of raw ACS query results (`cache = TRUE`): `acs_cache_dir()`, `acs_query()` (mockable seam around `tidycensus::get_acs()`), `cached_get_acs()` (one entry per geography × year × state × table, keyed by `rlang::hash()`; no expiry — published ACS estimates are immutable), and exported `clear_acs_cache()`. Cache dir is `tools::R_user_dir("urbnindicators", "cache")`, overridable via `options(urbnindicators.cache_dir = ...)` (tests use this). The per-table variable map comes from `collect_raw_variables_by_table()` in `R/table_registry.R`; the `cache = FALSE` path is unchanged (single combined query per state × year). 3. **`R/auto_percent.R`** - Auto-table support for raw ACS table codes: `is_raw_acs_code()`, `resolve_to_acs_table()`, `build_auto_table_entry()`, `generate_auto_definitions()`. 4. **`R/interpolate_acs.R`** - `interpolate_acs()` plus internal aggregation helpers; uses codebook attributes to dispatch per-variable aggregation strategy. 5. **`R/list_acs_variables.R`** - `get_acs_codebook()` (exported), and internal helpers `select_variables_by_name()` and `filter_variables()` used by the registry's `"select_variables"` path. (The deprecated `list_acs_variables()` stub was removed before 0.1.0.) @@ -121,7 +122,8 @@ Note: don't hard-code a list of block-group tables — derive it from the codebo ### Exported functions -- `compile_acs_data(tables, years, geography, states, counties, spatial, denominator, ...)` - Pull and compute ACS data. +- `compile_acs_data(tables, years, geography, states, counties, spatial, denominator, cache, ...)` - Pull and compute ACS data. `cache = TRUE` reuses raw ACS query results from the on-disk cache. +- `clear_acs_cache()` - Delete cached raw ACS query files. - `interpolate_acs(.data, target_geoid_column, weight = NULL, crosswalk = NULL, source_geoid = "GEOID", weight_variable = "total_population_universe")` - Aggregate or interpolate ACS data to custom geographies. `weight = NULL` for complete nesting (direct aggregation); pass a weight column name for fractional allocation via crosswalk. - `define_percent()`, `define_sum()`, `define_complement()`, `define_metadata()` - DSL constructors for derived variables. Use inside `register_table(definitions = list(...))` or pass directly to `compile_acs_data(tables = list(...))`. - `list_tables()` - Available registered table names for the `tables` parameter (construct-level names). diff --git a/DESCRIPTION b/DESCRIPTION index ecb343d..d6c5846 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -31,7 +31,8 @@ Imports: tigris, rlang, tibble, - sf + sf, + tools Suggests: arrow, bslib, @@ -49,7 +50,8 @@ Suggests: shiny, testthat (>= 3.0.0), scales, - urbnthemes (>= 0.0.2) + urbnthemes (>= 0.0.2), + withr Remotes: UI-Research/crosswalk, UrbanInstitute/urbnthemes diff --git a/NAMESPACE b/NAMESPACE index b7882aa..e54ebd0 100644 --- a/NAMESPACE +++ b/NAMESPACE @@ -1,6 +1,7 @@ # Generated by roxygen2: do not edit by hand export("%>%") +export(clear_acs_cache) export(compile_acs_data) export(define_complement) export(define_metadata) diff --git a/NEWS.md b/NEWS.md index f60c620..bf5c169 100644 --- a/NEWS.md +++ b/NEWS.md @@ -23,6 +23,11 @@ package's API and internals. income- and rent-band columns (e.g., from B25074/B25106) are syntactic: `household_income_..._less_than_$10000` becomes `household_income_..._less_than_10000`. +* The renter cost-burden measures now carry an explicit tenure label: + `cost_burdened_30percentormore_*` and `cost_burdened_50percentormore_*` + are now `cost_burdened_renter_30percentormore_*` and + `cost_burdened_renter_50percentormore_*`, disambiguating them from the + new owner-side measures. * Two variables were renamed for accuracy and consistency: `employment_civilian_labor_force_percent` is now `employment_civilian_labor_force_employed_percent` (it is the employment @@ -36,6 +41,13 @@ package's API and internals. `tidycensus::census_api_key("YOUR_KEY", install = TRUE)`. * `safe_divide()` now returns `NA` (not `0`) when a nonzero numerator is divided by zero; `0/0` still returns `0`. +* Raw ACS table codes are now **always auto-processed**, even when a + registered table covers the same code: `tables = "B22003"` returns the + auto-processed table with label-derived names, while `tables = "snap"` + returns the registered version (previously the raw code was silently + upgraded to the registered table). Requesting both in one call returns + the registered version and drops the auto-processed one with a warning, + since the two would fetch duplicate variables. ## New features @@ -55,6 +67,25 @@ package's API and internals. map results, interpolate to uploaded or hand-drawn target geographies, benchmark differences for statistical significance, and export data and images. +* **Caching**: `compile_acs_data(cache = TRUE)` caches raw ACS query results + on disk (one entry per geography-year-state-table) and reuses them across + calls and R sessions, so repeat and overlapping requests -- including + changed `tables` selections -- re-download only what is missing. Entries + never expire (published five-year ACS estimates do not change); a new + `clear_acs_cache()` deletes them to reclaim disk space. +* **Universe statements in the codebook.** The codebook gains a `universe` + column carrying the Census Bureau's published universe statement for each + variable (e.g., "Households", "Population 25 years and over"); derived + variables inherit their numerator's universe. Available for the 2020+ + vintages (the API does not publish universes for earlier years). +* **Owner cost-burden measures.** `tenure_by_housing_costs` (B25106) now + publishes owner cost-burden percentages (all incomes, incomes below + $35,000, incomes below $50,000) and an all-tenures headline measure; a + new `owner_cost_burden` table (B25091) adds 30-percent-or-more and + 50-percent-or-more (severe) burden shares by mortgage status, plus a + combined all-owners severe-burden measure. Denominators exclude + households whose cost ratio is not computable (zero/negative income; + renters paying no cash rent), consistent with the renter measures. * Unregistered ACS tables requested by code are auto-processed: the label hierarchy is parsed and percentages are computed against the nearest parent subtotal (configurable via the `denominator` parameter). @@ -79,6 +110,15 @@ package's API and internals. installed `tidycensus` geography lookup. * Margin-of-error calculations were vectorized (orders-of-magnitude faster on tract-level, multi-state pulls). +* The auto-table classifier now matches its skip keywords on word + boundaries, so tables whose concept contains "Means" (e.g., B08301, + Means of Transportation to Work) are no longer misclassified as + non-count tables ("mean") and receive auto-computed percentages. +* The auto-table classifier now requires a table's `_001` label to end + with `:` (or be exactly `Estimate!!Total`) to compute percentages, so + dollar-valued tables like B19080 (Household Income Quintile Upper + Limits) no longer produce nonsense percentages of one quintile limit + over another. * Fixed a typo in `make_pretty_names()` output ("renter-occuiped"). ## Infrastructure diff --git a/R/auto_percent.R b/R/auto_percent.R index abcaa3b..a1c2811 100644 --- a/R/auto_percent.R +++ b/R/auto_percent.R @@ -112,15 +112,17 @@ classify_acs_table = function(nodes) { concept_lower = tolower(concept) label_lower = tolower(total_label) - ## patterns that indicate non-percentage-amenable tables + ## patterns that indicate non-percentage-amenable tables; matched on word + ## boundaries so that, e.g., "mean" does not match "Means of Transportation" skip_patterns = c( "median", "aggregate", "average", "mean", "allocation of", "imputation of", "margin of error") has_skip_pattern = purrr::some(skip_patterns, function(pattern) { - stringr::str_detect(concept_lower, stringr::fixed(pattern)) || - stringr::str_detect(label_lower, stringr::fixed(pattern)) + pattern_regex = paste0("\\b", pattern, "\\b") + stringr::str_detect(concept_lower, pattern_regex) || + stringr::str_detect(label_lower, pattern_regex) }) if (has_skip_pattern) return("skip") @@ -128,8 +130,12 @@ classify_acs_table = function(nodes) { if (nrow(nodes) <= 1) return("skip") ## tables where the total is not a count (e.g., median income tables may have - ## a numeric label rather than "Estimate!!Total:") - if (!stringr::str_detect(total_label, ":") && + ## a numeric label rather than "Estimate!!Total:"). The _001 label of a count + ## table ends with ":" (or is exactly "Estimate!!Total" in older vintages); a + ## mid-label colon is not sufficient (e.g., B19080's _001 is + ## "Estimate!!Quintile Upper Limits:!!Lowest Quintile Upper Limit" — a dollar + ## value, not a total). + if (!stringr::str_detect(total_label, ":$") && !stringr::str_detect(total_label, "^Estimate!!Total$")) { return("skip") } diff --git a/R/cache.R b/R/cache.R new file mode 100644 index 0000000..882c7ee --- /dev/null +++ b/R/cache.R @@ -0,0 +1,61 @@ +## Internal: directory where cached raw ACS query results are stored. +## Overridable via options(urbnindicators.cache_dir = ...) (used by tests). +acs_cache_dir = function() { + getOption("urbnindicators.cache_dir", tools::R_user_dir("urbnindicators", which = "cache")) +} + +## Internal: thin wrapper around tidycensus::get_acs(). Exists as a seam so +## tests can mock the network via testthat::local_mocked_bindings(). +acs_query = function(args) { + do.call(tidycensus::get_acs, args) +} + +## Internal: fetch one table's raw estimates for a single query, reading from / +## writing to the on-disk cache when `cache = TRUE`. `args` holds the +## tidycensus::get_acs() arguments for this chunk, including the chunk's named +## `variables` vector, so a change to a table's variable definitions produces +## a new cache key. `cache_stats` is an environment with `hits` and `total` +## counters, reported to the user once per compile_acs_data() call. +cached_get_acs = function(args, table_name, cache, cache_stats) { + if (!cache) return(acs_query(args)) + + cache_stats$total = cache_stats$total + 1 + key = rlang::hash(list(table_name = table_name, args = args)) + path = file.path(acs_cache_dir(), paste0("acs_", key, ".rds")) + + if (file.exists(path)) { + ## a corrupt/unreadable file falls through to a refetch that overwrites it + cached = tryCatch(readRDS(path), error = function(e) NULL) + if (!is.null(cached)) { + cache_stats$hits = cache_stats$hits + 1 + return(cached) + } + } + + result = acs_query(args) + dir.create(acs_cache_dir(), recursive = TRUE, showWarnings = FALSE) + saveRDS(result, path) + result +} + +#' @title Clear the urbnindicators cache +#' @description Deletes all raw ACS query results cached by +#' \code{compile_acs_data(cache = TRUE)}. +#' @details Cached files live in +#' \code{tools::R_user_dir("urbnindicators", which = "cache")}, or in the +#' directory named by \code{options(urbnindicators.cache_dir = ...)} when +#' set. Because ACS estimates do not change, cached +#' entries never go stale; clearing the cache is only useful to reclaim +#' disk space. +#' @returns The number of cache files removed, invisibly. +#' @examples +#' \dontrun{ +#' clear_acs_cache() +#' } +#' @export +clear_acs_cache = function() { + cache_files = list.files(acs_cache_dir(), pattern = "^acs_.*\\.rds$", full.names = TRUE) + removed = sum(file.remove(cache_files)) + cli::cli_inform("Removed {removed} cached ACS quer{?y/ies} from {.path {acs_cache_dir()}}.") + invisible(removed) +} diff --git a/R/compile_acs_data.R b/R/compile_acs_data.R index 57233fb..b28ecbd 100644 --- a/R/compile_acs_data.R +++ b/R/compile_acs_data.R @@ -36,14 +36,43 @@ latest_acs_year = function() { ## Internal helper: fetch raw ACS estimates across years, states, and counties. ## One tidycensus call per (state, year), with vector `county =` when the user ## supplied a counties subset and the geography supports county filtering. +## When `cache = TRUE`, each (state, year) query is instead issued as one call +## per table in `table_variable_map` via cached_get_acs(), so raw results are +## cached and reused at the geography x year x state x table level. Chunks are +## joined on GEOID; because tidycensus orders wide output by the input variable +## order and the map concatenates to `variables`, the assembled columns match +## the single-call output. fetch_acs = function(geography, variables, years, states, counties, - county_codes, super_state_geographies) { + county_codes, super_state_geographies, + cache = FALSE, table_variable_map = NULL, cache_stats = NULL) { + + ## issue a single combined query (cache = FALSE) or one query per table + ## chunk, given fully-specified get_acs() args minus `variables` + fetch_one = function(base_args) { + if (!cache) { + return(acs_query(c(base_args, list(variables = variables)))) + } + chunks = purrr::imap(table_variable_map, function(table_variables, table_name) { + cached_get_acs( + args = c(base_args, list(variables = table_variables)), + table_name = table_name, + cache = TRUE, + cache_stats = cache_stats) + }) + ## join chunks on GEOID, dropping shared non-key columns (NAME) from later + ## chunks; all chunks cover the same geography universe + purrr::reduce(chunks, function(x, y) { + y = y %>% dplyr::select(-dplyr::any_of(setdiff(intersect(colnames(x), colnames(y)), "GEOID"))) + dplyr::full_join(x, y, by = "GEOID") + }) + } + if (geography %in% super_state_geographies) { return( purrr::map(years, function(year) { - tidycensus::get_acs( - geography = geography, variables = variables, - year = as.numeric(year), survey = "acs5", output = "wide") %>% + fetch_one(list( + geography = geography, + year = as.numeric(year), survey = "acs5", output = "wide")) %>% dplyr::mutate(data_source_year = year) }) %>% purrr::list_rbind()) } @@ -54,7 +83,7 @@ fetch_acs = function(geography, variables, years, states, counties, purrr::map(states, function(state) { purrr::map(years, function(year) { args = list( - geography = geography, variables = variables, + geography = geography, year = as.numeric(year), state = state, survey = "acs5", output = "wide") @@ -68,41 +97,40 @@ fetch_acs = function(geography, variables, years, states, counties, if (length(county_vec) > 0) args$county = county_vec } - do.call(tidycensus::get_acs, args) %>% + fetch_one(args) %>% dplyr::mutate(data_source_year = year) }) %>% purrr::list_rbind() }) %>% purrr::list_rbind() } -#' @title Analysis-ready social science measures -#' @description Construct measures frequently used in social sciences -#' research, leveraging \code{tidycensus::get_acs()} to acquire raw estimates from -#' the Census Bureau API. +#' @title Get analysis-ready estimates and errors from the ACS +#' @description Obtain raw and construct derived measures (primarily percentages) +#' from the ACS, along with appropriately-pooled margins of error. #' @param tables A character vector, list, or NULL specifying which data to #' include. Three kinds of elements are accepted and can be mixed freely #' inside a \code{list()}: #' \itemize{ #' \item \strong{Registered table names} (e.g., \code{"race"}, \code{"snap"}). -#' These are pre-built tables with curated variable definitions. Use +#' These are pre-built tables with non-standard variable definitions. Use #' \code{list_tables()} to see all available registered tables. #' \item \strong{Raw ACS table codes} (e.g., \code{"B25070"}, \code{"C15002B"}). #' Any valid ACS Detailed or Collapsed table code can be passed directly. -#' These are auto-processed at runtime: raw variables are fetched, the -#' label hierarchy is parsed, and percentages are computed automatically. -#' Use the \code{denominator} parameter to control how percentages are -#' calculated for these tables. -#' \item \strong{DSL definition objects} created with \code{\link{define_percent}}, +#' These return the full table along with percentage-based measures that are +#' calculated on the fly. Use the \code{denominator} parameter to control how +#' percentages are calculated. +#' \item \strong{Custom data specifications} created with \code{\link{define_percent}}, #' \code{\link{define_sum}}, \code{\link{define_complement}}, or #' \code{\link{define_metadata}}. These let you compute custom derived -#' variables from the columns produced by the tables you request. User -#' definitions are executed after all registered and auto-table -#' definitions, and their results appear in the codebook and have MOEs -#' computed automatically. +#' variables from the columns produced by the tables you request. #' } #' When mixing strings and definitions, wrap everything in \code{list()} #' (e.g., \code{list("snap", define_percent(...))}). -#' If an ACS code corresponds to an already-registered table, the registered -#' version is used automatically. +#' Raw ACS codes are always auto-processed, even when a registered table +#' covers the same code (e.g., \code{"B22003"} returns the auto-processed +#' table, not the registered \code{"snap"} table); registered tables are +#' returned only when requested by name. If a requested code overlaps a +#' registered table included in the same call, the registered version is +#' returned and the auto-processed version is dropped with a warning. #' When NULL (default), all registered tables are included (unregistered ACS #' tables must be requested explicitly). #' @param years A numeric vector of four-digit years for which to pull five-year @@ -127,6 +155,15 @@ fetch_acs = function(geography, variables, years, states, counties, #' \code{_001}). A specific ACS variable code (e.g., \code{"B25070_001"}) uses #' that variable. Only affects unregistered (auto) tables; registered tables #' always use their predefined definitions. +#' @param cache Boolean. When \code{TRUE}, raw ACS query results are cached on +#' disk and reused across calls and R sessions. Results are cached one file +#' per geography-year-state-table combination, so subsequent calls +#' re-download only what is not already cached -- including when the +#' \code{tables} selection changes. The cache lives at +#' \code{tools::R_user_dir("urbnindicators", which = "cache")} (override via +#' \code{options(urbnindicators.cache_dir = ...)}). Entries never expire, +#' because published five-year ACS estimates do not change; use +#' \code{\link{clear_acs_cache}()} to delete them and reclaim disk space. #' @param ... Deprecated arguments. If \code{variables} is passed, a deprecation #' warning is issued and the value is ignored. #' @seealso \code{tidycensus::get_acs()}, which this function wraps. @@ -206,6 +243,7 @@ compile_acs_data = function( counties = NULL, spatial = FALSE, denominator = "parent", + cache = FALSE, ...) { ## handle deprecated `variables` parameter and unknown arguments @@ -247,6 +285,11 @@ compile_acs_data = function( cli::cli_abort("{.arg denominator} must be {.val parent}, {.val total}, or a valid ACS variable code (e.g., {.val B25070_001}). Got: {.val {denominator}}.") } + ## validate cache parameter + if (!rlang::is_bool(cache)) { + cli::cli_abort("{.arg cache} must be TRUE or FALSE.") + } + ####----Partition tables into registry vs auto vs user definitions----#### auto_table_entries = list() registry_tables = tables @@ -275,42 +318,21 @@ compile_acs_data = function( census_variables_for_resolve = load_acs_variables(year = max(years), dataset = "acs5") })}) - ## collect all acs_tables from registered tables to detect overlap - registered_acs_codes = purrr::map(internal_names, function(tn) { - entry = get_table(tn) - if (!is.null(entry[["acs_tables"]])) entry[["acs_tables"]] else character(0) - }) %>% unlist() %>% unique() - - ## helper: find the registered table covering a given ACS code - find_covering_table = function(acs_code) { - purrr::detect(internal_names, function(tn) { - entry = get_table(tn) - acs_code %in% entry[["acs_tables"]] - }) - } - - ## classify each user-supplied table name + ## classify each user-supplied table name. Raw ACS codes are always served + ## as auto-processed tables, even when a registered table covers the same + ## code; registered tables are returned only when requested by name. classified = purrr::map(tables, function(tbl) { if (tbl %in% internal_names || tbl %in% names(construct_map)) { ## known registry table or construct name return(list(type = "registry", value = tbl)) } if (is_raw_acs_code(tbl)) { - ## raw ACS table code — check for overlap with registered tables - if (tbl %in% registered_acs_codes) { - covering = find_covering_table(tbl) - if (!is.null(covering)) return(list(type = "registry", value = covering)) - } return(list(type = "auto", value = tbl)) } ## try resolving as a cleaned variable name resolved_code = resolve_to_acs_table(tbl, year = max(years), census_variables = census_variables_for_resolve) if (!is.null(resolved_code)) { - if (resolved_code %in% registered_acs_codes) { - covering = find_covering_table(resolved_code) - if (!is.null(covering)) return(list(type = "registry", value = covering)) - } return(list(type = "auto", value = resolved_code)) } ## not resolvable — pass through to resolve_tables() which will error if invalid @@ -454,6 +476,26 @@ compile_acs_data = function( census_codebook = bg_codebook) })}) + ## a raw ACS code requested alongside a registered table that already pulls + ## (some of) its variables would duplicate ACS codes in the query; serve the + ## registered version and drop the auto entry with a warning + if (length(auto_table_entries) > 0 && length(variables) > 0) { + suppressWarnings({suppressMessages({ + registry_variable_map = collect_raw_variables_by_table( + resolved_tables = resolved_tables, year = max(years), + geography = geography, + census_codebook = bg_codebook) + })}) + auto_table_entries = purrr::imap(auto_table_entries, function(entry, code) { + overlapping = purrr::keep(registry_variable_map, ~ any(.x %in% entry[["raw_variables"]])) + if (length(overlapping) == 0) return(entry) + cli::cli_warn(c( + "ACS table {.val {code}} overlaps the registered table{?s} {.val {names(overlapping)}} included in this request; the registered version is returned.", + "i" = "To get the auto-processed version, request {.val {code}} without the overlapping registered table{?s}.")) + NULL + }) %>% purrr::compact() + } + ## append auto-table raw variables if (length(auto_table_entries) > 0) { auto_variables = purrr::map(auto_table_entries, ~ .x[["raw_variables"]]) %>% @@ -461,6 +503,23 @@ compile_acs_data = function( variables = c(variables, auto_variables) } + ## per-table variable map for the cache = TRUE fetch path (one cache entry + ## per table); its concatenation reproduces `variables` exactly + table_variable_map = NULL + if (cache) { + suppressWarnings({suppressMessages({ + table_variable_map = collect_raw_variables_by_table( + resolved_tables = resolved_tables, year = max(years), + geography = geography, + census_codebook = bg_codebook) + })}) + if (length(auto_table_entries) > 0) { + table_variable_map = c( + table_variable_map, + purrr::map(auto_table_entries, ~ .x[["raw_variables"]])) + } + } + ## resolve ACS codes in user definitions to clean column names if (length(user_definitions) > 0) { user_definitions = resolve_definition_variables(user_definitions, variables) @@ -587,6 +646,10 @@ compile_acs_data = function( })}) } + cache_stats = new.env(parent = emptyenv()) + cache_stats$hits = 0 + cache_stats$total = 0 + suppressMessages({ suppressWarnings({ df_raw_estimates = fetch_acs( geography = geography, @@ -595,10 +658,17 @@ compile_acs_data = function( states = states_for_fetch, counties = counties, county_codes = county_codes, - super_state_geographies = super_state_geographies) + super_state_geographies = super_state_geographies, + cache = cache, + table_variable_map = table_variable_map, + cache_stats = cache_stats) moes = df_raw_estimates %>% dplyr::select(GEOID, data_source_year, dplyr::matches("_M$")) })}) + if (cache && cache_stats$hits > 0) { + cli::cli_inform("Loaded {cache_stats$hits} of {cache_stats$total} ACS table quer{?y/ies} from the cache.") + } + ####----Compute derived variables----#### df_calculated_estimates = df_raw_estimates %>% dplyr::select(-dplyr::matches("_M$")) %>% diff --git a/R/generate_codebook.R b/R/generate_codebook.R index 858deaf..b5254b6 100644 --- a/R/generate_codebook.R +++ b/R/generate_codebook.R @@ -217,6 +217,44 @@ generate_codebook = function(.data, resolved_tables = NULL, auto_table_entries = TRUE ~ "unknown")) %>% ensure_aggregation_strategy() + ####----Universe----#### + ## Attach each ACS table's published universe statement (e.g., "Households", + ## "Population 25 years and over"). Raw variables resolve through the + ## variable crosswalk; derived variables inherit the universe of their first + ## numerator variable; a second pass lets derived-from-derived variables + ## (e.g., complements of percentages) inherit from their source variable. + ## The Census API publishes universes for vintages 2020+; earlier vintages + ## yield NA. + groups_metadata = load_acs_groups(year = year, dataset = "acs5") + universe_by_clean_name = variable_name_crosswalk %>% + dplyr::mutate(acs_table = stringr::str_remove(raw_name, "_[0-9]{3}$")) %>% + dplyr::left_join(groups_metadata, by = "acs_table") + universe_lookup = stats::setNames(universe_by_clean_name$universe, + universe_by_clean_name$clean_name) + + lookup_first_universe = function(source_vars, lookup) { + purrr::map_chr(source_vars, function(vars) { + matched = lookup[vars] + matched = matched[!is.na(matched)] + if (length(matched) > 0) matched[[1]] else NA_character_ + }) + } + + result1 = result1 %>% + dplyr::mutate(universe = dplyr::coalesce( + unname(universe_lookup[calculated_variable]), + lookup_first_universe(numerator_vars, universe_lookup))) + + self_lookup = stats::setNames(result1$universe, result1$calculated_variable) + result1 = result1 %>% + dplyr::mutate( + universe = dplyr::coalesce( + universe, lookup_first_universe(numerator_vars, self_lookup)), + ## metadata fields (GEOID, areas, etc.) have no ACS universe + universe = dplyr::if_else( + is.na(universe) & variable_type == "Metadata", + "Not applicable", universe)) + return(result1) } @@ -238,4 +276,5 @@ utils::globalVariables(c( "calculated_variable", "clean_name", "definition", "variable_type", "numerator_vars", "numerator_subtract_vars", "denominator_vars", "denominator_subtract_vars", - "se_calculation_type", "aggregation_strategy")) + "se_calculation_type", "aggregation_strategy", + "raw_name", "acs_table", "universe")) diff --git a/R/load_acs_variables.R b/R/load_acs_variables.R index 2500937..61c3b5c 100644 --- a/R/load_acs_variables.R +++ b/R/load_acs_variables.R @@ -94,3 +94,51 @@ load_acs_variables = function(year, dataset = "acs5") { .variables_cache[[cache_key]] = result result } + +## Internal: keyed, cached fetch of the Census ACS groups metadata, mapping +## each ACS table code to its published universe statement (e.g., +## "Households", "Population 25 years and over"). The groups.json endpoint +## does not require an API key. Universe statements are published for +## vintages 2020 and later; earlier vintages yield NA universes. +load_acs_groups = function(year, dataset = "acs5") { + if (!identical(dataset, "acs5")) { + cli::cli_abort(c( + "Only {.val acs5} is supported for {.arg dataset}.", + "i" = "Got {.val {dataset}}.")) + } + + cache_key = paste("groups", dataset, year, sep = "_") + if (!is.null(.variables_cache[[cache_key]])) { + return(.variables_cache[[cache_key]]) + } + + url = paste0("https://api.census.gov/data/", year, "/acs/acs5/groups.json") + + response = httr::GET(url) + if (httr::status_code(response) == 404L) { + cli::cli_abort(c( + "Census API endpoint not found.", + "i" = "Does the dataset exist for the specified year? See {.url https://api.census.gov/data.html}.")) + } + if (httr::http_status(response)$category != "Success") { + cli::cli_abort("Census API request failed: {httr::http_status(response)$message}") + } + + parsed = httr::content(response, as = "text", encoding = "UTF-8") %>% + jsonlite::fromJSON(simplifyVector = FALSE) + + result = purrr::map(parsed$groups, function(group_meta) { + ## the API returns the universe field name with a trailing space + ## ("universe "); accept either spelling + universe = group_meta[["universe "]] + if (is.null(universe)) universe = group_meta[["universe"]] + tibble::tibble( + acs_table = group_meta[["name"]], + universe = if (is.null(universe)) NA_character_ else universe) + }) %>% + purrr::list_rbind() %>% + dplyr::distinct(acs_table, .keep_all = TRUE) + + .variables_cache[[cache_key]] = result + result +} diff --git a/R/make_pretty_names.R b/R/make_pretty_names.R index 8f683bd..dce61ea 100644 --- a/R/make_pretty_names.R +++ b/R/make_pretty_names.R @@ -57,6 +57,9 @@ make_pretty_names = function(.data, .case = "title") { "types of computing devices household" = "computer access", "owneroccupied" = "owner-occupied", "renteroccupied" = "renter-occupied", + "withoutmortgage" = "without mortgage", + "withmortgage" = "with mortgage", + "alltenures" = "all tenures", "owner occupied" = "owner-occupied", "renter occupied" = "renter-occupied", "percentormore" = " percent or more", @@ -120,6 +123,10 @@ make_pretty_names = function(.data, .case = "title") { "1960 1969" = "1960-1969", "1950 1959" = "1950-1959", "1940 1949" = "1940-1949", + "10 0" = "10.0", + "14 9" = "14.9", + "15 0" = "15.0", + "19 9" = "19.9", "20 0" = "20.0", "24 9" = "24.9", "25 0" = "25.0", diff --git a/R/table_registry.R b/R/table_registry.R index bc44640..d2385b3 100644 --- a/R/table_registry.R +++ b/R/table_registry.R @@ -794,6 +794,31 @@ collect_raw_variables = function(resolved_tables, year = latest_acs_year(), geog return(all_variables) } +## Build per-table named ACS variable vectors for resolved tables (internal) +## Returns a named list mapping each table name to the variable subvector that +## collect_raw_variables() would contribute for it; tables with no raw +## variables of their own (e.g., population_density) are omitted. Concatenating +## the list in order reproduces the collect_raw_variables() vector exactly, +## which the cache = TRUE fetch path relies on to match single-call column +## order. Used to cache raw estimates one entry per table. +collect_raw_variables_by_table = function(resolved_tables, year = latest_acs_year(), geography = NULL, census_codebook = NULL) { + if (is.null(census_codebook)) { + census_codebook = load_acs_variables(year = year, dataset = "acs5") + } + + table_variables = purrr::map(resolved_tables, function(table_name) { + collect_table_variables(get_table(table_name), census_codebook) + }) %>% purrr::set_names(resolved_tables) + + ## at the block-group level, retain only variables published at that geography + if (!is.null(geography) && tolower(geography) == "block group") { + bg_variables = census_codebook$name[census_codebook$geography == "block group"] + table_variables = purrr::map(table_variables, ~ .x[.x %in% bg_variables]) + } + + purrr::compact(table_variables) %>% purrr::discard(~ length(.x) == 0) +} + ## Partition resolved tables by block-group availability (internal) ## Reads the codebook `geography` column (each variable's lowest published ## geography). Returns a list with: @@ -1594,27 +1619,27 @@ register_table(list( definitions = list( define_percent("household_income_by_gross_rent.*(30_0|35_0|40_0|50_0).*(pct|percent_more)$", denominator = "household_income_by_gross_rent.*([0-9]$|100000_more$)", - output = "cost_burdened_30percentormore_allincomes_percent", + output = "cost_burdened_renter_30percentormore_allincomes_percent", subtract_from_denominator = "household_income.*not_computed"), define_percent("household_income_by_gross_rent.*50_0.*(pct|percent_more)$", denominator = "household_income_by_gross_rent.*([0-9]$|100000_more$)", - output = "cost_burdened_50percentormore_allincomes_percent", + output = "cost_burdened_renter_50percentormore_allincomes_percent", subtract_from_denominator = "household_income.*not_computed"), define_percent("household_income_by_gross_rent.*(10000_|19999|34999).*(30_0|35_0|40_0|50_0).*(pct|percent_more)$", denominator = "household_income_by_gross_rent.*(10000|19999|34999)$", - output = "cost_burdened_30percentormore_incomeslessthan35000_percent", + output = "cost_burdened_renter_30percentormore_incomeslessthan35000_percent", subtract_from_denominator = "household_income.*(10000_|19999|34999).*not_computed"), define_percent("household_income_by_gross_rent.*(10000_|19999|34999).*50_0.*(pct|percent_more)$", denominator = "household_income_by_gross_rent.*(10000|19999|34999)$", - output = "cost_burdened_50percentormore_incomeslessthan35000_percent", + output = "cost_burdened_renter_50percentormore_incomeslessthan35000_percent", subtract_from_denominator = "household_income.*(10000_|19999|34999).*not_computed"), define_percent("household_income_by_gross_rent.*(10000_|19999|34999|49999).*(30_0|35_0|40_0|50_0).*(pct|percent_more)$", denominator = "household_income_by_gross_rent.*(10000|19999|34999|49999)$", - output = "cost_burdened_30percentormore_incomeslessthan50000_percent", + output = "cost_burdened_renter_30percentormore_incomeslessthan50000_percent", subtract_from_denominator = "household_income.*(10000_|19999|34999|49999).*not_computed"), define_percent("household_income_by_gross_rent.*(10000_|19999|34999|49999).*50_0.*(pct|percent_more)$", denominator = "household_income_by_gross_rent.*(10000|19999|34999|49999)$", - output = "cost_burdened_50percentormore_incomeslessthan50000_percent", + output = "cost_burdened_renter_50percentormore_incomeslessthan50000_percent", subtract_from_denominator = "household_income.*(10000_|19999|34999|49999).*not_computed")) )) @@ -1628,7 +1653,80 @@ register_table(list( calls = list( list(pattern = "B25106"))), raw_variables = NULL, - definitions = list() + ## Denominators exclude households for which a cost ratio is not computable + ## (zero/negative income; renters paying no cash rent): B25106 reports these + ## as standalone lines with no cost-band breakdown, so counting them in + ## denominators would classify all of them as un-burdened. + definitions = list( + define_percent("tenure_by_housing_costs.*owner_occupied.*30_percent_more$", + denominator = "tenure_by_housing_costs.*owner_occupied_housing_units$", + output = "cost_burdened_owner_30percentormore_allincomes_percent", + subtract_from_denominator = "tenure_by_housing_costs.*owner_occupied.*zero_negative_income$"), + define_percent("tenure_by_housing_costs.*owner_occupied.*(less_than_20000|20000_34999)_30_percent_more$", + denominator = "tenure_by_housing_costs.*owner_occupied.*(less_than_20000|20000_34999)$", + output = "cost_burdened_owner_30percentormore_incomeslessthan35000_percent"), + define_percent("tenure_by_housing_costs.*owner_occupied.*(less_than_20000|20000_34999|35000_49999)_30_percent_more$", + denominator = "tenure_by_housing_costs.*owner_occupied.*(less_than_20000|20000_34999|35000_49999)$", + output = "cost_burdened_owner_30percentormore_incomeslessthan50000_percent"), + define_percent("tenure_by_housing_costs.*(owner|renter)_occupied.*30_percent_more$", + denominator = "tenure_by_housing_costs.*universe$", + output = "cost_burdened_alltenures_30percentormore_allincomes_percent", + subtract_from_denominator = "tenure_by_housing_costs.*(zero_negative_income|no_cash_rent)$")) +)) + +register_table(list( + name = "owner_cost_burden", + description = "Owner cost burden by mortgage status (selected monthly owner costs as a percentage of household income)", + acs_tables = "B25091", + depends_on = character(0), + raw_variable_source = list(type = "manual"), + raw_variables = c( + owner_costs_universe_ = "B25091_001", + owner_costs_withmortgage_universe_ = "B25091_002", + owner_costs_withmortgage_less_than_10_0_ = "B25091_003", + owner_costs_withmortgage_10_0_14_9_ = "B25091_004", + owner_costs_withmortgage_15_0_19_9_ = "B25091_005", + owner_costs_withmortgage_20_0_24_9_ = "B25091_006", + owner_costs_withmortgage_25_0_29_9_ = "B25091_007", + owner_costs_withmortgage_30_0_34_9_ = "B25091_008", + owner_costs_withmortgage_35_0_39_9_ = "B25091_009", + owner_costs_withmortgage_40_0_49_9_ = "B25091_010", + owner_costs_withmortgage_50_0_more_ = "B25091_011", + owner_costs_withmortgage_not_computed_ = "B25091_012", + owner_costs_withoutmortgage_universe_ = "B25091_013", + owner_costs_withoutmortgage_less_than_10_0_ = "B25091_014", + owner_costs_withoutmortgage_10_0_14_9_ = "B25091_015", + owner_costs_withoutmortgage_15_0_19_9_ = "B25091_016", + owner_costs_withoutmortgage_20_0_24_9_ = "B25091_017", + owner_costs_withoutmortgage_25_0_29_9_ = "B25091_018", + owner_costs_withoutmortgage_30_0_34_9_ = "B25091_019", + owner_costs_withoutmortgage_35_0_39_9_ = "B25091_020", + owner_costs_withoutmortgage_40_0_49_9_ = "B25091_021", + owner_costs_withoutmortgage_50_0_more_ = "B25091_022", + owner_costs_withoutmortgage_not_computed_ = "B25091_023"), + ## "Not computed" (zero/negative income) households are excluded from + ## denominators, matching the treatment of B25074-based renter measures. + definitions = list( + define_percent("owner_costs_withmortgage_(30_0_34_9|35_0_39_9|40_0_49_9|50_0_more)$", + denominator = "owner_costs_withmortgage_universe$", + output = "cost_burdened_owner_withmortgage_30percentormore_percent", + subtract_from_denominator = "owner_costs_withmortgage_not_computed$"), + define_percent("owner_costs_withmortgage_50_0_more$", + denominator = "owner_costs_withmortgage_universe$", + output = "cost_burdened_owner_withmortgage_50percentormore_percent", + subtract_from_denominator = "owner_costs_withmortgage_not_computed$"), + define_percent("owner_costs_withoutmortgage_(30_0_34_9|35_0_39_9|40_0_49_9|50_0_more)$", + denominator = "owner_costs_withoutmortgage_universe$", + output = "cost_burdened_owner_withoutmortgage_30percentormore_percent", + subtract_from_denominator = "owner_costs_withoutmortgage_not_computed$"), + define_percent("owner_costs_withoutmortgage_50_0_more$", + denominator = "owner_costs_withoutmortgage_universe$", + output = "cost_burdened_owner_withoutmortgage_50percentormore_percent", + subtract_from_denominator = "owner_costs_withoutmortgage_not_computed$"), + define_percent("owner_costs_with(out)?mortgage_50_0_more$", + denominator = "owner_costs_with(out)?mortgage_universe$", + output = "cost_burdened_owner_50percentormore_allincomes_percent", + subtract_from_denominator = "owner_costs_with(out)?mortgage_not_computed$")) )) register_table(list( diff --git a/dev/release-decisions.md b/dev/release-decisions.md new file mode 100644 index 0000000..ccce14a --- /dev/null +++ b/dev/release-decisions.md @@ -0,0 +1,58 @@ +# Release decisions — handoff notes (updated 2026-07-04, second session) + +State when written: PRs #89–#98 merged; version 0.1.0 (no tag yet). Items 1–5 +below were decided on 2026-07-04 and implemented on branch +`release-prep-0.1.0`; item 6 remains open. + +## Still needs the user (blocked for the agent by permissions) + +1. ~~Add the Census API key as a repository secret~~ **DONE 2026-07-04** + (set by the user; pkgdown re-run triggered on PR #99). + +2. ~~Kill the overnight caffeinate~~ **DONE 2026-07-04**. + +## Decisions made (2026-07-04) — implemented on `release-prep-0.1.0` + +1. **`$` in column names — DECIDED: strip.** `clean_acs_names()` gained a + `"\\$" = ""` rule. Verified against the 2023 codebook: 3,175 names change + (exactly those with `$` in labels), no new duplicate names, all nine + cost_burden regexes match identical column sets before/after. +2. **Misleading names — DECIDED: rename both.** + `employment_civilian_labor_force_percent` → + `employment_civilian_labor_force_employed_percent`; + `median_household_income_nhpi_` → `median_household_income_nhpi_alone_`. +3. **`list_acs_variables()` stub — DECIDED: deleted** (function, test, + man page, pkgdown entry, NAMESPACE export). +4. **`se_*`/`cv` family — DECIDED: keep internal** for 0.1.0; exporting later + is non-breaking. +5. **R-CMD-check — tightened** to `error-on: '"warning"'`; watch the PR's CI + run for anything that surfaces. + +All three breaking changes are recorded in NEWS.md under 0.1.0. Fixtures +(`test_data_2026-02-08.rds`, `codebook_2026-02-08.rds`) were regenerated; the +block-group fixture was untouched (its tables have no `$` labels and none of +the renamed variables). + +## Still open + +(nothing — all handoff items are resolved) + +6. ~~`tenure_by_housing_costs` (B25106) indicators~~ **DECIDED & + IMPLEMENTED 2026-07-04** (PR #100, stacked on #99): owner 30%+ shares + (all incomes, <$35k, <$50k) + all-tenures headline from B25106; new + `owner_cost_burden` table (B25091) with 30%+/50%+ by mortgage status and + combined owner severe burden; zero/negative-income, no-cash-rent, and + not-computed households excluded from denominators; existing renter + measures renamed `cost_burdened_renter_*` (breaking, pre-tag). PR #100 + also adds a codebook `universe` column (Census universe statements, + 2020+ vintages). + +## Housekeeping + +- Three stashes exist (the original handoff said two): `stash@{0}` ("WIP on + pretty-names"), `stash@{1}` ("WIP on codebook"), `stash@{2}` ("WIP on cov"). + Review or drop. +- No 0.1.0 git tag yet; tag once `release-prep-0.1.0` merges (it contains the + breaking changes). +- renv reports the project out-of-sync (renv 1.1.8 loaded vs 1.2.3 in + lockfile) — pre-existing, not touched. diff --git a/man/clear_acs_cache.Rd b/man/clear_acs_cache.Rd new file mode 100644 index 0000000..8432fc6 --- /dev/null +++ b/man/clear_acs_cache.Rd @@ -0,0 +1,28 @@ +% Generated by roxygen2: do not edit by hand +% Please edit documentation in R/cache.R +\name{clear_acs_cache} +\alias{clear_acs_cache} +\title{Clear the urbnindicators cache} +\usage{ +clear_acs_cache() +} +\value{ +The number of cache files removed, invisibly. +} +\description{ +Deletes all raw ACS query results cached by +\code{compile_acs_data(cache = TRUE)}. +} +\details{ +Cached files live in +\code{tools::R_user_dir("urbnindicators", which = "cache")}, or in the +directory named by \code{options(urbnindicators.cache_dir = ...)} when +set. Because ACS estimates do not change, cached +entries never go stale; clearing the cache is only useful to reclaim +disk space. +} +\examples{ +\dontrun{ +clear_acs_cache() +} +} diff --git a/man/compile_acs_data.Rd b/man/compile_acs_data.Rd index 06f4aa1..e9fbd18 100644 --- a/man/compile_acs_data.Rd +++ b/man/compile_acs_data.Rd @@ -2,7 +2,7 @@ % Please edit documentation in R/compile_acs_data.R \name{compile_acs_data} \alias{compile_acs_data} -\title{Analysis-ready social science measures} +\title{Get analysis-ready estimates and errors from the ACS} \usage{ compile_acs_data( tables = NULL, @@ -12,6 +12,7 @@ compile_acs_data( counties = NULL, spatial = FALSE, denominator = "parent", + cache = FALSE, ... ) } @@ -21,26 +22,26 @@ include. Three kinds of elements are accepted and can be mixed freely inside a \code{list()}: \itemize{ \item \strong{Registered table names} (e.g., \code{"race"}, \code{"snap"}). -These are pre-built tables with curated variable definitions. Use +These are pre-built tables with non-standard variable definitions. Use \code{list_tables()} to see all available registered tables. \item \strong{Raw ACS table codes} (e.g., \code{"B25070"}, \code{"C15002B"}). Any valid ACS Detailed or Collapsed table code can be passed directly. -These are auto-processed at runtime: raw variables are fetched, the -label hierarchy is parsed, and percentages are computed automatically. -Use the \code{denominator} parameter to control how percentages are -calculated for these tables. -\item \strong{DSL definition objects} created with \code{\link{define_percent}}, +These return the full table along with percentage-based measures that are +calculated on the fly. Use the \code{denominator} parameter to control how +percentages are calculated. +\item \strong{Custom data specifications} created with \code{\link{define_percent}}, \code{\link{define_sum}}, \code{\link{define_complement}}, or \code{\link{define_metadata}}. These let you compute custom derived -variables from the columns produced by the tables you request. User -definitions are executed after all registered and auto-table -definitions, and their results appear in the codebook and have MOEs -computed automatically. +variables from the columns produced by the tables you request. } When mixing strings and definitions, wrap everything in \code{list()} (e.g., \code{list("snap", define_percent(...))}). -If an ACS code corresponds to an already-registered table, the registered -version is used automatically. +Raw ACS codes are always auto-processed, even when a registered table +covers the same code (e.g., \code{"B22003"} returns the auto-processed +table, not the registered \code{"snap"} table); registered tables are +returned only when requested by name. If a requested code overlaps a +registered table included in the same call, the registered version is +returned and the auto-processed version is dropped with a warning. When NULL (default), all registered tables are included (unregistered ACS tables must be requested explicitly).} @@ -72,6 +73,16 @@ the ACS label hierarchy. \code{"total"} uses the table total (variable that variable. Only affects unregistered (auto) tables; registered tables always use their predefined definitions.} +\item{cache}{Boolean. When \code{TRUE}, raw ACS query results are cached on +disk and reused across calls and R sessions. Results are cached one file +per geography-year-state-table combination, so subsequent calls +re-download only what is not already cached -- including when the +\code{tables} selection changes. The cache lives at +\code{tools::R_user_dir("urbnindicators", which = "cache")} (override via +\code{options(urbnindicators.cache_dir = ...)}). Entries never expire, +because published five-year ACS estimates do not change; use +\code{\link{clear_acs_cache}()} to delete them and reclaim disk space.} + \item{...}{Deprecated arguments. If \code{variables} is passed, a deprecation warning is issued and the value is ignored.} } @@ -109,9 +120,8 @@ estimates; they are an experimental feature and should be interpreted with care. } \description{ -Construct measures frequently used in social sciences -research, leveraging \code{tidycensus::get_acs()} to acquire raw estimates from -the Census Bureau API. +Obtain raw and construct derived measures (primarily percentages) +from the ACS, along with appropriately-pooled margins of error. } \examples{ \dontrun{ diff --git a/tests/testthat/fixtures/codebook_2026-02-08.rds b/tests/testthat/fixtures/codebook_2026-02-08.rds index cb22214..7752744 100644 Binary files a/tests/testthat/fixtures/codebook_2026-02-08.rds and b/tests/testthat/fixtures/codebook_2026-02-08.rds differ diff --git a/tests/testthat/fixtures/test_data_2026-02-08.rds b/tests/testthat/fixtures/test_data_2026-02-08.rds index 1baca70..4f3aa06 100644 Binary files a/tests/testthat/fixtures/test_data_2026-02-08.rds and b/tests/testthat/fixtures/test_data_2026-02-08.rds differ diff --git a/tests/testthat/test-auto_percent.R b/tests/testthat/test-auto_percent.R index 1504fbe..9ed9598 100644 --- a/tests/testthat/test-auto_percent.R +++ b/tests/testthat/test-auto_percent.R @@ -92,6 +92,28 @@ test_that("classify_acs_table correctly identifies count vs skip tables", { stringsAsFactors = FALSE ) expect_equal(classify_acs_table(singleton_nodes), "skip") + + ## skip keywords match on word boundaries: "mean" must not match "Means" + ## (B08301, Means of Transportation to Work, is a count table) + means_nodes = data.frame( + concept = "MEANS OF TRANSPORTATION TO WORK", + label = c("Estimate!!Total:", "Estimate!!Total:!!Car, truck, or van:"), + is_total = c(TRUE, FALSE), + stringsAsFactors = FALSE + ) + expect_equal(classify_acs_table(means_nodes), "count") + + ## a mid-label colon is not a count total: B19080's _001 is a dollar value + ## ("Lowest Quintile Upper Limit"), so no percentages should be computed + quintile_nodes = data.frame( + concept = "HOUSEHOLD INCOME QUINTILE UPPER LIMITS", + label = c( + "Estimate!!Quintile Upper Limits:!!Lowest Quintile Upper Limit", + "Estimate!!Quintile Upper Limits:!!Second Quintile Upper Limit"), + is_total = c(TRUE, FALSE), + stringsAsFactors = FALSE + ) + expect_equal(classify_acs_table(quintile_nodes), "skip") }) test_that("generate_auto_definitions produces correct definitions with parent mode", { @@ -254,18 +276,40 @@ test_that("compile_acs_data works with race-iterated table", { } }) -test_that("compile_acs_data detects overlap with registered tables", { +test_that("compile_acs_data serves the auto version for a raw code covered by a registered table", { skip_on_cran() skip_if_not(nchar(Sys.getenv("CENSUS_API_KEY")) > 0, "Census API key not available") - ## B22003 is the ACS table behind the registered "snap" table - ## passing the raw code should silently use the registered version + ## B22003 is the ACS table behind the registered "snap" table; passing the + ## raw code should return the auto-processed table, not the registered one result = compile_acs_data( tables = c("B22003"), years = 2022, geography = "state", states = "DC") - ## should have the registered snap variable + ## no registered snap variables + expect_false(any(grepl("^snap_", colnames(result)))) + ## auto-processed B22003 variables with label-derived names, plus percentages + auto_cols = grep("receipt_food_stamps", colnames(result), value = TRUE) + expect_true(length(auto_cols) > 0) + expect_true(any(grepl("_percent$", auto_cols))) +}) + +test_that("compile_acs_data warns and keeps the registered version when a raw code overlaps it", { + skip_on_cran() + skip_if_not(nchar(Sys.getenv("CENSUS_API_KEY")) > 0, "Census API key not available") + + ## requesting both the registered table and its underlying code would fetch + ## duplicate ACS variables; the registered version wins with a warning + expect_warning( + result <- compile_acs_data( + tables = c("snap", "B22003"), + years = 2022, + geography = "state", + states = "DC"), + regexp = "overlaps the registered table") + expect_true("snap_received_percent" %in% colnames(result)) + expect_false(any(grepl("receipt_food_stamps", colnames(result)))) }) diff --git a/tests/testthat/test-cache.R b/tests/testthat/test-cache.R new file mode 100644 index 0000000..edf2a50 --- /dev/null +++ b/tests/testthat/test-cache.R @@ -0,0 +1,189 @@ +# Tests for the raw-ACS-query cache (R/cache.R and the cache = TRUE fetch +# path in fetch_acs()). Unit tests mock acs_query() so no network or API key +# is needed; the integration tests at the bottom are gated on +# skip_if_no_census_key(). + +super_state_geographies = c( + "us", "region", "division", "metropolitan/micropolitan statistical area", + "metropolitan statistical area/micropolitan statistical area", + "cbsa", "urban area", "zip code tabulation area", "zcta") + +new_cache_stats = function() { + stats = new.env(parent = emptyenv()) + stats$hits = 0 + stats$total = 0 + stats +} + +cache_files = function(full.names = FALSE) { + list.files(acs_cache_dir(), pattern = "^acs_.*\\.rds$", full.names = full.names) +} + +## build a get_acs()-shaped wide tibble from a query's named variables vector +fake_acs_response = function(args) { + values = purrr::set_names( + purrr::map(seq_along(args$variables), ~ rep(.x, 2)), + paste0(names(args$variables), "E")) + dplyr::bind_cols( + tibble::tibble(GEOID = c("10", "34"), NAME = c("Delaware", "New Jersey")), + tibble::as_tibble(values)) +} + +test_that("cached_get_acs writes on a miss and reads on a hit", { + withr::local_options(urbnindicators.cache_dir = withr::local_tempdir()) + calls = 0 + local_mocked_bindings(acs_query = function(args) { + calls <<- calls + 1 + fake_acs_response(args) + }) + + stats = new_cache_stats() + args = list( + geography = "state", year = 2022, survey = "acs5", output = "wide", + variables = c(alpha_count_ = "B00001_001")) + + first = cached_get_acs(args, table_name = "alpha", cache = TRUE, cache_stats = stats) + expect_equal(calls, 1) + expect_length(cache_files(), 1) + + second = cached_get_acs(args, table_name = "alpha", cache = TRUE, cache_stats = stats) + expect_equal(calls, 1) + expect_identical(second, first) + expect_equal(stats$hits, 1) + expect_equal(stats$total, 2) +}) + +test_that("cache keys distinguish geography, year, and state", { + withr::local_options(urbnindicators.cache_dir = withr::local_tempdir()) + local_mocked_bindings(acs_query = fake_acs_response) + + base_args = list( + geography = "state", year = 2022, survey = "acs5", output = "wide", + variables = c(alpha_count_ = "B00001_001")) + year_args = base_args + year_args$year = 2021 + county_args = base_args + county_args$geography = "county" + state_args = c(base_args, list(state = "NJ")) + + purrr::walk( + list(base_args, year_args, county_args, state_args), + ~ cached_get_acs(.x, table_name = "alpha", cache = TRUE, cache_stats = new_cache_stats())) + expect_length(cache_files(), 4) +}) + +test_that("cache entries are per table: subset and added tables reuse prior downloads", { + withr::local_options(urbnindicators.cache_dir = withr::local_tempdir()) + fetched_tables = character(0) + local_mocked_bindings(acs_query = function(args) { + fetched_tables <<- c(fetched_tables, names(args$variables)[1]) + fake_acs_response(args) + }) + + map_ab = list( + alpha = c(alpha_count_ = "B00001_001"), + beta = c(beta_count_ = "B00002_001")) + call_fetch = function(table_variable_map) { + fetch_acs( + geography = "state", variables = unlist(unname(table_variable_map)), + years = 2022, states = "NJ", counties = c(), county_codes = NULL, + super_state_geographies = super_state_geographies, + cache = TRUE, table_variable_map = table_variable_map, + cache_stats = new_cache_stats()) + } + + first = call_fetch(map_ab) + expect_equal(fetched_tables, c("alpha_count_", "beta_count_")) + expect_length(cache_files(), 2) + ## assembled chunks follow the combined variable order, matching a single call + expect_identical( + colnames(first), + c("GEOID", "NAME", "alpha_count_E", "beta_count_E", "data_source_year")) + + ## a subset request triggers no new fetches + call_fetch(map_ab["alpha"]) + expect_equal(fetched_tables, c("alpha_count_", "beta_count_")) + + ## adding a table fetches only the new table + map_abc = c(map_ab, list(gamma = c(gamma_count_ = "B00003_001"))) + call_fetch(map_abc) + expect_equal(fetched_tables, c("alpha_count_", "beta_count_", "gamma_count_")) + expect_length(cache_files(), 3) +}) + +test_that("cache = FALSE issues one combined query and writes nothing", { + withr::local_options(urbnindicators.cache_dir = withr::local_tempdir()) + variable_counts = integer(0) + local_mocked_bindings(acs_query = function(args) { + variable_counts <<- c(variable_counts, length(args$variables)) + fake_acs_response(args) + }) + + variables = c(alpha_count_ = "B00001_001", beta_count_ = "B00002_001") + fetch_acs( + geography = "state", variables = variables, years = c(2021, 2022), + states = "NJ", counties = c(), county_codes = NULL, + super_state_geographies = super_state_geographies, cache = FALSE) + + ## one combined (two-variable) call per year, nothing cached + expect_equal(variable_counts, c(2L, 2L)) + expect_length(cache_files(), 0) +}) + +test_that("a corrupt cache file triggers a refetch that repairs it", { + withr::local_options(urbnindicators.cache_dir = withr::local_tempdir()) + calls = 0 + local_mocked_bindings(acs_query = function(args) { + calls <<- calls + 1 + fake_acs_response(args) + }) + + args = list( + geography = "state", year = 2022, survey = "acs5", output = "wide", + variables = c(alpha_count_ = "B00001_001")) + first = cached_get_acs(args, table_name = "alpha", cache = TRUE, cache_stats = new_cache_stats()) + + writeLines("not an rds file", cache_files(full.names = TRUE)) + second = cached_get_acs(args, table_name = "alpha", cache = TRUE, cache_stats = new_cache_stats()) + expect_equal(calls, 2) + expect_identical(second, first) + expect_identical(readRDS(cache_files(full.names = TRUE)), first) +}) + +test_that("clear_acs_cache removes cached files and returns the count", { + cache_dir = withr::local_tempdir() + withr::local_options(urbnindicators.cache_dir = cache_dir) + saveRDS(1, file.path(cache_dir, "acs_a.rds")) + saveRDS(2, file.path(cache_dir, "acs_b.rds")) + + expect_message(removed <- clear_acs_cache(), "Removed 2") + expect_equal(removed, 2) + expect_length(cache_files(), 0) + + expect_message(removed_again <- clear_acs_cache(), "Removed 0") + expect_equal(removed_again, 0) +}) + +test_that("compile_acs_data validates the cache argument", { + expect_error(compile_acs_data(cache = "yes"), "must be TRUE or FALSE") +}) + +test_that("cache = TRUE output matches cache = FALSE output", { + skip_if_no_census_key() + withr::local_options(urbnindicators.cache_dir = withr::local_tempdir()) + + compile_dc = function(cache) { + suppressMessages(suppressWarnings(compile_acs_data( + tables = c("snap", "race"), years = 2022, geography = "state", + states = "DC", cache = cache))) + } + + cached_cold = compile_dc(cache = TRUE) + expect_gt(length(cache_files()), 0) + + uncached = compile_dc(cache = FALSE) + expect_equal(cached_cold, uncached) + + cached_warm = compile_dc(cache = TRUE) + expect_equal(cached_warm, uncached) +}) diff --git a/tests/testthat/test-generate_codebook.R b/tests/testthat/test-generate_codebook.R index cdf79c1..0a9af88 100644 --- a/tests/testthat/test-generate_codebook.R +++ b/tests/testthat/test-generate_codebook.R @@ -17,6 +17,28 @@ test_codebook_path = test_path("fixtures", "codebook_2026-02-08.rds") testthat::expect_equal(results_missingness, 0) } ) +## Universe statements + testthat::test_that( + "Codebook includes ACS universe statements.", + { + testthat::skip_if_not(file.exists(test_codebook_path), "Test fixture not available") + codebook = readRDS(test_codebook_path) + + testthat::expect_true("universe" %in% colnames(codebook)) + testthat::expect_equal( + codebook$universe[codebook$calculated_variable == "snap_received_percent"], + "Households") + testthat::expect_equal( + codebook$universe[codebook$calculated_variable == "employment_civilian_labor_force_employed_percent"], + "Population 16 years and over") + ## complements inherit the universe of their source variable + testthat::expect_equal( + codebook$universe[codebook$calculated_variable == "ability_speak_english_less_than_very_well_percent"], + "Population 5 years and over") + ## metadata fields have no ACS universe + testthat::expect_true(all( + codebook$universe[codebook$variable_type == "Metadata"] == "Not applicable")) } ) + ## No transcribed function calls testthat::test_that( "No transcribed functions included in codebook output.", diff --git a/vignettes/acs-table-structures.Rmd b/vignettes/acs-table-structures.Rmd index 93df73a..916d9fb 100644 --- a/vignettes/acs-table-structures.Rmd +++ b/vignettes/acs-table-structures.Rmd @@ -103,12 +103,16 @@ to fetch and how to calculate derived measures from them. ### Auto-processed tables Any valid ACS table code can be passed directly (e.g., -`tables = "B25070"`). If the code corresponds to a registered -table, the registered version is used automatically. Otherwise, -the table is **auto-processed**: all of its variables are fetched, -names are generated algorithmically from the Census Bureau's -labels, and percentages are computed based on the table's label -hierarchy. +`tables = "B25070"`). The table is **auto-processed**: all of its +variables are fetched, names are generated algorithmically from +the Census Bureau's labels, and percentages are computed based on +the table's label hierarchy. This holds even when a registered +table covers the same code — `tables = "B22003"` returns the +auto-processed table, while `tables = "snap"` returns the +registered version. (Requesting both in the same call returns the +registered version and drops the auto-processed one with a +warning, since the two would otherwise fetch duplicate +variables.) Auto-processed tables are a flexible route for the thousands of ACS tables that `urbnindicators` doesn't have diff --git a/vignettes/codebook.Rmd b/vignettes/codebook.Rmd index af327b6..1be6615 100644 --- a/vignettes/codebook.Rmd +++ b/vignettes/codebook.Rmd @@ -59,7 +59,13 @@ The codebook's three primary columns are: `"Numerator = snap_received (B22003_002). Denominator = snap_universe (B22003_001)."` Subtracted components appear after a `-`. -The codebook also carries supporting columns used for margin-of-error +The codebook also carries a `universe` column giving the Census +Bureau's published universe statement for each variable--the +population the variable describes, e.g., `Households` or +`Population 25 years and over`. Derived variables inherit the +universe of their numerator. (The Census API publishes universe +statements for the 2020 and later vintages; earlier vintages yield +`NA`.) Supporting columns are used for margin-of-error calculation and re-aggregation--list-columns naming each variable's numerator/denominator components (`numerator_vars`, `denominator_vars`, and their `_subtract_` counterparts), an