Owner cost-burden measures, tenure-labeled renter measures, codebook universe statements

CLAUDE.md

@@ -58,7 +58,7 @@ The table registry is the central data structure that defines all ACS tables the
 - `raw_variables` - named vector of ACS variable codes (for manual sources)
 - `definitions` - list of DSL objects (`define_percent()`, `define_sum()`, `define_complement()`, `define_metadata()`) describing derived variables. Codebook entries and MOE-propagation strategy are derived from each object's `type`.
 
-There are 33 registered internal tables.
+There are 34 registered internal tables.
 
 ### Table selection API
 
@@ -82,7 +82,7 @@ Both construct names and internal names are accepted by `compile_acs_data(tables
 
 `tables` can contain three kinds of elements (mix freely inside a `list()`):
 - **Registered table names** (e.g., `"race"`, `"snap"`) — use `list_tables()` to see them all.
-- **Raw ACS table codes** (e.g., `"B25070"`, `"C15002B"`) — any valid Detailed/Collapsed code is auto-processed at runtime: raw variables are fetched, the label hierarchy is parsed, and percentages are computed automatically. The `denominator` parameter controls the percentage denominator (`"parent"` for nearest subtotal, `"total"` for `_001`, or a specific code like `"B25070_001"`). If a raw code is already covered by a registered table, the registered version is used.
+- **Raw ACS table codes** (e.g., `"B25070"`, `"C15002B"`) — any valid Detailed/Collapsed code is auto-processed at runtime: raw variables are fetched, the label hierarchy is parsed, and percentages are computed automatically. The `denominator` parameter controls the percentage denominator (`"parent"` for nearest subtotal, `"total"` for `_001`, or a specific code like `"B25070_001"`). Raw codes are always auto-processed, even when a registered table covers the same code (`"B22003"` returns the auto-processed table; `"snap"` returns the registered one). If a requested code overlaps a registered table included in the same call, the registered version wins and the auto entry is dropped with a warning.
 - **DSL definition objects** from `define_percent()`/`define_sum()`/`define_complement()`/`define_metadata()` — let users layer custom derived variables on top of the requested tables; results land in the returned data frame and the codebook with MOEs computed automatically.
 
 When `tables` are specified:
@@ -110,6 +110,7 @@ Note: don't hard-code a list of block-group tables — derive it from the codebo
 
 1. **`R/table_registry.R`** - Central registry, DSL constructors (`define_percent()`, `define_sum()`, `define_complement()`, `define_metadata()`), `validate_definition()`, `execute_definitions()`, `resolve_tables()`, `collect_raw_variables()`, `expand_codebook_entry()`, `list_tables()`, and all `register_table()` calls.
 2. **`R/compile_acs_data.R`** - `compile_acs_data()` (entry point), `fetch_acs()` (per-year/per-state tidycensus calls + ZCTA-style "super-state" geographies), `safe_divide()`.
+   **`R/cache.R`** - Opt-in disk cache of raw ACS query results (`cache = TRUE`): `acs_cache_dir()`, `acs_query()` (mockable seam around `tidycensus::get_acs()`), `cached_get_acs()` (one entry per geography × year × state × table, keyed by `rlang::hash()`; no expiry — published ACS estimates are immutable), and exported `clear_acs_cache()`. Cache dir is `tools::R_user_dir("urbnindicators", "cache")`, overridable via `options(urbnindicators.cache_dir = ...)` (tests use this). The per-table variable map comes from `collect_raw_variables_by_table()` in `R/table_registry.R`; the `cache = FALSE` path is unchanged (single combined query per state × year).
 3. **`R/auto_percent.R`** - Auto-table support for raw ACS table codes: `is_raw_acs_code()`, `resolve_to_acs_table()`, `build_auto_table_entry()`, `generate_auto_definitions()`.
 4. **`R/interpolate_acs.R`** - `interpolate_acs()` plus internal aggregation helpers; uses codebook attributes to dispatch per-variable aggregation strategy.
 5. **`R/list_acs_variables.R`** - `get_acs_codebook()` (exported), and internal helpers `select_variables_by_name()` and `filter_variables()` used by the registry's `"select_variables"` path. (The deprecated `list_acs_variables()` stub was removed before 0.1.0.)
@@ -121,7 +122,8 @@ Note: don't hard-code a list of block-group tables — derive it from the codebo
 
 ### Exported functions
 
-- `compile_acs_data(tables, years, geography, states, counties, spatial, denominator, ...)` - Pull and compute ACS data.
+- `compile_acs_data(tables, years, geography, states, counties, spatial, denominator, cache, ...)` - Pull and compute ACS data. `cache = TRUE` reuses raw ACS query results from the on-disk cache.
+- `clear_acs_cache()` - Delete cached raw ACS query files.
 - `interpolate_acs(.data, target_geoid_column, weight = NULL, crosswalk = NULL, source_geoid = "GEOID", weight_variable = "total_population_universe")` - Aggregate or interpolate ACS data to custom geographies. `weight = NULL` for complete nesting (direct aggregation); pass a weight column name for fractional allocation via crosswalk.
 - `define_percent()`, `define_sum()`, `define_complement()`, `define_metadata()` - DSL constructors for derived variables. Use inside `register_table(definitions = list(...))` or pass directly to `compile_acs_data(tables = list(...))`.
 - `list_tables()` - Available registered table names for the `tables` parameter (construct-level names).

DESCRIPTION

@@ -31,7 +31,8 @@ Imports:
     tigris,
     rlang,
     tibble,
-    sf
+    sf,
+    tools
 Suggests:
     arrow,
     bslib,
@@ -49,7 +50,8 @@ Suggests:
     shiny,
     testthat (>= 3.0.0),
     scales,
-    urbnthemes (>= 0.0.2)
+    urbnthemes (>= 0.0.2),
+    withr
 Remotes:
     UI-Research/crosswalk,
     UrbanInstitute/urbnthemes

NAMESPACE

@@ -1,6 +1,7 @@
 # Generated by roxygen2: do not edit by hand
 
 export("%>%")
+export(clear_acs_cache)
 export(compile_acs_data)
 export(define_complement)
 export(define_metadata)

NEWS.md

@@ -23,6 +23,11 @@ package's API and internals.
   income- and rent-band columns (e.g., from B25074/B25106) are syntactic:
   `household_income_..._less_than_$10000` becomes
   `household_income_..._less_than_10000`.
+* The renter cost-burden measures now carry an explicit tenure label:
+  `cost_burdened_30percentormore_*` and `cost_burdened_50percentormore_*`
+  are now `cost_burdened_renter_30percentormore_*` and
+  `cost_burdened_renter_50percentormore_*`, disambiguating them from the
+  new owner-side measures.
 * Two variables were renamed for accuracy and consistency:
   `employment_civilian_labor_force_percent` is now
   `employment_civilian_labor_force_employed_percent` (it is the employment
@@ -36,6 +41,13 @@ package's API and internals.
   `tidycensus::census_api_key("YOUR_KEY", install = TRUE)`.
 * `safe_divide()` now returns `NA` (not `0`) when a nonzero numerator is
   divided by zero; `0/0` still returns `0`.
+* Raw ACS table codes are now **always auto-processed**, even when a
+  registered table covers the same code: `tables = "B22003"` returns the
+  auto-processed table with label-derived names, while `tables = "snap"`
+  returns the registered version (previously the raw code was silently
+  upgraded to the registered table). Requesting both in one call returns
+  the registered version and drops the auto-processed one with a warning,
+  since the two would fetch duplicate variables.
 
 ## New features
 
@@ -55,6 +67,25 @@ package's API and internals.
   map results, interpolate to uploaded or hand-drawn target geographies,
   benchmark differences for statistical significance, and export data and
   images.
+* **Caching**: `compile_acs_data(cache = TRUE)` caches raw ACS query results
+  on disk (one entry per geography-year-state-table) and reuses them across
+  calls and R sessions, so repeat and overlapping requests -- including
+  changed `tables` selections -- re-download only what is missing. Entries
+  never expire (published five-year ACS estimates do not change); a new
+  `clear_acs_cache()` deletes them to reclaim disk space.
+* **Universe statements in the codebook.** The codebook gains a `universe`
+  column carrying the Census Bureau's published universe statement for each
+  variable (e.g., "Households", "Population 25 years and over"); derived
+  variables inherit their numerator's universe. Available for the 2020+
+  vintages (the API does not publish universes for earlier years).
+* **Owner cost-burden measures.** `tenure_by_housing_costs` (B25106) now
+  publishes owner cost-burden percentages (all incomes, incomes below
+  $35,000, incomes below $50,000) and an all-tenures headline measure; a
+  new `owner_cost_burden` table (B25091) adds 30-percent-or-more and
+  50-percent-or-more (severe) burden shares by mortgage status, plus a
+  combined all-owners severe-burden measure. Denominators exclude
+  households whose cost ratio is not computable (zero/negative income;
+  renters paying no cash rent), consistent with the renter measures.
 * Unregistered ACS tables requested by code are auto-processed: the label
   hierarchy is parsed and percentages are computed against the nearest
   parent subtotal (configurable via the `denominator` parameter).
@@ -79,6 +110,15 @@ package's API and internals.
   installed `tidycensus` geography lookup.
 * Margin-of-error calculations were vectorized (orders-of-magnitude faster
   on tract-level, multi-state pulls).
+* The auto-table classifier now matches its skip keywords on word
+  boundaries, so tables whose concept contains "Means" (e.g., B08301,
+  Means of Transportation to Work) are no longer misclassified as
+  non-count tables ("mean") and receive auto-computed percentages.
+* The auto-table classifier now requires a table's `_001` label to end
+  with `:` (or be exactly `Estimate!!Total`) to compute percentages, so
+  dollar-valued tables like B19080 (Household Income Quintile Upper
+  Limits) no longer produce nonsense percentages of one quintile limit
+  over another.
 * Fixed a typo in `make_pretty_names()` output ("renter-occuiped").
 
 ## Infrastructure

R/auto_percent.R

@@ -112,24 +112,30 @@ classify_acs_table = function(nodes) {
   concept_lower = tolower(concept)
   label_lower = tolower(total_label)
 
-  ## patterns that indicate non-percentage-amenable tables
+  ## patterns that indicate non-percentage-amenable tables; matched on word
+  ## boundaries so that, e.g., "mean" does not match "Means of Transportation"
   skip_patterns = c(
     "median", "aggregate", "average", "mean",
     "allocation of", "imputation of",
     "margin of error")
 
   has_skip_pattern = purrr::some(skip_patterns, function(pattern) {
-    stringr::str_detect(concept_lower, stringr::fixed(pattern)) ||
-      stringr::str_detect(label_lower, stringr::fixed(pattern))
+    pattern_regex = paste0("\\b", pattern, "\\b")
+    stringr::str_detect(concept_lower, pattern_regex) ||
+      stringr::str_detect(label_lower, pattern_regex)
   })
   if (has_skip_pattern) return("skip")
 
   ## singleton tables (only one variable) — no meaningful percentages
   if (nrow(nodes) <= 1) return("skip")
 
   ## tables where the total is not a count (e.g., median income tables may have
-  ## a numeric label rather than "Estimate!!Total:")
-  if (!stringr::str_detect(total_label, ":") &&
+  ## a numeric label rather than "Estimate!!Total:"). The _001 label of a count
+  ## table ends with ":" (or is exactly "Estimate!!Total" in older vintages); a
+  ## mid-label colon is not sufficient (e.g., B19080's _001 is
+  ## "Estimate!!Quintile Upper Limits:!!Lowest Quintile Upper Limit" — a dollar
+  ## value, not a total).
+  if (!stringr::str_detect(total_label, ":$") &&
       !stringr::str_detect(total_label, "^Estimate!!Total$")) {
     return("skip")
   }

R/cache.R

@@ -0,0 +1,61 @@
+## Internal: directory where cached raw ACS query results are stored.
+## Overridable via options(urbnindicators.cache_dir = ...) (used by tests).
+acs_cache_dir = function() {
+  getOption("urbnindicators.cache_dir", tools::R_user_dir("urbnindicators", which = "cache"))
+}
+
+## Internal: thin wrapper around tidycensus::get_acs(). Exists as a seam so
+## tests can mock the network via testthat::local_mocked_bindings().
+acs_query = function(args) {
+  do.call(tidycensus::get_acs, args)
+}
+
+## Internal: fetch one table's raw estimates for a single query, reading from /
+## writing to the on-disk cache when `cache = TRUE`. `args` holds the
+## tidycensus::get_acs() arguments for this chunk, including the chunk's named
+## `variables` vector, so a change to a table's variable definitions produces
+## a new cache key. `cache_stats` is an environment with `hits` and `total`
+## counters, reported to the user once per compile_acs_data() call.
+cached_get_acs = function(args, table_name, cache, cache_stats) {
+  if (!cache) return(acs_query(args))
+
+  cache_stats$total = cache_stats$total + 1
+  key = rlang::hash(list(table_name = table_name, args = args))
+  path = file.path(acs_cache_dir(), paste0("acs_", key, ".rds"))
+
+  if (file.exists(path)) {
+    ## a corrupt/unreadable file falls through to a refetch that overwrites it
+    cached = tryCatch(readRDS(path), error = function(e) NULL)
+    if (!is.null(cached)) {
+      cache_stats$hits = cache_stats$hits + 1
+      return(cached)
+    }
+  }
+
+  result = acs_query(args)
+  dir.create(acs_cache_dir(), recursive = TRUE, showWarnings = FALSE)
+  saveRDS(result, path)
+  result
+}
+
+#' @title Clear the urbnindicators cache
+#' @description Deletes all raw ACS query results cached by
+#'    \code{compile_acs_data(cache = TRUE)}.
+#' @details Cached files live in
+#'    \code{tools::R_user_dir("urbnindicators", which = "cache")}, or in the
+#'    directory named by \code{options(urbnindicators.cache_dir = ...)} when
+#'    set. Because ACS estimates do not change, cached
+#'    entries never go stale; clearing the cache is only useful to reclaim
+#'    disk space.
+#' @returns The number of cache files removed, invisibly.
+#' @examples
+#' \dontrun{
+#' clear_acs_cache()
+#' }
+#' @export
+clear_acs_cache = function() {
+  cache_files = list.files(acs_cache_dir(), pattern = "^acs_.*\\.rds$", full.names = TRUE)
+  removed = sum(file.remove(cache_files))
+  cli::cli_inform("Removed {removed} cached ACS quer{?y/ies} from {.path {acs_cache_dir()}}.")
+  invisible(removed)
+}