[jknowles]

Gitea issue #77 (R side). Step 4, cog-api, is out of scope, so the commits say Refs #77 rather than Closes #77.

Summary

year is the Census survey year, which covers different calendar months for different governments. This PR does three things:

• Measure first. It measures how usable long.fiscal_year_end is (step 1, results below). That came before anything was built on it; the draft PR carried those results alone.
• Per-row periods, opt-in. It returns per-row fiscal periods from the verbs behind a new fiscal_period = FALSE argument (step 2).
• Two helpers. It adds cog_fiscal_periods() and cog_survey_year() (step 3), plus docs (step 5).

Step 1: how usable is fiscal_year_end?

What I measured: the bundled fixture's long partitions, FY2011, FY2012, FY2019 and FY2020, read directly with DuckDB. data-raw/regenerate_fixture_corpus.R copies each partition byte-for-byte from the publish tree (build 2026-08-03, schema 6), so these are full published years for types 0–3. I could not reach any other year. The corpus parquet files redirect to us.aws.cdn.hf.co, which the sandbox's network policy blocks, and the public API does not expose fiscal_year_end. Nothing below describes FY1967–FY2010, FY2013–FY2018 or FY2021+.

Reproduce with Rscript data-raw/measure_fiscal_year_end.R (commit 533b80d).

Survey year	Type	Gov-years	Missing	Zero	Implausible MMDD	Conflicting within the year
2011	0 state	50	0	0	0	0
2011	1 county	1,781	2	0	0	0
2011	2 municipal	3,533	54	0	0	0
2011	3 township	1,058	2	0	0	0
2012	0	50	0	0	0	0
2012	1	3,031	0	0	0	0
2012	2	19,515	46	0	0	0
2012	3	16,352	2	0	0	0
2019	0	50	0	0	0	0
2019	1	1,706	0	0	0	0
2019	2	3,875	18	0	0	0
2019	3	750	0	0	0	0
2020	0	50	0	0	0	0
2020	1	1,706	0	0	0	0
2020	2	3,879	21	0	0	0
2020	3	752	2	0	0	0

• Missing: 0.12% to 0.9% of government-years per survey year, nearly all type 2. No government-year has the value on only some of its rows.
• Most common values: 1231, 630 and 930 cover 84% to 95% of government-years per year. Then 430, 331, 228 and 531.
• Changes between consecutive years:
  • FY2011 → FY2012: 221 of 6,314 counties, municipalities and townships change value, mostly 1231↔630. That is also where the source switches from the legacy to the modern files, and I can't tell real changes from artifacts.
  • FY2019 → FY2020: none change.
• State governments: 46 states are 630 in all four years. Alabama and Michigan are 930, Texas 831, New York 331.
• District of Columbia: type 2, 930 in all four years.

Conclusion: coverage in these four years supports steps 2 and 3. Government-years with no usable value get NA periods. No other year is assumed to look the same.

Open questions

Does the same rule hold for state governments (type 0)? No, for the three states whose fiscal year ends after June 30. For them, survey year Y is the fiscal year ending in calendar year Y.

• Census documentation.

  • The 2006 technical documentation, extracted from the PDF: "Data in this report pertain to government fiscal years that ended between July 1, 2005, and June 30, 2006. The following governments are exceptions, and are included as though they were part of the June 30 group: ending September 30, 2006, the state governments of Alabama and Michigan, the District of Columbia and Alabama school districts; and ending August 31, 2006, the state government of Texas and Texas and Nebraska school districts."
  • The 2004 edition says the same, per web-search extracts.
  • The 2022 state technical documentation names the same four states but not which calendar year their September or August end falls in.

• Data test, survey year 2020 (public API, build 2026-10-26). CARES Act relief reached governments in April 2020. New York (March 31) is a control: its survey-year 2020 is the fiscal year ending March 31, 2020 under either reading, before most of the aid arrived.

  IG Federal revenue, state govt	2018→19	2019→20	2020→21
  Alabama (930)	+6.7%	+20.9%	+15.8%
  Michigan (930)	+6.5%	+17.6%	+9.4%
  Texas (831)	+6.3%	+36.1%	+37.3%
  New York (331)	+4.5%	+5.7%	+17.3%
  June-30 states, median (46)	+2.4%	+25.6%	+15.4%

  Under the plain rule, Alabama's and Michigan's survey-year 2020 would end September 30, 2019 and Texas's August 31, 2019. Neither period contains April 2020, so their jump would land in 2021, as New York's does. It lands in 2020 instead, with the June-30 states.

Do the older exceptions still apply, and do any reach types 0–3?

• Alabama, Michigan and Texas state governments (type 0): yes, at least in survey year 2020.
• District of Columbia (type 2 here): unresolved. The 2004 and 2006 documentation lists it. The 2020 test is inconclusive (+49.7%, +16.9%, +21.8%). It is implemented with the state rule on the strength of the documentation, and flagged as unconfirmed in the docs.
• Alabama, Texas and Nebraska school districts: type 5, not in the published corpus.

Ordinary local governments with a September 30 year end follow the plain rule. Across 15 large 930 cities and counties (San Antonio, Dallas, Austin, Fort Worth, Jacksonville; Miami-Dade, Broward, Palm Beach, Hillsborough, Orange FL, Pinellas, Bexar, Tarrant, Dallas County, Travis), median IG Federal growth is +6.2% in survey-year 2020 and +39.1% in 2021. Six of the 15 exceed +50% in 2021, against two in 2020.

What changed

Steps 2 and 3, R/fiscal.R (new):

• .survey_period() implements the rule:
  • Local governments: the fiscal year ending July 1 of Y-1 through June 30 of Y.
  • State governments and D.C.: the fiscal year ending in calendar year Y.
  • February 29 year ends are clamped in common years. Missing, zero or impossible values give NA.
• .fiscal_year_lookup() reads fiscal_year_end per government-year from long. A government-year whose rows disagree gets NA, which never happens in the fixture.
• cog_spending(), cog_revenue(), cog_balances() each gain a trailing fiscal_period = FALSE. With TRUE, every row gains fiscal_year_end, period_start and period_end, plus provenance$transformations$fiscal_period. For holdings, period_end is the balance date.
• cog_fiscal_periods(govid, years): the govid × years grid of fiscal_year_end, period_start and period_end, NA where unknown.
• cog_survey_year(govid, date): the survey year whose period contains each date. It tries Y, Y+1 and Y+2 with each candidate year's own fiscal_year_end, returns NA rather than borrowing another year's value, and takes the earlier year on overlap.
• NAMESPACE, man/, and _pkgdown.yml (new "Fiscal years" reference section).

Step 5, docs:

• "The survey-year rule" section on ?cog_fiscal_periods, inherited by ?cog_survey_year, with a worked example for survey year 2017 and the state and D.C. exception.
• total-spending.Rmd (evaluated chunk): the Florida rollup mixes a June-30 state (July 2019–June 2020) with two September-30 counties (October 2018–September 2019) under the same 2020.
• coverage-counters.Rmd (evaluated chunk): FY2012 year ends of the 566 Georgia cities: 273 December 31, 196 June 30, 36 September 30, small groups, and 31 with no value.
• population-denominators.Rmd: a note on survey years. The file is eval = FALSE.
• NEWS.md: a new # uscogdata (development version) entry.

Shape changes

• Default calls: none. With fiscal_period unset, all three verbs return the same columns and provenance as before. A test checks the names and the absence of the provenance block.
• Signatures: cog_spending(), cog_revenue() and cog_balances() each gain one trailing argument, fiscal_period = FALSE. Positional calls are unaffected, and cog-api can feature-detect it with formals(), as with limit.
• With fiscal_period = TRUE:
  • Three columns are appended after every existing column: fiscal_year_end (integer), period_start (Date), period_end (Date).
  • provenance$transformations$fiscal_period is added. provenance-v1.json allows additional properties.
• Not changed: cog_geographic_rollup() and cog_peer_compare() do not take fiscal_period.
• New exports: cog_fiscal_periods() and cog_survey_year().
• Merge note: PR feat(api): export cog_suggestions() (#75) #1 also adds a # uscogdata (development version) heading to NEWS.md. Whichever lands second has a one-line conflict at the top of the file.

Tests

• tests/testthat/test-fiscal-period.R: 85 expectations, all against the bundled fixture or synthetic data:
  • the rule for local and state governments, D.C., February and invalid values
  • conflicting values within a government-year, on a synthetic long table
  • default-unchanged verbs, plus each verb with TRUE
  • composition with per_capita, adjust_to_year, limit and complete
  • the grid, the date mapping and its boundaries
  • agreement between the helpers and the verb columns
  • input validation
• Full suite: 1,193 expectations, 0 failed, 0 warnings, 2 skipped (opt-in live-corpus tests). main gives 1,108.
• R CMD build --no-manual + R CMD check --no-manual: all three vignettes build and run, and tests pass. Status is 1 WARNING and 2 NOTEs, the same as on main:
  • WARNING: the sandbox has no en_US.UTF-8 locale.
  • NOTE: .roborev.toml is a hidden file.
  • NOTE: installed size.
• Local deviation, as on earlier PRs: the sandbox cannot reach extensions.duckdb.org, so local runs drop cog_open()'s INSTALL httpfs; LOAD httpfs; line in a scratch copy. That is not in any commit.
• roxygen2 7.3.3 matches RoxygenNote, so DESCRIPTION is untouched.

Verified

• The step 1 table and statistics, on the fixture's four years.
• The state exception in survey year 2020 for Alabama, Michigan and Texas, and the plain rule for 15 large September-30 local governments, both through the API (build 2026-10-26).
• The 2006 and 2022 Census documentation wording, from the PDFs.
• Every period in the tests and in the vignette prose against fixture output. For example:
  • Broward County 2020 covers 2018-10-01 to 2019-09-30.
  • Alabama state 2020 covers 2019-10-01 to 2020-09-30.
  • Texas state 2019 covers 2018-09-01 to 2019-08-31.
  • New York state 2020 covers 2019-04-01 to 2020-03-31.

Not verified

• Coverage outside the four fixture years. The early years are the gap the issue worries about, and I could not measure them.
• The state exception in any survey year other than 2020.
• D.C.'s rule, which rests on the 2004 and 2006 documentation only.
• Whether the FY2011→FY2012 fiscal-year-end changes are real.
• Carried-forward rows from FY2023 on. The source marks them with data_year < year, a schema 7 column that the fixture lacks. Their periods describe the survey year, not the year the data pertain to. The docs say so, but nothing adjusts for it.
• The 2004 Census edition, cited from search extracts only.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qbe6u2h83wgZdTgJgE4Xh5

[github-actions]

Thanks for this — and one thing worth knowing before it happens.

This repository is a mirror. Development happens on Gitea at
gitea.civilytics.org/Civilytics/uscogdata. Your pull request will be fetched
from here, landed there, and synced back.

Because that merge preserves your commits at their original SHAs, GitHub will
mark this pull request "Merged" on its own — without anyone visibly clicking
the Merge button, and possibly without a review comment on this page first.

If your pull request closes as "Merged" and nobody appears to have merged it,
that is the normal, successful outcome — not a rejection.

If it is not going to be merged, you will get an actual reply saying so.

Substantial contributions get a ctb entry in DESCRIPTION, which surfaces in
citation("uscogdata"). There is no CLA and no DCO sign-off.

Full details: CONTRIBUTING.md.