Missing vars in 2010 census tracts pessoa02

[rafapereirabr]

The "pessoa02" table of 2010 census tracts should have more variables, up to "pessoa02_V255". However, it stops at "pessoa02_V170". See page 87 of the data dictionary

# data dictionary
censobr::data_dictionary(year = 2010, dataset = "tracts")

# try to get the data 
df <- censobr::read_tracts(
  year = 2010, 
  dataset = "Pessoa"
  ) |>
  dplyr::select(
    code_tract, code_muni,
    pop_litmen   = pessoa02_V086,
    pop_litwomen = pessoa02_V171
  )

[antrologos]

Hi @rafapereirabr — we dug into this and the cause turned out to be unusual. It's not truncation — IBGE shipped the SP files with shifted column names.

In 26 of the UFs, Pessoa02_<UF>.xls has 170 columns: V001 (Men 5+), V002 (Men aged 5), ..., V086 (Women 5+), ..., V170 (women live-in companions). Matches the official dictionary.

In SP1 and SP2, Pessoa02_SP*.xls has the SAME 170 columns, SAME values, SAME order. But the names are shifted by +85: what should be V001 shows up as V086; what should be V086 shows up as V171; what should be V170 shows up as V255.

That's why the parquet looked like it had "extra variables that truncated at V170": V001-V085 were empty in SP rows (SP never published columns with those names), and V171-V255 were empty in the other 26 UFs' rows (those UFs never published columns with those names). But the VARIABLES are the same — only the names IBGE attached to SP are wrong.

How we proved it: the identity Pessoa01_V_i = Pessoa02_men_i + Pessoa02_women_i (Total = Men + Women) must hold in any UF. In SP, testing this using V_(i+85) and V_(i+170) as Men/Women, 84 of 85 age cuts match with ZERO absolute difference. The only exception is V001 (overall 5+ count) with ~0.04% diff across all UFs — IBGE's "X" censorship at the aggregate level, not SP-specific.

Fix (commit 29fee76 on censobr_prep_data main): when reading Pessoa02_SP1.xls and Pessoa02_SP2.xls, the pipeline subtracts 85 from every V_n name with n >= 86. Result:

• v0.5.0: PESSOA parquet had 2094 columns, data scrambled across names.
• v0.6.0: PESSOA parquet has 2008 columns, canonical schema in every UF.

Full crosswalk (170 rows) and empirical analysis in references/relatorio_pessoa02_sp_shift.md.

Heads up about your snippet: you used pessoa02_V086 = pop_litmen and pessoa02_V171 = pop_litwomen. That worked on v0.5.0 because the parquet preserved SP's shifted names. In v0.6.0 it becomes inconsistent because names go back to canonical. Updated:

df <- censobr::read_tracts(year = 2010, dataset = "Pessoa") |>
  dplyr::select(
    code_tract, code_muni,
    pop_litmen   = pessoa02_V001,  # Men 5+ literate
    pop_litwomen = pessoa02_V086   # Women 5+ literate
  )

pessoa02_V171–pessoa02_V255 no longer exist in v0.6.0.

Note about IBGE: they fixed a similar issue in GO on 2025-09-15 (V01-V99 missing zeros, logged in Atualizacoes_20250915.txt on the FTP). They didn't catch the SP shift. We'll email them.

Keeping this issue open until censobr v0.6.0 ships, then closing with release notes link.