Support quanteda 4.5.0 (fix token type-table handling)

[kbenoit]

Why

quanteda 4.5.0 is currently held back on CRAN because it breaks its reverse dependency sentopics. This PR fixes sentopics so that quanteda 4.5.0 can be released.

The root cause is a behaviour change in quanteda 4.5.0: subsetting a tokens object no longer compacts its type table. A subset such as ECB_press_conferences_tokens[1:10] now retains all types inherited from the parent object (e.g. 1168) instead of only the types that actually appear in the subset (e.g. 53).

sentopics builds a model's vocabulary directly from attr(toks, "types"), so these phantom types inflated every model's vocabulary. This surfaced as:

• chains_scores() / coherence() errors — computeFcm() indexed the full type table into a dfm that only contains the appearing features, giving Error: Subscript out of bounds.
• Bloated vocabularies in models built from subset tokens, with corresponding test failures (types: Lengths (53, 1168) differ).

What this PR does

1. Recompile the token type table at model creation. A new internal helper recompileTypes() drops types that no longer appear, applied at the single point where user-supplied tokens/dfm enter a model (sentopicsmodel(), the shared backend for LDA(), JST(), rJST()). This restores the pre-4.5.0 vocabulary.

   • It is deliberately not applied in makeVocabulary(): the as.LDA / as.rJST / ... conversion helpers build tokens from an external vocabulary whose zero-count entries must be preserved to keep the topic-word matrices aligned.
   • The helper is guarded to remain backward compatible with quanteda (>= 3.2.0), where subsetting already compacts the type table.

2. Fix an unqualified docvars<- call in sentopics_sentiment<-. Only the docvars getter is imported from quanteda, so clearing stale per-topic sentiment docvars raised could not find function 'docvars<-'. Now qualified as quanteda::docvars(...) <- NULL.

3. Update a stale test fixture (as.tokens.dfm) that relied on the pre-4.5.0 behaviour of a subset tokens object.

Verification

R CMD check passes on both examples and tests against quanteda 4.5.0:

• checking examples ... OK
• checking tests ... OK (351 passing, 0 failures)

No minimum-version bump is required; the fix works across quanteda 3.2.0 – 4.5.0+.

[odelmarcelle]

Hi @kbenoit, thank you for the pull request. I'm not merging it yet as i'm not sure this is the good fix. This will solve the issue for new models but the same errors with computeFcm() could still surface from converted models. So i'd like to investigate a bit further and either:

• fix computeFcm and accept that a user passing a tokens object with inflated types remains that way.
• systematically recompile types and accept that alignment from converted models may fail

But digging further I encountered a compilation error trying to install quanteda 4.5.0. I'm on a Mac with M4 processor (ARM).

clang++ -std=gnu++20 -I"/opt/homebrew/Cellar/r/4.6.1/lib/R/include" -DNDEBUG  -I'/opt/homebrew/lib/R/4.6/site-library/Rcpp/include' -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/xz/include -I/opt/homebrew/include   -I../inst/include  -fPIC  -g -O2  -Wall -pedantic -fdiagnostics-color=always  -c tokens_segment.cpp -o tokens_segment.o
tokens_segment.cpp:125:33: error: use of undeclared identifier 'texts'
  125 |     for (std::size_t h = 0; h < texts.size(); h++) {
      |                                 ^~~~~
1 error generated.
make: *** [tokens_segment.o] Error 1
ERROR: compilation failed for package ‘quanteda’

Could it be that the identifier texts at https://github.com/quanteda/quanteda/blob/25e88e983753ad3e17573c0d722616114cc80e44/src/tokens_segment.cpp#L127-L131 should have been modified as well in the commit 25e88e983753ad3e17573c0d722616114cc80e44 ?

[kbenoit]

Hi @odelmarcelle thanks for pointing that out. I also work on an M4 Mac, but with TBB installed I did not encounter the problem. I uninstalled TBB and confirmed that it generates that error. This is fixed in quanteda/quanteda#2511, pending approval by @koheiw.

On the backward compatibility, @koheiw is this something that can be disabled by sentopics to ensure backward compatibility, or should it recompile older objects when loading them. What's your recommendation for a fix?

[odelmarcelle]

After digging into it, this looks like the most desirable solution. Because makeVocabulary() uses quanteda::tokens_*() functions to process the input tokens when instanciating a JST or rJST model, the tokens types are systematically recompiled. This creates alignment issues between LDA and JST (see test "from LDA works").

I'll merge soon into master and prepare a CRAN release for version 0.7.7.

[odelmarcelle]

@kbenoit version 0.7.7 is on CRAN