refactor!: tighten lexer API surface and relocate WordSpan to ast

[mpecan]

Summary

Final PR of the v0.2.0 refactoring cycle (#61). Tightens the library's API surface so visibility declarations match actual reachability, relocates WordSpan from lexer/word_builder.rs to src/ast.rs, and removes the dead PendingHereDoc.quoted field.

Visibility tightening

The lexer module is already pub(crate) mod lexer at lib.rs, so nothing inside was externally reachable. This PR makes the markers match:

• PendingHereDoc → pub(crate) with pub(crate) fields; dropped unused quoted field and the #[allow(dead_code)] attribute.
• WordBuilder, QuotingContext, WordSpanKind → pub(crate).
• Lexer, LexerContext, LexerMode, LexerCheckpoint → pub(crate) struct + pub(crate) fields; all impl Lexer methods → pub(crate).
• heredoc / word_builder submodules → pub(crate) / pub(super).
• Added #![allow(clippy::redundant_pub_crate)] to src/lexer/mod.rs and src/lexer/word_builder.rs — pedantic clippy warns about pub(crate) inside an already-pub(crate) module, but the markers are intentional for documenting the visibility boundary.

WordSpan relocation

WordSpan now lives in src/ast.rs (alongside Span, Node, NodeKind). Rationale: Rust's private_in_public check requires WordSpan to have pub visibility since it's a field type of the public NodeKind::Word { spans: Vec<WordSpan> } variant. The struct is pub struct with pub(crate) fields — external consumers see the type as opaque (can pattern-match and hold references but cannot read fields or construct instances). WordSpanKind and QuotingContext stay in lexer/word_builder.rs as pub(crate) (they're only used in the pub(crate) fields of WordSpan).

A pub(crate) use crate::ast::WordSpan re-export in word_builder.rs preserves existing crate::lexer::word_builder::WordSpan imports across the crate.

Semver

This PR uses a refactor!: commit prefix and a BREAKING CHANGE: trailer. Per .release-please-config.json with bump-minor-pre-major: true, release-please will emit a release PR bumping Cargo.toml and pyproject.toml from 0.1.15 → 0.2.0 when this lands on main. No manual version edit in this PR.

External observable change: the canonical path for WordSpan is now rable::ast::WordSpan. The lexer module was already crate-private at the public boundary, so no consumer had access via any stable path. rippy (uses Node/NodeKind/ListItem/ListOperator only) and tokf (uses ast::* only) are unaffected.

Test plan

• cargo fmt
• cargo clippy --all-targets -- -D warnings — no warnings
• cargo test — 252 passed (197 unit + 54 integration + 1 tree-sitter)
• cargo test --test integration oracle_ — 12 passed
• PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 cargo check --features python — clean
• Downstream ../rippy (rable 0.1.13) — 1332/1332 tests pass with local rable patch
• Downstream ../tokf (rable 0.1.15) — 145/147 tests pass with local rable patch; the 2 failures are pre-existing rustls-native-certs TLS env issues that reproduce against origin/main rable and are unrelated to this change

Stack

Part of the v0.2.0 refactoring cycle (#61). This is PR 10 of 10 — final PR of the cycle. After merge, release-please will emit the 0.2.0 release PR.

Closes #60

🤖 Generated with Claude Code