Skip to content

feat(ecosystems): add TOML reader plus 9 ecosystems and 20+ new sources - #2

Merged
Niicolaa merged 1 commit into
claude/windows-support-planning-tb6aefrom
claude/ecosystem-sources-tb6ae
Aug 4, 2026
Merged

Niicolaa merged 1 commit into
claude/windows-support-planning-tb6aefrom
claude/ecosystem-sources-tb6ae

Conversation

@Niicolaa

@Niicolaa Niicolaa commented Aug 4, 2026

Copy link
Copy Markdown
Owner

Closes most of the gap between what Bumblebee collects and what a developer endpoint actually has on disk, including every ecosystem docs/inventory-sources.md listed under "Not currently covered".

This is PR 2 of 2 and is stacked on #1 — its base is claude/windows-support-planning-tb6ae, not main. Merge #1 first; GitHub will then retarget this to main automatically. The diff shown here is only this PR's own changes.

The blocker was TOML

The stdlib has no TOML support and this project ships zero non-stdlib dependencies. That single gap was what blocked Cargo, Poetry, uv, PEP 751, Julia, and the Codex MCP host config — which is why config.toml was listed as an explicit v0.1 non-goal despite ~/.codex already being a scanned root.

internal/toml is a ~590-line restricted reader covering what lockfiles actually use: bare/quoted/dotted keys, [table], [[array of tables]], basic and literal strings (including multi-line), integers, floats, booleans, arrays, and inline tables. Dates are returned as raw text rather than typed — no consumer here needs temporal comparison, and skipping it keeps the reader small. Everything goes through a small facade (Parse, Table, Tables, String, Strings, Get), so swapping in a library later is a one-file change rather than a refactor of six parsers.

New ecosystems

Emitted values follow the lowercased OSV identifiers, so an exposure catalog derived from an advisory feed needs no translation table.

Ecosystem Sources
crates.io Cargo.lock
nuget packages.lock.json, packages.config, *.deps.json, Directory.Packages.props / Packages.props, installed <id>/<version>/<id>.nuspec
maven pom.xml, *gradle.lockfile, *.sbt.lock, META-INF/maven/*/pom.properties inside .jar/.war/.ear/.par
swift Package.resolved (v1 and v2/v3)
cocoapods Podfile.lock
pub pubspec.lock
hex mix.lock (:hex entries only)
conan conan.lock (v1 graph and v2 flat)
julia Manifest.toml (format 1.0 and 2.0)

Extended existing ecosystems

  • pypirequirements*.txt, Pipfile.lock, poetry.lock, uv.lock, pylock.toml. This closes a real asymmetry: npm had declared-state coverage (lockfiles) while Python had installed-state only (dist-info), even though PyPI is a constant target for typosquat campaigns and the repo already ships threat_intel/auto-osv-malicious-pypi.json.
  • gogo.work.sum, vendor/modules.txt, and build info read from compiled binaries via stdlib debug/buildinfo. go install writes tools into ~/go/bin and nothing else on disk records what went into them.
  • mcp~/.codex/config.toml. env values and key names are dropped, same as every other MCP source.

Confidence reflects what each file proves

  • high — resolved locks, installed metadata, embedded JAR coordinates.
  • medium — declarations: pom.xml, Packages.props, a requirements.txt == pin. Exact versions, but a request rather than a record of an install.
  • low — a requirements entry with any other specifier: no single version, so the raw spec goes in requested_spec and version is left empty, matching how MCP server references are handled.

Unresolvable inputs are skipped rather than guessed: MSBuild $(Property) references, Maven ${...} that doesn't resolve against the pom's own <properties>, version ranges, wildcard pins, (devel) Go builds, and .nuspec $token$ versions.

Notes on scope

Baseline roots gain only ~/.nuget/packages and ~/.julia/environments — the two that actually hold metadata a scanner can read. ~/.m2/repository and ~/.gradle/caches are deliberately not baseline roots: both are already in walk.DefaultExcludes as high-cost dependency caches, and opening every cached JAR is exactly what a profile meant to run 6-hourly should avoid. The Pub/Hex/Conan/Swift caches hold extracted sources or tarballs rather than any file this scanner parses. All remain reachable through an explicit --root or a deep sweep.

Java archive scanning reads identity only from the embedded META-INF/maven/*/pom.properties. A shaded uber-jar without one yields nothing rather than a filename-derived guess.

Not adopted from Trivy: OS packages, container images, IaC misconfiguration, and secret scanning. The last would directly contradict the stated privacy posture — MCP env values and key names are both dropped, and .env/.envrc are skipped even outside excluded directories.

Dispatch wiring

The walker's dispatch switch and the worker's handler switch were linked only by matching string literals, so a typo on either side compiled cleanly and silently dropped an entire ecosystem. All 43 job kinds are now constants referenced by both, making a mismatch a compile error.

That matters because tests can't cover everything here: go-binary cannot be covered by a record-level test in this repo. This module has zero non-stdlib dependencies, so a binary built from it reports len(Deps)==0, and its own Main.Version is (devel), which ScanGoBinary deliberately skips. Verified directly:

Main.Path="github.com/perplexityai/bumblebee" Main.Version="(devel)" len(Deps)=0

dispatch_test.go still walks a tree with one fixture per new source and asserts each produces a record, including the three gated on file shape rather than basename (cached .nuspec, Java archive, Go binary).

Also fixed

Editor-extension roots were missing .vscode-insiders-server and .vscodium-server while their local counterparts were present, so remote/SSH installs of those two editors were invisible.

Verification

  • go test -race ./... — green across all 30 packages
  • gofmt -l . and go vet ./... — clean
  • GOOS=windows amd64 and arm64 — build and vet clean
  • bumblebee selftest — OK
  • --ecosystem now accepts: npm,pypi,go,rubygems,packagist,crates.io,nuget,maven,swift,cocoapods,pub,hex,conan,julia,mcp,editor-extension,browser-extension

🤖 Generated with Claude Code

https://claude.ai/code/session_01H9erz849YcDcunP7sWteMu


Generated by Claude Code

@Niicolaa
Niicolaa force-pushed the claude/ecosystem-sources-tb6ae branch 2 times, most recently from 5bcd8cb to 9b2b58b Compare August 4, 2026 14:24
@Niicolaa
Niicolaa force-pushed the claude/windows-support-planning-tb6ae branch from 3f055c4 to e62c2c6 Compare August 4, 2026 14:31
Adds a dependency-free restricted TOML reader (internal/toml) and uses
it to unblock the ecosystem coverage previously listed in
docs/inventory-sources.md as "not currently covered". The stdlib has no
TOML support and this project ships zero non-stdlib dependencies, so
that single gap was what blocked Cargo, Poetry, uv, PEP 751, Julia, and
the Codex MCP host config.

New ecosystems. Emitted values follow the lowercased OSV identifiers so
an advisory-derived exposure catalog needs no translation table:

  crates.io  Cargo.lock
  nuget      packages.lock.json, packages.config, *.deps.json,
             Directory.Packages.props / Packages.props, and installed
             <id>/<version>/<id>.nuspec in the global packages folder
  maven      pom.xml, *gradle.lockfile, *.sbt.lock, and
             META-INF/maven/*/pom.properties inside jar/war/ear/par
  swift      Package.resolved (v1 and v2/v3)
  cocoapods  Podfile.lock
  pub        pubspec.lock
  hex        mix.lock (:hex entries only)
  conan      conan.lock (v1 graph and v2 flat)
  julia      Manifest.toml (format 1.0 and 2.0)

Extended existing ecosystems:

  pypi  requirements*.txt, Pipfile.lock, poetry.lock, uv.lock,
        pylock.toml — closes the asymmetry where npm had declared-state
        coverage and Python had installed-state only
  go    go.work.sum, vendor/modules.txt, and build info read from
        compiled binaries via debug/buildinfo (stdlib), the only record
        of what went into a `go install`-ed tool
  mcp   ~/.codex/config.toml

Confidence reflects what each file proves: resolved locks are high,
declarations (pom.xml, Packages.props, pinned requirements) are medium,
and unpinned requirement specifiers record the spec with no version at
low confidence, matching how MCP server references are handled.

Baseline roots gain only ~/.nuget/packages and ~/.julia/environments —
locations that actually hold metadata a scanner can read. Deliberately
excluded are ~/.m2/repository and ~/.gradle/caches, which are already
in walk.DefaultExcludes as high-cost dependency caches; opening every
cached JAR is far too expensive for a profile meant to run 6-hourly.
Both stay reachable via an explicit --root or a deep sweep.

Also fixes an editor-extension gap found while mapping coverage: the
roots were missing .vscode-insiders-server and .vscodium-server while
their local counterparts were present, so remote/SSH installs of those
two editors were invisible.

The walker's dispatch switch and the worker's handler switch were
linked only by matching string literals, so a typo on either side
compiled cleanly and silently dropped an ecosystem. All 43 job kinds
are now constants referenced by both, making a mismatch a compile
error. dispatch_test.go additionally walks a tree with one fixture per
new source and asserts each produces a record.
@Niicolaa
Niicolaa force-pushed the claude/ecosystem-sources-tb6ae branch from 9b2b58b to 4edb711 Compare August 4, 2026 14:34
@Niicolaa
Niicolaa merged commit c30ac87 into claude/windows-support-planning-tb6ae Aug 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants