-
Notifications
You must be signed in to change notification settings - Fork 12
Description
Problem
ddprof bundles blazesym 0.2.0-rc.5 (via libdatadog v26), which does not support finding split debug files via build-id paths (/usr/lib/debug/.build-id/XX/rest.debug). It only supports GNU debuglink-style paths (/usr/lib/debug/<binary-path>/<debuglink-name>).
On modern Debian/Ubuntu systems, debug symbols are distributed as automatic -dbgsym packages which exclusively use build-id paths. This is the standard mechanism since debhelper compat level 9+ (see Debian AutomaticDebugPackages and Ubuntu Debug Symbol Packages).
dh_strip creates the -dbgsym packages with debug files at:
/usr/lib/debug/.build-id/XX/YYYYYY.debug
It also sets .gnu_debuglink in the stripped binary, but the debuglink filename is the build-id hash (e.g. 2436a298407b13414d190e466b5dcd5df8f462.debug), stored only at the build-id path. There is no file at the debuglink search paths (/usr/lib/debug/usr/bin/<name>), so blazesym's debuglink search silently fails.
The result is that ddprof flame graphs show the binary name (e.g. "IndexBuilderServer") instead of function names for stripped binaries, even with DD_PROFILING_INLINED_FUNCTIONS=true. Dynamic library symbols (libc, grpc, etc.) work because they have .dynsym entries — but the main binary's own functions are only in .symtab inside the debug file that blazesym cannot find.
Fix
blazesym v0.2.3 (Jan 29, 2025) added build-id debug file lookup in libbpf/blazesym#1415 (commit 8db66bc). From the v0.2.3 changelog:
Added
- Added support for
/usr/lib/debug/.build-iddebug link target directory
Updating the bundled blazesym (via libdatadog) to >= 0.2.3 would fix this issue.
Workaround
Until the update, users can create symlinks from the debuglink search path to the build-id files, e.g. by overriding dh_strip in debian/rules to add symlinks like:
/usr/lib/debug/usr/bin/<debuglink-name> → ../../.build-id/XX/rest.debug
References
- blazesym v0.2.3 CHANGELOG — build-id lookup added
- libbpf/blazesym#1415 — the PR implementing build-id lookup
- Debian AutomaticDebugPackages — how
-dbgsympackages work - Ubuntu Debug Symbol Packages — Ubuntu's debug symbol distribution
- Distribution of debug information (MaskRay) — detailed analysis of build-id vs debuglink
- GDB Separate Debug Files — GDB's debug file search algorithm