diff --git a/docs/zh/05-features/04-continuous-profiling/01-auto-profiling.md b/docs/zh/05-features/04-continuous-profiling/01-auto-profiling.md index f66497a8..b6ee1e47 100644 --- a/docs/zh/05-features/04-continuous-profiling/01-auto-profiling.md +++ b/docs/zh/05-features/04-continuous-profiling/01-auto-profiling.md @@ -15,36 +15,41 @@ permalink: /features/continuous-profiling/auto-profiling 支持的 eBPF Profiling 数据类型： -| 类型 | 支持语言/库 | 社区版 | 企业版 | -| --------- | ---------------- | ------ | ------ | -| on-cpu | Java | ✔ | ✔ | -| | C/C++ | ✔ | ✔ | -| | Rust | ✔ | ✔ | -| | Golang | ✔ | ✔ | -| | CUDA | ✔ | ✔ | -| | Lua | | ✔ | -| | Python `***` | | ✔ | -| off-cpu | Java | | ✔ | -| | C/C++ | | ✔ | -| | Rust | | ✔ | -| | Golang | | ✔ | -| | Python `***` | | ✔ | -| | CUDA | | ✔ | -| | Lua `*` | | ✔ | -| on-gpu | CUDA `*` | | ✔ | -| mem-alloc | Java `**` | | ✔ | -| | Rust | | ✔ | -| | Golang `*` | | ✔ | -| | Python `*` `***` | | ✔ | -| mem-inuse | Rust | | ✔ | -| hbm-alloc | CUDA `*` | | ✔ | -| hbm-inuse | CUDA `*` | | ✔ | -| rdma | C/C++ `*` | | ✔ | +| 类型 | 支持语言/库 | 社区版 | 企业版 | +| ---------- | ----------- | ------ | ------ | +| on-cpu | Java | ✔ | ✔ | +| | C/C++ | ✔ | ✔ | +| | Rust | ✔ | ✔ | +| | Golang | ✔ | ✔ | +| | CUDA | ✔ | ✔ | +| | Node.js/V8 | | ✔ | +| | PHP | | ✔ | +| | Lua | | ✔ | +| | Python | | ✔ | +| off-cpu | Java | | ✔ | +| | C/C++ | | ✔ | +| | Rust | | ✔ | +| | Golang | | ✔ | +| | CUDA | | ✔ | +| | Node.js/V8 | | ✔ | +| | PHP | | ✔ | +| | Python | | ✔ | +| on-gpu | CUDA `*` | | ✔ | +| mem-alloc | Java `**` | | ✔ | +| | Rust | | ✔ | +| | Golang `*` | | ✔ | +| | Python | | ✔ | +| mem-inuse | Rust | | ✔ | +| | Python | | ✔ | +| hbm-alloc | CUDA `*` | | ✔ | +| | Python（调用 CUDA） | | ✔ | +| hbm-inuse | CUDA `*` | | ✔ | +| | Python（调用 CUDA） | | ✔ | +| rdma | C/C++ `*` | | ✔ | 说明： - `*`: features in development - `**`: 运行 Java 程序的 JVM 须有符号表，参考[检查方法](#jvm-符号表检查) -- `***`: 当前支持版本为 Python 3.10 - 类型： - on-cpu：函数在 CPU 上消耗的时间 - off-cpu：函数等待 CPU 的时间 @@ -56,7 +61,29 @@ permalink: /features/continuous-profiling/auto-profiling - 语言： - 编译为 ELF 格式可执行文件的语言：Golang、Rust、C/C++ - 使用 JVM 虚拟机的语言：Java - - 解释型语言：Python + - 解释器运行时：Node.js/V8、PHP、Lua、Python + +### 解释器运行时 + +解释器脚本函数栈展开属于企业版能力，当前支持范围如下： + +| 运行时 | 支持版本 | 支持架构 | Profile 类型 | +| ------ | -------- | -------- | ------------ | +| Node.js/V8 | Node.js 16～23（V8 9～12） | x86_64、AArch64 | On-CPU、Off-CPU | +| PHP | 7.4～8.3 | x86_64、AArch64 | On-CPU、Off-CPU | +| Lua | Lua 5.1～5.4、LuaJIT 2.1 | x86_64、AArch64 | On-CPU | +| Python | CPython 3.10～3.13 | x86_64、AArch64 | On-CPU、Off-CPU、mem-alloc、mem-inuse、hbm-alloc、hbm-inuse | + +Python 的 HBM Profile 来自目标进程调用的 CUDA API，而不是 CPython 内存分配器。PyTorch、vLLM 等 Python 应用通过受支持的 CUDA API 分配和释放显存时，可生成 `hbm-alloc` 和 `hbm-inuse`，具体采集范围和启动时序限制请参考[配置方法](./configuration/#cuda-hbm-profiling)。 + +脚本函数栈展开依赖 Agent 加载增强型 Continuous Profiler： + +- 标准 Linux 内核版本须为 5.2 或更高。 +- Kylin V10 SP3 v2207 的 `4.19.90-*.v2207.ky10.*` 内核也支持该能力，包括 x86_64 和 AArch64 架构。 +- 普通 Linux 4.19 内核和 Kylin V10 SP3 v2101 内核不支持脚本函数栈展开。Agent 仍可使用通用 Profiler 采集满足条件的 Native 栈。 +- 如果增强型 Profiler 加载失败，Agent 会自动降级为通用 Profiler；降级后不再提供 DWARF 和解释器脚本函数栈展开。 + +Agent 必须能够识别目标解释器版本，并读取目标进程的可执行文件及已加载的解释器共享库。不支持的运行时版本、识别失败或文件不可访问时，Profile 中仍可能包含 Native/运行时函数，但不会显示脚本函数。Python 内存剖析通过 CPython 的 `PyObject_Malloc/Free/Realloc` 和 `PyMem_RawMalloc/Free/Realloc` 公共符号采集对象分配与释放事件。 通过通用 eBPF On-CPU/Off-CPU Profiling 获取调用栈时，需满足以下两个前提条件： diff --git a/docs/zh/05-features/04-continuous-profiling/02-configuration.md b/docs/zh/05-features/04-continuous-profiling/02-configuration.md index 0c730e6b..287f1450 100644 --- a/docs/zh/05-features/04-continuous-profiling/02-configuration.md +++ b/docs/zh/05-features/04-continuous-profiling/02-configuration.md @@ -18,11 +18,31 @@ inputs: only_in_container: false rewrite_name: $3 enabled_features: [ebpf.profile.on_cpu, proc.gprocess_info] + - match_regex: \bjava( +\S+)* +-(?:cp|classpath) +\S+ +(?P[$_A-Za-z][$_0-9A-Za-z]*(?:\.[$_A-Za-z][$_0-9A-Za-z]*)*) + match_type: cmdline_with_args + only_in_container: false + rewrite_name: ${CLASS_NAME} + enabled_features: [ebpf.profile.on_cpu, proc.gprocess_info] - match_regex: \bpython(\S)*( +-\S+)* +(\S*/)*([^ /]+) match_type: cmdline_with_args only_in_container: false rewrite_name: $4 enabled_features: [ebpf.profile.on_cpu, proc.gprocess_info] + - match_regex: \b(?:lua|luajit)(\S)*( +-\S+)* +(\S*/)*([^ /]+) + match_type: cmdline_with_args + only_in_container: false + rewrite_name: $5 + enabled_features: [ebpf.profile.on_cpu, proc.gprocess_info] + - match_regex: \bphp(\d+)?(-fpm|-cli|-cgi)?( +-\S+)* +(\S*/)*([^ /]+\.php) + match_type: cmdline_with_args + only_in_container: false + rewrite_name: $5 + enabled_features: [ebpf.profile.on_cpu, proc.gprocess_info] + - match_regex: \b(node|nodejs)( +--\S+)* +(\S*/)*([^ /]+\.js) + match_type: cmdline_with_args + only_in_container: false + rewrite_name: $4 + enabled_features: [ebpf.profile.on_cpu, proc.gprocess_info] - match_regex: ^deepflow- only_in_container: false enabled_features: [ebpf.profile.on_cpu, proc.gprocess_info] @@ -33,9 +53,9 @@ inputs: 上述配置的含义如下： - **match_regex**: 进程匹配的正则表达式，匹配规则如下: - - 第一条规则匹配 Java 进程，例如 `java -jar app.jar`，并将进程名重写为 jar 包名 - - 第二条规则匹配 Python 进程，例如 `python app.py`，并将进程名重写为 Python 脚本名 - - 第三条规则匹配以 `deepflow-` 开头的进程 + - 前两条规则分别匹配以 JAR 包和主类启动的 Java 进程，并将进程名重写为 JAR 包名或主类名 + - 后续规则匹配 Python、Lua/LuaJIT、PHP 和 Node.js 进程，并将进程名重写为脚本名 + - `^deepflow-` 规则匹配以 `deepflow-` 开头的进程 - 最后一条规则匹配所有进程 - **match_type**: 匹配类型，可选值: - `cmdline_with_args`: 匹配完整命令行（包含参数） @@ -49,6 +69,8 @@ inputs: - `ebpf.profile.off_cpu`: 开启 Off-CPU 剖析，需要配置 `inputs.ebpf.profile.off_cpu.disabled: false` - `ebpf.profile.memory`: 开启内存剖析，需要配置 `inputs.ebpf.profile.memory.disabled: false` +默认 Process Matcher 已为 Java、Python、Lua/LuaJIT、PHP、Node.js 和 DeepFlow 进程开启 `ebpf.profile.on_cpu`。如需采集 Off-CPU 或 Memory Profile，必须同时在目标进程的 `enabled_features` 中增加相应功能，并开启对应的全局 Profile 类型。Memory Profile 的解释器函数栈当前仅支持 Python。 + 同时可以使用 `inputs.proc.process_blacklist` 来忽略某些进程，其优先级比 `process_matcher` 高。 ```yaml @@ -77,6 +99,28 @@ inputs: - **refresh_defer_duration**: Java 符号表的刷新延迟，避免高频刷新。 - **max_symbol_file_size**: Java 符号表占用的最大空间大小，单位为 GB，避免占用过大的 `/tmp` 空间。 +# 解释器 Profiling + +Node.js/V8、PHP、Lua 和 Python 的脚本函数栈展开默认启用。若主机上不运行某种解释器，可通过以下配置禁用对应的展开能力，以减少内核内存占用： + +```yaml +inputs: + ebpf: + profile: + languages: + python_disabled: false + php_disabled: false + nodejs_disabled: false + lua_disabled: false +``` + +- 四项配置均默认为 `false`，表示启用对应解释器的函数栈展开。 +- 配置项设为 `true` 后，目标进程的通用 Native Profile 仍可继续采集，但不再展开对应语言的脚本函数。 +- 修改这些配置后需重启 Agent 生效。 +- 全部启用时预计占用约 17～20 MB 内核内存。可关闭未使用的语言以降低内存占用。 + +语言开关只控制脚本函数栈展开。目标进程仍须由 `inputs.proc.process_matcher` 选中，并开启所需的全局 Profile 类型。支持版本、架构和内核条件请参考[能力和限制](./auto-profiling/#解释器运行时)。 + # eBPF On-CPU Profiling eBPF On-CPU Profiling 是默认开启的，但需要修改 `inputs.proc.process_matcher` 来指定进程列表。Agent 支持的配置参数如下： @@ -145,6 +189,19 @@ inputs: - **queue_size**：内存剖析组件内部的队列大小。 - 配置该选项可以参考采集器性能统计 `deepflow_agent_ebpf_memory_profiler` 中 `overwritten` 和 `pending` 指标，增大该配置使得前者为 0，后者不高于该配置即可。 +## CUDA/HBM Profiling + +HBM Profiling 不限制目标进程的编程语言。对于 Python/PyTorch/vLLM，必须确保真正执行 GPU 分配的 Worker 子进程也被 `inputs.proc.process_matcher` 匹配，并启用 `ebpf.profile.memory`；仅匹配父进程无法采集子进程的显存事件。`inputs.ebpf.profile.languages.python_disabled` 只控制 Python 脚本函数栈展开，不控制 HBM 事件采集。 + +当前 Agent 通过 uprobe 采集 `cudaMalloc`、`cudaFree`、`cuMemAlloc_v2` 和 `cuMemFree_v2` 调用，并据此生成： + +- `hbm-alloc`：uprobe 生效后观测到的显存分配量及调用栈。 +- `hbm-inuse`：根据已观测到的分配地址及后续释放事件计算的当前显存用量及调用栈。 + +Agent 约每 10 秒扫描一次进程。对于新匹配的非 Agent 进程，当前还会等待约 120 秒再解析已加载的 CUDA 库并挂载 Memory uprobe。uprobe 生效前发生的显存分配不会补采，也不会计入后续的 `hbm-inuse`。挂载完成后，如果进程持续调用上述受支持的 CUDA 分配和释放 API，则可正常生成新的 HBM Profile。 + +PyTorch/vLLM 等框架可能通过 Caching Allocator 预留大块显存，再在池内完成对象分配和释放；未再次调用上述 CUDA API 的池内操作不会产生 HBM 事件。当前也不采集 `cudaMallocAsync`、`cudaFreeAsync`、`cuMemAllocAsync` 等其他 CUDA 分配接口。 + # Java CPU Profiling Java CPU Profiling 通过 Java Agent 的 AsyncGetCallTrace（AGCT）持续采集 JVM 方法调用栈，并补全 Java JIT 方法符号。该功能独立于 eBPF On-CPU Profiling，必须同时满足以下两个条件才会采集目标进程： diff --git a/docs/zh/05-features/04-continuous-profiling/03-data.md b/docs/zh/05-features/04-continuous-profiling/03-data.md index 912fdc03..3f0ea6f4 100644 --- a/docs/zh/05-features/04-continuous-profiling/03-data.md +++ b/docs/zh/05-features/04-continuous-profiling/03-data.md @@ -36,8 +36,8 @@ curl -X POST http://${deepflow_server_node_ip}:$port/v1/profile/ProfileTracing \ API 请求参数说明： - **app_service**：进程名 -- **profile_language_type**：获取 eBPF Profile 数据时使用 `eBPF` -- **profile_event_type**：对于 eBPF On-CPU Profile 数据赋值为 `on-cpu` 即可 +- **profile_language_type**：获取 Agent 采集的 eBPF Profile 数据时使用 `eBPF`。Node.js/V8、PHP、Lua 和 Python 的解释器函数栈也统一使用 `eBPF`，不会按运行时拆分为不同的 Language Type +- **profile_event_type**：对于本文涉及的解释器 Profile，根据支持类型使用 `on-cpu`、`off-cpu`、`mem-alloc`、`mem-inuse`、`hbm-alloc` 或 `hbm-inuse` - **tag_filter**：当进程名冲突时，可使用其他 Tag 过滤 - 例如 `"tag_filter": "pod_cluster='prod-cluster' AND pod_ns='app'"` - **time_start**、**time_end**：时间范围 @@ -45,7 +45,7 @@ API 请求参数说明： `profile.in_process` 表支持使用的 `tag_filter` 字段如下： [csv-profile-tag-filters](https://raw.githubusercontent.com/deepflowio/deepflow/main/server/querier/db_descriptions/clickhouse/tag/profile/in_process.ch) -API 返回结果示例： +API 返回结果示例（以下以 CPU Profile 为例）： ```json { @@ -99,6 +99,8 @@ API 返回结果说明： - `[k] function_name`：Linux 内核函数、CUDA 动态链接库函数（[libcuda](https://developer.nvidia.com/cuda-toolkit)、[libcublas](https://developer.nvidia.com/cublas) 等） - `[l] function_name`：动态链接库中的函数 - `function_name`：表示应用程序的业务函数 + - `function_name [JS]`：Node.js/V8 的 JavaScript 函数，例如 `workLoop [JS]` + - `Class::method:line [PHP]` 或 `function:line [PHP]`：PHP 方法或函数及其源码行号 - `$app_service`：火焰图最顶层的节点，名字为进程名 - 除此之外，当函数名未成功翻译时，可能显示为如下几种形式之一 - `[/tmp/perf-29887.map]`：方括号中为进程号 29887 的 Java 进程符号文件名，函数地址未能在该文件中找到。Java 进程符号文件会自动周期性生成，此时一般由于该符号文件生成时该函数尚未加载导致。 @@ -117,6 +119,19 @@ API 返回结果说明： 使用 API 的返回结果，可以绘制**指定进程**的 CPU 火焰图。 +## 确认解释器函数栈 + +Node.js/V8 和 PHP 脚本帧分别带有 `[JS]`、`[PHP]` 后缀。Python 和 Lua 的脚本函数也会与 Native/运行时函数一起出现在火焰图中，但展示格式可能随运行时版本和符号内容变化。即使查询解释器进程，`profile_language_type` 仍须使用 `eBPF`。 + +如果 Profile 中只有 Native 或解释器运行时函数，没有业务脚本函数，请依次检查： + +1. 运行时版本和主机架构是否位于[支持范围](./auto-profiling/#解释器运行时)内。 +2. 内核是否支持增强型 Continuous Profiler，以及 Agent 是否出现 `falling back to common (no DWARF/unwind)` 降级日志。 +3. 目标进程是否已由 Process Matcher 开启所需 Profile 类型，且对应的 `inputs.ebpf.profile.languages` 开关未禁用。 +4. Agent 是否能读取目标进程的可执行文件及已加载的解释器共享库，日志中是否存在版本识别或符号读取失败。 + +配置方法请参考[配置方法](./configuration/)。 + # 获取指定主机的 Profile ## Grafana Panel @@ -185,7 +200,7 @@ API 返回结果说明： | Function Type | 含义 | Profile Event Type | 特征 | | ------------- | -------------- | ------------------ | ---------------------------------------- | -| O | 对象类型 | `mem-*` | Memory Profile 的叶子节点 | +| O | 对象类型 | `mem-*`、`hbm-*` | Memory/HBM Profile 的叶子节点 | | H | 云主机 | `*` | 等于 `Total` 的根节点 | | P | 进程 | `*` | `[p] ` 开头，以及不等于 `Total` 的根节点 | | T | 线程 | `*` | `[t] ` 开头 | diff --git a/translate/translated/05-features/04-continuous-profiling/01-auto-profiling.md b/translate/translated/05-features/04-continuous-profiling/01-auto-profiling.md index c9806db8..1df0e8c0 100644 --- a/translate/translated/05-features/04-continuous-profiling/01-auto-profiling.md +++ b/translate/translated/05-features/04-continuous-profiling/01-auto-profiling.md @@ -13,53 +13,82 @@ By using eBPF to capture snapshots of an application's function call stack, Deep # Capabilities and Limitations +## eBPF Profiling + Supported eBPF profiling data types: -| Type | Supported Languages/Libraries | Community Edition | Enterprise Edition | -| --------- | ----------------------------- | ----------------- | ------------------ | -| on-cpu | Java | ✔ | ✔ | -| | C/C++ | ✔ | ✔ | -| | Rust | ✔ | ✔ | -| | Golang | ✔ | ✔ | -| | Python `***` | ✔ | ✔ | -| | CUDA | ✔ | ✔ | -| | Lua `*` | ✔ | ✔ | -| off-cpu | Java | | ✔ | -| | C/C++ | | ✔ | -| | Rust | | ✔ | -| | Golang | | ✔ | -| | Python `***` | | ✔ | -| | CUDA | | ✔ | -| | Lua `*` | | ✔ | -| on-gpu | CUDA `*` | | ✔ | -| mem-alloc | Java `**` | | ✔ | -| | Rust | | ✔ | -| | Golang `*` | | ✔ | -| | Python `*` `***` | | ✔ | -| mem-inuse | Rust | | ✔ | -| hbm-alloc | CUDA `*` | | ✔ | -| hbm-inuse | CUDA `*` | | ✔ | -| rdma | C/C++ `*` | | ✔ | +| Type | Supported Languages/Libraries | Community Edition | Enterprise Edition | +| ---------- | ----------------------------- | ----------------- | ------------------ | +| on-cpu | Java | ✔ | ✔ | +| | C/C++ | ✔ | ✔ | +| | Rust | ✔ | ✔ | +| | Golang | ✔ | ✔ | +| | CUDA | ✔ | ✔ | +| | Node.js/V8 | | ✔ | +| | PHP | | ✔ | +| | Lua | | ✔ | +| | Python | | ✔ | +| off-cpu | Java | | ✔ | +| | C/C++ | | ✔ | +| | Rust | | ✔ | +| | Golang | | ✔ | +| | CUDA | | ✔ | +| | Node.js/V8 | | ✔ | +| | PHP | | ✔ | +| | Python | | ✔ | +| on-gpu | CUDA `*` | | ✔ | +| mem-alloc | Java `**` | | ✔ | +| | Rust | | ✔ | +| | Golang `*` | | ✔ | +| | Python | | ✔ | +| mem-inuse | Rust | | ✔ | +| | Python | | ✔ | +| hbm-alloc | CUDA `*` | | ✔ | +| | Python (using CUDA) | | ✔ | +| hbm-inuse | CUDA `*` | | ✔ | +| | Python (using CUDA) | | ✔ | +| rdma | C/C++ `*` | | ✔ | Notes: -- `*`: features in development -- `**`: The JVM running the Java program must have a symbol table, see [check method](#jvm-symbol-table-check) -- `***`: Currently supports Python 3.10 -- Types: - - on-cpu: Time a function spends on the CPU - - off-cpu: Time a function waits for the CPU - - on-gpu: Time a function spends on the GPU - - mem-alloc: Total memory allocated by objects and the function call stack - - mem-inuse: Current memory usage of objects and the function call stack - - hbm-alloc: Total GPU memory allocated by objects and the function call stack - - hbm-inuse: Current GPU memory usage of objects and the function call stack -- Languages: - - Languages compiled into ELF format executables: Golang, Rust, C/C++ - - Languages using the JVM: Java - - Interpreted languages: Python - -Two prerequisites must be met to obtain profiling data: +- `*`: features in development +- `**`: The JVM running the Java program must have a symbol table. See the [check method](#jvm-symbol-table-check). +- Types: + - on-cpu: Time a function spends on the CPU + - off-cpu: Time a function waits for the CPU + - on-gpu: Time a function spends on the GPU + - mem-alloc: Total memory allocated by objects and the function call stack + - mem-inuse: Current memory usage of objects and the function call stack + - hbm-alloc: Total GPU memory allocated by objects and the function call stack + - hbm-inuse: Current GPU memory usage of objects and the function call stack +- Languages: + - Languages compiled into ELF format executables: Golang, Rust, C/C++ + - Languages using the JVM: Java + - Interpreter runtimes: Node.js/V8, PHP, Lua, Python + +### Interpreter Runtimes + +Interpreter-level script stack unwinding is an Enterprise Edition capability. The following runtimes are currently supported: + +| Runtime | Supported Versions | Architectures | Profile Types | +| ------- | ------------------ | ------------- | ------------- | +| Node.js/V8 | Node.js 16–23 (V8 9–12) | x86_64, AArch64 | On-CPU, Off-CPU | +| PHP | 7.4–8.3 | x86_64, AArch64 | On-CPU, Off-CPU | +| Lua | Lua 5.1–5.4, LuaJIT 2.1 | x86_64, AArch64 | On-CPU | +| Python | CPython 3.10–3.13 | x86_64, AArch64 | On-CPU, Off-CPU, mem-alloc, mem-inuse, hbm-alloc, hbm-inuse | + +Python HBM Profiles come from CUDA APIs called by the target process, not from the CPython allocator. Python applications such as PyTorch and vLLM can generate `hbm-alloc` and `hbm-inuse` when they allocate and release GPU memory through supported CUDA APIs. See [Configuration](./configuration/#cuda-hbm-profiling) for the collection scope and startup timing limitations. + +Script stack unwinding requires the Agent to load the enhanced Continuous Profiler: + +- Standard Linux kernels must be version 5.2 or later. +- Kylin V10 SP3 v2207 kernels matching `4.19.90-*.v2207.ky10.*` are also supported on both x86_64 and AArch64. +- Generic Linux 4.19 kernels and Kylin V10 SP3 v2101 kernels do not support script stack unwinding. The Agent can still use the common profiler to collect eligible native stacks. +- If the enhanced profiler fails to load, the Agent automatically falls back to the common profiler. DWARF and interpreter-level script stack unwinding are unavailable after fallback. + +The Agent must be able to identify the target interpreter version and read the process executable and loaded interpreter libraries. With an unsupported runtime, a failed identification, or inaccessible files, a Profile may still contain native/runtime frames but no script functions. Python Memory Profiling captures object allocation and release events through the public CPython symbols `PyObject_Malloc/Free/Realloc` and `PyMem_RawMalloc/Free/Realloc`. + +Two prerequisites must be met when the common eBPF On-CPU/Off-CPU profiler obtains call stacks: - The application process must enable Frame Pointer or enable the Agent's DWARF stack unwinding capability - Enable Frame Pointer (frame pointer register) for the application process: @@ -78,6 +107,25 @@ The Off-CPU profiling feature **only** collects the following call stacks: - Call stacks containing **at least one** user-space function - Call stacks where the CPU wait time is **no more than** 1 hour +## Java Profiling + +Supported Java Profiling data types: + +| Type | Supported Language/Library | Community Edition | Enterprise Edition | +| ---- | -------------------------- | ----------------- | ------------------ | +| cpu | Java | | ✔ | + +Notes: + +- `cpu` measures the CPU time consumed by Java methods and captures their call stacks. +- A Java Agent continuously samples JVM method stacks using HotSpot AsyncGetCallTrace (AGCT) and resolves JIT method symbols. +- Java CPU Profiling and eBPF On-CPU Profiling are independent: + - Java CPU Profiling samples Java method stacks inside the JVM. It uses `java.profile.cpu` to select processes and does not require Frame Pointer. + - eBPF On-CPU Profiling samples user-space and kernel-space stacks through eBPF/perf. It uses `ebpf.profile.on_cpu` to select processes. + - Both can profile the same Java process, or either one can be enabled independently. If only clear Java method stacks are needed, enable Java CPU Profiling alone to avoid collecting two profiles from different sources. + +See [Configuration Method](./configuration/#java-cpu-profiling) for details. + # FAQ ## JVM Symbol Table Check diff --git a/translate/translated/05-features/04-continuous-profiling/02-configuration.md b/translate/translated/05-features/04-continuous-profiling/02-configuration.md index e618523e..76771f23 100644 --- a/translate/translated/05-features/04-continuous-profiling/02-configuration.md +++ b/translate/translated/05-features/04-continuous-profiling/02-configuration.md @@ -20,11 +20,31 @@ inputs: only_in_container: false rewrite_name: $3 enabled_features: [ebpf.profile.on_cpu, proc.gprocess_info] + - match_regex: \bjava( +\S+)* +-(?:cp|classpath) +\S+ +(?P[$_A-Za-z][$_0-9A-Za-z]*(?:\.[$_A-Za-z][$_0-9A-Za-z]*)*) + match_type: cmdline_with_args + only_in_container: false + rewrite_name: ${CLASS_NAME} + enabled_features: [ebpf.profile.on_cpu, proc.gprocess_info] - match_regex: \bpython(\S)*( +-\S+)* +(\S*/)*([^ /]+) match_type: cmdline_with_args only_in_container: false rewrite_name: $4 enabled_features: [ebpf.profile.on_cpu, proc.gprocess_info] + - match_regex: \b(?:lua|luajit)(\S)*( +-\S+)* +(\S*/)*([^ /]+) + match_type: cmdline_with_args + only_in_container: false + rewrite_name: $5 + enabled_features: [ebpf.profile.on_cpu, proc.gprocess_info] + - match_regex: \bphp(\d+)?(-fpm|-cli|-cgi)?( +-\S+)* +(\S*/)*([^ /]+\.php) + match_type: cmdline_with_args + only_in_container: false + rewrite_name: $5 + enabled_features: [ebpf.profile.on_cpu, proc.gprocess_info] + - match_regex: \b(node|nodejs)( +--\S+)* +(\S*/)*([^ /]+\.js) + match_type: cmdline_with_args + only_in_container: false + rewrite_name: $4 + enabled_features: [ebpf.profile.on_cpu, proc.gprocess_info] - match_regex: ^deepflow- only_in_container: false enabled_features: [ebpf.profile.on_cpu, proc.gprocess_info] @@ -35,9 +55,9 @@ inputs: The meaning of the above configuration is as follows: - **match_regex**: The regular expression for process matching. The matching rules are as follows: - - The first rule matches Java processes, e.g., `java -jar app.jar`, and rewrites the process name to the JAR filename. - - The second rule matches Python processes, e.g., `python app.py`, and rewrites the process name to the Python script name. - - The third rule matches processes starting with `deepflow-`. + - The first two rules match Java processes started with a JAR or a main class, and rewrite the process name to the JAR filename or class name. + - The following rules match Python, Lua/LuaJIT, PHP, and Node.js processes and rewrite the process name to the script filename. + - The `^deepflow-` rule matches processes starting with `deepflow-`. - The last rule matches all processes. - **match_type**: The matching type. Optional values are: - `cmdline_with_args`: Matches the full command line (including arguments). @@ -46,10 +66,13 @@ The meaning of the above configuration is as follows: - **only_in_container**: Whether to match only processes within containers. - **rewrite_name**: The rule for rewriting the process name, supporting references to regex capture groups. - **enabled_features**: The list of features enabled for matched processes: + - `java.profile.cpu`: Enables Java CPU Profiling, requires `inputs.java.profile.cpu.enabled: true`, and does not depend on `ebpf.profile.on_cpu` - `ebpf.profile.on_cpu`: Enables On-CPU profiling, requires `inputs.ebpf.profile.on_cpu.disabled: false` - `ebpf.profile.off_cpu`: Enables Off-CPU profiling, requires `inputs.ebpf.profile.off_cpu.disabled: false` - `ebpf.profile.memory`: Enables memory profiling, requires `inputs.ebpf.profile.memory.disabled: false` +The default Process Matcher enables `ebpf.profile.on_cpu` for Java, Python, Lua/LuaJIT, PHP, Node.js, and DeepFlow processes. To collect Off-CPU or Memory Profiles, add the corresponding feature to `enabled_features` for the target process and enable the matching global Profile type. Interpreter-level Memory Profiling currently supports Python only. + Additionally, `inputs.proc.process_blacklist` can be used to ignore specific processes. It has higher priority than `process_matcher`. ```yaml @@ -64,7 +87,7 @@ Symbol table related settings can be configured for specific languages. These se ```yaml inputs: - ebpf: + proc: symbol_table: golang_specific: enabled: false @@ -78,6 +101,28 @@ The meaning of the above configuration is as follows: - **refresh_defer_duration**: The refresh deferral duration for the Java symbol table, to avoid high-frequency refreshing. - **max_symbol_file_size**: The maximum disk space occupied by the Java symbol table, in GB, to avoid consuming excessive `/tmp` space. +# Interpreter Profiling + +Script stack unwinding for Node.js/V8, PHP, Lua, and Python is enabled by default. If a runtime is not used on a host, disable its unwinder to reduce kernel memory usage: + +```yaml +inputs: + ebpf: + profile: + languages: + python_disabled: false + php_disabled: false + nodejs_disabled: false + lua_disabled: false +``` + +- All four options default to `false`, which enables stack unwinding for the corresponding runtime. +- When an option is set to `true`, common native profiles of the target process can still be collected, but its script functions are no longer unwound. +- Restart the Agent after changing these options. +- Enabling all runtime unwinders uses approximately 17–20 MB of kernel memory. Disable unused runtimes to reduce this overhead. + +The language switches only control script stack unwinding. The target process must still be selected by `inputs.proc.process_matcher`, and the required global Profile type must be enabled. See [Capabilities and Limitations](./auto-profiling/#interpreter-runtimes) for supported versions, architectures, and kernel requirements. + # eBPF On-CPU Profiling eBPF On-CPU Profiling is enabled by default, but requires modifying `inputs.proc.process_matcher` to specify the target process list. The configuration parameters supported by the Agent are as follows: @@ -145,4 +190,57 @@ The meaning of the above configuration is as follows: - When configuring this option, refer to the collector performance metric `deepflow_agent_ebpf_memory_profiler`, specifically the `time_backtracked` metric. Increase this parameter until the metric becomes 0. Note that it might be necessary to correspondingly increase the `sort_length` parameter. - **queue_size**: The internal queue size of the memory profiling component. - When configuring this option, refer to the collector performance metrics `deepflow_agent_ebpf_memory_profiler`, specifically the `overwritten` and `pending` metrics. Increase this configuration until the former is 0 and the latter does not exceed this configuration value. + +## CUDA/HBM Profiling + +HBM Profiling is independent of the target process's programming language. For Python/PyTorch/vLLM, the worker process that actually performs GPU allocations must also match `inputs.proc.process_matcher` and have `ebpf.profile.memory` enabled. Matching only the parent process does not collect GPU memory events from its children. `inputs.ebpf.profile.languages.python_disabled` controls Python script stack unwinding only; it does not disable HBM event collection. + +The Agent currently traces calls to `cudaMalloc`, `cudaFree`, `cuMemAlloc_v2`, and `cuMemFree_v2` with uprobes and generates: + +- `hbm-alloc`: GPU memory allocations and call stacks observed after the uprobes are active. +- `hbm-inuse`: Current GPU memory usage and call stacks calculated from observed allocation addresses and subsequent free events. + +The Agent scans processes approximately every 10 seconds. For a newly matched non-Agent process, it currently waits approximately another 120 seconds before resolving loaded CUDA libraries and attaching Memory uprobes. GPU allocations made before the uprobes are active are not backfilled and are not included in subsequent `hbm-inuse` data. After attachment, new HBM Profiles are generated when the process continues to call the supported CUDA allocation and free APIs listed above. + +Frameworks such as PyTorch and vLLM may reserve large GPU memory blocks through a caching allocator and then allocate and free objects within that pool. Pool operations that do not call the CUDA APIs listed above do not generate HBM events. Other CUDA allocation APIs, including `cudaMallocAsync`, `cudaFreeAsync`, and `cuMemAllocAsync`, are not currently traced. + +# Java CPU Profiling + +Java CPU Profiling continuously samples JVM method stacks through a Java Agent using AsyncGetCallTrace (AGCT), and resolves Java JIT method symbols. This feature is independent of eBPF On-CPU Profiling. Both of the following conditions must be met before a target process is profiled: + +- Set `inputs.java.profile.cpu.enabled: true` to enable the Java CPU Profiler. +- Match the target process with `inputs.proc.process_matcher` and include `java.profile.cpu` in `enabled_features`. + +Start with a small number of business processes matched by JAR filename or full command line, and verify the resource overhead before expanding the scope. Merge the following example into the existing Agent group configuration; do not replace existing Process Matchers or unrelated settings: + +```yaml +inputs: + proc: + process_matcher: + - match_regex: '.*my-order-service\.jar.*' + match_type: cmdline_with_args + only_in_container: false + enabled_features: + - java.profile.cpu + - proc.gprocess_info + java: + profile: + cpu: + enabled: true + frequency: 99 + max_depth: 98 + sample_ring_size: 512 + method_cache_size: 256 ``` + +To also collect ordinary eBPF On-CPU Profiles from the same process, retain `ebpf.profile.on_cpu` in `enabled_features` and ensure `inputs.ebpf.profile.on_cpu.disabled: false`. The two features use independent sampling paths and process lists; neither is a prerequisite for the other. + +Configuration parameters: + +- **enabled**: Defaults to false. When set to true, the Agent prepares the Java CPU Profiler at startup. Restart the Agent after changing this option. +- **frequency**: Sampling frequency in Hz. The default is 99 and the allowed range is 1–1000. For resource-sensitive environments, start at 49. Use 199 only for short diagnostics and after load testing. +- **max_depth**: Maximum number of frames retained in a Java stack. The default is 98 and the allowed range is 1–128. A larger value preserves deeper call paths but increases sample size and processing overhead. +- **sample_ring_size**: Per-JVM sample ring capacity. The default is 512 and the allowed range is 64–8192. A larger ring can absorb burst sampling or temporary sender backpressure, at the cost of additional JVM memory. +- **method_cache_size**: Per-JVM method cache capacity. The default is 256 and the allowed range is 64–8192. Increase it when a JVM has many methods or repeatedly resolves symbols, at the cost of additional JVM memory. + +Changes to `enabled` and the sampling parameters require an Agent restart. Process Matcher supports hot updates; adding or removing `java.profile.cpu` does not restart the target JVM. diff --git a/translate/translated/05-features/04-continuous-profiling/03-data.md b/translate/translated/05-features/04-continuous-profiling/03-data.md index 476012ca..b2eefdae 100644 --- a/translate/translated/05-features/04-continuous-profiling/03-data.md +++ b/translate/translated/05-features/04-continuous-profiling/03-data.md @@ -38,8 +38,8 @@ curl -X POST http://${deepflow_server_node_ip}:$port/v1/profile/ProfileTracing \ Explanation of API request parameters: - **app_service**: Process name -- **profile_language_type**: Use `eBPF` when retrieving eBPF Profile data -- **profile_event_type**: For eBPF On-CPU Profile data, set the value to `on-cpu` +- **profile_language_type**: Use `eBPF` for Profiles collected by the Agent's eBPF profiler. Interpreter frames for Node.js/V8, PHP, Lua, and Python also use `eBPF`; they are not split into runtime-specific Language Types +- **profile_event_type**: For the interpreter Profiles covered here, use `on-cpu`, `off-cpu`, `mem-alloc`, `mem-inuse`, `hbm-alloc`, or `hbm-inuse`, according to the supported Profile type - **tag_filter**: When process names conflict, other tags can be used for filtering - For example, `"tag_filter": "pod_cluster='prod-cluster' AND pod_ns='app'"` - **time_start**, **time_end**: Time range @@ -47,7 +47,7 @@ Explanation of API request parameters: The `profile.in_process` table supports the following `tag_filter` fields: [csv-profile-tag-filters](https://raw.githubusercontent.com/deepflowio/deepflow/main/server/querier/db_descriptions/clickhouse/tag/profile/in_process.en) -Example of API return result: +Example API response (using a CPU Profile): ```json { @@ -101,6 +101,8 @@ Explanation of API return result: - `[k] function_name`: Linux kernel functions, CUDA dynamic library functions ([libcuda](https://developer.nvidia.com/cuda-toolkit), [libcublas](https://developer.nvidia.com/cublas), etc.) - `[l] function_name`: Functions in dynamic libraries - `function_name`: Represents business functions of the application + - `function_name [JS]`: A Node.js/V8 JavaScript function, for example `workLoop [JS]` + - `Class::method:line [PHP]` or `function:line [PHP]`: A PHP method or function and its source line - `$app_service`: The top node of the flame graph, named after the process name - Additionally, when the function name translation fails, it may appear in one of the following forms: - `[/tmp/perf-29887.map]`: The file name of the Java process symbol file for process number 29887 in square brackets, the function address was not found in this file. Java process symbol files are automatically generated periodically, usually because the function was not loaded when the symbol file was generated. @@ -119,6 +121,19 @@ Explanation of API return result: Using the API return result, you can draw a CPU flame graph for the **specified process**. +## Verify Interpreter Frames + +Node.js/V8 and PHP script frames use the `[JS]` and `[PHP]` suffixes, respectively. Python and Lua script functions also appear alongside native/runtime frames, although their rendering can vary with the runtime version and available symbols. Even when querying an interpreter process, set `profile_language_type` to `eBPF`. + +If a Profile contains only native or runtime functions and no business script functions, check the following: + +1. Verify that the runtime version and host architecture are within the [supported range](./auto-profiling/#interpreter-runtimes). +2. Verify that the kernel supports the enhanced Continuous Profiler and check the Agent log for `falling back to common (no DWARF/unwind)`. +3. Verify that Process Matcher enables the required Profile type for the target process and that the corresponding `inputs.ebpf.profile.languages` switch is not disabled. +4. Verify that the Agent can read the target process executable and loaded interpreter libraries, and check the log for version detection or symbol-reading failures. + +See [Configuration Method](./configuration/) for configuration details. + # Retrieve Profile for a Specific Host ## Grafana Panel @@ -187,7 +202,7 @@ Using the API return result, you can draw an On-CPU flame graph for the **specif | Function Type | Meaning | Profile Event Type | Characteristics | | ------------- | --------------- | ------------------ | ---------------------------------------- | -| O | Object type | `mem-*` | Leaf node of Memory Profile | +| O | Object type | `mem-*`, `hbm-*` | Leaf node of Memory/HBM Profile | | H | Cloud host | `*` | Root node equal to `Total` | | P | Process | `*` | Starts with `[p] `, and root nodes not equal to `Total` | | T | Thread | `*` | Starts with `[t] ` |