Add integration with rust-jemalloc-pprof for easy mem perf

[jiacai2050]

Describe This Problem

rust-jemalloc-pprof is a cool project to do memory perf.

Proposal

Add a new HTTP endpoint /debug/pprof/heap to serve the profiling.

Additional Context

We can remove old /debug/profile/heap/{seconds} endpoint when this is ready.

horaedb/src/server/src/http.rs

Line 607 in 7962799

fn profile_heap(

[jiacai2050]

@zuston Thanks, assigned.

[zuston]

@zuston Thanks, assigned.

From this crate doc, I found this way is not better than the vaillina jeprof way. It just generates the pprof format, if you want to dig it with human format, you have to execute in the host machine instead of any remote machine to call api directly. Because it still depends on the execute binary.

[zuston]

PTAL @jiacai2050

[jiacai2050]

you have to execute in the host machine instead of any remote machine to call api directly.

What do you mean by call api directly?

With pprof, we can explore the perf data in various format, such as flamegraph

• https://github.com/google/pprof/blob/main/doc/README.md#graph

[vmandke]

@jiacai2050 is this still available to pick ?

[jiacai2050]

@vmandke Yeah, it's still available, but you should merge the PR to https://github.com/apache/horaedb/tree/analytic-engine branch.

[BaldDemian]

Hey @jiacai2050, I’m trying to implement this issue, but I’m unclear about the purpose of this endpoint.
The existing endpoint profile_heap accepts a seconds parameter, meaning that the heap profiling is activated only during that specified time duration.
However, the endpoint proposed in this issue lacks a seconds parameter. Then when should the profiling be activated and deactivated in this case? Should the profiling starts from the beginning of the system’s execution, i.e. export MALLOC_CONF="prof:true,prof_active:true,lg_prof_sample:19"? But the comment says this will hurt the performance:

horaedb/src/components/profile/src/lib.rs

Lines 73 to 75 in fa64050

	/// ProfLockGuard hold the profile lock and take responsibilities for
	/// (de)activating heap profiling. NOTE: Keeping heap profiling on may cause
	/// some extra runtime cost so we choose to activating it dynamically.

Thanks for your response😀

[jiacai2050]

Hi, thanks for you interests.

The new API should also contains a seconds params, sorry for misleading.

[BaldDemian]

I made a PR here, but there is a problem about the usage of the returned heap.pb.gz file🤔. You can see #1627 (comment) for more description.

[jiacai2050]

👍

I will review it in next couple of days.