perf(metrics): partition ValueMap into per-thread shards to reduce lock contention

[souyang-dev]

Metrics ValueMap is protected by a RwLock. It can be a source of contention under high concurrency. What's more, the cost to update a histogram is even higher because you need to grab a mutex lock after the read lock before you can update a bucket.
There is a previous attempt to shard the ValueMap (#1564), but that PR partitions metrics updates based on attribute-sets, therefore updates to the same attribute-set still land on the same shard, so it's not effective in reducing locking contention when most updates are focused on a small set of attributes.

Fixes #
This PR partitions the single HashMap into shards. Each thread update different shards based on thread id to minimize locking contention. If the number of shards >= number of CPUs you won't see locking contentions when updating metrics.

Here are Benchcmark results (metrics_histogram, metrics) with AMD EPYC 9555 CPUs (255 CPU threads).
The numbers in the tables are updates/second in millions. "shards=16" is this PR with 16 shards, "shards=32" is this PR with 32 shards.

Counter update throughput (cargo run --release --package stress --bin metrics -- N)

number of threads	1	2	4	8	16
upstream	10	5.7	6.8	6	5.5
shards=16	10	22	32	40	43
shards=32	10	22	31	48	100

Histogram update throughput (cargo run --release --package stress --bin metrics_histogram -- N)

number of threads	1	2	4	8	16
upstream	9.5	4.8	5.2	6	5.5
shards=16	9.5	19	28	40	42
shards=32	9.5	19	28	45	84

Changes

Two changes are made:
1, Partition ValueMap into shards and let different threads update different shards to minimize locking contention.
2, Implement merge function for Aggregators, so collector can merge the shards.

Merge requirement checklist

• [* ] CONTRIBUTING guidelines followed
• [* ] Unit tests added/updated (if applicable)
• Appropriate CHANGELOG.md files updated for non-trivial, user-facing changes
• Changes in public API reviewed (if applicable)

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: souyang-dev / name: Shawn (4aa7aa9, 6dfac19, 9676b42, ff20752)

[codecov]

Codecov Report

❌ Patch coverage is 93.90756% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.9%. Comparing base (c992379) to head (ff20752).

Files with missing lines	Patch %	Lines
...-sdk/src/metrics/internal/exponential_histogram.rs	92.5%	19 Missing ⚠️
opentelemetry-sdk/src/metrics/internal/mod.rs	94.9%	10 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #3473     +/-   ##
=======================================
+ Coverage   83.7%   83.9%   +0.1%     
=======================================
  Files        126     126             
  Lines      25386   25784    +398     
=======================================
+ Hits       21255   21635    +380     
- Misses      4131    4149     +18     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[scottgerring]

Pinging @cijothomas because metrics is really his area :) !

I'm not sure about this part but it might be a gap in my mental model. The metrics SDK spec defines aggregation_cardinality_limit as "the maximum number of data points allowed to be emitted in a collection cycle by a single instrument" and says that, for a given metric, the cardinality limit is a "hard limit" on the number of metric points collected during a collection cycle:

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#cardinality-limits

It looks like with this change, the effective limit becomes roughly configured_limit * num_shards where num_shards is the smaller of CPU count or 64, so a configured limit of 2000 can produce many more than 2000 points.

I think sharding needs to remain an internal implementation detail and preserve the configured stream cardinality limit. The user-visible impact is that a limit chosen to cap SDK memory usage and downstream time-series cardinality would no longer provide that cap; the same configuration could produce very different numbers of exported
series depending on CPU count.

[cijothomas]

Previous shard attempts were also blocked on this same problem. It is non-trivial to solve with sharding approach, unfortunately. We'd need to solve contention issue differently where consciously opts in to this model. (I'll write what I have in my mind soon)

[souyang-dev]

Thank you @scottgerring for the comments.
I have updated the patch to enforce cardinality limit across all shards.