Metrics instrumentation across all DataStoreTypes (ZK, HTTP, DROVE) by jitendradhawan · Pull Request #108 · appform-io/ranger

jitendradhawan · 2026-06-01T05:14:39Z

Summary

Adds Dropwizard metrics instrumentation across Ranger's ZooKeeper, HTTP, and Drove discovery paths. Metrics now cover upstream availability, remote calls, parsing/serialization, registry refreshes, stale-data retention, health checks, provider updates, and node/service counts.

This change also introduces stable upstream IDs for metric attribution, wires metrics through Dropwizard bundles, adds backend-focused integration tests, and prepares release 2.0.0-RC7.

Why

Ranger previously exposed limited visibility into discovery failures and data quality. Operators could not reliably distinguish upstream outages, malformed responses, stale-node retention, zombie filtering, or registry refresh latency.

These metrics make those states observable and separate them by datastore type and configured upstream.

What Changed

Metrics foundation

Added shared MetricRecorder, backed by Dropwizard MetricRegistry and safe no-op behavior before initialization.
Added DataStoreType (ZK, HTTP, DROVE) and upstream ID dimensions.
Standardized names under io.appform.ranger.
Used meters for events/status observations, timers for refresh latency, gauges for latest node counts, and 64-sample UniformReservoir histograms for API response counts.
Added metrics enablement to discovery and server bundles. YAML/no-arg discovery configuration and server-bundle configuration default to enabled; ServiceDiscoveryConfiguration.builder() callers must explicitly set .metricsEnabled(true).

Instrumented flows

Area	Coverage
Registry refresh	Success/failure latency, fetched nodes, accepted nodes, zombie filtering, stale-node retention
Upstream health	Active/inactive observations for ZooKeeper, HTTP, and Drove
Remote operations	Response status, empty/null response, parse failure, transport/unknown failure
Service providers	ZooKeeper and HTTP registration/update, serialization, and write failures
Health checks	Healthy/unhealthy results and checker exceptions
Drove cache	Event-triggered application/instance cache updates
Ranger server API	Services returned and nodes returned per service

Representative metric hierarchy:

io.appform.ranger.dataStoreType.<ZK|HTTP|DROVE>.dataSource.<upstreamId>...

Data flow

Bundle MetricRegistry
  -> MetricRecorder initialization
  -> configured upstream ID
  -> client/factory/builder
  -> NodeDataSource or NodeDataSink
  -> operation/status/latency/count metric

Bundle and configuration wiring

Discovery, server, and hub-server bundles initialize instrumentation from their Dropwizard MetricRegistry.
HTTP, Drove, and Ranger Hub ZooKeeper upstream configurations now carry a required non-blank id.
Finder/provider builders and backend factories propagate the ID to sources and sinks.
Added metricsEnabled configuration and server-bundle enablement hooks.
Discovery bundles use the fixed upstream ID zkServiceDiscovery.
MetricRecorder is process-global. Disabling one bundle skips initialization but does not clear a registry initialized by another bundle.

Breaking Changes / Migration

Before	After / action required
HTTP or Drove upstream without an ID	Add a unique, stable `id`
Ranger Hub ZooKeeper `zookeepers: [...]`	Use `zookeeper: "connection-string"` and add `id`
Multiple ZooKeeper upstreams in one list	Configure separate upstream entries, each with a unique ID
Finder/provider builder without upstream metadata	Call the corresponding `withUpstreamId(...)` API
Custom `NodeDataSource` / `NodeDataSink`	Implement `getUpstreamId()` and `getDataStoreType()`
Overridden source/sink factory hook	Accept and propagate the new upstream ID argument
Direct backend constructor calls	Supply the new upstream ID parameter
Direct hub-client builder	Add `.upstreamId(...)`, including `SimpleRangerZKClient.builder()`
Discovery-bundle default constants	Import `DEFAULT_NAMESPACE` / `DEFAULT_HOST` from discovery-core

HTTP upstream migration

Each entry under httpClientConfigs now requires a non-blank id. Use a unique, stable value for correct metric attribution. Other HTTP client fields remain unchanged. The ID is propagated to the HTTP hub client and appears in metric names as <upstreamId>.

# Before
- type: HTTP
  nodeRefreshTimeMs: 5000
  serviceRefreshTimeoutMs: 300000
  hubStartTimeoutMs: 210000
  httpClientConfigs:
    - host: ranger-a.example.com
      port: 443
      secure: true

# After
- type: HTTP
  nodeRefreshTimeMs: 5000
  serviceRefreshTimeoutMs: 300000
  hubStartTimeoutMs: 210000
  httpClientConfigs:
    - id: primary-http
      host: ranger-a.example.com
      port: 443
      secure: true

Add an ID to every configured endpoint when multiple HTTP clients exist:

httpClientConfigs:
  - id: primary-http
    host: ranger-a.example.com
    port: 443
    secure: true
  - id: secondary-http
    host: ranger-b.example.com
    port: 443
    secure: true

Programmatic configuration must also set the ID:

HttpClientConfig config = HttpClientConfig.builder()
        .id("primary-http")
        .host("ranger-a.example.com")
        .port(443)
        .secure(true)
        .connectionTimeoutMs(5_000)
        .operationTimeoutMs(5_000)
        .refreshIntervalMillis(10_000)
        .build();

Direct callers of the generated all-arguments constructor must pass id as its new first argument. Prefer the builder to avoid positional migration mistakes.

Resulting metrics use the configured value, for example:

io.appform.ranger.dataStoreType.HTTP.dataSource.primary-http...

Drove upstream migration

Each DroveUpstreamConfig now requires a non-blank id. Use a unique, stable value for correct metric attribution. Add it to YAML and programmatic builders:

- type: DROVE
  droveClusters:
    - id: primary-drove
      endpoints:
        - drove-a.example.com:443

DroveUpstreamConfig config = DroveUpstreamConfig.builder()
        .id("primary-drove")
        .endpoints(List.of("drove-a.example.com:443"))
        .build();

Direct DroveOkHttpTransport callers must change (okHttpClient) to (upstreamId, okHttpClient). Remove explicit generic type witnesses from RangerDroveUtils.buildDroveClient(...), which is no longer generic.

ZooKeeper upstream migration

Each Ranger Hub ZooKeeper upstream now requires a non-blank id. Configuration changes from plural zookeepers (List<String>) to singular zookeeper (String). A ZooKeeper ensemble remains one comma-separated connection string; configure separate upstream entries when connecting to multiple independent ensembles.

# Before
- type: ZK
  zookeepers:
    - zk-a:2181,zk-b:2181

# After
- type: ZK
  id: primary-zk
  zookeeper: zk-a:2181,zk-b:2181

Multiple independent ensembles now require separate entries:

- type: ZK
  id: primary-zk
  zookeeper: zk-a:2181,zk-b:2181
- type: ZK
  id: secondary-zk
  zookeeper: zk-c:2181,zk-d:2181

Programmatic hub configuration must call setId(...) and setZookeeper(...) instead of setZookeepers(...). Direct ZooKeeper clients, finder/provider builders, factories, data sources, and sinks must also receive and propagate upstreamId.

Resulting metrics use the configured value, for example:

io.appform.ranger.dataStoreType.ZK.dataSource.primary-zk...

Upstream IDs become metric-name segments. Validation enforces non-blank values but not uniqueness or metric-safe characters; choose unique, stable, low-cardinality IDs.

Programmatic extension migration

Custom implementations and direct callers must update these signatures:

Previous	New
`dataSource(Service)`	`dataSource(String upstreamId, Service)`
`dataSink(Service)`	`dataSink(String upstreamId, Service)`
`getDefaultDataSource()`	`getDefaultDataSource(String upstreamId)`
HTTP/ZooKeeper source, service-source, and sink constructors	Prepend `upstreamId`
ZooKeeper finder-factory builders/constructors	Supply `upstreamId`

Behavioral Notes

Active/inactive metrics are meters incremented when isActive() is observed, not current-state gauges or transition counters.
Refresh success is recorded for any non-null result, including an empty node list. Exceptions record refresh failure and retain only currently healthy stale nodes with refreshed timestamps.
An active source returning Optional.empty() records neither refresh success/failure nor stale retention.
HTTP status/parse metrics and unknownFailure metrics can overlap because handled failures are rethrown as communication exceptions.
ZooKeeper missing paths, empty children, and child-level KeeperExceptions are classified as empty responses; broader read failures use zkRead.*.unknownFailure.
Drove is source-only; HTTP and ZooKeeper instrument both discovery sources and provider sinks.
Health-check metrics are process-global and do not include datastore, upstream, or service dimensions.

Operational Considerations

One metric object is retained per complete metric name. Bounded histogram reservoirs limit samples per histogram, not number of service/upstream metrics.
Metric names include raw upstream IDs, service names, exception classes, HTTP statuses, and Drove event names. Keep configured IDs and discovered service sets bounded.
Reinitializing the process-global recorder changes the registry for all Ranger producers. Review shared-process deployments and multiple bundle initialization carefully.
Namespace is not included in service-level metric names; identical service names across namespaces aggregate.

Dependency Impact

ranger-core gains production dependency on Dropwizard Metrics.
ranger-http gains production dependency on javax.validation-api for HttpClientConfig.id validation.
ranger-zookeeper gains WireMock as test dependency.

Testing

Added 12 metrics-focused unit/integration test suites across ranger-core, ranger-http, ranger-drove, and ranger-zookeeper.
Covered metric naming/counting, latest-value gauges, refresh success/failure latency, stale retention, zombie filtering, health checks, upstream status, response handling, parse failures, and ZooKeeper reads/writes.
Updated existing client, factory, builder, bundle, and hub tests for upstream IDs and datastore metadata.
Replaced PingCheckMonitorTest external network dependency with local WireMock.
Focused core tests and affected-module compilation pass; full reactor test suite was not run during this review.
Not directly covered: bundle enable/disable and registry replacement, Ranger API count histograms, HTTP sink failure metrics, Drove event-cache metrics, and missing/blank-ID configuration migration.

Suggested verification:

mvn -pl ranger-core,ranger-http,ranger-drove,ranger-zookeeper -am test
mvn test

Reviewer Guide

Recommended review order:

MetricRecorder metric taxonomy, registry lifecycle, and cardinality behavior.
ServiceRegistryUpdater, HealthChecker, and FinderUtils instrumentation semantics.
HTTP, Drove, and ZooKeeper source/sink failure classification.
Upstream ID propagation through configs, clients, factories, and builders.
Bundle initialization and metricsEnabled behavior.
Metrics integration tests and migration coverage.

Please pay particular attention to:

Global registry behavior when multiple Ranger bundles share one process.
Gauge registration/reinitialization and concurrent registry access.
Metric cardinality for service names, upstream IDs, event names, and exception classes.
Distinction between empty responses, transport failures, parse failures, and stale-data retention.
Backward compatibility of public constructors, extension hooks, custom sources/sinks, and YAML configuration.
HTTP activity now follows HttpCommunicator.healthy() instead of always reporting active.

Additional Changes

Bumped project version from 2.0.0-RC6 to 2.0.0-RC7 and documented migration in CHANGELOG.md.
Included Sonar-oriented cleanup and test reliability changes.
Updated ID-generator performance snapshots, changed one invalid-state exception to IllegalStateException, and added Sonar suppressions in ID model classes.
Updated server-bundle README examples from Guava list creation to List.of(...).

…ailability

…ng instead of a list

…r-fix [Fix] Collision Check concurrency handling

…nanoseconds-fix [Fix] Prevent duplicate IDs by switching to monotonic clock (nanoTime)

Version Bump

…d classes

…rding

…ce metrics

… recording

…ding

…ependency

… metrics recording

…ceRegistryUpdaterMetricsIntegrationTest

…date log messages for clarity

ToOnlyGaurav

Please go through comments

Let's call metric_id to be something meaningful like upstream_id or dataSourceId etc.

ToOnlyGaurav · 2026-06-02T07:10:27Z

 @SuperBuilder
 public abstract class AbstractRangerHubClient<T, R extends ServiceRegistry<T>, D extends Deserializer<T>> implements RangerHubClient<T,R> {

+    private final String metricId;


Let's keep it generic name closer to functional. Keeping non-functional attribute with functional group looks a bit off

renamed to upstreamId

ToOnlyGaurav · 2026-06-02T07:13:43Z

+
+        // Wait for initial update and a brief period for metric recording to complete
+        awaitRefresh(registry);
+        sleep(100); // Allow time for MetricRecorder call after updateNodes


Possible to use any synchronisation mechanism which gives better guarantees than sleep. Most of the time these tests fails if we have slow environments.

Using awaitility now instead of sleep to check periodically with few ms wait

ToOnlyGaurav · 2026-06-02T07:15:59Z

            List<ServiceNode<T>> nodes = new ArrayList<>(children.size());
            log.debug("Found {} nodes for [{}]", children.size(), serviceName);
+            if(children.isEmpty()){
+                MetricRecorder.recordNullOrEmptyListNodeResponse(DataStoreType.ZK, metricId);


Can we have wrapper method for these two.

extracted new method recordNullOrEmptyResponse to encapsulate both methods

ToOnlyGaurav · 2026-06-02T08:38:42Z

+  public static final String STALE_DATA_RETAINED = "staleDataRetained";
+  public static final String ZK_READ = "zkRead";
+
+  private static MetricRegistry metricRegistry = new MetricRegistry();


new MetricRegistry(); is of no use.

ToOnlyGaurav · 2026-06-02T08:46:13Z

+
+  public static void recordNodeDataSinkUpdateStatus(DataStoreType dataStoreType, String metricId, String status) {
+    if (metricRegistry != null) {
+      metricRegistry.meter(MetricRegistry.name(PACKAGE_PREFIX, DATA_SOURCE, dataStoreType.name(),


Looks a bit confusion when read with line#110

Resolved by using DATA_STORE_TYPE, dataStoreType.name(), DATA_SOURCE, upstreamId in all MetricRecorder methods

ToOnlyGaurav · 2026-06-02T08:59:15Z

+                                        ? oldV
+                                        : newV))
+                .values();
+        MetricRecorder.recordServiceNodesReturned(service.getServiceName(), serviceNodes.size());


If we are getting these from upstreams, how we will figure out which upstream is returning wrong value?

Now we're pushing node count also which are fetched from any upstream in ServiceRegistryUpdater

ToOnlyGaurav · 2026-06-02T09:08:37Z

        dataLock.lock();
        try {
-            long resolvedTime = resolution.convert(timeInMillis, TimeUnit.MILLISECONDS);
+            long resolvedTime = resolution.convert(


Didn't get the need of doing this?

ToOnlyGaurav · 2026-06-02T09:09:48Z

    }

-    public boolean check(long timeInMillis, int location) {
+    public long checkAndGetTime(int location) {


This method looks overloaded from the response perspective.

ToOnlyGaurav · 2026-06-02T09:11:43Z

 * Checks collisions between ids in given period
 */
 @Slf4j
 public class CollisionChecker {


@r0goyal Please take a look.

ToOnlyGaurav · 2026-06-02T09:16:50Z

+        val serviceRegistrationResponse = registerService(service.getServiceName(), httpUrl, requestBody).orElse(null);
        if(null == serviceRegistrationResponse || !serviceRegistrationResponse.valid()){
            log.warn("Http call to {} returned a failure response {}", httpUrl, serviceRegistrationResponse);
+            MetricRecorder.recordNullOrEmptyRegisterServiceResponse(DataStoreType.HTTP, metricId, service.getServiceName());


let's shrink these two methods to one.
Don't want metrics related code to be surfaced more.

extracted to recordNullOrEmptyRegisterServiceResponse

…or improved clarity

…s; simplify metric recording

…tention

sonarqubecloud · 2026-06-02T15:39:39Z

Quality Gate passed

Issues
7 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.7% Duplication on New Code

See analysis details on SonarQube Cloud

@JsonProperty

- MatchingShardSelector: remove unused Collectors import (S1128) - HealthcheckStatus: suppress S115 - renaming healthy/unhealthy breaks 145 call sites - ServiceDiscoveryBundle: suppress S1172 on overridable protected methods - param required by subclasses - Domain: rename field 'domain' -> 'domainName' to avoid S1700 (field same name as class) - Id: rename field 'id' -> 'idValue' to avoid S1700; add @JsonProperty to preserve JSON serialization - IdGeneratorBase: replace RuntimeException with IllegalStateException (S112) - RangerDroveUtils: add explanatory comments to intentional no-op SSL methods (S1186/S4830); upgrade SSLContext from 'SSL' to 'TLSv1.2' (S4423); remove unused type param <T> (S2326); remove commented-out code block (S125) - RangerHubServerBundle: remove explicit type param from buildDroveClient call - HttpApiCommunicator: add comment to empty close() method (S1186) - PathBuilder: make REGISTERED_SERVICES_PATH static final (S1104/S1444/S3008)

@SuppressWarnings

…r suppressions - Domain.java: revert domainName->domain rename; add @SuppressWarnings(java:S1700) on class - Id.java: revert idValue->id rename and @JsonProperty; add @SuppressWarnings(java:S1700) on class - RangerDroveUtils.java: revert TLSv1.2->SSL; add @SuppressWarnings(java:S4423) on SSLContext line

…ith WireMock

…ery-dw5-bundle ServiceDiscoveryBundle

- Bump all Maven modules to 2.0.0-RC7 - Document metrics instrumentation and fixes - Call out breaking API and configuration migrations

Cap node-count reservoirs and remove unbounded service zombie metrics.\nMove standalone metric initialization behind configurable bundle lifecycle.

Replace inventory node-count histograms with latest-value gauges to reduce retained samples and update overhead. Keep service response histograms bounded.

…anch

sonarqubecloud · 2026-09-21T11:25:06Z

Quality Gate passed

Issues
9 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.0% Duplication on New Code

See analysis details on SonarQube Cloud

ToOnlyGaurav

LGTM

jitendradhawan and others added 24 commits April 8, 2026 08:48

Add metrics instrumentation for service health checks and upstream av…

35e7957

…ailability

Refactor RangerZkUpstreamConfiguration to use a single zookeeper stri…

b22adac

…ng instead of a list

[Fix] Collision Check concurrency handling

07cb447

Merge pull request appform-io#102 from himneesh9/fix/collision-checke…

79378f7

…r-fix [Fix] Collision Check concurrency handling

[Fix] Use of Nanoseconds in Id generator

ac60486

Merge pull request appform-io#103 from himneesh9/fix/collision-check-…

50dbb53

…nanoseconds-fix [Fix] Prevent duplicate IDs by switching to monotonic clock (nanoTime)

Version Bump

a59e0e1

Merge pull request appform-io#105 from himneesh9/fix/version-bump

3dddf43

Version Bump

Removed legacy nexus plugin

4945fbc

Add metrics instrumentation to service data sources and update relate…

e990fc7

…d classes

resolve merge conflicts with main branch

9b41df9

Fix the unit tests

7991b11

Add integration tests for Drove API and Node Data Source metrics reco…

85888b2

…rding

Add integration tests for HttpApiCommunicator and HttpServiceDataSour…

9fbfe1c

…ce metrics

Add integration tests for ZkNodeDataSink and ZkNodeDataSource metrics…

2cd0e22

… recording

Add integration tests for FinderUtils and HealthChecker metrics recor…

adbcf41

…ding

Update performance test mean operations for IdGenerator

59dfd9c

reverse merge with parent branch 2.x and resolve merge conflicts

101e545

remove redundant MonitoredFunction annotations and function metrics d…

bf89c32

…ependency

Refactor HealthChecker and MetricRecorder to include DataStoreType in…

3e2df50

… metrics recording

Update performance test metrics and enhance timer assertions in Servi…

647ea5a

…ceRegistryUpdaterMetricsIntegrationTest

Fix all the sonar issues

6287778

Add aggregate meters for null/empty responses and parse failures

f08051d

Enhance metrics recording for service nodes and services returned; up…

7b6941b

…date log messages for clarity

ToOnlyGaurav suggested changes Jun 2, 2026

View reviewed changes

jitendradhawan added 5 commits June 2, 2026 19:30

Refactor metrics recording to include DataStoreType in metric names f…

ac4485f

…or improved clarity

Refactor HealthChecker to remove DataStoreType and metricId parameter…

e612bff

…s; simplify metric recording

Address code review comments

6106154

Add node count metrics for service registry updates and stale data re…

8cb0427

…tention

Add unit tests for MetricRecorder methods including node count metrics

44e92ba

jitendradhawan added 14 commits July 31, 2026 09:29

fix: replace external google.com dependency in PingCheckMonitorTest w…

da7d056

…ith WireMock

fix: suppress S1172 on overridable protected methods in ranger-discov…

c8ca503

…ery-dw5-bundle ServiceDiscoveryBundle

resolve merge conflicts with sonar quality fix branch

1fedd85

fix: update DroveOkHttpTransport initialization to include config ID

cab7d28

Address code review comments

f2f60ff

fix: make metric registry reference thread-safe

6a5fdb6

Merge branch '2.x' into metrics_instrumentation

0b0b8c2

chore(release): prepare 2.0.0-RC7

12bd605

- Bump all Maven modules to 2.0.0-RC7 - Document metrics instrumentation and fixes - Call out breaking API and configuration migrations

fix(metrics): bound metric memory growth

cb8d3da

Cap node-count reservoirs and remove unbounded service zombie metrics.\nMove standalone metric initialization behind configurable bundle lifecycle.

fix(metrics): retain service zombie metrics

84ece2e

perf(metrics): use gauges for node counts

bacd039

Replace inventory node-count histograms with latest-value gauges to reduce retained samples and update overhead. Keep service response histograms bounded.

remove redundant change logs as a result of reverse merge with 2.x br…

95602b9

…anch

ToOnlyGaurav approved these changes Sep 21, 2026

View reviewed changes

jitendradhawan changed the title ~~Draft: Metrics instrumentation across all DataStoreTypes (ZK, HTTP, DROVE)~~ Metrics instrumentation across all DataStoreTypes (ZK, HTTP, DROVE) Sep 21, 2026

jitendradhawan closed this Sep 22, 2026

Conversation

jitendradhawan commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

What Changed

Metrics foundation

Instrumented flows

Data flow

Bundle and configuration wiring

Breaking Changes / Migration

HTTP upstream migration

Drove upstream migration

ZooKeeper upstream migration

Programmatic extension migration

Behavioral Notes

Operational Considerations

Dependency Impact

Testing

Reviewer Guide

Additional Changes

Uh oh!

ToOnlyGaurav left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jitendradhawan Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Jun 2, 2026

Quality Gate passed

Uh oh!

sonarqubecloud Bot commented Sep 21, 2026

Quality Gate passed

Uh oh!

ToOnlyGaurav left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

jitendradhawan commented Jun 1, 2026 •

edited

Loading

jitendradhawan Jun 3, 2026 •

edited

Loading