Skip to content

Spike: OTel disk buffer for metrics, 24 h on HyperOS, OpAMP on Android, OTLP into VictoriaMetrics #1

Closed

Description

@groupsky

Goal

Before we build the app, prove the four risky parts with a small test app. The answers decide the design of the MVP (#2).

The app's plan: an Android app that collects device health (battery, power, boot, network, storage, thermal) and sends it over OTLP to any OTLP receiver. Built on opentelemetry-android (OTLP export, disk buffering). Settings decide where, what and how often; remote config uses OpAMP.

Questions to answer, with evidence

  1. Disk buffer for metrics. Does opentelemetry-android's disk buffering keep metrics (not only spans and logs) across a network cut of at least 30 min and an app restart? If not, what is the smallest own queue in front of the OTLP metric exporter (for example Room or files)?
  2. Staying alive. Does a foreground service that reads the battery every 60 s keep running for 24 h with the screen off on a Xiaomi phone with HyperOS 2 (Android 15)? Note which phone settings were needed (Autostart, battery saver "No restrictions"). WorkManager is not enough: its shortest period is 15 min.
  3. OpAMP on Android. Does the opamp-client module of opentelemetry-java-contrib (io.opentelemetry.contrib:opentelemetry-opamp-client) run in the app and apply a config from a test server? opamp-go's example server is enough. Does the app report the effective config hash back?
  4. Receiver. Do the metrics arrive in VictoriaMetrics through its OTLP endpoint (/opentelemetry/v1/metrics)? How do metric names, units, types and resource attributes turn into series names and labels?

Done when

  • A short report docs/spike.md answers 1–4, each with the evidence (logs, a query result, screenshots of settings), and the versions used.
  • For 1: a graph or query with no gap over the network cut, and one over the app restart.
  • For 2: 24 h of samples, at most 2 min apart, with the screen off.
  • For 3: the app log shows a config received, applied (the interval changed) and its hash reported.
  • For 4: one query result per metric type (gauge, counter) with the labels shown.
  • The report ends with a decision for MVP: battery, power, boot, network, device and thermal collectors; local config; OTLP export with disk buffer #2: buffer (library or own), service model, OpAMP client yes/no.
  • The spike code is in a branch or a spike/ folder, not in the app module.

Activity

  1. groupsky commented on Nov 11, 2026

    @groupsky
    ContributorAuthor

    hq-jobs aoda-1-spike (run r-20261111-013003-a7e1)

    Spike status, 2026-11-11 (first part done on the emulator; the phone part waits for one install).

    The test app is in spike/ on branch hq-job/aoda-1-spike/r-20261111-013003-a7e1. It runs two metric pipelines side by side, so one network cut tests both: opentelemetry-android's disk buffering and a small own file queue.

    First results (Android 15 emulator):

    1. Disk buffer: opentelemetry-android 1.8.1 does write metrics to disk, but a batch whose export fails is dropped, not retried. A 200 s receiver outage left a 276 s gap. The own queue has no gap over a cut with an app restart in the middle. It is one file per export (the OTLP protobuf body), and a file is deleted only after a 2xx answer.
    2. OpAMP: the java-contrib opamp-client runs in the app over WebSocket. It got a config from the opamp-go example server, applied it (interval 20 s → 30 s) and reported APPLIED with the config hash. Over plain HTTP, the example server forgets the agent between polls.
    3. VictoriaMetrics: OTLP works with no setup. Names keep their dots, units are dropped, counters get no _total, and resource and point attributes become labels.

    Next: the tests on a Redmi Note 12S on HyperOS 2. They are a 35 min cut, a cut with a restart, the OpAMP change, and 24 h for question 2. They start when the app is installed at the phone. Follow-up jobs then finish docs/spike.md and the PR.

  2. groupsky commented on Nov 11, 2026

    @groupsky
    ContributorAuthor

    Phone tests moved to a OnePlus Nord CE 2 Lite on OxygenOS (Android 12) (2026-11-11). The 24 h test on HyperOS 2 moves to #2.

    Findings from the setup, before the tests started:

    • adb install worked with no tap on this phone.
    • With default settings, OxygenOS froze the app about 2 min after start, even with a foreground service (OplusHansManager: freeze uid … scene: LcdOn). Over the next hour it made only 4 samples, 25–30 min apart. Adding the app to the Doze allow-list (cmd deviceidle whitelist +…) did not stop the freezing. What worked: App info → Battery → Allow background activity (a tap in the phone's settings).
    • A VPN with per-app split tunneling (here Tailscale in "include only selected apps" mode) sent the app's traffic outside the VPN, so the receiver could not be reached. The app's own file queue kept all batches and sent them as soon as the app was added to the VPN.
    • Both belong in MVP: battery, power, boot, network, device and thermal collectors; local config; OTLP export with disk buffer #2's setup checklist: "allow background activity" on OnePlus/OPPO, and "if you use a VPN with split tunneling, include this app".

    Tests (OpAMP change, 35 min cut, cut with a restart, 24 h) started 2026-11-11 08:44 UTC.

  3. groupsky commented on Nov 11, 2026

    @groupsky
    ContributorAuthor

    hq-jobs aoda-1-check (run r-20261111-160004-c9bd)

    Spike follow-up 1 (2026-11-11): Q1, Q3 and Q4 are answered in docs/spike.md (branch hq-job/aoda-1-spike/r-20261111-013003-a7e1, commit 8a86bf3).

  4. groupsky commented on Nov 12, 2026

    @groupsky
    ContributorAuthor

    hq-jobs aoda-1-final (run r-20261112-114504-2326)

    Spike done: the report is in docs/spike.md (PR #6, merged 2026-11-12).

    Answers

    1. Disk buffer: opentelemetry-android keeps metrics over a 35 min cut, but loses everything not yet sent when the app restarts during a cut (15 min gap). A small own file queue (one OTLP body per file, deleted after a 2xx) kept every point.
    2. Staying alive (OnePlus Nord CE 2 Lite, OxygenOS, Android 12, 2026-11-11 08:43 → 2026-11-12 08:43 UTC): the foreground service lived the full 24 h with "Allow background activity" on, and was never killed. But the 60 s tick was not kept on battery with the screen off: 145 gaps over 120 s (at most 196 s), 22–30 ticks per hour in the worst hours. On the charger every tick came on time. The likely cause is the Handler.postDelayed timer, which stops while the CPU sleeps. So the "at most 2 min apart" goal is not met with the spike code. The fix (exact alarm or a wake lock) and the HyperOS 24 h test move to MVP: battery, power, boot, network, device and thermal collectors; local config; OTLP export with disk buffer #2.
    3. OpAMP: the java-contrib opamp-client works on Android over WebSocket: config received, applied (interval 60 → 30 → 60 s), hash reported as APPLIED.
    4. VictoriaMetrics: OTLP arrives. Names keep their dots, units and types are dropped, and every resource attribute becomes a label (drop app.installation.id).

    Decision for #2: own file queue (with a size cap); one foreground service whose timer wakes the CPU; OpAMP client yes.

    Not done: the final pull of the phone's samples.csv and of the phone state at the end. The phone's wireless debugging port had changed, and no new port came in 90 min. The 24 h result above comes from the receiver's raw points.

  5. groupsky commented on Nov 12, 2026

    @groupsky
    ContributorAuthor

    Phone-side check (2026-11-12, by hand after the final job)

    The phone's own samples.csv is now pulled. It agrees with the receiver's points:

    • 24 h window (2026-11-11 08:43 → 2026-11-12 08:43 UTC): 1170 samples, screen off 19.2 h of 24 h, 147 gaps over 120 s (receiver: 145), largest 196 s, all with the screen off. One process start besides the first, at 09:49 UTC: the planned force-stop of the restart cut. So no kill by the system.
    • Same pattern continues to 12:59 UTC (28.3 h, 182 gaps, max 199 s).
    • Phone state at the end: standby bucket 5 (exempt), on the Doze allow-list, RUN_ANY_IN_BACKGROUND: allow, foreground service still running.
    • The own queue holds 59 files. All are from after 11:28 UTC, when the test receiver was removed. So nothing was stuck before that.

    The decision for #2 stands: a timer that wakes the CPU (exact alarm or wake lock).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions