You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Spike: OTel disk buffer for metrics, 24 h on HyperOS, OpAMP on Android, OTLP into VictoriaMetrics #1
Before we build the app, prove the four risky parts with a small test app. The answers decide the design of the MVP (#2).
The app's plan: an Android app that collects device health (battery, power, boot, network, storage, thermal) and sends it over OTLP to any OTLP receiver. Built on opentelemetry-android (OTLP export, disk buffering). Settings decide where, what and how often; remote config uses OpAMP.
Questions to answer, with evidence
Disk buffer for metrics. Does opentelemetry-android's disk buffering keep metrics (not only spans and logs) across a network cut of at least 30 min and an app restart? If not, what is the smallest own queue in front of the OTLP metric exporter (for example Room or files)?
Staying alive. Does a foreground service that reads the battery every 60 s keep running for 24 h with the screen off on a Xiaomi phone with HyperOS 2 (Android 15)? Note which phone settings were needed (Autostart, battery saver "No restrictions"). WorkManager is not enough: its shortest period is 15 min.
OpAMP on Android. Does the opamp-client module of opentelemetry-java-contrib (io.opentelemetry.contrib:opentelemetry-opamp-client) run in the app and apply a config from a test server? opamp-go's example server is enough. Does the app report the effective config hash back?
Receiver. Do the metrics arrive in VictoriaMetrics through its OTLP endpoint (/opentelemetry/v1/metrics)? How do metric names, units, types and resource attributes turn into series names and labels?
Done when
A short report docs/spike.md answers 1–4, each with the evidence (logs, a query result, screenshots of settings), and the versions used.
For 1: a graph or query with no gap over the network cut, and one over the app restart.
For 2: 24 h of samples, at most 2 min apart, with the screen off.
For 3: the app log shows a config received, applied (the interval changed) and its hash reported.
For 4: one query result per metric type (gauge, counter) with the labels shown.
Spike status, 2026-11-11 (first part done on the emulator; the phone part waits for one install).
The test app is in spike/ on branch hq-job/aoda-1-spike/r-20261111-013003-a7e1. It runs two metric pipelines side by side, so one network cut tests both: opentelemetry-android's disk buffering and a small own file queue.
First results (Android 15 emulator):
Disk buffer: opentelemetry-android 1.8.1 does write metrics to disk, but a batch whose export fails is dropped, not retried. A 200 s receiver outage left a 276 s gap. The own queue has no gap over a cut with an app restart in the middle. It is one file per export (the OTLP protobuf body), and a file is deleted only after a 2xx answer.
OpAMP: the java-contrib opamp-client runs in the app over WebSocket. It got a config from the opamp-go example server, applied it (interval 20 s → 30 s) and reported APPLIED with the config hash. Over plain HTTP, the example server forgets the agent between polls.
VictoriaMetrics: OTLP works with no setup. Names keep their dots, units are dropped, counters get no _total, and resource and point attributes become labels.
Next: the tests on a Redmi Note 12S on HyperOS 2. They are a 35 min cut, a cut with a restart, the OpAMP change, and 24 h for question 2. They start when the app is installed at the phone. Follow-up jobs then finish docs/spike.md and the PR.
Phone tests moved to a OnePlus Nord CE 2 Lite on OxygenOS (Android 12) (2026-11-11). The 24 h test on HyperOS 2 moves to #2.
Findings from the setup, before the tests started:
adb install worked with no tap on this phone.
With default settings, OxygenOS froze the app about 2 min after start, even with a foreground service (OplusHansManager: freeze uid … scene: LcdOn). Over the next hour it made only 4 samples, 25–30 min apart. Adding the app to the Doze allow-list (cmd deviceidle whitelist +…) did not stop the freezing. What worked: App info → Battery → Allow background activity (a tap in the phone's settings).
A VPN with per-app split tunneling (here Tailscale in "include only selected apps" mode) sent the app's traffic outside the VPN, so the receiver could not be reached. The app's own file queue kept all batches and sent them as soon as the app was added to the VPN.
Spike follow-up 1 (2026-11-11): Q1, Q3 and Q4 are answered in docs/spike.md (branch hq-job/aoda-1-spike/r-20261111-013003-a7e1, commit 8a86bf3).
Q1 disk buffer: over a 35 min network cut, both pipelines kept every point. Over a cut with an app restart, the opentelemetry-android disk buffer lost 15 min (gap of 895 s). The own file queue (one OTLP/HTTP body per file, about 120 lines) lost nothing.
Q3 OpAMP: the opamp-client (1.62.1-alpha, WebSocket) received interval_seconds: 30 and then 60. The app changed its interval 60 → 30 → 60 and reported APPLIED with the sha256 of each config.
Q2 early sign: the service is still alive. But since about 10:50 UTC, with the screen off, the points come every 120–200 s instead of 60 s. The tick uses Handler.postDelayed without a wake lock. If the phone's own samples show the same, the 24 h test misses the "at most 2 min apart" goal, and MVP: battery, power, boot, network, device and thermal collectors; local config; OTLP export with disk buffer #2 needs a wake lock or an exact alarm. Q2 and the decision come in the final follow-up.
Spike done: the report is in docs/spike.md (PR #6, merged 2026-11-12).
Answers
Disk buffer: opentelemetry-android keeps metrics over a 35 min cut, but loses everything not yet sent when the app restarts during a cut (15 min gap). A small own file queue (one OTLP body per file, deleted after a 2xx) kept every point.
Staying alive (OnePlus Nord CE 2 Lite, OxygenOS, Android 12, 2026-11-11 08:43 → 2026-11-12 08:43 UTC): the foreground service lived the full 24 h with "Allow background activity" on, and was never killed. But the 60 s tick was not kept on battery with the screen off: 145 gaps over 120 s (at most 196 s), 22–30 ticks per hour in the worst hours. On the charger every tick came on time. The likely cause is the Handler.postDelayed timer, which stops while the CPU sleeps. So the "at most 2 min apart" goal is not met with the spike code. The fix (exact alarm or a wake lock) and the HyperOS 24 h test move to MVP: battery, power, boot, network, device and thermal collectors; local config; OTLP export with disk buffer #2.
OpAMP: the java-contrib opamp-client works on Android over WebSocket: config received, applied (interval 60 → 30 → 60 s), hash reported as APPLIED.
VictoriaMetrics: OTLP arrives. Names keep their dots, units and types are dropped, and every resource attribute becomes a label (drop app.installation.id).
Decision for #2: own file queue (with a size cap); one foreground service whose timer wakes the CPU; OpAMP client yes.
Not done: the final pull of the phone's samples.csv and of the phone state at the end. The phone's wireless debugging port had changed, and no new port came in 90 min. The 24 h result above comes from the receiver's raw points.
Phone-side check (2026-11-12, by hand after the final job)
The phone's own samples.csv is now pulled. It agrees with the receiver's points:
24 h window (2026-11-11 08:43 → 2026-11-12 08:43 UTC): 1170 samples, screen off 19.2 h of 24 h, 147 gaps over 120 s (receiver: 145), largest 196 s, all with the screen off. One process start besides the first, at 09:49 UTC: the planned force-stop of the restart cut. So no kill by the system.
Same pattern continues to 12:59 UTC (28.3 h, 182 gaps, max 199 s).
Phone state at the end: standby bucket 5 (exempt), on the Doze allow-list, RUN_ANY_IN_BACKGROUND: allow, foreground service still running.
The own queue holds 59 files. All are from after 11:28 UTC, when the test receiver was removed. So nothing was stuck before that.
The decision for #2 stands: a timer that wakes the CPU (exact alarm or wake lock).
Activity
groupsky commentedon Nov 11, 2026
hq-jobs
aoda-1-spike(runr-20261111-013003-a7e1)Spike status, 2026-11-11 (first part done on the emulator; the phone part waits for one install).
The test app is in
spike/on branchhq-job/aoda-1-spike/r-20261111-013003-a7e1. It runs two metric pipelines side by side, so one network cut tests both: opentelemetry-android's disk buffering and a small own file queue.First results (Android 15 emulator):
opamp-clientruns in the app over WebSocket. It got a config from the opamp-go example server, applied it (interval 20 s → 30 s) and reportedAPPLIEDwith the config hash. Over plain HTTP, the example server forgets the agent between polls._total, and resource and point attributes become labels.Next: the tests on a Redmi Note 12S on HyperOS 2. They are a 35 min cut, a cut with a restart, the OpAMP change, and 24 h for question 2. They start when the app is installed at the phone. Follow-up jobs then finish
docs/spike.mdand the PR.groupsky commentedon Nov 11, 2026
Phone tests moved to a OnePlus Nord CE 2 Lite on OxygenOS (Android 12) (2026-11-11). The 24 h test on HyperOS 2 moves to #2.
Findings from the setup, before the tests started:
adb installworked with no tap on this phone.OplusHansManager: freeze uid … scene: LcdOn). Over the next hour it made only 4 samples, 25–30 min apart. Adding the app to the Doze allow-list (cmd deviceidle whitelist +…) did not stop the freezing. What worked: App info → Battery → Allow background activity (a tap in the phone's settings).Tests (OpAMP change, 35 min cut, cut with a restart, 24 h) started 2026-11-11 08:44 UTC.
Spike report: Q1, Q3, Q4 answers with evidence (#1)
groupsky commentedon Nov 11, 2026
hq-jobs
aoda-1-check(runr-20261111-160004-c9bd)Spike follow-up 1 (2026-11-11): Q1, Q3 and Q4 are answered in docs/spike.md (branch
hq-job/aoda-1-spike/r-20261111-013003-a7e1, commit 8a86bf3).opamp-client(1.62.1-alpha, WebSocket) receivedinterval_seconds: 30and then60. The app changed its interval 60 → 30 → 60 and reportedAPPLIEDwith the sha256 of each config._totaland no unit suffix. Units and types are dropped. Every resource attribute becomes a label, and that includes the library'sapp.installation.id, which MVP: battery, power, boot, network, device and thermal collectors; local config; OTLP export with disk buffer #2 should drop.Handler.postDelayedwithout a wake lock. If the phone's own samples show the same, the 24 h test misses the "at most 2 min apart" goal, and MVP: battery, power, boot, network, device and thermal collectors; local config; OTLP export with disk buffer #2 needs a wake lock or an exact alarm. Q2 and the decision come in the final follow-up.Spike: OTel disk buffer, staying alive, OpAMP, VictoriaMetrics (#1) (#6)
groupsky commentedon Nov 12, 2026
hq-jobs
aoda-1-final(runr-20261112-114504-2326)Spike done: the report is in
docs/spike.md(PR #6, merged 2026-11-12).Answers
Handler.postDelayedtimer, which stops while the CPU sleeps. So the "at most 2 min apart" goal is not met with the spike code. The fix (exact alarm or a wake lock) and the HyperOS 24 h test move to MVP: battery, power, boot, network, device and thermal collectors; local config; OTLP export with disk buffer #2.opamp-clientworks on Android over WebSocket: config received, applied (interval 60 → 30 → 60 s), hash reported as APPLIED.app.installation.id).Decision for #2: own file queue (with a size cap); one foreground service whose timer wakes the CPU; OpAMP client yes.
Not done: the final pull of the phone's
samples.csvand of the phone state at the end. The phone's wireless debugging port had changed, and no new port came in 90 min. The 24 h result above comes from the receiver's raw points.groupsky commentedon Nov 12, 2026
Phone-side check (2026-11-12, by hand after the final job)
The phone's own
samples.csvis now pulled. It agrees with the receiver's points:RUN_ANY_IN_BACKGROUND: allow, foreground service still running.The decision for #2 stands: a timer that wakes the CPU (exact alarm or wake lock).