Skip to content

Bug Report: TAKAware-CIV — Uncontrolled Parallel Reconnection on Connection Loss #215

Description

@Humble-Helper-96

Product: TAKAware-CIV
Version Observed: 1.7.3.233
Platform: iOS / Android
TAK Server Version: 5.6-RELEASE-22-HEAD, 5.7-RELEASE-8-HEAD
Date: 2026-05-16
Severity: High — causes server-side resource exhaustion
Status: Open


Summary

When a TAKAware-CIV client loses its TLS connection to a TAK Server — whether due to network disruption, server restart, certificate revocation, or connectivity instability — the client opens a large number of simultaneous parallel reconnection attempts instead of a single sequential retry with backoff. Each attempt is accepted by TAK Server as a new subscription before the authentication/CRL check completes, resulting in dozens to hundreds of zombie sessions accumulating in the TAK Server Client Dashboard. This behavior has been observed to generate 240+ simultaneous subscriptions from a single device within a 2-second window.


Steps to Reproduce

  1. Connect a TAKAware-CIV client (v1.7.3.233) to a TAK Server 5.6+ instance via TLS on port 8089.
  2. Confirm clean single-session connection in the TAK Server Client Dashboard.
  3. Trigger a connection disruption by any of the following:
    • Restart the TAK Server (sudo systemctl restart takserver)
    • Revoke the client's certificate on the server
    • Cause network instability (e.g. cellular/WiFi handoff, brief signal loss)
  4. Observe the TAK Server Client Dashboard and takservermessaging.log.

Expected Behavior

The client should:

  1. Detect the lost connection
  2. Wait a brief backoff interval
  3. Attempt one reconnection
  4. On failure, wait again and retry (exponential backoff or fixed interval)

At no point should more than one concurrent connection attempt be in flight from the same device.


Actual Behavior

The client immediately opens multiple simultaneous TCP/TLS connections to the server. TAK Server's DistributedSubscriptionManager accepts and registers each as a new subscription before the TLS handshake and CRL check complete, resulting in a flood of entries:

2026-05-16-03:58:39.632 [epollEventLoopGroup-4-11] INFO c.b.m.s.DistributedSubscriptionManager - Added Subscription: id=tls:169 source=166.205.116.1
2026-05-16-03:58:39.633 [epollEventLoopGroup-4-3]  INFO c.b.m.s.DistributedSubscriptionManager - Added Subscription: id=tls:168 source=166.205.116.1
2026-05-16-03:58:39.689 [epollEventLoopGroup-4-15] INFO c.b.m.s.DistributedSubscriptionManager - Added Subscription: id=tls:170 source=166.205.116.1
...
[70+ additional entries within 2 seconds, all from the same source IP]

All resulting subscriptions show:

  • Last Report: 1969-12-31 14:00:00 -1000 (Unix epoch zero — never successfully reported)
  • Pending Writes: 0
  • No callsign assigned — raw tls:NNN identifiers only
  • Health indicator: orange/red (unhealthy)

The zombie subscriptions persist indefinitely until the TAK Server process is restarted. They do not time out or self-clean under default TAK Server configuration.


Impact

  • Server resource exhaustion: Each zombie subscription holds open a Netty channel, consuming file descriptors, heap, and thread pool slots.
  • Client Dashboard pollution: Administrators cannot distinguish real clients from zombie sessions.
  • Operational confusion: During active incidents, a flooded dashboard obscures situational awareness of who is actually connected.
  • Cascading effect: If the flood is large enough, TAK Server's thread pool becomes saturated, degrading performance for all connected clients.
  • Reproduced on two separate TAK Server instances (5.6 and 5.7) with the same TAKAware client.

Observed Trigger Conditions

Trigger Observed Storm?
TAK Server restart while client connected Yes
Client certificate revoked Yes
Cellular / WiFi connectivity instability Yes
Clean client disconnect (user-initiated) No

Attachments

  • TAK Server messaging log excerpt (provided above)
  • TAK Server Client Dashboard screenshots showing zombie flood
  • Client Dashboard screenshots post-restart showing clean state

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions