Script Runner cannot reconnect after ~8-12 min in some deployments

[jalexanderqed]

Describe the bug

• A Python script that has been running for roughly 8-12 minutes can no longer be
  reconnected to from Script Runner. The script keeps executing, but none of
  its output is visible in the editor, and it is not possible to interact
  with prompts.
• 7.2.0 symptom: spinner ("Connecting...") forever.
• 7.3.0 symptom: connect succeeds, editor is blank (occasionally a green
  current-line marker on empty space), logs only appear if the script was
  started recently; prompts appear if joining before the ~8-12 min threshold
  and stop appearing after that threshold passes.
  • The blank editor portion of this issue may have been fixed in 7.4.0.

To Reproduce

• I observed this bug on an OpenC3 instance deployed in Kubernetes
  with a high-latency connection between the OpenC3 server and the web client.
• The issue reproduces reliably in the environment above on v7.3.0, but I have not
  reproduced it in a Docker environment, and I have not managed to update
  our Kubernetes-deployed version of OpenC3 to v7.4.0 yet.
• Trigger: from a script that has been running for at least 8-12 minutes,
  reload the page, then Script -> Execution Status -> Running Scripts
  -> Connect. This is a completely fresh subscribe on a fresh WebSocket.
• The issue reproduces even with a script that only calls wait(...) and produces no
  logging.

Expected behavior

After clicking "Connect," the script runner displays the currently execute line, the logs, and any prompts from the target script.

Screenshots

No response

OS

Windows 11

OpenC3 COSMOS Version

7.3.0

Browser

Firefox

[jalexanderqed]

Additional Details and Hypothesis

Example Script:

import os
import time

start = time.time()

while time.time() < start + 3600 * 24:
  wait(0.5)
  print(f"slept for {time.time() - start}s so far. {os.urandom(1000)}")

Hypothesis

The two anycable-go processes are configured asymmetrically:

openc3-cosmos-cmd-tlm-api/Procfile:3
anycable-go ... --max_message_size=1000000 --ws_max_pending_size=50000000

openc3-cosmos-script-runner-api/Procfile:3
anycable-go ... (no size overrides)

The CMD-TLM Procfile was patched in OpenC3/cosmos PR #2938 to raise
ws_max_pending_size to 50 MB. The same override was never applied to the
script-runner Procfile.

Without the override, the script-runner cable falls back to anycable-go's
compiled-in default from ws/config.go:

MaxMessageSize: 65536,          // 64 KB (incoming, from client)
MaxPendingSize: 1024 * 1024,    // 1 MB (outgoing write queue)

MaxPendingSize is a per-session cap on outstanding bytes in the
WebSocket write queue. From node/session.go, sendFrame:

switch s.writeQueue.Push(item, maxPending) {
case utils.PushOverflow:
    s.maxPendingSize.Store(0)
    s.Log.Debug("slow client detected, disconnecting", "max", maxPending)
    s.metrics.CounterIncrement(metricsDisconnectedSlowClients)
    s.DisconnectWithMessage(
        common.NewDisconnectMessage(common.SLOW_CLIENT_REASON, true),
        common.SLOW_CLIENT_REASON,
    )
}

When a Push exceeds MaxPendingSize the session is dropped with
SLOW_CLIENT_REASON.

How the backlog crosses 1 MB in 8-12 min for a wait-only Python script

7.2.0 subscribe path (openc3-cosmos-script-runner-api/app/channels/
running_script_channel.rb:48):

OpenC3::Topic.xrange(topic, '-', '+').each do |msg_id, msg_hash|
  ...
  transmit(event)
end

The whole backlog is drained inside one RPC. Anycable-rails-core
accumulates every transmit() into CommandResponse.transmissions, and
anycable-go receives one gRPC response containing all of them. It then
iterates them into sendFrame in a tight loop, faster than the WebSocket
drain goroutine can push frames out.

Each server-pushed frame is not just the OpenC3 event payload. The
ActionCable-protocol wrapper echoes the client's subscription identifier
on every server message:

{"identifier":"{\"channel\":\"RunningScriptChannel\","
               "\"id\":\"42\","
               "\"token\":\"<big JWT>\"}",
 "message":<event>}

openc3-cosmos-init/plugins/packages/openc3-js-common/src/services/cable.js:49
puts token: localStorage.openc3Token into the subscription params. On
Enterprise / Keycloak deployments (BRME's case) that is a JWT, typically
1.5-2 KB of base64. Every server-pushed frame carries that identifier
JSON, so per-frame overhead is ~1.6-2 KB before you count the event
payload itself.

Arithmetic for wait(3600) (running_script.py:98 emits one :line state=
waiting event every ~1 s from within _openc3_script_sleep):

Frame size ~= 1.6 KB (identifier) + 100 B (payload) ~= 1.7 KB
MaxPendingSize / frame size = 1 MB / 1.7 KB ~= 588 frames
588 frames at 1/sec ~= 9.8 minutes

That falls squarely inside the observed 8-12 min window. The exact minute
depends on the JWT length and the initial :file event's script text size:
a larger script file drags the crossing point earlier, still inside the
same window.

Sequence when Connect is clicked at ~10 min:

1. Fresh page's WS opens, fresh subscription is created.
2. Server RunningScriptChannel#subscribed runs, xrange returns the
   entire backlog, transmit() is called for each entry.
3. anycable-rails-core bundles them into one CommandResponse.
4. anycable-go receives it, iterates transmissions into sendFrame.
5. The write queue fills faster than it drains. Total queued bytes
   crosses 1 MB.
6. sendFrame hits PushOverflow. Session is dropped with
   SLOW_CLIENT_REASON.

Why 7.2.0 stays on a spinner

7.2.0's openc3-js-common/services/cable.js (see git show v7.2.0:openc3-cosmos-init/plugins/packages/openc3-js-common/src/services/ cable.js) has no ResilientSubscription. It just calls
this._cable.subscriptions.create(...). When the WS is dropped mid-
subscribe:

• The subscription's connected callback never fires.
• ScriptRunner.vue:1878 set this.state = 'Connecting...' in
  initScriptStart. Nothing overwrites it.
• The user sees the spinner forever.

Why 7.3.0 shows the "blank editor + occasional cursor" symptom

7.3.0 shipped the fixes from PRs #3610 (async races, arm/ready
mechanism) and #3645 (ResilientSubscription cable.js rewrite and the
per-script subscription_key). The ResilientSubscription catches the
disconnect via the disconnected(data) callback with
data.allowReconnect === false, tears down the consumer, and re-
subscribes on a fresh consumer.

But the re-subscribe hits the same wall: server ships the same backlog,
anycable-go queues the same ~1 MB, may or may not drain fast enough this
round. If the disconnect happens after the initial :file event's bytes
have gone into the pending queue but before they're drained on the wire,
the client ends up with some of the :line frames but not the file text.

processLine at ScriptRunner.vue:2039 uses the :line events to set
this.state and place a marker, but it only calls editor.setValue(...)
when this.files[data.filename] is already populated. Without the :file
frame that population never happens — hence blank editor with an
occasional green marker on empty space. handleScript for prompts also
depends on that same file context, which is why prompts stop appearing
past the threshold.

For "young" scripts (before the backlog crosses 1 MB) the backlog fits
without overflow, everything is delivered, and the tool works normally —
matching the observation that recently-started scripts still connect
cleanly on 7.3.0.

[crawlchat]

Likely Root Cause

Based on the replay mechanism, this looks like a Redis stream TTL/expiration issue, not a websocket/gRPC transport bug (that's issue #3240, which is different).

The comments in running_script_replay_thread.rb state the per-script Redis stream (running-script-channel:<id>:replay) is explicitly "capped + short TTL" . When a client subscribes:

1. RunningScriptChannel#subscribed does an xrange over that stream to replay the backlog (current line, logs, prompts) via transmit().
2. If the script has been idle (e.g., in a long wait()) and hasn't written any new events recently, the initial state events (current-line, output) written near script start can age out of the stream's TTL before a client reconnects.
3. Since nothing new is being written while the script waits, there's nothing to "refresh" that expired backlog — a fresh subscribe gets an empty xrange, hence blank editor / no current line / no prompt, even though the script process is still running fine.

This matches your ~8-12 minute threshold and the fact that it reproduces even with a no-logging wait(...) script — a bare wait doesn't emit any events to keep the stream/TTL alive.

What I couldn't confirm (need engineer follow-up)

I hit tool limits before locating:

• The exact TTL/MAXLEN value configured for the :replay stream on the writer side (running_script.rb/running_script.py).
• Whether prompts specifically are re-emitted periodically (which would explain why prompts still worked right up to the threshold, then stopped).

Could you confirm: are you running default OpenC3 config, or have you customized any Redis/Valkey maxmemory/eviction policy in your Kubernetes deployment? That would help confirm whether this is stream TTL expiration vs. a memory-pressure eviction, and would help an engineer pinpoint the exact fix (e.g., extending TTL or periodic heartbeat events).

[jalexanderqed]

I have opened PR #3905 with a fix for this bug.

[jmthomas]

Your fix will help but we should probably not keep the entire script history and send that to the front-end. We should investigate further and fix the root cause.