Skip to content

fix(nodeagent): forget a log stream whose directory never appeared - #2471

Open
acodercat wants to merge 3 commits into
opensandbox-group:mainfrom
acodercat:fix/nodeagent-prune-unadopted-streams
Open

acodercat wants to merge 3 commits into
opensandbox-group:mainfrom
acodercat:fix/nodeagent-prune-unadopted-streams

Conversation

@acodercat

@acodercat acodercat commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

The container-logs source adopts a stream only once it can watch the container's log directory. That is when coverage_started_at is set, and a stream can only end after it has been adopted. Pruning (pruneEndedRuntime) only looks at streams that have ended, been acknowledged and passed their repair deadline.

When a sandbox Pod is deleted before kubelet creates the container's log directory, the first scan creates the stream and persists it to bbolt, but the stream is never adopted. So it never ends and is never pruned:

  • the runtime stays in s.streams;
  • the bbolt record stays;
  • the resource stays in the store, because Forget is only called on prune;
  • every reconcile scans the stream again.

After a restart, restoreStreams loads it from bbolt and the same thing continues. TestMissingPodDirectoryDefersCoverageUntilLeafWatch shows the deliberate part: a terminated stream must not end just because the directory is missing, since kubelet may still create it (OSEP-0019: directory disappearance does not prove the end). But nothing ever lets the stream go.

Summary

  • A terminated stream that has not ended, was never adopted and has read nothing becomes prunable once NODEAGENT_ENDED_STATE_RETENTION has passed since it was first seen in that state, provided its log directory still does not exist. It then goes through the existing prune path: the runtime is dropped and the store resource is forgotten.
  • Its bbolt record is deleted with every sink, including the durable-file sink. That sink normally keeps ended state for its own file cleanup, but the cleanup only looks at ended streams and a never-adopted stream sent nothing, so there is nothing for it to collect.
  • If kubelet creates the directory within the window, the stream is adopted and continues as before. Adopted streams are not affected by this change.
  • Nothing was observed, so no StreamEnd or marker is emitted. This keeps the rule that data the agent could not observe is never claimed as covered.
  • The deadline is saved on the source stream as an optional unadopted_deadline the first time the stream is seen this way, so restarts do not start it over. It is cleared when the stream is adopted. The field is optional, so the schema version stays the same and older records simply start the window on the next scan.
  • TestMissingPodDirectoryDefersCoverageUntilLeafWatch now sets EndedStateRetention. It used the zero value, which the agent never runs with (config validation requires a positive duration), and with it the stream would be forgotten on the first scan.

Testing

  • Not run (explain why)

  • Unit tests

  • Integration tests

  • e2e / manual verification

  • New table test, TestNeverAdoptedTerminatedStreamIsPrunedAfterRetention. Cases: within the retention window, after it, after it with the durable-file sink, after it across a restart, after it but with the directory created, an active resource, an adopted stream, and a stream that read something before adoption. Only the three "after retention" cases are pruned; they check that the bbolt record, the runtime and the store resource are gone and that no event is emitted. The directory case also checks that adoption clears the saved deadline. The test fails on current main.

  • Mutation checks: removing the wait, the terminated check, the adoption check, the progress check, saving the deadline, deleting state for the durable-file sink, or clearing the deadline on adoption each fails a case.

  • I verified that TestMissingPodDirectoryDefersCoverageUntilLeafWatch fails without its new retention setting and passes with it.

  • make fmt-check, make vet, golangci-lint (no new findings), the Windows and macOS builds and make integration-compile are clean. My workstation is out of inotify instances, so I ran go test ./... for nodeagent in a container; every package passes, including the watcher-based tests.

Breaking Changes

  • None
  • Yes (describe impact and migration path)

Checklist

  • Linked Issue or clearly described motivation
  • Added/updated docs (if needed)
  • Added/updated tests (if needed)
  • Security impact considered
  • Backward compatibility considered

A container-logs stream is adopted only once its log directory can be
watched, and it can end only after adoption. When a Pod is deleted
before kubelet creates the container's log directory, the stream is
created and persisted on the first scan but never adopted, so it never
ends, and pruning only considers ended streams. Its runtime, its bbolt
record and its store resource were kept forever, and every reconcile
scanned it again. A restart restored it from bbolt and the loop
continued.

Treat a terminated stream that was never adopted as prunable once the
ended-state retention has passed since it was first seen that way and
its log directory still does not exist. The wait keeps the existing
rule that a missing directory does not prove the end, since kubelet may
still create it; if it does, the stream is adopted as before. Nothing
was observed, so no StreamEnd or marker is emitted. The start of the
wait is kept in memory, so a restart only restarts the wait.

TestMissingPodDirectoryDefersCoverageUntilLeafWatch now sets a
retention: it used the zero value, which the agent never runs with
(the setting must be positive), and with it the stream would be
forgotten on the first scan.
@github-actions github-actions Bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Oct 29, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T02:50:50.705273Z 2e22971 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2e22971512

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

pruneEndedRuntime computed the ended-stream condition and then
overwrote it for never-adopted streams, which read as if one rule
replaced the other. Move the never-adopted rule into
unadoptedExpiredLocked and combine the two conditions explicitly.
The deadline for forgetting a stream whose log directory never appeared
lived only in memory, so every restart started it over. It is now saved
with the source stream the first time the stream is seen that way, and
cleared once the stream is adopted.

The state of such a stream is also deleted when the durable-file sink is
in use. That sink keeps ended state for its own file cleanup, which only
looks at ended streams, so a never-adopted record would otherwise stay in
bbolt forever. Streams that already read something before adoption are
left alone, since that data may have reached a sink.
@github-actions github-actions Bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 29, 2026

@Pangjiping Pangjiping left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants