Repository navigation
Conversation
A container-logs stream is adopted only once its log directory can be watched, and it can end only after adoption. When a Pod is deleted before kubelet creates the container's log directory, the stream is created and persisted on the first scan but never adopted, so it never ends, and pruning only considers ended streams. Its runtime, its bbolt record and its store resource were kept forever, and every reconcile scanned it again. A restart restored it from bbolt and the loop continued. Treat a terminated stream that was never adopted as prunable once the ended-state retention has passed since it was first seen that way and its log directory still does not exist. The wait keeps the existing rule that a missing directory does not prove the end, since kubelet may still create it; if it does, the stream is adopted as before. Nothing was observed, so no StreamEnd or marker is emitted. The start of the wait is kept in memory, so a restart only restarts the wait. TestMissingPodDirectoryDefersCoverageUntilLeafWatch now sets a retention: it used the zero value, which the agent never runs with (the setting must be positive), and with it the stream would be forgotten on the first scan.
acodercat
requested review from
Pangjiping,
hittyt,
jwx0925 and
ninan-nn
as code owners
October 29, 2026 02:45
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2e22971512
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
pruneEndedRuntime computed the ended-stream condition and then overwrote it for never-adopted streams, which read as if one rule replaced the other. Move the never-adopted rule into unadoptedExpiredLocked and combine the two conditions explicitly.
The deadline for forgetting a stream whose log directory never appeared lived only in memory, so every restart started it over. It is now saved with the source stream the first time the stream is seen that way, and cleared once the stream is adopted. The state of such a stream is also deleted when the durable-file sink is in use. That sink keeps ended state for its own file cleanup, which only looks at ended streams, so a never-adopted record would otherwise stay in bbolt forever. Streams that already read something before adoption are left alone, since that data may have reached a sink.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The container-logs source adopts a stream only once it can watch the container's log directory. That is when
coverage_started_atis set, and a stream can only end after it has been adopted. Pruning (pruneEndedRuntime) only looks at streams that have ended, been acknowledged and passed their repair deadline.When a sandbox Pod is deleted before kubelet creates the container's log directory, the first scan creates the stream and persists it to bbolt, but the stream is never adopted. So it never ends and is never pruned:
s.streams;Forgetis only called on prune;After a restart,
restoreStreamsloads it from bbolt and the same thing continues.TestMissingPodDirectoryDefersCoverageUntilLeafWatchshows the deliberate part: a terminated stream must not end just because the directory is missing, since kubelet may still create it (OSEP-0019: directory disappearance does not prove the end). But nothing ever lets the stream go.Summary
NODEAGENT_ENDED_STATE_RETENTIONhas passed since it was first seen in that state, provided its log directory still does not exist. It then goes through the existing prune path: the runtime is dropped and the store resource is forgotten.unadopted_deadlinethe first time the stream is seen this way, so restarts do not start it over. It is cleared when the stream is adopted. The field is optional, so the schema version stays the same and older records simply start the window on the next scan.TestMissingPodDirectoryDefersCoverageUntilLeafWatchnow setsEndedStateRetention. It used the zero value, which the agent never runs with (config validation requires a positive duration), and with it the stream would be forgotten on the first scan.Testing
Not run (explain why)
Unit tests
Integration tests
e2e / manual verification
New table test,
TestNeverAdoptedTerminatedStreamIsPrunedAfterRetention. Cases: within the retention window, after it, after it with the durable-file sink, after it across a restart, after it but with the directory created, an active resource, an adopted stream, and a stream that read something before adoption. Only the three "after retention" cases are pruned; they check that the bbolt record, the runtime and the store resource are gone and that no event is emitted. The directory case also checks that adoption clears the saved deadline. The test fails on current main.Mutation checks: removing the wait, the terminated check, the adoption check, the progress check, saving the deadline, deleting state for the durable-file sink, or clearing the deadline on adoption each fails a case.
I verified that
TestMissingPodDirectoryDefersCoverageUntilLeafWatchfails without its new retention setting and passes with it.make fmt-check,make vet, golangci-lint (no new findings), the Windows and macOS builds andmake integration-compileare clean. My workstation is out of inotify instances, so I rango test ./...for nodeagent in a container; every package passes, including the watcher-based tests.Breaking Changes
Checklist