Deduplicate Kubernetes observer state checks and diagnosis logs for unchanged Pending pods by devin-ai-integration[bot] · Pull Request #22965 · PrefectHQ/prefect

devin-ai-integration · 2026-08-27T19:45:21Z

This PR stops the Kubernetes observer from repeating flow run state checks and diagnosis logs when a pod stays in the Pending phase.

Kubernetes sends many watch events for a pod that does not change. Before this change, each event caused a new read_flow_run call, a new propose_state call, and a possible duplicate flow run log. This PR makes these operations occur one time for each unchanged condition:

The observer deduplicates diagnoses by normalized cause. For Unschedulable diagnoses, it normalizes only complete, recognized Kubernetes scheduler formats. Supported filter counts and reason ordering, the Kubernetes v1.35 NodeDeclaredFeatures requirement list, and default-preemption summaries are canonicalized while general post-filter text remains exact. Unknown or malformed scheduler output uses the exact original detail, so upstream format changes can produce an extra log instead of suppressing a meaningfully changed diagnosis.
One lifecycle-state owner retains each pod's completed state check and last diagnosis key together. Completed checks are cleared when a pod leaves Pending, diagnosis keys are cleared on recovery, and empty records are removed. Deletion and shutdown release the whole record. A periodic, paginated Kubernetes listing also releases state for pods whose deletion event was missed, but only after the full watched namespace and label scope is listed successfully. Live pods and UIDs observed concurrently with reconciliation remain retained.
The observer records a completed flow run state check for each pod while the pod stays in the Pending phase. Transient failures, timeouts, and rejected proposals are not retained, so the observer retries them on the next event.
The observer inspects the state that propose_state() returns. It records the check as completed only when the returned state is InfrastructurePending or is running, final, paused, or cancelling. Any other replacement state, such as a Scheduled state after a rejected proposal, is retried on the next event.

Implementation details

diagnostics.py: adds InfrastructureDiagnosis._dedupe_key(). Recognized Unschedulable* scheduler messages use a namespaced normalized key; unfamiliar or malformed messages use a namespaced exact key. The parser requires the expected scheduler scaffold, fully consumes counted reason histograms, and accepts only the Kubernetes reason shapes covered by the observer's scheduler-message corpus. It tags scheduler stages, preserves arbitrary post-filter text, validates nested default-preemption summaries independently, and sorts NodeDeclaredFeatures requirements only when that complete reason matches. All other diagnosis categories remain exact.
observer.py:
- _PodLifecycleStates owns one record per relevant pod UID, containing the completed Pending-state check and last successfully handed-off diagnosis key.
- Lifecycle observation clears completed checks when a pod leaves Pending and clears diagnosis keys on recovery. Deletion releases the whole pod record even if event handling fails, and observer shutdown clears the owner after reconciliation stops.
- While pod event replication is enabled, a private reconciliation task lists pods every five minutes using the observer's exact namespaces and label filters. It follows every Kubernetes continuation token and mutates lifecycle state only after every page succeeds. Failed, partial, or invalid listings leave state unchanged.
- Reconciliation snapshots eligible UIDs before Kubernetes I/O and tracks UIDs observed by concurrent pod events, so neither newly added state nor a refreshed existing UID can be removed by an older listing. Overlapping passes keep independent observation sets.
- Observer shutdown cancels and awaits reconciliation before running the existing Prefect client cleanup path and releasing all lifecycle state. Kubernetes client-close errors cannot mask an in-flight cancellation.
- Startup events reconcile phase and diagnosis recovery before the existing-event deduplication return.
- _is_completed_flow_run_state() is used both for the state read before the proposal and for the state that propose_state() returns.
- ObjectNotFound from read_flow_run completes the check because a missing flow run is a terminal observation for an immutable flow run ID.
- propose_state() raises Pause instead of returning a paused replacement state, and Pause is a BaseException subclass. The observer catches it so the signal does not escape the event handler and records the check as completed.
- A diagnosis key is retained only after the configured flow_run_logger call returns successfully. Durable API delivery remains owned by the existing Prefect logging pipeline rather than the observer.

Kopf multiplexes the watch stream into one queue for each object UID and processes each queue with one worker, so two events for the same pod cannot run in the handler at the same time. Reconciliation separately protects any UID touched while its Kubernetes listing is in flight.

Checklist

This pull request references any related issue by including "closes <link to issue>"
If this is a complex change, a maintainer has confirmed the proposed approach on the linked issue.
If this pull request adds or changes functionality, it includes tests or explains why tests are not needed.
If this pull request changes user-facing behavior, it updates documentation or explains why documentation is not needed.
- No documentation change is needed: the change reduces duplicate logs and API calls and does not change any public interface.
If this pull request removes docs files, it includes redirect settings in mint.json.
If this pull request adds functions or classes, it includes helpful docstrings.

Link to Devin session: https://app.devin.ai/sessions/a3dd1abe570d4a97b6ab97350a8a06fb
Open in Devin Desktop: https://app.devin.ai/desktop/session/a3dd1abe570d4a97b6ab97350a8a06fb?variant=devin

…nchanged Pending pods Deduplicate diagnoses by normalized causes so scheduler count changes do not produce duplicate flow run logs, cache completed Pending state checks per pod until the phase changes while retrying failures and timeouts, and do not cache a proposal rejected with a Scheduled replacement. OSS-8181 Co-authored-by: Alexander Streed <alex.s@prefect.io>

devin-ai-integration · 2026-08-27T19:45:23Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment, CI, and merge conflict monitoring

chuqCTC

I found three correctness gaps compared with the issue's intended behavior.

Co-authored-by: Alexander Streed <alex.s@prefect.io>

chuqCTC

I found two remaining edge cases in the scheduler-message normalization.

chuqCTC

One additional state-transition edge case and one non-blocking simplification from a cleanup pass.

Co-authored-by: Alexander Streed <alex.s@prefect.io>

chatgpt-codex-connector · 2026-08-29T19:46:01Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-01T21:14:59.507794Z	`5522a21`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Co-authored-by: Alexander Streed <alex.s@prefect.io>

devin-ai-integration

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 2 new potential issues.

devin-ai-integration · 2026-08-31T20:08:59Z

+@pytest.fixture(autouse=True)
+def clear_observer_lifecycle_caches() -> Iterator[None]:
+    _completed_state_check_cache.clear()
+    _last_diagnosis_cache.clear()
+    observer._pod_cache_reconciliation_touches.clear()
+    yield
+    _completed_state_check_cache.clear()
+    _last_diagnosis_cache.clear()
+    observer._pod_cache_reconciliation_touches.clear()


🔍 Tests expose private cache structure

Several tests mutate private caches and reconciliation registries directly. This conflicts with the contribution guide's preference for behavior-focused tests and increases refactoring cost.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration

Devin Review found 1 new potential issue.

devin-ai-integration · 2026-08-31T21:35:08Z

+            else:
+                _last_diagnosis_cache[uid] = key


🟡 Queued log failures suppress retries

When the background API request fails, _last_diagnosis_cache still marks its queued diagnosis as delivered. Unchanged pod events never retry that missing log.

Was this helpful? React with 👍 or 👎 to provide feedback.

github-actions Bot added the bug Something isn't working label Aug 27, 2026

chuqCTC reviewed Aug 27, 2026

View reviewed changes

Comment thread src/integrations/prefect-kubernetes/prefect_kubernetes/diagnostics.py Outdated

Comment thread src/integrations/prefect-kubernetes/prefect_kubernetes/observer.py Outdated

Comment thread src/integrations/prefect-kubernetes/prefect_kubernetes/observer.py Outdated

chuqCTC reviewed Aug 27, 2026

View reviewed changes

Comment thread src/integrations/prefect-kubernetes/prefect_kubernetes/observer.py Outdated

Fix OSS-8181 Kubernetes observer review issues

e675bf9

Co-authored-by: Alexander Streed <alex.s@prefect.io>

chuqCTC reviewed Aug 28, 2026

View reviewed changes

Comment thread src/integrations/prefect-kubernetes/prefect_kubernetes/diagnostics.py Outdated

Comment thread src/integrations/prefect-kubernetes/prefect_kubernetes/diagnostics.py Outdated

chuqCTC reviewed Aug 28, 2026

View reviewed changes

Comment thread src/integrations/prefect-kubernetes/prefect_kubernetes/observer.py

Comment thread src/integrations/prefect-kubernetes/prefect_kubernetes/observer.py Outdated

Address OSS-8181 second review round

df10b2c

Co-authored-by: Alexander Streed <alex.s@prefect.io>

desertaxle marked this pull request as ready for review August 29, 2026 19:45

desertaxle requested review from chrisguidry, desertaxle and zzstoatzz as code owners August 29, 2026 19:45

This comment was marked as resolved.

Sign in to view

devin-ai-integration Bot and others added 2 commits August 29, 2026 19:55

Fix OSS-8181 observer cleanup and diagnosis dedupe

6aac7aa

Co-authored-by: Alexander Streed <alex.s@prefect.io>

Fix OSS-8181 lifecycle and scheduler deduplication

993dff9

This comment was marked as resolved.

Sign in to view

desertaxle added 2 commits August 31, 2026 12:11

Guarantee Kubernetes observer cleanup

1bc4e7c

Reconcile Kubernetes observer lifecycle caches

3c83cfd

devin-ai-integration Bot commented Aug 31, 2026

View reviewed changes

Harden Kubernetes scheduler diagnosis deduplication

22db944

This comment was marked as resolved.

Sign in to view

Retry Kubernetes diagnosis logs after failures

6749cac

This comment was marked as resolved.

Sign in to view

Preserve pod events when diagnosis logging fails

9e0c0a4

devin-ai-integration Bot commented Aug 31, 2026

View reviewed changes

Centralize Kubernetes pod lifecycle state

5522a21

Uh oh!

Conversation

devin-ai-integration Bot commented Aug 27, 2026 • edited by desertaxle Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

devin-ai-integration Bot commented Aug 27, 2026

🤖 Devin AI Engineer

Uh oh!

chuqCTC left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chuqCTC left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chuqCTC left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Aug 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

devin-ai-integration Bot Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

devin-ai-integration Bot commented Aug 27, 2026 •

edited by desertaxle

Loading

chatgpt-codex-connector Bot commented Aug 29, 2026 •

edited

Loading

devin-ai-integration Bot left a comment •

edited

Loading

devin-ai-integration Bot Aug 31, 2026 •

edited

Loading

devin-ai-integration Bot Aug 31, 2026 •

edited

Loading