Skip to content

feat: task ingest dialog#1685

Merged
ricofurtado merged 30 commits into
mainfrom
task-ingest-dialog
May 28, 2026
Merged

feat: task ingest dialog#1685
ricofurtado merged 30 commits into
mainfrom
task-ingest-dialog

Conversation

@Wallgau
Copy link
Copy Markdown
Collaborator

@Wallgau Wallgau commented May 26, 2026

issue: https://github.ibm.com/lakehouse/tracker/issues/71806 (task dialog part)
Add full retry-ingestion workflow across backend and frontend for task files, including retry APIs, retry mutation hooks, task-detail querying, and richer task-file metadata from enhanced task endpoints.

Screen.Recording.2026-05-27.at.11.35.07.PM.mov
Screen.Recording.2026-05-27.at.11.50.17.PM.mov

Summary by CodeRabbit

  • New Features

    • Retry failed file ingestions from a task dialog (per-file or all) with filters, search, tabs, status/category chips, and detailed pipeline-step failure views.
    • Backend/API endpoints added to initiate retries and surface retry/skipped outcomes.
  • Bug Fixes

    • Improved error classification and clearer user-facing diagnostics.
    • More reliable selection, overlay reconciliation, and task-list refreshes during retries and cancellations.
  • Tests

    • New unit and integration tests covering retry workflows and failure classifications.

Review Change Stack

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 26, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 70f0d673-357a-4b00-8d04-2da5f57b8741

📥 Commits

Reviewing files that changed from the base of the PR and between b3d1ad1 and 5e94758.

📒 Files selected for processing (3)
  • frontend/app/api/mutations/useCancelTaskMutation.ts
  • src/app/routes/internal.py
  • src/config/settings.py
🚧 Files skipped from review as they are similar to previous changes (2)
  • frontend/app/api/mutations/useCancelTaskMutation.ts
  • src/app/routes/internal.py

Walkthrough

Adds backend retry API and service, enhances task queries/types and failure analysis, implements a TaskDialog UI and useTaskDialog hook for selecting/retrying files, updates context/knowledge-page overlay and deletion logic, extends styling tokens and Tailwind, and adds tests and misc wiring.

Changes

Task Ingestion Retry

Layer / File(s) Summary
Backend retry endpoint & service
src/api/tasks.py, src/services/task_service.py, src/app/routes/internal.py
Adds RetryTaskBody, POST /tasks/{task_id}/retry, TaskService.retry_failed_files, _resolve_upload_task_store, and updated _infer_failure_metadata to classify retryability and schedule background processing.
Backend unit tests & schemas
tests/unit/test_task_service_retry_failed_files.py, tests/unit/test_task_service_get_task_status2.py, src/api/schemas/tasks.py
New/extended tests covering retry flows and additional _infer_failure_metadata cases; new Pydantic response models for retry responses and skipped-file details.
Frontend queries & types
frontend/app/api/queries/useGetTasksQuery.ts, frontend/app/api/queries/useGetTaskQuery.ts
Adds TASKS_QUERY_KEY, TASK_DETAIL_QUERY_KEY, useGetTaskQuery, fetches enhanced endpoints, and extends TaskFileEntry with failure metadata types.
Frontend mutations & helpers
frontend/app/api/mutations/useRetryTaskMutation.ts, frontend/app/api/mutations/useCancelTaskMutation.ts, frontend/app/api/mutations/useDeleteDocument.ts
Adds useRetryTaskMutation (types, POST to /tasks/{id}/retry, invalidations), updates cancel mutation to use TASKS_QUERY_KEY, and extracts deleteDocumentByFilename.
Task utilities & overlays
frontend/lib/task-utils.ts
Adds naming/type helpers, predicates (isTaskFileFailed/isTaskFileRetryable), filtering/sorting/counts, overlay helpers (isTaskInProgressStatus, pendingTaskFileSourceUrl, findTaskFileOverlayIndex, getEnhancedListDisappearedFilePaths, finalizeProcessingOverlaysForEnhancedTask).
Structured error analysis
frontend/lib/task-error-display.ts
Introduces pipeline-step model types and functions (analyzeTaskFileIngestionFailure, resolveTaskFileError, buildPipelineStepsFromFailurePhase, formatApiComponent) replacing raw-string display helpers.
Task dialog UI
frontend/components/task-dialog/*
New dialog system: CATEGORY_CHIPS, TaskDialogCategoryChips, TaskDialogFilters, TaskDialogHeader, TaskDialogFileErrorDetails, TaskDialogFileList, TaskDialog, and index barrel for browsing, filtering, inspecting, selecting, and retrying files.
Dialog state & orchestration
frontend/components/task-dialog/use-task-dialog.ts
Hook driving dialog: fetch task detail, compute retryable entries, selection toggles, retry handlers that mark processing, call retry mutation, refetch, show toasts, and reset selection.
Task error content & tasks details
frontend/components/task-error-content.tsx, frontend/components/tasks_details.tsx
Refactors failed-file rendering to use new analysis APIs and open the TaskDialog; removes brand-specific styling in tasks details.
Context & knowledge page integration
frontend/contexts/task-context.tsx, frontend/app/knowledge/page.tsx
Adds markTaskFilesProcessing, switches to TASKS_QUERY_KEY invalidation, uses enhanced overlay reconciliation helpers, enforces deletable-only selection and Promise.allSettled bulk-deletes, and syncs AG Grid selection via identity-aware logic.
Styling & tokens
frontend/app/globals.css, frontend/tailwind.config.ts
Adds task-dialog CSS custom properties and Tailwind theme tokens for dialog sizing, colors, spacing, and z-index.
Tests / mocks / telemetry / misc
frontend/tests/core/tasks-unified-panel.spec.ts, src/utils/telemetry/message_id.py, src/config/settings.py, src/models/processors.py
Playwright mock adjusted to continue non-GET /api/tasks; telemetry file whitespace normalized; Langflow client request gains bounded retry loop via LANGFLOW_REQUEST_RETRIES; Langflow processor sets IngestionPhase.LANGFLOW.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Possibly related PRs

Suggested reviewers

  • mfortman11
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 23.91% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'feat: task ingest dialog' clearly summarizes the main change—adding a task ingestion dialog UI component with retry workflow. It accurately reflects the primary objective of this pull request.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch task-ingest-dialog

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions github-actions Bot added the frontend 🟨 Issues related to the UI/UX label May 26, 2026
@Wallgau Wallgau changed the title Task ingest dialog feat: task ingest dialog May 26, 2026
- Add POST /tasks/{task_id}/retry to re-queue failed RETRYABLE files on the
  existing task processor (optional file_paths subset).
- Implement TaskService.retry_failed_files with skip reasons (not_failed,
  not_retryable, source_file_missing, task_in_progress) and retry_count tracking.
- Classify non-retryable file errors vs transient Docling failures.
- Add tests/unit/test_task_service_retry_failed_files.py.

Frontend — retry
- Add useRetryTaskMutation calling the retry endpoint.
- Extend enhanced task types (actionable_by, retry_count, failure component/
  phase) in useGetTasksQuery and single-task fetch in useGetTaskQuery.
- Task dialog: retry all/selected, retry ingestions tab, row selection, polling
  while retry runs, and toasts for retried/skipped files.
- task-utils: retryable file helpers, filters, and dialog status labels.
- task-context: markTaskFilesProcessing when retry starts; improved overlay
  sync/finalization with enhanced task payloads and knowledge refetch on
  completio
- task-error-display: use structured API failure metadata for file errors.
- task-error-content: open task dialog from failure panel; show API component
  and resolved error text.
- useCancelTaskMutation: invalidate tasks queries with exact: false.
- Playwright: mock GET /api/tasks** only so retry POST is not intercepted.

Frontend — knowledge bulk delete
- Fix delete modal listing wrong files after partial ingest/retry: AG Grid kept
  failed rows selected after checkboxes disappeared (processing → failed).
- Restrict selection and delete to active rows present in listFiles/search
  (indexed knowledge only, not overlay-only rows without chunks).
- Prune non-deletable selection when grid data changes; add isRowSelectable.
- Delete with Promise.allSettled via deleteDocumentByFilename so one file
  without chunks does not abort the batch; resolve indexed filename for delete.

Tests
- Extend tests/unit/test_task_service_get_task_status2.py for enhanced status.
@github-actions github-actions Bot added backend 🔷 Issues related to backend services (OpenSearch, Langflow, APIs) tests labels May 27, 2026
@github-actions github-actions Bot added the enhancement 🔵 New feature or request label May 27, 2026
@github-actions github-actions Bot added enhancement 🔵 New feature or request and removed enhancement 🔵 New feature or request labels May 28, 2026
@github-actions github-actions Bot added enhancement 🔵 New feature or request and removed enhancement 🔵 New feature or request labels May 28, 2026
@github-actions github-actions Bot added enhancement 🔵 New feature or request and removed enhancement 🔵 New feature or request labels May 28, 2026
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/config/settings.py`:
- Around line 866-868: The auth-retry path is passing the original caller kwargs
(which may still contain 'headers') into the HTTP call causing "got multiple
values for keyword argument 'headers'"; in the retry where you call
self.langflow_http_client.request(method=method, url=url, headers=headers,
**kwargs) replace **kwargs with **request_kwargs (the same dict used earlier
where headers was popped) so only the sanitized request_kwargs and the explicit
headers variable are passed into the request.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 9b12b4a7-814b-4ca3-982b-fdc55c5ba559

📥 Commits

Reviewing files that changed from the base of the PR and between 7a3dbb1 and cb4722a.

📒 Files selected for processing (4)
  • src/config/settings.py
  • src/models/processors.py
  • src/services/task_service.py
  • tests/unit/test_task_service_retry_failed_files.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests/unit/test_task_service_retry_failed_files.py

Comment thread src/config/settings.py Outdated
@github-actions github-actions Bot added enhancement 🔵 New feature or request and removed enhancement 🔵 New feature or request labels May 28, 2026
@github-actions github-actions Bot added enhancement 🔵 New feature or request and removed enhancement 🔵 New feature or request labels May 28, 2026
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/api/schemas/tasks.py (1)

16-22: ⚡ Quick win

Consider using a string literal or enum for the status field.

The status field accepts any string value, which could lead to inconsistent status values across the codebase. Using a Literal type or Enum would provide type safety and ensure consistent status strings.

🔒 Suggested fix using Literal types
-from pydantic import BaseModel, Field
+from typing import Literal
+from pydantic import BaseModel, Field


 class TaskRetryResponse(BaseModel):
     task_id: str
     retried: int = 0
     skipped: list[TaskRetrySkippedFile] = Field(default_factory=list)
-    status: str
+    status: Literal["success", "partial", "failed"]  # Adjust values based on actual usage
     message: str | None = None
     error: str | None = None
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/api/schemas/tasks.py` around lines 16 - 22, The TaskRetryResponse model's
status: str allows arbitrary values; change it to a constrained type (either a
Literal union of allowed strings or an Enum) to enforce consistency. Update the
TaskRetryResponse definition to replace status: str with status:
Literal["pending","in_progress","completed","failed"] (or an Enum class like
TaskRetryStatus with those members) and adjust any code that constructs or
checks TaskRetryResponse.status to use the chosen literals/enum members; also
import typing.Literal or the Enum type and update any tests/serializers that
expect a plain string.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/api/schemas/tasks.py`:
- Around line 16-22: The TaskRetryResponse model's status: str allows arbitrary
values; change it to a constrained type (either a Literal union of allowed
strings or an Enum) to enforce consistency. Update the TaskRetryResponse
definition to replace status: str with status:
Literal["pending","in_progress","completed","failed"] (or an Enum class like
TaskRetryStatus with those members) and adjust any code that constructs or
checks TaskRetryResponse.status to use the chosen literals/enum members; also
import typing.Literal or the Enum type and update any tests/serializers that
expect a plain string.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: e989b78a-5d04-4f29-9b05-74f2a385d106

📥 Commits

Reviewing files that changed from the base of the PR and between cb4722a and b3d1ad1.

📒 Files selected for processing (7)
  • frontend/app/knowledge/page.tsx
  • frontend/components/task-dialog/use-task-dialog.ts
  • src/api/schemas/tasks.py
  • src/api/tasks.py
  • src/app/routes/internal.py
  • src/config/settings.py
  • src/services/task_service.py
🚧 Files skipped from review as they are similar to previous changes (6)
  • src/app/routes/internal.py
  • src/api/tasks.py
  • src/config/settings.py
  • frontend/app/knowledge/page.tsx
  • frontend/components/task-dialog/use-task-dialog.ts
  • src/services/task_service.py

Olfa Maslah added 2 commits May 28, 2026 07:59
limit auth-refresh retry to once per request call
retry boundedly on persistent 401/403 instead of immediately returning
@github-actions github-actions Bot added enhancement 🔵 New feature or request and removed enhancement 🔵 New feature or request labels May 28, 2026
@github-actions github-actions Bot added enhancement 🔵 New feature or request and removed enhancement 🔵 New feature or request labels May 28, 2026
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 28, 2026

Actionable comments posted: 0

@Wallgau Wallgau requested review from lucaseduoli and mpawlow May 28, 2026 13:18
@github-actions github-actions Bot added the lgtm label May 28, 2026
@ricofurtado ricofurtado merged commit 56dd575 into main May 28, 2026
19 of 21 checks passed
@github-actions github-actions Bot deleted the task-ingest-dialog branch May 28, 2026 13:59
Wallgau added a commit that referenced this pull request May 28, 2026
* Revert "feat: task ingest dialog (#1685)"

This reverts commit 56dd575.

* style: ruff autofix (auto)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend 🔷 Issues related to backend services (OpenSearch, Langflow, APIs) enhancement 🔵 New feature or request frontend 🟨 Issues related to the UI/UX lgtm tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants