[codex] Complete Dash/Plotly migration, parity remediation, and runtime/library stabilization#14
Merged
Merged
Conversation
Add dash_app/ with combined FastAPI + Dash server, sidebar navigation, import page (file upload + one-click analysis), project workspace (save/load), export page (CSV/DOCX), and about page. Backend endpoints are consumed via httpx API client. No changes to existing backend or core logic. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add the missing WSGI bridge dependency required by the Dash server, fix sample-data resolution to use real repo fixtures, and cover the combined app with a minimal startup/import smoke test. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Carry the non-analysis MaterialScope workflow into Dash so import, project, compare, and report tasks can run end-to-end before the analysis-page migration. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…rors - Add backend-driven DSC analysis page with dataset selection, workflow template picker, execution via /analysis/run, and result display with Plotly figure, peak table, Tg markers, and processing details - Enable DSC nav link in sidebar, rename section from 'Analysis (coming soon)' to 'Analysis' - Fix stale project-id after server restart: ensure_project callback now validates the stored id against the backend before reusing it - Fix import 400 Bad Request: validate column mapping values exist in the current file's columns before sending to backend; add clear error in read_thermal_data when mapped columns are missing from the DataFrame - Add workspace_result_detail to api_client for fetching result details Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Expose DSC analysis-state curves so result views can show overlays and keep related workspace pages in sync after runs. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…tion - New TGA Dash page at /tga with dataset selection, unit mode selector (auto/percent/absolute_mass), workflow template picker, and run button - Adds unit_mode passthrough: AnalysisRunRequest -> execution_engine -> batch_runner -> _execute_tga_batch, applying set_tga_unit_mode before unit resolution - Extends AnalysisStateCurvesResponse with dtg/has_dtg fields so the TGA page can overlay the DTG curve on a secondary y-axis - Displays result summary cards (step count, total mass loss, residue), per-step cards with onset/midpoint/endset, mass vs temperature figure with DTG overlay and step markers, step data table, processing details - Enables TGA nav link in sidebar; auto-refreshes workspace on success Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Extract duplicated patterns into dash_app.components.analysis_page: - metric_card / metrics_row for KPI rendering - eligible_datasets / dataset_options / dataset_selector_block for dataset selection logic and UI - dataset_selection_card / workflow_template_card / execute_card / result_placeholder_card / analysis_page_stores for layout blocks - interpret_run_result for run-result status interpretation - processing_details_section with extra_lines hook - empty_result_msg / no_data_figure_msg helpers Refactor DSC and TGA pages to consume shared components; modality- specific logic (figures, specialised cards, extra selectors) remains in each page. Add 16 targeted tests for shared behavior. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Add resolve_sample_name() with fallback chain: summary.sample_name -> fallback_display_name -> stripped dataset_key -> N/A; prevents showing None or raw filenames when metadata has a usable name - DSC figure: suppress peak/Tg text labels when within 15 C of an already-annotated temperature; only show Tg onset/endset labels when not overlapping the midpoint; position annotations top-left - TGA figure: suppress step midpoint labels when within 15 C; hide onset/endset vline annotations when >4 steps (thin guide lines still drawn); use compact On/End labels - 7 new tests for resolve_sample_name Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…tion Add the first Dash DTA analysis page following the established DSC/TGA backend-driven pattern. Users can select an eligible DTA dataset, choose a workflow template (General DTA / Thermal Event Screening), run analysis through the existing backend, and view result summary cards, DTA curve figure with raw/smoothed/baseline/corrected overlay, direction-colored peak markers, detected event cards, event data table, and processing details. Workspace/report/compare state auto-refreshes on success. Also includes prior improvements to api_client error reporting, data_io spectral source hints and headerless-file handling, and import error hints for spectral data. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Improve DTA readability by making the figure the primary result surface, deriving event counts from saved result data, and resolving sample labels from dataset metadata. Add a repo rule that keeps future work aligned with the incremental Dash migration architecture. Made-with: Cursor
Add a backend-driven Dash Raman analysis slice with dataset/template selection, run flow, summary metrics, primary spectrum figure, library match views, and focused page tests. Include a root .cursorignore tuned for faster indexing while keeping migration-critical code visible. Made-with: Cursor
Deliver a backend-driven XRD Dash slice so users can run qualitative phase screening without leaving the stable analysis surface. This wires XRD routing, result rendering, and focused tests while reusing existing run/detail/state endpoints. Made-with: Cursor
Add end-to-end API tests for import, analysis run, result detail, analysis-state, workspace context/results, export preparation, compare PUT, project save/load, error paths, and key route smoke. Point dash server smoke test at the new module. Made-with: Cursor
- Add workspace_batch_run client and compare_curve_utils (axis titles, pick_best_series). - Compare page: analysis-state overlay with raw fallback, batch UI and persisted summary. - Tests: compare phase2 module and workflow regression batch path; combined-app overlay data path. Made-with: Cursor
- Harden react-select and DataTable styling for dark theme (headers, scrollbars, placeholders). - Add EN/TR i18n for sidebar and shared chrome; locale dropdown and clientside theme hook. - Apply figure theme tokens across analysis pages and data preview quick plots. - Align Import Active Dataset row; add chrome i18n tests; adjust page tests for ui_theme. Made-with: Cursor
Cover both legacy and modern react-select selector paths and keep targeted fallback selectors for locale, compare, and export controls so dark mode no longer leaks white control/menu/search surfaces. Made-with: Cursor
Remount non-analysis compare/export dropdown components on ui-theme changes so surfaces refresh reliably between light and dark modes, and add dedicated guidance block classes for readable, theme-consistent workflow callouts. Made-with: Cursor
- Add core/modality_specs.py with per-technique contracts (DSC/TGA/DTA/FTIR/RAMAN/XRD) defining allowed units, column aliases, suspicious combos, and spectral flags - Update guess_columns() to accept modality parameter that restricts signal detection to only the selected technique's patterns and locks data_type deterministically - Update read_thermal_data() to pass data_type through to guess_columns(modality=...) - Add 4-tier validation status: pass / pass_with_review / warn / fail - Add modality-specific suspicious unit combo review flags in validation - Rewrite dash_app/pages/home.py as a 6-step modality-first import wizard: Step 1: Technique selection, Step 2: Upload, Step 3: Raw preview, Step 4: Modality-aware column mapping, Step 5: Unit/metadata review, Step 6: Validation summary + confirm - Add dash_app/components/stepper.py reusable wizard UI component - Update import_preview.py to accept modality parameter - Add 41 new tests covering modality specs, modality-aware parsing, 4-tier validation, import preview, and backend integration Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
… management Add a compact recent-history panel to the sidebar, below the Management nav section and separated by a horizontal rule. The panel auto-refreshes on workspace changes (imports, analysis runs, etc.) and shows the last 8 events with timestamp, action and detail. The duplicate history card is removed from the Project Workspace page. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add verified Dash 4 dash-dropdown selector coverage for sidebar locale and non-analysis dropdown surfaces, and theme sidebar scrollbar colors to remove remaining white UI leaks in dark mode. Made-with: Cursor
Constrain sidebar locale dark-surface overrides to dark theme only and add explicit light-theme trigger/menu/option states so locale dropdown remains readable without leaking dark colors into light mode. Made-with: Cursor
When render_sidebar rebuilds the theme-toggle button (on locale change or theme change), Dash fires the toggle callback with n_clicks=0. Guard against falsy n_clicks with PreventUpdate to avoid unintended theme flips.
- Unify copy on translate_ui / utils.i18n catalog (dash.home, dash.export, dash.compare). - Hero and guidance slots plus locale-driven workbench where needed. - Thread ui-locale through callbacks that emit user-facing strings. - Thin dash_app.i18n wrapper; optional locale on page guidance helpers. - Extend test_dash_chrome_i18n for chrome behavior. Made-with: Cursor
…i-locale - Extend utils.i18n with dash.analysis.* (shared metrics, figure strings, DSC/TGA/DTA/FTIR/Raman/XRD templates and copy). - Refactor analysis_page primitives for translate_ui and locale-aware dataset selector, metrics, alerts, processing block, sample name fallback. - DSC/TGA/DTA/FTIR/Raman/XRD: hero slots, locale chrome callbacks, ui-locale on load/run/display, translated figures and cards where applicable. - Clear stale dash_app.pages.* PAGE_REGISTRY entries in create_dash_app to avoid duplicate routes after tests import page modules. Made-with: Cursor
…PDF hint
- Gate export/branding/support actions when commercial mode and license disallows write
- Preview batch summary with totals, filter, and table (parity with Streamlit)
- GET /workspace/{id}/exports/support-snapshot plus Dash prepare/download flow
- Hide PDF format when reportlab missing; i18n strings for new UI
- Test: support snapshot returns valid JSON
Made-with: Cursor
Remove the FTIR Setup Raw Data Quality card and dead explore helpers. Add ftir_literature_query_builder and _compare_ftir_result_to_literature with FTIR-specific queries, relevance scoring, and honest context states. Add _build_ftir_reasoning and FTIR experiment recommendations. Fix LiteratureContext.executed_queries through normalize_literature_context. Update tests and .ai SESSION/TASK/DECISIONS.
- Add dash.analysis.ftir.* i18n for presets, processing, baseline (cm^-1), quality, raw metadata, and empty-state hint; stop borrowing TGA strings. - FTIR results: library_unavailable uses info alert and hides match table; quieter pre-run placeholders; cards mainly for summary and figure. - finalized_validation_warning_issue_counts + interpret_run_result uses list length so run banner matches the validation panel. - Tests for count alignment, library-unavailable panels, interpret_saved. - Update .ai SESSION, TASK, DECISIONS.
…tion - Defer backend import in dash_app.server; sanitize hosted/mirror env and align MATERIALSCOPE_LIBRARY_CLOUD_URL with combined server listen port (default 8050 vs docker-style :8000). - Detect Windows-style path leaks on POSIX in path_env, resolve_hosted_root, and mirror feed resolution; reset library cloud client after env changes. - Add spectral_library_diagnostics and extend ftir_library_diagnostics CLI; document in README and .env.example; add tests.
- Lead with MATERIALSCOPE_* and combined local dev (8050); comment optional paths. - Document split stack (8000) and legacy THERMOANALYZER_* as secondary, commented. - README: point to .env.example ordering; de-emphasize Windows paths in split snippet.
- Replace XRD Dash page with Setup/Processing/Run plus results surface aligned with other modalities. - Add figure registration helper, saved-figures panel, GET figure endpoint with optional max_edge (Pillow), and api_client wiring. - Extend result detail with figure_artifacts; batch_runner XRD context; i18n and tests. - Remove obsolete .sisyphus planning artifacts from the repo tree. Made-with: Cursor
- Restructure result figure: toolbar for overlay and snapshot/report; artifacts under Details. - Collapse plot settings under Processing; default match overlays off; optional smoothed/baseline traces. - Compact candidate cards and literature compare (compact_toolbar + collapsed reference block). - Document Slice 7 in TASK and refresh SESSION. Made-with: Cursor
Normalize results-column spacing, compact figure toolbar with ButtonGroup, quieter artifacts and evidence bands, denser processing-tab cards, lighter candidate cards and literature chrome. Soften XRD-visible strings. Update .ai SESSION/TASK and XRD dash page test for registry summary translation. Made-with: Cursor
Made-with: Cursor
P0-2 + P0-3 from the repo-wide parity audit. DSC/DTA processing history labels borrowed TGA i18n keys; TGA quality/metadata/summary labels borrowed DSC keys. Added 28 modality-native keys (TR+EN), swapped 32 references, added 6 regression tests (source-grep + monkeypatch) to prevent recurrence. Made-with: Cursor
- Rename shared result-surface/structural classes from modality-specific dsc-*/dta-*
to generic ms-* prefix across all 6 Dash pages + CSS:
ms-results-surface, ms-result-section, ms-result-{context,hero,support,secondary},
ms-result-figure-shell, ms-result-graph, ms-meta-{term,def,value}
- Add per-modality root page hooks (dsc-page, dta-page, tga-page, ftir-page,
raman-page, xrd-page) for future per-modality styling flexibility
- Fix TGA derivative class leakage: dsc-derivative-graph/helper -> tga-derivative-*
- Preserve DTA-only debug classes (dta-figure-stack, dta-result-debug, dta-debug-shell,
dta-debug-graph) untouched
- Merge duplicate DSC CSS section into shared generic rules
- Update 6 test files to assert new generic class names
- Update .ai/SESSION.md and .ai/TASK.md
Verification: 260 passed, 1 pre-existing unrelated failure
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e9d5a9eb3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| _merge_result_artifacts(state, result_id, record, artifacts_update=artifacts) | ||
| return {"figure_key": key, "reused": True, "status": "captured"} | ||
|
|
||
| label = f"{str(analysis_type or '').upper()} Analysis - {dataset_key}" |
There was a problem hiding this comment.
Generate the auto snapshot label with only analysis_type + dataset_key here, and then store PNG bytes under that key in state["figures"]. If a user saves multiple results for the same dataset/modality (e.g., different processing templates), later runs overwrite the same figure key, while older result records still reference that key; exporting or viewing older results then shows the newest plot, corrupting historical result fidelity.
Useful? React with 👍 / 👎.
Comment on lines
+914
to
+915
| a = np.where(np.isfinite(a), a, None) | ||
| return a.tolist() |
There was a problem hiding this comment.
This conversion replaces non-finite values with None, but the response model fields are typed as list[float]. When analysis state contains NaN/Inf (which can occur in derived curves), FastAPI/Pydantic response validation can fail and turn the request into a 500 instead of returning a usable payload. The sanitizer should emit only valid floats (or filter invalid entries consistently with axis length handling) rather than None.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch lands the full
web-dash-plotly-migrationeffort againstmain.At a high level it replaces the earlier partial Dash surface with a product-grade Dash + Plotly application, brings all six analysis modalities to near-full parity with the existing Streamlit experience, adds the backend and export/reporting support those pages require, and closes the remaining parity/stability gaps that showed up during end-to-end validation.
The last two branch-level follow-ups were:
c2d7472— runtime/library test stabilization8e9d5a9— removal of generated test artifacts from git trackingAfter the artifact cleanup, the branch-level diff against
mainis down to 180 changed files.What Changed
1. Dash application foundation and workflow surfaces
2. Full analysis-page migration across modalities
3. Shared UI primitives and figure/report behavior
4. Backend and API support needed for Dash parity
5. Parity remediation and polish after the core migration
This branch also closes the follow-up parity backlog identified after the initial migration:
ms-*structural classes.cosine/pearson) and propagated the selected metric through local matching and cloud search.6. Runtime/library stabilization in tests
The final stabilization slice keeps production behavior strict and fixes the tests instead of weakening runtime behavior.
Specifically:
tests/test_backend_api.py,tests/test_backend_batch.py, andtests/test_reference_library.pynow clear both MaterialScope and legacy ThermoAnalyzer library/runtime environment variables before each test.MATERIALSCOPE_HOMEvalues so ambient developer-machine state and.envvalues cannot leak into expectations.no_matchregression test now provisions fallback-library state intentionally and uses a deterministic low-similarity FTIR signal instead of depending on incidental local library availability.7. PR cleanup: generated artifact removal
pytest_temp/content from the branch.python3root artifact..gitignoreto coverpytest_temp/,.pytest_temp/, andpython3explicitly.build/reference_library_ingest_live/because it is an intentional checked-in fixture used bytests/test_reference_library_ingest.pyas the seed XRD normalized corpus.Why
This branch exists to finish the Streamlit-to-Dash migration as an actual product surface rather than a partial prototype.
Before these changes, the Dash app had major modality gaps, repeated page-specific boilerplate, inconsistent result/figure behavior, and several places where the frontend could claim capability that the backend did not yet support end-to-end. After parity cleanup, the remaining failures were mostly in test isolation around runtime/library configuration. A later review also found that generated local test artifacts had been accidentally committed into the branch and needed to be removed before review.
The intent of this PR is therefore threefold:
Root Cause for the Final Stabilization Fix
The last failing full-suite cases were not caused by a production logic regression. They came from test contamination:
The fix in this branch treats those as test-fixture problems first. It does not broaden dev overrides, weaken cloud-auth expectations, or relax scientific matching behavior to make the tests pass.
User / Developer Impact
User-facing
Developer-facing
pytest_temp/clutter..aidecision/session/task records now reflect the completed parity and stabilization state.Validation
Recorded validation on this branch includes:
1116 passed, 9 skipped6 passed116 passed67 passed303 passed309 passed189 passedArtifact cleanup verification:
1116 passed, 9 skippedAdditional checks recorded in the branch session notes:
Review Notes
dash_app/pages/*)8e9d5a9