fix(disambiguate): mapeo por contenido en lugar de índice de array (#78)#86
Conversation
Reviewer's GuideRefactors the disambiguation result mapping to match response items to paragraphs by text content using a consumption queue, and preserves predictions for paragraphs that the backend omits, instead of relying on brittle array index-based mapping that could misalign labels or lose annotations. File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Pull request overview
This PR fixes incorrect paragraph-to-disambiguation-result alignment in the /anonymizer/disambiguate flow by matching backend response items back to the original paragraphs by paragraph content (text) instead of array index, and it avoids losing existing annotations when the backend drops paragraphs from the response.
Changes:
- Replace index-based mapping (
paragraphs[i]) with a per-text “consumption queue” to match response items to the first unconsumed paragraph with the same text. - Ensure returned
PredictLabel.paragraphIdis always the stable paragraphid(not the raw paragraph text from the backend). - Preserve original predictions for any paragraph omitted by the backend response.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -207,10 +225,18 @@ export const disambiguate = (file: DocFile) => | |||
| aymurai_label_instance: l.attrs.aymurai_label_instance ?? null, | |||
| aymurai_disambiguation: l.attrs.aymurai_disambiguation ?? null, | |||
| }, | |||
| paragraphId: paragraph?.id ?? item.document, | |||
| paragraphId: paragraph.id, | |||
| } satisfies PredictLabel; | |||
| }); | |||
| }); | |||
|
|
|||
| // Preserve raw predictions for any paragraph the backend did not return, | |||
| // so a backend count drop never silently erases annotations. | |||
| const unmatched = predictions.filter( | |||
| (p) => !matchedParagraphIds.has(p.paragraphId), | |||
| ); | |||
|
|
|||
| return [...disambiguated, ...unmatched]; | |||
* chore(routing): migrate to tanstack router * build: move tanstack router dependencies to devDependencies * fix(styling): import styled function from stitches instance * fix(routing): used wrong parms in route, switched to correct path * fix(schema): make predict alt values nullable * chore: fix pr comments * chore: remove comments * fix: using react query to download anonymized file * chore: remove deprecated code and functions * fix: don't revoke api response blob url * chore: add format package script and ignore route tree gen file * chore: format code * chore: lint code * fix: add file name to anonymize query key * chore: remove global axios interceptor * chore: type global router context * chore: set stale time to default * chore: kill cache when entering home layout * docs: update readme * build: add support for environmental variables for both web and electron apps * build: install and configure pandacss * chore: flag stitches as deprecated * feat: add loading page and updated branding images * chore: configure view transition for all pages * feat: add loading screen on boot, timer of 1.5s * chore: re-implement button and partially input * chore: fix some tokens in panda and move stitches global styles * chore: add changes to router file * chore: change to named export on local store * chore: replace custom use mutation hook with base on connect to host hook * fix: add lineheight to text styles and adjust font weight * fix: add fixed height to button and auto adjust icon size * feat: create components to render in home screen * feat: create link card tool for features * feat: redesign home * fix: extra character in home layout and rename the component * chore(styles): add animation semantic tokens * feat: add more brand images * feat(components): create brand, layout and ui components * fix: replace brand images with correct ones and set proper heights * feat: create modern tooltip component * chore: flag card as deprecated * chore: create modern ui components * fix: remove fadeIn scaling animation * build: update react and add radix dependencies * chore: export constants * fix: adjust stepper styles (sizing and colors) * chore: refactor header so we can correctly position all elements * chore: use constants and base card element on feature selector * chore: migrate hidden input * build: add web or electron run modes * fix: typo on anonymizer label * chore: add tutorial seen to local storage store * feat: make card clickable * chore: adjust button sizes and enum import * chore: simplify main app layout * feat: rework onboarding page * feat(ui): create and/or adjust components * fix: update enum import * chore: rework layout components * chore: ignore personal analysis folder * feat: improve topbar accessibility with semantic icons and aria labels * refactor(layout): hoist Topbar and Stepper to global wizard route * build: add i18n * feat: add i18n support for the whole app * fix(layout): address PR review on header and icon changes - Replace House icon with DotsNine in FeaturesMenu - Replace Info icon with Question in HowItWorksModal - Remove Header from shared route layout (route.tsx) - Restore Header to onboarding.tsx and add it to each feature page - Each page now controls its own Header, Stepper step, and right section * feat: add api base url protection and apply it * chore: restructure HOC to be a regular component * chore: cleanup * fix: redo home layout * feat: add more copies to locale file * feat: use a11y on finish page * fix: add missing feature param call on route.tsx * chore: kill unused old hidden input component * feat: add more copies for the finish page * feat: create label manager component * feat: create switch component * chore: minor changes * fix: title in anonymizer locale * chore: remove stepper component * chore: refactor Searchbar * feat: create initial label manager * feat: add feature icon record * add disambiguate and predict react query options * finish onboarding page * finish preview page * improve layout header component * connect rest of label manager * add label manager to file annotator * make search bar static and follow scroll * use translation on droparea * clear files on features menu * make dataset validation file annotator not annotable * add missing built by in features page * drop file queries when resetting the progress * add className prop to footer * simplify finish dataset and anonymizer * update decision tabs to panda * update file processing component * hide file stepper selector arrows depending on cursor * fix gaps on finish page * add checked variant to button * update suggestion mark component's styles * improve suggestion component * create toast component * create callout from toast, and then apply a11y to toast * remove old component implementations * fix title in process copy etxt * adjust callout styling * adjust styling and positioning on preview page * create better select component * kill more unused components * minor styling fixes + new select imports * add retro compatibility features to select * add custom icons to callout, toast and showToast * add input size variants * remove old tooltip implementation * add radix's select as dependency * initial mark component update * improve ui dialog component * add small version of select component * create search tagger * extract tagger popover logic * update button icons * swap replace button icons * deprecate usage of useSchemedMutation * deprecate usage of useSchemedQuery and useSchemedQueries * add odt to pdf conversion * update tanstack react query version * add className to SectionTitle * improve accessibility on host page * finish major pages and add file protection feature * finish mark and tag annotation feature * adjust styles on callout * fix select viewport and add scroll * add size variant to suggestion and adjust display * add max width to toast and remove instad of dismiss it * move createAnnotationData, tag and suffix to context * disambiguate hook * adjust spacing on dataset validation * move suggestion label and add mark props * store label manager config in local storage * add more anonymizer copy locale texts * add metadata to search and tag annotations * simplify annotation components * add remove dialog to label manager's entity tab * smaller fixes on file annotation components * fix TS issues * adjust predicting and file parser flows * add useShallow to local storage default getters * simplify and remove unused code on fille annotator component * add value as undefined on ui/input component * add random cannonical id on search add event * add update by cannonical id function * add more canonical id operations on reducers * adjust typings * remove axios' default api url * add fixed height to validation dataset page * make suggestion clickable on uncontrolled text input * more fixes in height to validate dataset * include label policies and annotations when anonymizing document * fix ts clone element issue * add knip * make OS taskbar API safe to call in web * add missing anonymize value on label attrs * fix electron ts issues * make how it works modal bigger * restore useNotify feature in process page * fix issue regarding annotation keyboard navigation * use base_url when dealing with public assets * 1.25.0 * copy changes on label manager config tab * copy change on label manager tab * responsiveness for screens less than 1280px in width * create pdf to odt service * use extension to check if file conversion is needed * feat: add back-to-home button on features page Add a BackButton (arrow left) next to the greeting title on /home/features that navigates back to /home/host (Local/Servidor selection). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use House icon in header instead of BackButton arrow Place House icon button next to FeaturesMenu (DotsNine) in the header right section, matching the original design where the home button was in the top-right area. Navigates to /home/host. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: increase spacing between home and features menu buttons Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * limit concurrent predict requests to avoid connection exhaustion * add error handling to finish file conversion * prevent semaphores underflow * copy change * remove copies and functions referencing .doc files * change NINO to NIÑO * add "Entidad" placeholder to tagger select * create link component * Restore delete-one and delete-all hover actions on annotations (#57) * feat: restore delete-one and delete-all hover actions on annotations - Create RemoveDialog confirmation component for tag-annotation - Add delete-one and delete-all buttons to Tagger popover - Wire callbacks through AnnotationPopover to TagAnnotation - Uses existing remove/removeByText from useAnnotation context Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(annotations): clarify hover remove/replace scope to text-based matching - Remove canonical_entity_id branch from confirmReplaceAll; hover actions now always operate by exact text match (removeByText / updateByText) - Update remove-dialog copy: title and body now reflect text-based removal - Update replace-dialog copy: body shows affected text alongside new label - Pass annotation text to both dialogs via new `text` prop --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: jansaldo <julianansaldo@gmail.com> * Fix text overflow in HowItWorksModal (#58) * fix: responsive maxW on HowItWorksModal to prevent text overflow Add maxWidth and overflowY to the modal content so card text stops overflowing at viewport widths < 900px. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use 90vw width (max 1200px) for HowItWorksModal The previous 900px constraint was too narrow for the content. Now the modal uses 90% of viewport width up to 1200px. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: move howItWorks steps to feature namespaces Co-authored-by: Copilot <copilot@github.com> * fix: align HowItWorksModal dimensions and card layout with onboarding Co-authored-by: Copilot <copilot@github.com> * fix: correct Spanish language typos in anonymizer and common localization files * chore: update anonymizer howItWorks step 2 subtitle --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: jansaldo <julianansaldo@gmail.com> Co-authored-by: Copilot <copilot@github.com> * feat: add tooltips to tagger label and suffix inputs * fix: prevent select caret rotation from leaking ancestor data-state * create home button component * make aymurai's logo a link in the header * add home button to header on all flow's pages * remove slot checks on header * make hover effect in button work for anchor tanstack link wrapper * add a "config" button in features menu * chore: add .npmrc to configure public hoist pattern for @types * Add drag-and-drop reordering and inline rename to label manager (#60) * feat: add drag-and-drop reordering and inline rename to label manager - Install @dnd-kit/core, @dnd-kit/sortable, @dnd-kit/utilities - Add groupOrder and categoryAssignments to Zustand store (persisted) - Create SortableGroup component with drag handle - Wire DndContext + SortableContext per category in entity-tab - Add inline rename: pencil icon toggles input, Enter saves, Escape cancels Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(anonymizer): track stable mention ids * feat(anonymizer): centralize active prediction filtering * feat(label-manager): group labels by anonymizer category * feat(label-manager): derive canonical entity groups * feat(label-manager): support group edits and merges * feat(label-manager): improve panel layout and hover feedback * feat(anonymizer): add entity normalization and similarity helpers * feat(anonymizer): resolve manual annotations into entity groups * feat(label-manager): split mentions into new entity groups * feat(anonymizer): validate annotations before export * chore(anonymizer): recategorize USUARIX label * feat(anonymizer): add searchable grouped label select * feat(label-manager): allow clearing excluded terms * fix(annotation-popover): stabilize hover and focus behavior * feat(search): manage active matches in file annotator * feat(search): preserve search state on tagged annotations * fix(annotations): support selections across split spans * feat(annotations): apply bulk edits to normalized occurrences * fix(entity-groups): detect token-order duplicate aliases * fix(aymurai): scope query cache keys by API base URL * style(entity-resolution): tighten dialog width * fix(dependencies): move dnd-kit packages from devDependencies to dependencies --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: jansaldo <julianansaldo@gmail.com> * feat(validation): persist and restore predictions via backend validation endpoint (#68) * refactor: minor biome formatting and variable naming cleanup * test: set up vitest with @-alias path resolution * feat(schema): export predictLabelSchema for validation reuse * feat(validation): load stored validation before predict to skip re-processing * fix(anonymizer): preserve excluded entities in anonymize-document payload * test: add unit tests for normalizeValidationLabel and predictions utils * feat(dependencies): add dnd-kit packages for drag-and-drop functionality * fix(file-annotator): fix upward autoscroll on search previous navigation (#76) * Fix/homogenize file check ui (#77) * fix(spinner): add viewBox attribute to SVG for proper scaling * fix(styles): homogenize maxWidth and dimensions for file check components * fix(components): change subtitle from h3 to h2 for consistency with preview screen * fix(file-check): add size prop to Text component for consistent styling * Fix/responsive home layout (#80) * fix(home-layout): replace absolute-positioned BuiltBy with flex in-flow layout * fix(features): move BuiltBy from absolute position to Footer component * fix(main-content): add overflow-y auto to prevent content overflow on small screens * fix(home-layout): adjust MainContent and HomeLayout styles for better responsiveness * fix(home-layout): restore padding-bottom to inner layout for improved spacing * Feat/entity manager mention feedback (#81) * feat(entity-manager): enhance text item styles and group creation logic * feat(file-annotator): initialize label manager state based on isAnnotable prop * fix(entity-tab): preserve native context menu for single-mention groups * Fix/invalid entity offsets (#82) * feat(export-validation): add issue summary and table to offset error log * chore(usePredict): add debug logging for restored labels and model predictions * fix(validation): add offset integrity checks for stored validation labels * fix(offsets): validate alternative offsets for prediction labels to prevent invalid ranges * fix(validation): correct indentation for offset validation checks in getStoredValidation * fix(predict): use strict null checks for alternative offsets validation * fix(useFileParse): use position-based paragraph ID to avoid key collisions (#85) * fix(useFileParse): use position-based paragraph ID to avoid key collisions * fix(useFileParse): update paragraph ID mapping to use descriptive variable names * Mover configuración de pnpm de `package.json` a `pnpm-workspace.yml` (#83) * chore(pnpm): remove pnpm configuration from package.json and add pnpm-workspace.yml * chore(pnpm): rename pnpm-workspace.yaml * fix(disambiguate): match response items by text instead of array index (#78) (#86) * fix(useLocal): stop persisting groupOrder and remove dead categoryAssignments (#78) (#87) * chore(package): switch from npm to pnpm for typecheck, validate, package, and make scripts * refactor: update import paths and clean up code structure - Adjusted import paths for components to improve clarity and maintainability. - Refactored various components to use specific imports instead of bulk imports. - Cleaned up formatting and spacing for better readability. - Updated ignore patterns in biome.json to exclude additional files and directories. - Enhanced the showToast function for better readability. - Made minor adjustments to validation and prediction logic for consistency. * fix(routes): standardize '/home' path to '/home/' for consistency * fix(biome): add 'dist' to ignore list for build artifacts * chore: update package.json and pnpm-lock.yaml * 1.5.0 * fix: downgrade electron-builder to version 26.0.12 for compatibility --------- Co-authored-by: AymurAI Frontend <front-end@datagenero.org> Co-authored-by: Luciano Lapenna <lucianoglapenna@gmail.com> Co-authored-by: front-end2-lgtm <front-end2@datagenero.org> Co-authored-by: lio <lio@collectiveai.io> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Lio <lionel.chamorro85@gmail.com> Co-authored-by: Copilot <copilot@github.com>
El endpoint
/anonymizer/disambiguatedevuelve un array de párrafos procesados. El código anterior asumía que el backend respondía en el mismo orden y con la misma cantidad de elementos que el request, mapeando por índice. Si el backend reordenaba o descartaba algún párrafo, las etiquetas quedaban asignadas al párrafo incorrecto, o elparagraphIdse seteaba con el texto plano del párrafo (que tras el fix del PR #85 ya no coincide con ningún ID en el store de Redux).Cambios:
flatMap((item, i) => paragraphs[i])por una consumption queue keyed por texto: cada item del response se resuelve buscando el primer párrafo no consumido con ese texto, tolerando reordenamiento y párrafos duplicados.Resuelve parcialmente #78.
Summary by Sourcery
Align disambiguation results with their original paragraphs based on paragraph content instead of response index, and preserve predictions for paragraphs omitted by the backend.
Bug Fixes: