Skip to content

fix(disambiguate): mapeo por contenido en lugar de índice de array (#78)#86

Merged
jansaldo merged 1 commit into
developfrom
fix/disambiguate-text-based-mapping
May 26, 2026
Merged

fix(disambiguate): mapeo por contenido en lugar de índice de array (#78)#86
jansaldo merged 1 commit into
developfrom
fix/disambiguate-text-based-mapping

Conversation

@jansaldo

@jansaldo jansaldo commented May 26, 2026

Copy link
Copy Markdown
Contributor

El endpoint /anonymizer/disambiguate devuelve un array de párrafos procesados. El código anterior asumía que el backend respondía en el mismo orden y con la misma cantidad de elementos que el request, mapeando por índice. Si el backend reordenaba o descartaba algún párrafo, las etiquetas quedaban asignadas al párrafo incorrecto, o el paragraphId se seteaba con el texto plano del párrafo (que tras el fix del PR #85 ya no coincide con ningún ID en el store de Redux).

Cambios:

  • Se reemplaza el flatMap((item, i) => paragraphs[i]) por una consumption queue keyed por texto: cada item del response se resuelve buscando el primer párrafo no consumido con ese texto, tolerando reordenamiento y párrafos duplicados.
  • Si el backend omite un párrafo, sus predicciones originales se preservan en lugar de descartarse silenciosamente.

Resuelve parcialmente #78.

Summary by Sourcery

Align disambiguation results with their original paragraphs based on paragraph content instead of response index, and preserve predictions for paragraphs omitted by the backend.

Bug Fixes:

  • Match disambiguation response items to paragraphs by their text content, tolerating backend reordering and duplicate paragraphs to avoid misassigned labels and invalid paragraph IDs.
  • Keep existing predictions for paragraphs that the backend omits from the disambiguation response so annotations are not silently lost.

@jansaldo jansaldo requested review from Copilot and llapenna May 26, 2026 17:33
@sourcery-ai

sourcery-ai Bot commented May 26, 2026

Copy link
Copy Markdown

Reviewer's Guide

Refactors the disambiguation result mapping to match response items to paragraphs by text content using a consumption queue, and preserves predictions for paragraphs that the backend omits, instead of relying on brittle array index-based mapping that could misalign labels or lose annotations.

File-Level Changes

Change Details Files
Match disambiguation response items to paragraphs by content using a per-text consumption queue instead of relying on array indices.
  • Replace index-based mapping over parsed disambiguation data with a lookup queue keyed by paragraph text to handle reordering and drops from the backend response.
  • Build a Map from paragraph text to an ordered list of Paragraphs to support duplicate texts and track consumption per text key.
  • Track consumed items per text value and skip response items that cannot be matched to any remaining paragraph.
src/renderer/src/services/aymurai/queries.ts
Ensure paragraph identifiers and predictions remain stable and preserve annotations when the backend omits paragraphs.
  • Always use the matched paragraph.id when building the origByPos lookup and assigning paragraphId on resulting PredictLabel objects, removing fallbacks to the plain text document field.
  • Capture a set of matched paragraph IDs while mapping disambiguation results.
  • After mapping, re-append any original predictions for paragraphs not returned by the backend so those annotations are not silently discarded.
src/renderer/src/services/aymurai/queries.ts

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've reviewed your changes and they look great!


Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes incorrect paragraph-to-disambiguation-result alignment in the /anonymizer/disambiguate flow by matching backend response items back to the original paragraphs by paragraph content (text) instead of array index, and it avoids losing existing annotations when the backend drops paragraphs from the response.

Changes:

  • Replace index-based mapping (paragraphs[i]) with a per-text “consumption queue” to match response items to the first unconsumed paragraph with the same text.
  • Ensure returned PredictLabel.paragraphId is always the stable paragraph id (not the raw paragraph text from the backend).
  • Preserve original predictions for any paragraph omitted by the backend response.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 170 to +239
@@ -207,10 +225,18 @@ export const disambiguate = (file: DocFile) =>
aymurai_label_instance: l.attrs.aymurai_label_instance ?? null,
aymurai_disambiguation: l.attrs.aymurai_disambiguation ?? null,
},
paragraphId: paragraph?.id ?? item.document,
paragraphId: paragraph.id,
} satisfies PredictLabel;
});
});

// Preserve raw predictions for any paragraph the backend did not return,
// so a backend count drop never silently erases annotations.
const unmatched = predictions.filter(
(p) => !matchedParagraphIds.has(p.paragraphId),
);

return [...disambiguated, ...unmatched];
@jansaldo jansaldo merged commit 47fb1fb into develop May 26, 2026
2 checks passed
@jansaldo jansaldo deleted the fix/disambiguate-text-based-mapping branch May 26, 2026 19:50
jansaldo added a commit that referenced this pull request Jun 11, 2026
* chore(routing): migrate to tanstack router

* build: move tanstack router dependencies to devDependencies

* fix(styling): import styled function from stitches instance

* fix(routing): used wrong parms in route, switched to correct path

* fix(schema): make predict alt values nullable

* chore: fix pr comments

* chore: remove comments

* fix: using react query to download anonymized file

* chore: remove deprecated code and functions

* fix: don't revoke api response blob url

* chore: add format package script and ignore route tree gen file

* chore: format code

* chore: lint code

* fix: add file name to anonymize query key

* chore: remove global axios interceptor

* chore: type global router context

* chore: set stale time to default

* chore: kill cache when entering home layout

* docs: update readme

* build: add support for environmental variables for both web and electron apps

* build: install and configure pandacss

* chore: flag stitches as deprecated

* feat: add loading page and updated branding images

* chore: configure view transition for all pages

* feat: add loading screen on boot, timer of 1.5s

* chore: re-implement button and partially input

* chore: fix some tokens in panda and move stitches global styles

* chore: add changes to router file

* chore: change to named export on local store

* chore: replace custom use mutation hook with base on connect to host hook

* fix: add lineheight to text styles and adjust font weight

* fix: add fixed height to button and auto adjust icon size

* feat: create components to render in home screen

* feat: create link card tool for features

* feat: redesign home

* fix: extra character in home layout and rename the component

* chore(styles): add animation semantic tokens

* feat: add more brand images

* feat(components): create brand, layout and ui components

* fix: replace brand images with correct ones and set proper heights

* feat: create modern tooltip component

* chore: flag card as deprecated

* chore: create modern ui components

* fix: remove fadeIn scaling animation

* build: update react and add radix dependencies

* chore: export constants

* fix: adjust stepper styles (sizing and colors)

* chore: refactor header so we can correctly position all elements

* chore: use constants and base card element on feature selector

* chore: migrate hidden input

* build: add web or electron run modes

* fix: typo on anonymizer label

* chore: add tutorial seen to local storage store

* feat: make card clickable

* chore: adjust button sizes and enum import

* chore: simplify main app layout

* feat: rework onboarding page

* feat(ui): create and/or adjust components

* fix: update enum import

* chore: rework layout components

* chore: ignore personal analysis folder

* feat: improve topbar accessibility with semantic icons and aria labels

* refactor(layout): hoist Topbar and Stepper to global wizard route

* build: add i18n

* feat: add i18n support for the whole app

* fix(layout): address PR review on header and icon changes

- Replace House icon with DotsNine in FeaturesMenu
- Replace Info icon with Question in HowItWorksModal
- Remove Header from shared route layout (route.tsx)
- Restore Header to onboarding.tsx and add it to each feature page
- Each page now controls its own Header, Stepper step, and right section

* feat: add api base url protection and apply it

* chore: restructure HOC to be a regular component

* chore: cleanup

* fix: redo home layout

* feat: add more copies to locale file

* feat: use a11y on finish page

* fix: add missing feature param call on route.tsx

* chore: kill unused old hidden input component

* feat: add more copies for the finish page

* feat: create label manager component

* feat: create switch component

* chore: minor changes

* fix: title in anonymizer locale

* chore: remove stepper component

* chore: refactor Searchbar

* feat: create initial label manager

* feat: add feature icon record

* add disambiguate and predict react query options

* finish onboarding page

* finish preview page

* improve layout header component

* connect rest of label manager

* add label manager to file annotator

* make search bar static and follow scroll

* use translation on droparea

* clear files on features menu

* make dataset validation file annotator not annotable

* add missing built by in features page

* drop file queries when resetting the progress

* add className prop to footer

* simplify finish dataset and anonymizer

* update decision tabs to panda

* update file processing component

* hide file stepper selector arrows depending on cursor

* fix gaps on finish page

* add checked variant to button

* update suggestion mark component's styles

* improve suggestion component

* create toast component

* create callout from toast, and then apply a11y to toast

* remove old component implementations

* fix title in process copy etxt

* adjust callout styling

* adjust styling and positioning on preview page

* create better select component

* kill more unused components

* minor styling fixes + new select imports

* add retro compatibility features to select

* add custom icons to callout, toast and showToast

* add input size variants

* remove old tooltip implementation

* add radix's select as dependency

* initial mark component update

* improve ui dialog component

* add small version of select component

* create search tagger

* extract tagger popover logic

* update button icons

* swap replace button icons

* deprecate usage of useSchemedMutation

* deprecate usage of useSchemedQuery and useSchemedQueries

* add odt to pdf conversion

* update tanstack react query version

* add className to SectionTitle

* improve accessibility on host page

* finish major pages and add file protection feature

* finish mark and tag annotation feature

* adjust styles on callout

* fix select viewport and add scroll

* add size variant to suggestion and adjust display

* add max width to toast and remove instad of dismiss it

* move createAnnotationData, tag and suffix to context

* disambiguate hook

* adjust spacing on dataset validation

* move suggestion label and add mark props

* store label manager config in local storage

* add more anonymizer copy locale texts

* add metadata to search and tag annotations

* simplify annotation components

* add remove dialog to label manager's entity tab

* smaller fixes on file annotation components

* fix TS issues

* adjust predicting and file parser flows

* add useShallow to local storage default getters

* simplify and remove unused code on fille annotator component

* add value as undefined on ui/input component

* add random cannonical id on search add event

* add update by cannonical id function

* add more canonical id operations on reducers

* adjust typings

* remove axios' default api url

* add fixed height to validation dataset page

* make suggestion clickable on uncontrolled text input

* more fixes in height to validate dataset

* include label policies and annotations when anonymizing document

* fix ts clone element issue

* add knip

* make OS taskbar API safe to call in web

* add missing anonymize value on label attrs

* fix electron ts issues

* make how it works modal bigger

* restore useNotify feature in process page

* fix issue regarding annotation keyboard navigation

* use base_url when dealing with public assets

* 1.25.0

* copy changes on label manager config tab

* copy change on label manager tab

* responsiveness for screens less than 1280px in width

* create pdf to odt service

* use extension to check if file conversion is needed

* feat: add back-to-home button on features page

Add a BackButton (arrow left) next to the greeting title on /home/features
that navigates back to /home/host (Local/Servidor selection).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use House icon in header instead of BackButton arrow

Place House icon button next to FeaturesMenu (DotsNine) in the header
right section, matching the original design where the home button was
in the top-right area. Navigates to /home/host.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: increase spacing between home and features menu buttons

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* limit concurrent predict requests to avoid connection exhaustion

* add error handling to finish file conversion

* prevent semaphores underflow

* copy change

* remove copies and functions referencing .doc files

* change NINO to NIÑO

* add "Entidad" placeholder to tagger select

* create link component

* Restore delete-one and delete-all hover actions on annotations (#57)

* feat: restore delete-one and delete-all hover actions on annotations

- Create RemoveDialog confirmation component for tag-annotation
- Add delete-one and delete-all buttons to Tagger popover
- Wire callbacks through AnnotationPopover to TagAnnotation
- Uses existing remove/removeByText from useAnnotation context

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(annotations): clarify hover remove/replace scope to text-based matching

- Remove canonical_entity_id branch from confirmReplaceAll; hover actions
  now always operate by exact text match (removeByText / updateByText)
- Update remove-dialog copy: title and body now reflect text-based removal
- Update replace-dialog copy: body shows affected text alongside new label
- Pass annotation text to both dialogs via new `text` prop

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: jansaldo <julianansaldo@gmail.com>

* Fix text overflow in HowItWorksModal (#58)

* fix: responsive maxW on HowItWorksModal to prevent text overflow

Add maxWidth and overflowY to the modal content so card text
stops overflowing at viewport widths < 900px.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use 90vw width (max 1200px) for HowItWorksModal

The previous 900px constraint was too narrow for the content.
Now the modal uses 90% of viewport width up to 1200px.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: move howItWorks steps to feature namespaces

Co-authored-by: Copilot <copilot@github.com>

* fix: align HowItWorksModal dimensions and card layout with onboarding

Co-authored-by: Copilot <copilot@github.com>

* fix: correct Spanish language typos in anonymizer and common localization files

* chore: update anonymizer howItWorks step 2 subtitle

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: jansaldo <julianansaldo@gmail.com>
Co-authored-by: Copilot <copilot@github.com>

* feat: add tooltips to tagger label and suffix inputs

* fix: prevent select caret rotation from leaking ancestor data-state

* create home button component

* make aymurai's logo a link in the header

* add home button to header on all flow's pages

* remove slot checks on header

* make hover effect in button work for anchor tanstack link wrapper

* add a "config" button in features menu

* chore: add .npmrc to configure public hoist pattern for @types

* Add drag-and-drop reordering and inline rename to label manager (#60)

* feat: add drag-and-drop reordering and inline rename to label manager

- Install @dnd-kit/core, @dnd-kit/sortable, @dnd-kit/utilities
- Add groupOrder and categoryAssignments to Zustand store (persisted)
- Create SortableGroup component with drag handle
- Wire DndContext + SortableContext per category in entity-tab
- Add inline rename: pencil icon toggles input, Enter saves, Escape cancels

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(anonymizer): track stable mention ids

* feat(anonymizer): centralize active prediction filtering

* feat(label-manager): group labels by anonymizer category

* feat(label-manager): derive canonical entity groups

* feat(label-manager): support group edits and merges

* feat(label-manager): improve panel layout and hover feedback

* feat(anonymizer): add entity normalization and similarity helpers

* feat(anonymizer): resolve manual annotations into entity groups

* feat(label-manager): split mentions into new entity groups

* feat(anonymizer): validate annotations before export

* chore(anonymizer): recategorize USUARIX label

* feat(anonymizer): add searchable grouped label select

* feat(label-manager): allow clearing excluded terms

* fix(annotation-popover): stabilize hover and focus behavior

* feat(search): manage active matches in file annotator

* feat(search): preserve search state on tagged annotations

* fix(annotations): support selections across split spans

* feat(annotations): apply bulk edits to normalized occurrences

* fix(entity-groups): detect token-order duplicate aliases

* fix(aymurai): scope query cache keys by API base URL

* style(entity-resolution): tighten dialog width

* fix(dependencies): move dnd-kit packages from devDependencies to dependencies

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: jansaldo <julianansaldo@gmail.com>

* feat(validation): persist and restore predictions via backend validation endpoint (#68)

* refactor: minor biome formatting and variable naming cleanup

* test: set up vitest with @-alias path resolution

* feat(schema): export predictLabelSchema for validation reuse

* feat(validation): load stored validation before predict to skip re-processing

* fix(anonymizer): preserve excluded entities in anonymize-document payload

* test: add unit tests for normalizeValidationLabel and predictions utils

* feat(dependencies): add dnd-kit packages for drag-and-drop functionality

* fix(file-annotator): fix upward autoscroll on search previous navigation (#76)

* Fix/homogenize file check ui (#77)

* fix(spinner): add viewBox attribute to SVG for proper scaling

* fix(styles): homogenize maxWidth and dimensions for file check components

* fix(components): change subtitle from h3 to h2 for consistency with preview screen

* fix(file-check): add size prop to Text component for consistent styling

* Fix/responsive home layout (#80)

* fix(home-layout): replace absolute-positioned BuiltBy with flex in-flow layout

* fix(features): move BuiltBy from absolute position to Footer component

* fix(main-content): add overflow-y auto to prevent content overflow on small screens

* fix(home-layout): adjust MainContent and HomeLayout styles for better responsiveness

* fix(home-layout): restore padding-bottom to inner layout for improved spacing

* Feat/entity manager mention feedback (#81)

* feat(entity-manager): enhance text item styles and group creation logic

* feat(file-annotator): initialize label manager state based on isAnnotable prop

* fix(entity-tab): preserve native context menu for single-mention groups

* Fix/invalid entity offsets (#82)

* feat(export-validation): add issue summary and table to offset error log

* chore(usePredict): add debug logging for restored labels and model predictions

* fix(validation): add offset integrity checks for stored validation labels

* fix(offsets): validate alternative offsets for prediction labels to prevent invalid ranges

* fix(validation): correct indentation for offset validation checks in getStoredValidation

* fix(predict): use strict null checks for alternative offsets validation

* fix(useFileParse): use position-based paragraph ID to avoid key collisions (#85)

* fix(useFileParse): use position-based paragraph ID to avoid key collisions

* fix(useFileParse): update paragraph ID mapping to use descriptive variable names

* Mover configuración de pnpm de `package.json` a `pnpm-workspace.yml` (#83)

* chore(pnpm): remove pnpm configuration from package.json and add pnpm-workspace.yml

* chore(pnpm): rename pnpm-workspace.yaml

* fix(disambiguate): match response items by text instead of array index (#78) (#86)

* fix(useLocal): stop persisting groupOrder and remove dead categoryAssignments (#78) (#87)

* chore(package): switch from npm to pnpm for typecheck, validate, package, and make scripts

* refactor: update import paths and clean up code structure

- Adjusted import paths for components to improve clarity and maintainability.
- Refactored various components to use specific imports instead of bulk imports.
- Cleaned up formatting and spacing for better readability.
- Updated ignore patterns in biome.json to exclude additional files and directories.
- Enhanced the showToast function for better readability.
- Made minor adjustments to validation and prediction logic for consistency.

* fix(routes): standardize '/home' path to '/home/' for consistency

* fix(biome): add 'dist' to ignore list for build artifacts

* chore: update package.json and pnpm-lock.yaml

* 1.5.0

* fix: downgrade electron-builder to version 26.0.12 for compatibility

---------

Co-authored-by: AymurAI Frontend <front-end@datagenero.org>
Co-authored-by: Luciano Lapenna <lucianoglapenna@gmail.com>
Co-authored-by: front-end2-lgtm <front-end2@datagenero.org>
Co-authored-by: lio <lio@collectiveai.io>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Lio <lionel.chamorro85@gmail.com>
Co-authored-by: Copilot <copilot@github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants