We should extend the quality control in src/mdner_llm/annotations/validate_annotations.py to verify textual proximity between annotations.
This would help detect near-duplicate or overly similar texts that may reduce dataset quality.
Suggestion:
Use text embeddings (e.g. cosine similarity) to flag pairs of annotations above a similarity threshold.
We should extend the quality control in
src/mdner_llm/annotations/validate_annotations.pyto verify textual proximity between annotations.This would help detect near-duplicate or overly similar texts that may reduce dataset quality.
Suggestion:
Use text embeddings (e.g. cosine similarity) to flag pairs of annotations above a similarity threshold.