feat(annotation): configurable per-field markdown rendering#232
Draft
henrycgbaker wants to merge 21 commits into
Draft
feat(annotation): configurable per-field markdown rendering#232henrycgbaker wants to merge 21 commits into
henrycgbaker wants to merge 21 commits into
Conversation
Hoist the four binary implication rules out of three hand-written check_* functions into a frozen ConstraintRule dataclass list. The check_* helpers now return derived violation strings, preserving existing export CSV and constraint_summary key formats. Adds to_widget_payload() so the same definitions can drive an annotator-time JS widget without duplication.
The term "rule" was too generic for a binary-implication constraint between question values; "logical constraint" reflects intent more directly. Renames: - ConstraintRule → LogicalConstraint - CONSTRAINT_RULES → LOGICAL_CONSTRAINTS - rule_id field → constraint_id - constraint_rules.py → logical_constraints.py - violated_rules() → violated_constraints()
- LogicalConstraint.applies/violated_by: row: Any → row: object (matches constraints.py and reflects the actual surface — getattr-only access) - Replace `is`/`is not` with `==`/`\!=` for boolean comparisons (idiomatic value comparison; ruff E712-safe)
Introduced as a structured-form alternative for callers needing severity, but never used. The same shape is available by walking LOGICAL_CONSTRAINTS directly when a real caller appears.
…et PR Move severity, message, Severity Literal, and to_widget_payload off LogicalConstraint until the constraint widget actually lands. The fields had no consumer in this PR and the docstring referenced a constraints_field.html template that does not yet exist. Also drop misleading "backwards-compat" framing from constraints module docstrings and align LogicalConstraint docstring with the runtime: the export uses violation_string() as the constraint_summary key, not constraint_id.
…ks.py The module name was ambiguous now that constraint validation runs in two places (live UI widget at annotation time, post-hoc audit at export time). Rename to make the file's purpose explicit and expand the docstring to spell out the dual-half story.
Adds an annotation-time UI surface for the logical constraints already validated at export time. A new constraints_panel rg.CustomField (srcdoc iframe + same-origin parent DOM access, mirroring the existing discard widget pattern) re-evaluates each rule against the current chip state on every observed change. - Warn-severity rules (heuristic): yellow banner; submission allowed. - Block-severity rules (logical implications): red banner; native Submit button is disabled until the violation is resolved. The panel is injected above .footer-form so the warning is adjacent to the action area. Inline styles + DOM are inserted into the parent doc, the iframe's own host box is collapsed. Argilla v2 record-field contract: the frontend silently skips rendering a CustomField when the record has no value for that field, so every record now carries a placeholder for both constraints_panel and discard_flow via WIDGET_FIELD_PLACEHOLDERS. Argilla DOM contract documented inline in the widget (selector targets verified against argilla-frontend v2.8.0 source). Maintenance surface matches the existing discard widget.
- argilla_task_definitions imports LOGICAL_CONSTRAINTS / drops CONSTRAINT_RULES - constraints_field.html renames JS RULES → CONSTRAINTS, rule_id → constraint_id, and the .rule-msg/.rule-cite CSS classes → .constraint-msg/.constraint-cite - tests updated to the new names
- Submit-gate state-machine: detect drift between visible-state signature and the actual `dataset.constraintBlocked` on the submit button (covers the case where the very first tick evaluated a block before the button was mounted in parent DOM) - Cite line renders human question titles from QUESTION_TITLES rather than raw snake_case names - hideHostBox catch now logs via console.warn so future Argilla DOM changes that hide the host class are visible in dev tools - cardFor docstring: note dependence on one-visible-record invariant (Argilla v2.8.0 shows records one at a time)
…get PR Re-add severity, message, Severity Literal, and to_widget_payload on LogicalConstraint where the widget actually consumes them. These fields were lifted out of the SSOT PR (01) because they had no consumer there. Also fix stale design-doc reference (constraint_rules.CONSTRAINT_RULES to logical_constraints.LOGICAL_CONSTRAINTS) and tighten to_widget_payload to a direct dict literal (drops unused task field from the wire payload, removes asdict import).
Severity is a runtime setting, not a property of the rule, so its defaults live in AnnotationSettings.constraint_severity alongside other deployment-scope defaults (production_min_submitted, calibration_min_submitted). - LogicalConstraint no longer carries a severity field; the dataclass is now pure rule definition. - AnnotationSettings.constraint_severity ships full defaults via _DEFAULT_CONSTRAINT_SEVERITY. A model_validator(mode="before") overlays user-supplied entries on top, so configs need only specify the constraint_ids they want to override (sparse overlay). - Unknown constraint_ids are rejected at construction time. - build_task_settings(settings) and _render_constraints_template take settings; the widget reads severity from settings.constraint_severity. - @functools.cache dropped from build_task_settings since the result depends on settings; callers hold the dict for the import duration.
…rop module constant
…lections with bold values
Extract the per-task fields-list assembly (content + constraints_panel + discard_flow) and rg.Settings construction into a single inner assemble() helper, so the three task return entries become a single content_fields argument rather than repeating the discard+constraints wiring per task. Pure refactor: no behaviour change.
Adds per-workspace overrides on top of the deployment-scope defaults introduced in the widget PR. - WorkspaceSettings.constraint_severity: sparse constraint_id to severity map, empty default. Listed constraint_ids override deployment values for that workspace only; omitted constraint_ids fall through. - AnnotationSettings.resolved_severity(workspace_name, constraint_id) walks workspace then deployment. Unknown constraint_ids raise KeyError. - AnnotationSettings.task_to_workspace() inverts the topology, consolidating the lookup that record_builder.fan_out_records and build_task_settings both need. _invert_workspace_map removed from record_builder. - The constraint widget now renders with the workspace-resolved severity per task, so per-workspace overrides reach the annotator UI. - build_task_settings only builds Settings for tasks present in the workspaces topology (avoids resolving severity against a missing workspace; the unused rg.Settings had no consumer either way). - Validator rejects unknown constraint_ids at workspace scope too.
CONSTRAINT_CHECKERS and the check_retrieval/check_grounding/check_generation helpers now return list[LogicalConstraint] instead of list[str]. The per-row CSV constraint_details column is unchanged - export_runner generates it via LogicalConstraint.violation_string() at write time. export_meta.json's constraint_summary is now keyed by the stable short constraint_id (e.g. "evidence_requires_relevance") instead of the verbose violation message, matching how AnnotationSettings.constraint_severity overrides are keyed.
Adds AnnotationSettings.field_render_mode (deployment / workspace / task scopes, sparse-dict overlay) that controls use_markdown on each rg.TextField. Mirrors the constraint_severity pattern with the per-task scope extended (since each field name belongs to specific tasks). Defaults to "plain" everywhere; users opt fields into "markdown" via YAML to let Argilla render pipeline-emitted HTML (e.g. publikationsbot output) in the annotator UI. Schema (which fields exist) stays hardcoded per ADR-0009; rendering is an operational layer above it (see ADR addendum). TEXT_FIELDS registry in core/schemas/annotation_task.py is the source of truth for which fields can be configured; a unit test asserts it matches the rg.TextField names actually constructed in build_task_settings.
This was referenced May 27, 2026
…tants Pull the all-known-field-names set and the all-plain default dict out of inline comprehensions inside Field(default_factory=...) and _validate_field_render_mode_keys. The values are derived from the immutable TEXT_FIELDS registry and never change at runtime; recomputing them on every AnnotationSettings() instantiation and on every key validation is wasted work. Also: - correct AnnotationSettings.field_render_mode docstring to mention the per-task scope and the three-scope resolution walk - correct TEXT_FIELDS docstring: drift detection is via the test suite (TestTextFieldsRegistryMatchesActualTextFields), not a module-load assertion (no such assertion exists in argilla_task_definitions.py).
Match the established codebase convention for module-level Literal type aliases (Severity = Literal[...] in logical_constraints.py). PEP 695 'type X = ...' would also work but is inconsistent with existing aliases in core/.
9fe6999 to
8b9eda6
Compare
53ba408 to
f9b6faa
Compare
Base automatically changed from
feat/annotation-logical-constraints/05-export-summary-by-id
to
main
July 3, 2026 13:56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
AnnotationSettings.field_render_mode(deployment / workspace / task scopes, sparse-dict overlay) controllinguse_markdownon eachrg.TextField. Lets a deployment opt selected fields into Argilla's markdown renderer — which also handles inline raw HTML — so content from pipelines that emit HTML (e.g. publikationsbot) displays as HTML in the annotator UI rather than escaping.Mirrors the
constraint_severitypattern from #216 very closely:constraint_severityfield_render_modedict[str, Severity]dict[str, FieldRenderMode]"plain"resolved_severity(ws, constraint_id)resolved_render_mode(ws, task, field_name)The one difference:
field_render_modeadds a per-task scope (TaskSettings.field_render_mode) since"answer"belongs to different tasks (grounding vs. generation) and a deployment might legitimately want them rendered differently.constraint_severityskips per-task because severity is per-constraint, not per-task — that reason doesn't apply here.Files changed
core/schemas/annotation_task.py— addsFieldRenderModetype +TEXT_FIELDSregistrycore/settings/annotation_settings.py— addsfield_render_modeto all 3 scope classes, plusresolved_render_mode(),_merge_field_render_mode_defaults,_validate_field_render_mode_keyscore/annotation/argilla_task_definitions.py— localmd()helper insidebuild_task_settings, wiresuse_markdown=md(task, name)into the 6rg.TextFieldcallsdocs/decisions/0009-annotation-schema-configurability.md— addendum: rendering is operational, not schema (in the same layer asmin_submitted,constraint_severity,locale); the hardcoded constraint on fields/questions/labels still holdstests/unit/core/settings/test_annotation_settings.py—TestFieldRenderModeDefaults,TestResolvedRenderMode,TestWorkspaceFieldRenderModeValidation,TestTaskFieldRenderModeValidationtests/unit/core/annotation/test_argilla_task_definitions.py—TestTextFieldsRegistryMatchesActualTextFields(drift check),TestUseMarkdownWireUp(propagation across 3 scopes)Example YAML
Dependency chain
Branched on
feat/annotation-logical-constraints/05-export-summary-by-id(#217 head). Depends on #213 → #214 → #215 → #216 → #217 landing first. The work itself is independent of those changes' content but uses:build_task_settings(settings)signature from refactor(annotation): DRY build_task_settings via assemble() helper #215task_to_workspace()helper + sparse-dict-overlay pattern from feat(annotation): configurable constraint severity at deployment + workspace scopes #216Test plan
pytest tests/unit/— 754 pass, no regressionsfield_render_mode: {answer: markdown}in YAML, runpragmata annotation setup, verify resulting Argilla datasets haveuse_markdown=Trueon grounding/generation answer fields (already demoed manually on a downstream demo branch).