Skip to content

feat(annotation): configurable per-field markdown rendering#232

Draft
henrycgbaker wants to merge 21 commits into
mainfrom
feat/configurable-field-rendering-textfields
Draft

feat(annotation): configurable per-field markdown rendering#232
henrycgbaker wants to merge 21 commits into
mainfrom
feat/configurable-field-rendering-textfields

Conversation

@henrycgbaker

Copy link
Copy Markdown
Collaborator

Summary

Adds AnnotationSettings.field_render_mode (deployment / workspace / task scopes, sparse-dict overlay) controlling use_markdown on each rg.TextField. Lets a deployment opt selected fields into Argilla's markdown renderer — which also handles inline raw HTML — so content from pipelines that emit HTML (e.g. publikationsbot) displays as HTML in the annotator UI rather than escaping.

Mirrors the constraint_severity pattern from #216 very closely:

constraint_severity field_render_mode
Shape dict[str, Severity] dict[str, FieldRenderMode]
Default factory populated with all constraint IDs populated with all known field names → "plain"
Merge validator overlays user values onto defaults overlays user values onto defaults
Key validator rejects unknown constraint IDs rejects unknown field names (scope-aware)
Resolver method resolved_severity(ws, constraint_id) resolved_render_mode(ws, task, field_name)
Scope deployment + workspace deployment + workspace + task

The one difference: field_render_mode adds a per-task scope (TaskSettings.field_render_mode) since "answer" belongs to different tasks (grounding vs. generation) and a deployment might legitimately want them rendered differently. constraint_severity skips per-task because severity is per-constraint, not per-task — that reason doesn't apply here.

Files changed

  • core/schemas/annotation_task.py — adds FieldRenderMode type + TEXT_FIELDS registry
  • core/settings/annotation_settings.py — adds field_render_mode to all 3 scope classes, plus resolved_render_mode(), _merge_field_render_mode_defaults, _validate_field_render_mode_keys
  • core/annotation/argilla_task_definitions.py — local md() helper inside build_task_settings, wires use_markdown=md(task, name) into the 6 rg.TextField calls
  • docs/decisions/0009-annotation-schema-configurability.md — addendum: rendering is operational, not schema (in the same layer as min_submitted, constraint_severity, locale); the hardcoded constraint on fields/questions/labels still holds
  • Tests:
    • tests/unit/core/settings/test_annotation_settings.pyTestFieldRenderModeDefaults, TestResolvedRenderMode, TestWorkspaceFieldRenderModeValidation, TestTaskFieldRenderModeValidation
    • tests/unit/core/annotation/test_argilla_task_definitions.pyTestTextFieldsRegistryMatchesActualTextFields (drift check), TestUseMarkdownWireUp (propagation across 3 scopes)

Example YAML

# Deployment-wide: render HTML answers as markdown
field_render_mode:
  answer: markdown

# Or with per-workspace / per-task overrides:
workspaces:
  grounding:
    field_render_mode:
      context_set: markdown
    tasks:
      grounding:
        field_render_mode:
          answer: plain  # override the deployment default just here

Dependency chain

Branched on feat/annotation-logical-constraints/05-export-summary-by-id (#217 head). Depends on #213#214#215#216#217 landing first. The work itself is independent of those changes' content but uses:

Test plan

  • pytest tests/unit/ — 754 pass, no regressions
  • Manual: on a fresh test deployment, set field_render_mode: {answer: markdown} in YAML, run pragmata annotation setup, verify resulting Argilla datasets have use_markdown=True on grounding/generation answer fields (already demoed manually on a downstream demo branch).

Hoist the four binary implication rules out of three hand-written check_*
functions into a frozen ConstraintRule dataclass list. The check_* helpers
now return derived violation strings, preserving existing export CSV and
constraint_summary key formats. Adds to_widget_payload() so the same
definitions can drive an annotator-time JS widget without duplication.
The term "rule" was too generic for a binary-implication constraint between
question values; "logical constraint" reflects intent more directly. Renames:
- ConstraintRule → LogicalConstraint
- CONSTRAINT_RULES → LOGICAL_CONSTRAINTS
- rule_id field → constraint_id
- constraint_rules.py → logical_constraints.py
- violated_rules() → violated_constraints()
- LogicalConstraint.applies/violated_by: row: Any → row: object
  (matches constraints.py and reflects the actual surface — getattr-only access)
- Replace `is`/`is not` with `==`/`\!=` for boolean comparisons
  (idiomatic value comparison; ruff E712-safe)
Introduced as a structured-form alternative for callers needing severity,
but never used. The same shape is available by walking LOGICAL_CONSTRAINTS
directly when a real caller appears.
…et PR

Move severity, message, Severity Literal, and to_widget_payload off
LogicalConstraint until the constraint widget actually lands. The fields
had no consumer in this PR and the docstring referenced a
constraints_field.html template that does not yet exist.

Also drop misleading "backwards-compat" framing from constraints module
docstrings and align LogicalConstraint docstring with the runtime: the
export uses violation_string() as the constraint_summary key, not
constraint_id.
…ks.py

The module name was ambiguous now that constraint validation runs in two
places (live UI widget at annotation time, post-hoc audit at export time).
Rename to make the file's purpose explicit and expand the docstring to
spell out the dual-half story.
Adds an annotation-time UI surface for the logical constraints already
validated at export time. A new constraints_panel rg.CustomField (srcdoc
iframe + same-origin parent DOM access, mirroring the existing discard
widget pattern) re-evaluates each rule against the current chip state on
every observed change.

- Warn-severity rules (heuristic): yellow banner; submission allowed.
- Block-severity rules (logical implications): red banner; native
  Submit button is disabled until the violation is resolved.

The panel is injected above .footer-form so the warning is adjacent to
the action area. Inline styles + DOM are inserted into the parent doc,
the iframe's own host box is collapsed.

Argilla v2 record-field contract: the frontend silently skips rendering
a CustomField when the record has no value for that field, so every
record now carries a placeholder for both constraints_panel and
discard_flow via WIDGET_FIELD_PLACEHOLDERS.

Argilla DOM contract documented inline in the widget (selector targets
verified against argilla-frontend v2.8.0 source). Maintenance surface
matches the existing discard widget.
- argilla_task_definitions imports LOGICAL_CONSTRAINTS / drops CONSTRAINT_RULES
- constraints_field.html renames JS RULES → CONSTRAINTS, rule_id → constraint_id,
  and the .rule-msg/.rule-cite CSS classes → .constraint-msg/.constraint-cite
- tests updated to the new names
- Submit-gate state-machine: detect drift between visible-state signature
  and the actual `dataset.constraintBlocked` on the submit button (covers
  the case where the very first tick evaluated a block before the button
  was mounted in parent DOM)
- Cite line renders human question titles from QUESTION_TITLES rather
  than raw snake_case names
- hideHostBox catch now logs via console.warn so future Argilla DOM
  changes that hide the host class are visible in dev tools
- cardFor docstring: note dependence on one-visible-record invariant
  (Argilla v2.8.0 shows records one at a time)
…get PR

Re-add severity, message, Severity Literal, and to_widget_payload on
LogicalConstraint where the widget actually consumes them. These fields
were lifted out of the SSOT PR (01) because they had no consumer there.

Also fix stale design-doc reference (constraint_rules.CONSTRAINT_RULES
to logical_constraints.LOGICAL_CONSTRAINTS) and tighten to_widget_payload
to a direct dict literal (drops unused task field from the wire payload,
removes asdict import).
Severity is a runtime setting, not a property of the rule, so its
defaults live in AnnotationSettings.constraint_severity alongside other
deployment-scope defaults (production_min_submitted, calibration_min_submitted).

- LogicalConstraint no longer carries a severity field; the dataclass
  is now pure rule definition.
- AnnotationSettings.constraint_severity ships full defaults via
  _DEFAULT_CONSTRAINT_SEVERITY. A model_validator(mode="before") overlays
  user-supplied entries on top, so configs need only specify the
  constraint_ids they want to override (sparse overlay).
- Unknown constraint_ids are rejected at construction time.
- build_task_settings(settings) and _render_constraints_template take
  settings; the widget reads severity from settings.constraint_severity.
- @functools.cache dropped from build_task_settings since the result
  depends on settings; callers hold the dict for the import duration.
Extract the per-task fields-list assembly (content + constraints_panel +
discard_flow) and rg.Settings construction into a single inner
assemble() helper, so the three task return entries become a single
content_fields argument rather than repeating the discard+constraints
wiring per task.

Pure refactor: no behaviour change.
Adds per-workspace overrides on top of the deployment-scope defaults
introduced in the widget PR.

- WorkspaceSettings.constraint_severity: sparse constraint_id to severity
  map, empty default. Listed constraint_ids override deployment values
  for that workspace only; omitted constraint_ids fall through.
- AnnotationSettings.resolved_severity(workspace_name, constraint_id)
  walks workspace then deployment. Unknown constraint_ids raise KeyError.
- AnnotationSettings.task_to_workspace() inverts the topology, consolidating
  the lookup that record_builder.fan_out_records and build_task_settings
  both need. _invert_workspace_map removed from record_builder.
- The constraint widget now renders with the workspace-resolved severity
  per task, so per-workspace overrides reach the annotator UI.
- build_task_settings only builds Settings for tasks present in the
  workspaces topology (avoids resolving severity against a missing
  workspace; the unused rg.Settings had no consumer either way).
- Validator rejects unknown constraint_ids at workspace scope too.
CONSTRAINT_CHECKERS and the check_retrieval/check_grounding/check_generation
helpers now return list[LogicalConstraint] instead of list[str]. The per-row
CSV constraint_details column is unchanged - export_runner generates it via
LogicalConstraint.violation_string() at write time. export_meta.json's
constraint_summary is now keyed by the stable short constraint_id (e.g.
"evidence_requires_relevance") instead of the verbose violation message,
matching how AnnotationSettings.constraint_severity overrides are keyed.
Adds AnnotationSettings.field_render_mode (deployment / workspace / task
scopes, sparse-dict overlay) that controls use_markdown on each
rg.TextField. Mirrors the constraint_severity pattern with the per-task
scope extended (since each field name belongs to specific tasks).

Defaults to "plain" everywhere; users opt fields into "markdown" via YAML
to let Argilla render pipeline-emitted HTML (e.g. publikationsbot output)
in the annotator UI. Schema (which fields exist) stays hardcoded per
ADR-0009; rendering is an operational layer above it (see ADR addendum).

TEXT_FIELDS registry in core/schemas/annotation_task.py is the source
of truth for which fields can be configured; a unit test asserts it
matches the rg.TextField names actually constructed in build_task_settings.
…tants

Pull the all-known-field-names set and the all-plain default dict out
of inline comprehensions inside Field(default_factory=...) and
_validate_field_render_mode_keys. The values are derived from the
immutable TEXT_FIELDS registry and never change at runtime; recomputing
them on every AnnotationSettings() instantiation and on every key
validation is wasted work.

Also:
- correct AnnotationSettings.field_render_mode docstring to mention the
  per-task scope and the three-scope resolution walk
- correct TEXT_FIELDS docstring: drift detection is via the test suite
  (TestTextFieldsRegistryMatchesActualTextFields), not a module-load
  assertion (no such assertion exists in argilla_task_definitions.py).
Match the established codebase convention for module-level Literal
type aliases (Severity = Literal[...] in logical_constraints.py).
PEP 695 'type X = ...' would also work but is inconsistent with
existing aliases in core/.
@henrycgbaker henrycgbaker added annotation Changes affecting the annotation tool config Changes user-facing settings, defaults, configuration resolution, or runtime behavior contracts Changes schemas, type contracts, or validation logic for inputs, outputs, and persisted artifacts labels May 27, 2026
@henrycgbaker henrycgbaker marked this pull request as draft May 28, 2026 07:31
@henrycgbaker henrycgbaker force-pushed the feat/annotation-logical-constraints/05-export-summary-by-id branch from 9fe6999 to 8b9eda6 Compare June 3, 2026 13:45
@henrycgbaker henrycgbaker force-pushed the feat/annotation-logical-constraints/05-export-summary-by-id branch 5 times, most recently from 53ba408 to f9b6faa Compare July 3, 2026 12:06
Base automatically changed from feat/annotation-logical-constraints/05-export-summary-by-id to main July 3, 2026 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

annotation Changes affecting the annotation tool config Changes user-facing settings, defaults, configuration resolution, or runtime behavior contracts Changes schemas, type contracts, or validation logic for inputs, outputs, and persisted artifacts feature Adds or expands user-facing functionality size: M 200-499 LOC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant