Feature: Add Custom Regex Builder UI with Sample-Based Generation

[goobz22]

Summary

Add a regex builder tool to the UI that allows users to generate custom regex patterns from sample data (log lines, UUIDs, etc.). Uses a sophisticated custom-built regex inference algorithm - no external packages.

Use Case

Users often encounter unique data formats in their logs or systems that aren't covered by built-in patterns. Instead of requiring regex expertise, the tool should:

1. Accept sample data (e.g., a log line, a custom UUID format)
2. Generate a matching regex pattern using our custom inference engine
3. Allow users to test/refine the pattern
4. Submit both the regex AND the original sample data to the feedback system

Regex Inference Algorithm (Custom Implementation)

Build a sophisticated regex generator from scratch:

• Tokenization: Break sample into character classes (digits, letters, special chars, whitespace)
• Pattern Detection: Identify repeating structures, delimiters, fixed vs variable length segments
• Generalization: Convert specific values to regex character classes
• Optimization: Simplify and combine adjacent similar patterns
• Validation: Ensure generated regex matches the original sample

No external packages - pure JavaScript implementation.

Requirements

Regex Builder UI

• Input field for pasting sample data (single line or multi-line)
• Auto-generate regex from the sample using custom algorithm
• Live preview showing what the regex matches
• Ability to highlight/select specific parts of the sample to target
• Editable regex output for manual refinement
• Test area to validate the regex against additional samples

Integration with Feedback Loop

• 'Submit Pattern' button that sends to POST /api/feedback
• Payload should include:
  • regex: The generated/refined regex pattern
  • sampleData: The original data used to create the pattern
  • patternName: User-suggested name for the pattern
  • category: Suggested category (PII, System, Financial, Custom)

UI Location

• New tab/section in the main UI: 'Pattern Builder'
• Accessible from the main redaction view

Tasks

• Create regex inference engine in packages/core/src/regex-builder/
• Implement tokenization and pattern detection algorithms
• Create regex builder component in packages/ui
• Add live preview/testing functionality
• Integrate with feedback API endpoint (Issue Feature: Implement Feedback Loop for Missed Redactions #5)
• Add pattern submission flow
• Include validation to prevent submitting overly broad patterns

Dependencies

• Issue Feature: Implement Feedback Loop for Missed Redactions #5 (Feedback Loop) should be implemented first or in parallel

[goobz22]

Implementation Notes Based on Codebase Review

Current UI Architecture

• Single-page app in packages/ui/app/page.tsx (~1,098 lines)
• Three tabs: Simple Config, JSON Editor, Output Format
• Uses React hooks for state management
• Patriotic theme: Navy (#002868), Red (#BF0A30), Gold (#FFD700)

Suggested Implementation

1. Add new tab to the UI
Extend TabType to include 'builder' tab

2. Create Regex Builder Component

Key features:

• Sample input textarea with selection detection
• Auto-generate regex when text is pasted or selected
• Editable regex output field for manual refinement
• Test inputs array with live match preview
• Pattern name and category inputs
• Submit button to send to feedback API

3. Regex Inference Logic

The inferRegexFromSample function should:

• Detect common patterns (UUID, IP, email formats)
• For unknown patterns: escape special chars and generalize digits/letters
• Support partial selection to target specific portions

4. Live Testing
Show green checkmark with match count or red X for each test input

Integration with Feedback API (Issue #5)

Submit payload:

• regex: The generated/refined pattern
• sampleData: Original sample text
• patternName: User-suggested name
• category: pii | system | financial | custom

Dependencies

• Issue Feature: Implement Feedback Loop for Missed Redactions #5 (Feedback Loop) for submission endpoint
• Consider using a library like regexgen for smarter pattern inference

UI Location

Add as 4th tab: Pattern Builder - accessible alongside Simple, JSON, and Output tabs

[goobz22]

Implementation Complete

Created custom regex builder engine and UI:

Core Engine (packages/core/src/regex-builder/)

• tokenizer.ts - Classifies input into tokens (digits, letters, hex, special chars)
• pattern-detector.ts - Detects known patterns (UUID, IPv4, MAC) and generates segments
• optimizer.ts - Optimizes and validates generated regex
• index.ts - Main API exports

API

import { generateFromSample, refineFromSamples, testPattern } from '@data-redactor/core'

const result = generateFromSample('ABC-12345-XYZ')
// result.regex = '\b[A-Z]{3}-\d{5}-[A-Z]{3}\b'
// result.valid = true
// result.suggestedName = 'alphanumeric-mixed-pattern'

UI (Pattern Builder Tab)

• Sample input textarea (supports multiple samples)
• Word boundary and case-insensitive options
• Generated regex display with copy button
• Validation status indicator
• Pattern structure visualization (segment chips)
• Live pattern testing
• Add to configuration button

Files Created/Modified

• packages/core/src/regex-builder/*.ts - Engine files
• packages/ui/index.html - Pattern Builder tab
• packages/ui/main.js - Builder handlers
• packages/ui/styles.css - Builder styles

Ready to close.

[goobz22]

Correction

The previous comment incorrectly stated 'Ready to close'. There are still 2 tasks remaining:

Still Pending:

• Integrate with feedback API endpoint (Issue Feature: Implement Feedback Loop for Missed Redactions #5)
• Add pattern submission flow

The core regex builder engine and UI are complete, but the integration with the feedback API for pattern submission is not yet implemented. This issue should remain open until those tasks are completed.