Skip to content

Custom plugins are silently dropped on schedule-fuzzed runs #50

Description

@twof

Summary

When scheduleFuzzing: true, any custom plugins passed to fuzz(...) are silently ignored. Observer/analysis plugins (stop conditions, issue recording, custom telemetry) don't run on schedule-fuzzed campaigns, with no error or warning — only a doc note.

Why it happens

Schedule fuzzing reframes the campaign over an extended input pack ([UInt8], repeat each Input) (schedule bytes prepended as element 0), run by runFlattenedSchedule:

  • Sources/PropertyTestingKit/Fuzzing/ScheduleFlatten.swift:138-155 — runs the extended pack and passes makeHandlers: { [] } (no plugins).
  • runFlattenedSchedule has no plugins parameter (ScheduleFlatten.swift:101-114); the caller in FuzzAPI.swift:205-219 never forwards it.

The user's plugins are typed FuzzPlugin<repeat each Input> (over the user pack), but the extended engine needs FuzzPlugin<[UInt8], repeat each Input>. The test closure is easy to bridge — peelTest drops element 0 on the way in (ScheduleFlatten.swift:130-132) — but a FuzzPlugin is bidirectional: it reads the iteration context and emits actions (e.g. selectForMutation) that carry an input of its own pack type back into the engine. Lifting a plugin therefore requires peeling element 0 on every context in, and re-synthesizing a valid schedule byte-string for every input the plugin re-injects out. That adapter was never built, so scheduled runs just drop plugins.

(The scheduler is threaded through — ScheduleFlatten.swift:147 — because post-#43 it's engine-owned and operates on the extended pack natively. So the gap is specifically observer/analysis plugins, not mutation scheduling.)

Impact

  • parallelism is already forced to 1 on scheduled runs; combined with dropped plugins, a user relying on a stop plugin or a custom analysis plugin gets different behavior than they asked for, silently.
  • Documented but not enforced: FuzzAPI.swift:111 and ScheduleFlatten.swift:99 note the limitation; nothing fails or warns if a caller passes plugins anyway.

Suggested directions

  1. Lift plugins onto the extended pack. Build a FuzzPlugin<repeat each Input> -> FuzzPlugin<[UInt8], repeat each Input> adapter that:
    • peels element 0 from each IterationContext before calling the user handler, and
    • re-extends any input-carrying action the handler emits (selectForMutation, etc.) by prepending a schedule byte-string (the current input's element 0, or a freshly generated one).
      Analysis-only plugins (AnalysisPlugin, which emit just stop/recordIssue and don't carry inputs) are the easy subset — lift those first; they cover the common stop/record use cases without the re-injection problem.
  2. At minimum, fail loudly. If full lifting is out of scope, throw or log when scheduleFuzzing is set and a non-empty plugins factory is supplied, instead of silently ignoring them.

Acceptance

  • Custom plugins (at least analysis-only ones) run on schedule-fuzzed campaigns, OR passing plugins with scheduleFuzzing: true produces a clear error rather than silent no-op.
  • Plugin-emitted actions that re-inject inputs remain valid on the extended pack (each re-injected input carries a usable schedule byte-string).

Context: surfaced reviewing the engine-owned scheduler path (PR #43). Related coarse-mechanism issues: #48 (Character mutator), #49 (Double mutator).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions