Skip to content

schemathesis_datagen_pipeline: state the Target 1 module interface - #54

Draft
findalexli wants to merge 1 commit into
ByteDance-Seed:mainfrom
findalexli:fix/datagen-module-interface
Draft

findalexli wants to merge 1 commit into
ByteDance-Seed:mainfrom
findalexli:fix/datagen-module-interface

Conversation

@findalexli

Copy link
Copy Markdown
Contributor

Task finding: #49.

What and why

Hidden Phase 1 imports names from schemathesis.specs.openapi.headers and checks literal format keys and Range forms that TODO.md doesn't give. A module with other names fails the import and loses all of Phase 1, 15.38 of 100 points. This PR adds a "Target 1 Module Interface" section to instruction.md and adjusts one rule line to allow it.

It goes in instruction.md because environment/Dockerfile pins TODO.md to v1's commit. Putting it in TODO.md would mean updating the pin and accepting new image tags.

Old vs new judge

Both trees build the same images: judge 484f15fd657f, work 8911231cbfba; 18 of 18 files under environment/ and judge/ have the same sha256. One worker, tools/judge_local.py, task limits. "Names" is an alias layer we added to the real archive, standing in for a solver told them; we didn't measure how a solver reacts to this text.

Submission Old New Phase 1
Opus 5.5 solver, unchanged 72.31 (3 of 3) 72.31 0 of 15
Same, with names and keys 86.67, 85.64, 86.67 85.64 13 or 14 of 15
That plus the three literal Range forms 87.69 (3 of 3) 87.69 15 of 15
Empty starter plus headers.py from Schemathesis 4.15.0 47.73 (3 of 3) 47.73 15 of 15
Empty starter 32.35 (3 of 3) not run 0 of 15

The spread in the second row is an existing unseeded test (below), not the patch. Over all six jobs on this image: 72.31 in 19 of 19 grades, 85.64 or 86.67 in 15, 87.69 in 13, 47.73 in 11 of 12 (46.71 once).

In the public rollouts at huggingface.co/datasets/adafadaq/edgebench-opus-5-5-xhigh-rollouts, all 96 graded Kimi K3 and Qwen 3.5 submissions score 0 of Phase 1. We didn't check each one's cause.

Not in this PR. test_negative_mode_range_has_three_tiers needs the exact value bytes= among 200 unseeded examples, so one file can lose 1.03 points on one grade. HTTP_DATE_RE accepts impossible dates. Fixing either can move scores, so each needs its own PR. Details: https://github.com/findalexli/EdgeBench/tree/da540564098c8bbc0ad5b9b4cfce0ce53045f8d6/schemathesis_datagen_pipeline.

Checklist for task changes

  • version in task.toml is bumped.
  • NOTES.md: the change is under Changes; Source notes the instruction change. It isn't a listed Known issue.
  • Old vs new judge on the same submissions, above (both trees give the same images).
  • New blobs: none.

@Odysseusqs
Odysseusqs changed the base branch from community to main November 6, 2026 17:11

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant