Skip to content

juliet_vulnerability_analyzer: accept the release as a CWE-416 source - #55

Draft
findalexli wants to merge 1 commit into
ByteDance-Seed:mainfrom
findalexli:fix/juliet-cwe416-release
Draft

findalexli wants to merge 1 commit into
ByteDance-Seed:mainfrom
findalexli:fix/juliet-cwe416-release

Conversation

@findalexli

Copy link
Copy Markdown
Contributor

Task finding: #50.

What and why

The README says source_stmt "must identify the source/allocation/release witness", but trace_valid rejects a CWE-416 finding that starts at the release. For CWE-416 only, it now also accepts a free or delete of a value the sink uses, in the sink's function, between the labelled allocation and the sink, with [release, sink] as the required witness. The public scorer is the same file and gets the same change.

Decision for maintainers

This PR follows the README and can't lower a score. Or change the README so CWE-416 names the allocation: no score moves, and the analyzer below stays at 85.66. We prefer this PR because the README is what solvers were told. Someone who knows the field should check that the release is an acceptable source for a use-after-free trace.

It extends a looseness the judge already has: trace_valid accepts any existing statements between source and sink, so [release, statement, sink] passes as [allocation, statement, sink] does. Tightening that would be a separate change.

Old vs new judge

tools/judge_local.py, f92857a (1.1.0) against this branch, three grades per judge per row on one worker, reversing order, no spread.

Submission Old Valid traces New Valid traces
Opus 5.5 analyzer, release as source 85.66 365 of 540 89.00 485 of 540
Same, flag off, allocation as source 89.00 485 of 540 89.00 485 of 540
Best analyzer from that run, allocation as source 90.70 511 of 551 90.70 511 of 551
That analyzer with the flag on 87.43 391 of 551 90.70 511 of 551
That analyzer, statement before the sink as source (wrong) 87.43 391 of 551 87.43 391 of 551
Empty starter 10 0 10 0

For each of the 723 hidden labels, every statement in its case as source_stmt, with trace [statement, sink], old against new trace_valid: 118255 comparisons, 0 accepted traces rejected, 120 newly accepted, one release per CWE-416 label. Of 38 distinct Opus 5.5 analyzers, one uses the release and 37 don't, so their scores don't move. Census and public-label results: https://github.com/findalexli/EdgeBench/tree/da540564098c8bbc0ad5b9b4cfce0ce53045f8d6/juliet_vulnerability_analyzer.

Checklist for task changes

  • version in task.toml is bumped (any change that can move scores).
  • NOTES.md: the change and its score effect are under Changes. It isn't a listed Known issue.
  • Old vs new judge on the same submissions, on the same host, results above.
  • New blobs: none.

@Odysseusqs
Odysseusqs changed the base branch from community to main November 6, 2026 17:11

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant