Add Shipwright official offline benchmark results

[kpuru88]

Martian submission requirements

• Fork all 50 benchmark PRs into a GitHub organization where Shipwright is installed.
• Let Shipwright review every PR. All 50 reviews completed.
• Add Shipwright to the download configuration and run the pipeline. Shipwright is now explicitly configured; the pipeline must be rerun from the forked PR reviews after the organization is provided.
• Generate dashboard results. Candidate, deduplication, evaluation, dashboard JSON, and dashboard HTML artifacts are included.

Current generated results

Profile	TP	FP	FN	Precision	Recall	F1	F2
Core	75	68	83	52.4%	47.5%	49.8%	48.4%
All categories	81	68	92	54.4%	46.8%	50.3%	48.2%

These results are not presented as a completed leaderboard submission until the two unchecked Martian requirements are satisfied.

Validation

• 50 completed Shipwright reviews
• 147 candidates
• 50 evaluations
• Zero judge errors
• uv run --frozen pytest — 29 passed
• uv run --frozen ruff check . — passed

[ashleyzhang01]

Hey @kpuru88, can you provide the official site for Shipwright? And the Github org under which these PRs were cloned for Shipwright's CRB run?

[kpuru88]

Hi @ashleyzhang01 ,
Thank you for the repsonse.
Official site for Shipwright- https://shipwright.work/
Github Org - https://github.com/ShipWright-Review
Let me know if you have additional questions.

[ashleyzhang01]

Hi @ashleyzhang01 , Thank you for the repsonse. Official site for Shipwright- https://shipwright.work/ Github Org - https://github.com/ShipWright-Review Let me know if you have additional questions.

@kpuru88 can you make the repos in that org public?

[kpuru88]

Hi @ashleyzhang01,

Would it be okay if I added you as a member of the organization? We’d like to keep this private for now.
Once we gain sufficient traction with Shipwright,my cofounder and I plan to start fundraising.
Please let me know if that works for you.

[kpuru88]

Hi @ashleyzhang01 ,
Please let me know the best way to collaborate on this.
If you’re available, I’d also be happy to hop on a call and talk it through.

[ashleyzhang01]

@kpuru88 yes, you can add me to the org

[kpuru88]

@ashleyzhang01 : I've added you.
Looking forward to the next steps

[kpuru88]

@ashleyzhang01 : Please let me know if you haven’t received access yet.
We’re continuing to work on improving the score and would love to hear your thoughts on the next steps.

[ashleyzhang01]

@kpuru88 i just joined the org but i can't see the repos/PRs that were used for the runs for the benchmark. to add you guys to the benchmark, i would need to be able to validate those, run your review tool on the benchmark myself, and verify that you have enough online usage.

[kpuru88]

Thanks.
@ashleyzhang01 : what does enough online usage mean ?

[ashleyzhang01]

@kpuru88 at least 600 public PRs across different repos that our online benchmark would be able to pick up

[kpuru88]

@ashleyzhang01 :
Thanks for clarifying. Do the 600 public PRs need to be from repositories owned by different organizations, or can they include multiple public repositories within the same organization? Do the 600 public PR reviews need to come from real customer usage, or can they include benchmark and test PRs?

[ashleyzhang01]

@kpuru88 it should be real customer usage from different organizations

[kpuru88]

Hi @ashleyzhang01 : Regarding the 600 PRs, could we use Shipwright to review open-source ones?

[kpuru88]

Hi @ashleyzhang01 ,
We've run it on 600 PRs, can you please let us know the next steps ?

[ashleyzhang01]

@kpuru88 I checked what our pipeline picks up from Shipwright, and this is what we see from shipwright-agent[bot]:

• reviewed 531 public PRs all time
• every review lives in one org, ShipWright-Review
• every review is in a fork — 53 forks of popular OSS projects (vscode, TypeScript, sentry, supabase, ClickHouse, n8n, novu, mattermost…)
• was authored by a single account, anurag6569201 (anurag singh, company listed as withclario.com)
• there are zero PRs in any repo they don't own

Let me know if this is incorrect or I'm missing anything.

Previously I said:

it should be real customer usage from different organizations

But this seems like you guys are just running it yourself, rather than reflecting real usage, meaning we can't actually score and assess. Also, none of these are merged, and if there isn't human developer activity acting on the review comments, along with a merge, we can't analyze and score the reviews.