Question about SkillFortifyBench dataset availability

[kexinoh]

Hi SkillFortify authors,

Thanks for releasing SkillFortify. I’m trying to reproduce the benchmark results and wanted to ask: is the SkillFortifyBench dataset (the 540-skill benchmark mentioned in the paper) publicly available yet? If so, could you point me to the exact link/path (GitHub directory, release asset, or Zenodo file)?

Thank you!

[varun369]

Hi @kexinoh, thank you for your patience and apologies for the delayed response.

The SkillFortifyBench dataset (540 skills) is not yet available in the repository — we're in the process of preparing it for public release as a standalone, reproducible artifact. We expect to push it to tests/benchmark/ and upload it to Zenodo within the next 7-10 days.

We'll update this issue when it's live. Thank you for your interest in reproducing the results — that's exactly why we built the benchmark.

[wahaha0112]

Dear Dr. Bhardwaj,

I hope you are well.

I am currently reading your SkillFortify paper and trying to reproduce the benchmark setup. I wanted to ask whether the SkillFortifyBench dataset described in the paper, especially the 540-skill corpus, is publicly available.

I checked the paper, Zenodo page, GitHub repository, and PyPI package, but I could not find a downloadable release of the benchmark dataset. If there is an official link or an access procedure, could you please share it with me?

Thank you very much for your help.

Best regards.

[varun369]

The SkillFortifyBench dataset is now publicly available:

Standalone repository (MIT): https://github.com/qualixar/skillfortifybench
Main repository (benchmarks/ subtree): https://github.com/qualixar/skillfortify/tree/main/benchmarks

The repository contains:

• 540 pre-generated skills (270 malicious across 13 attack types + 270 benign across 5 categories)
• Three formats: Claude (.md), MCP (.json), OpenClaw (.yaml)
• manifest.json with SHA-256 per skill
• attack_taxonomy.json
• Deterministic generator (seed=42, PYTHONHASHSEED=0)
• Evaluation results with Wilson 95% confidence intervals

To reproduce from source:

pip install skillfortify
PYTHONHASHSEED=0 python -m benchmarks.generator --output ./skills --seed 42

Full results in RESULTS.md.