Hi,
I have recently read your paper on https://arxiv.org/pdf/2602.16901. Currently, I am trying to replicate the result for GPT-5.1 for TaskInjection, which reported an ASR of 21.5% (no defense). However, my run (on slack) reported a ~4% (4/105) ASR. Could you let me know how should I run the experiment to achieve this result?
Currently, here is my launch script:
python -m agentdojo.attacks.search_attack_pipeline
--suite slack
--user-task-id user_task_0 user_task_1 user_task_2 user_task_3 user_task_4 user_task_5 user_task_6 user_task_7 user_task_8 user_task_9 user_task_10 user_task_11 user_task_12 user_task_13 user_task_14 user_task_15 user_task_16 user_task_17 user_task_18 user_task_19 user_task_20
--injection-task-id injection_task_1 injection_task_2 injection_task_3 injection_task_4 injection_task_5
--agent-model-name gpt-5.1
--attack-model-name gpt-5.1
--max-workers 6
--max-rewrites 5
Let me know if anything is incorrect.
Also, if possible, can we get a breakdown of the ASR over different suites? (Sometimes attacks work well for one suite, and poorly on another.)
Hi,
I have recently read your paper on https://arxiv.org/pdf/2602.16901. Currently, I am trying to replicate the result for GPT-5.1 for TaskInjection, which reported an ASR of 21.5% (no defense). However, my run (on slack) reported a ~4% (4/105) ASR. Could you let me know how should I run the experiment to achieve this result?
Currently, here is my launch script:
python -m agentdojo.attacks.search_attack_pipeline
--suite slack
--user-task-id user_task_0 user_task_1 user_task_2 user_task_3 user_task_4 user_task_5 user_task_6 user_task_7 user_task_8 user_task_9 user_task_10 user_task_11 user_task_12 user_task_13 user_task_14 user_task_15 user_task_16 user_task_17 user_task_18 user_task_19 user_task_20
--injection-task-id injection_task_1 injection_task_2 injection_task_3 injection_task_4 injection_task_5
--agent-model-name gpt-5.1
--attack-model-name gpt-5.1
--max-workers 6
--max-rewrites 5
Let me know if anything is incorrect.
Also, if possible, can we get a breakdown of the ASR over different suites? (Sometimes attacks work well for one suite, and poorly on another.)