Generation on AdversarialQA Testing and PCW Usage

[naknak-choi]

Hi there!

I hope this message finds you well. I have a couple of questions regarding the AdversarialQA testing:

1. Did you use the code snippet provided in the README for testing AdversarialQA?
2. If so, did you simply add the context of AdversarialQA to the contexts parameter?

Below, I've included a sample code snippet demonstrating the usage of PCW for generation:

PCW Usage Example

Using PCW for generation:

output = wrapper.pcw_generate(
contexts=[
"Review: Great movie!\n",
"Review: Horrible film\n"
],
task_text="Review:",
temperature=1,
do_sample=True,
max_new_tokens=16
)
print(output)

Please let me know if there are any additional steps or considerations I should be aware of when testing AdversarialQA.

Thank you in advance!

Best regards,
Jacob Choi

[nirrai21]

Hi Jacob,
You can check out the run_evaluation script which enables replicating our classification results: https://github.com/AI21Labs/Parallel-Context-Windows/blob/main/run_evaluation.py

We publish the code used for the adversarial QA experiment, but it is quite similar to the classification flow, with the exception of different evaluation metric.