Tracking Technical Debt

[sdhossain]

Creating a thread / issue to track technical debt.

[sdhossain]

For infra:

• more of an open issue, but we need to save inferences, then run autograder (might be better for batching)
• Potentially move to hydra
• Create attack registry
• We need pytest with good test coverage
• Proper CONTRIBUTING.md
• Update README.md

For embedding_attack:

• we should make a version of strong_reject dataset (or any other dataset) with targets so that targeted embedding attacks work
• we should look into a separation of evaluator (auto-grader) and dataset. (my initial proposal would be to have the autograder as a config)

For finetune attacks in general:

• Look into using apply_chat_template and seeing how to potentially use that -> would probably be good to change to that for future attacks (the trl library also automatically applies this if the dataset is in a conversational format). I tried this out a bit, and realized it’s not always completely configurable to ensure loss is computed only on the completion / generation part of responses, requires a specific type of chat template. Also tried converting datasets to instruction format (which is more similar to the custom way done) and had different results than before (I think this is fix-able, just need to debug a bit). I think using one of these features would be really helpful, but needs a bit more debugging - which I’d like to do after merging - will also look to add unit tests when / if that is added in.
• My initial investigation into using SFT directly with apply_chat_template left me a bit on the fence on whether to directly use that, I think DataCollatorForCompletionOnlyLM might make more sense (it is similar to our current approach but more standardized)
• We should have our instruction_template and assistant_template tailored to the models that are being fine-tuned / attacked.
• We should refactor our fine-tuning attacks to inherit from a BaseFinetune class.
• We should have sanity checks for all fine-tune methods (to make sure it is a fine-tune method that actually works) -> this should probably by some task specific dataset that always has a performance improvement if fine-tuned on that
• We need some sanity check tests for our collators and tokenizers
• We need a more standardized way to handle tokenizers (when adding pad_token and other special tokens)

[sdhossain]

I feel like most of these have been resolved with PR's #94 and #97 open as well as the merging of #91 and #53 . Probably makes sense to make issues for others as they come.