Padding

[rayendito]

so far the input sentences need to be at least context_length. need to have padding to make sure model takes in at least a context_length (esp. GPT)

[rayendito]

fixed in commit 131a961
turns out i don't really need padding. initially i set the attention mask using register_buffers, which means that if i set context _length = 8, the attention mask will also be of size 8, 8 although the current input is maybe less than that

[rayendito]

turns out i need it for finetuning lol

[rayendito]

2 things needs to be considered:

• make sure the padding embedding doesn't get updated (as simple as setting padding_idx in the embedding layer)
• mask positions that are padding tokens during attention layer (this one might be alittle bit tricky)

currently, it's only padded (runs, but not right), gonna prioritize on running finetuning parallel data first