Transformer Programs Training and Evaluation

[purvapruthi]

• Implement transformer programs with absolute and relative position embeddings.
• Train on smaller composition sizes (K=3) for uniform and diverse benchmarks with a smaller vocabulary size.
• Evaluate the models within K (and cross-K later) for the uniform and diverse benchmark.
• Analyze the patterns in the learned programs to understand the mechanism behind learning composition equivalence.
• See if programs are the same across equivalent compositions.