[CIKM`19]BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer

[Peiyance]

The main goal of reading paper is not just understanding it. Try to understand the key concept, but we need to get new ideas and research directions from the paper.

Paper information

• title: BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
• authors: Sun, Fei, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang
• venue: In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 1441–50. CIKM ’19. New York, NY, USA: Association for Computing Machinery.
• pdf link: link
• github: tensorflow pytorch
• abstract: Modeling users’ dynamic preferences from their historical behav- iors is challenging and crucial for recommendation systems. Previ- ous methods employ sequential neural networks to encode users’ historical interactions from left to right into hidden representations for making recommendations. Despite their effectiveness, we argue that such left-to-right unidirectional models are sub-optimal due to the limitations including: a) unidirectional architectures restrict the power of hidden representation in users’ behavior sequences; b) they often assume a rigidly ordered sequence which is not always practical. To address these limitations, we proposed a sequential rec- ommendation model called BERT4Rec, which employs the deep bidirectional self-attention to model user behavior sequences. To avoid the information leakage and efficiently train the bidirectional model, we adopt the Cloze objective to sequential recommendation, predicting the random masked items in the sequence by jointly conditioning on their left and right context. In this way, we learn a bidirectional representation model to make recommendations by allowing each item in user historical behaviors to fuse infor- mation from both left and right sides. Extensive experiments on four benchmark datasets show that our model outperforms various state-of-the-art sequential models consistently.

Summary: problems to address, key ideas, quick results

presentation link

Questions about the paper?

What do you like?

Flexibly apply classical pre-trained model on sequential data into Recsys
Utilize Cloze objective to avoid the information leakage when training

What you don't like?

How to improve?

Any new ideas?

According to the paradigm that applying pre-trained model on sequential data into Recsys, we can also apply pre-trained model on graph data into Recsys when we transform the interaction data from sequence into graph.

Reproducing results (if any)