Consultation on RAG Method

[DenceChen]

Thank you for your work, which has brought me good ideas for my research.

but I have a question, could you please help me? In the following code rag method, should we sort it first and then intercept it, so as to ensure that the data ragged each time is the most relevant?

[siyan-zhao]

Hi, thanks for the interest! for the RAG baseline we sort by similarities and then choose the top k to insert. For example, we sort it here: https://github.com/amazon-science/PrefEval/blob/main/generation_task/baselines_handling.py#L81.
Let me know if this helps!

[DenceChen]

Sorry I missed the code screenshot. What I meant is, as shown in the red box in the screenshot, should the order be swapped? Should they be sorted first and then intercepted, especially when inter_turns is relatively small?

[DenceChen]

@siyan-zhao Hi, is my understanding correct?

[siyan-zhao]

Hi
The rag_data contains precomputed similarity scores for all inter-turn conversations we collected from the LMSys dataset (benchmark_dataset/filtered_inter_turns.json). We stored them so we don't need to repeatedly calling the sentence embedding model at runtime. In the main script, inter-turns are inserted from top to bottom based on the order in filtered_inter_turns.json. In the RAG baseline, we should only consider the inter-turns that have actually been inserted. That’s why we use the left-to-right order here and only sort them.

[kangrui-yu]

Hi, following up on this:

as the note says idx == 0 (the user stated preference) or idx == 1 (the system response) is the ground truth exchange for retrieval, for example, in the dataset {simcse_explicit_pref/education_learning_styles_overall300_topk_history.json}, the first entry's preference ('I have a strong aversion to digital or online learning environments' ...) is indeeed at idx 0 on the corresponding message index set {simcse_explicit_pref/msg_index_education_learning_styles_overall300_topk_history.json},

however, for the second entry's preference, ('I find flashcards ineffective and prefer storytelling and narrative-based approaches'), I cannot find it in any msg files, am I missing anything here?

Thank you for your time and help!