Skip to content

Evaluate(partially) reasoning enhanced hotpot.#2

Open
ConstFr wants to merge 7 commits into
mainfrom
cot_eval
Open

Evaluate(partially) reasoning enhanced hotpot.#2
ConstFr wants to merge 7 commits into
mainfrom
cot_eval

Conversation

@ConstFr

@ConstFr ConstFr commented Apr 23, 2025

Copy link
Copy Markdown
Owner

Enhanced hotpot questions by:
cot_prompt = """QUESTION
Let's think step by step.
"""

HF dataset: denis1699/hotpot_cot

Added 8bit quantization due to the lack of memory locally.

@ConstFr

ConstFr commented May 5, 2025

Copy link
Copy Markdown
Owner Author

Added benchmark for reasoning enhanced HotpotQA. Evaluated through:

CUDA_VISIBLE_DEVICES=0 polygraph_eval \
    --config-dir=./examples/configs/ \
    --config-name=polygraph_eval_cot_hotpot.yaml \
    model.path=meta-llama/Llama-3.2-3B-Instruct

from root.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant