Skip to content

Integration with Ragas in not working #222

@Jeerhz

Description

@Jeerhz

Question
I am just asking if Ragas is still working with Tonic Validate since the code provided in the documentation is not working: https://validate.tonic.ai/

Below is a code that is supposed to calculate the score on a benchmark composed of one question.

Code Examples

from ragas import evaluate
import json
from datasets import Dataset
from tonic_ragas_logger import RagasValidateApi

#Case of no restriction
data_samples = {"question": [], "answer": [], "contexts": [], "ground_truth": []}

with open("reponse_rag_testset_one_question.json", "r", encoding="utf-8") as file:
    reponse_data = json.load(file)

for item in reponse_data:
    data_samples["answer"].append(item["answer"][0])
    data_samples["question"].append(item["question"])
    data_samples["contexts"].append(item["context"])
    data_samples["ground_truth"].append(item["ground_truth"])

dataset = Dataset.from_dict(data_samples)
scores = evaluate(
        dataset, metrics=metrics, llm=azure_model, embeddings=azure_embeddings
    )

#send the score to Tonic Validate
validate_api = RagasValidateApi("G6ph3yz5hk47bQIUtRKMqfF7y8GgiJ-uBLrv6-GNmtU")
validate_api.upload_results("f5aa069e-924b-4723-a5b5-ced06f9b8ab7", scores)

Error
I also provide here the error message when running the code:

Traceback (most recent call last):
  File "c:\Users\adle.bensalem\Downloads\RAG-HES-SO\ragas_evaluate.py", line 90, in <module>
    validate_api.upload_results("f5aa069e-924b-4723-a5b5-ced06f9b8ab7", scores)
  File "C:\Users\adle.bensalem\AppData\Local\Programs\Python\Python312\Lib\site-packages\tonic_ragas_logger\ragas_validate_api.py", line 56, in upload_results
    run = self.__convert_to_run(results)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\adle.bensalem\AppData\Local\Programs\Python\Python312\Lib\site-packages\tonic_ragas_logger\ragas_validate_api.py", line 124, in __convert_to_run
    return Run(
           ^^^^
  File "C:\Users\adle.bensalem\AppData\Local\Programs\Python\Python312\Lib\site-packages\pydantic\_internal\_dataclasses.py", line 141, in __init__
    s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
pydantic_core._pydantic_core.ValidationError: 1 validation error for Run
llm_evaluator
  Field required [type=missing, input_value=ArgsKwargs((), {'overall_...onner.'])], 'id': None}), input_type=ArgsKwargs]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions