add generate metrics

[Wenshansilvia]

@QianHaosheng @bugtig6351 @yuanpcr you can list all potential metrics for the generate task in this issue. For more details about the generate task, you can refer to issue #12 .

[faneshion]

This issue is to define new feature about the answer correctness.

[QianHaosheng]

There are some open source libraries of metrics that we may be able to use in our projects. For example, Rouge and MAUVE.
Rouge
Rouge-chinese
MAUVE

[bugtig6351]

I add Rouge metrics using rouge-score, referencing huggingface/evaluate and stanford-crfm/helm.
For non-Latin languages like Chinese and others, it is optional to use a word cutter defined by user, similar to what rouge-chinese has done. If necessary, these tokenizers can be added later.

[faneshion]

I add Rouge metrics using rouge-score, referencing huggingface/evaluate and stanford-crfm/helm. For non-Latin languages like Chinese and others, it is optional to use a word cutter defined by user, similar to what rouge-chinese has done. If necessary, these tokenizers can be added later.

There are many high-quality implemented metrics we can use in datasets. Besides, we can learn from them for those uncovered metrics.

[QianHaosheng]

Here are some metrics related to the answer and the papers mention them.

1. Answer/Query
   DisambigF1 Active retrieval augmented generation
   Answer Relevance RAGAS: Automated Evaluation of Retrieval Augmented Generation

2. Answer/Contexts
   FActScore FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
   D-FActScore Merging Facts, Crafting Fallacies: Evaluating the Contradictory Nature of Aggregated Factual Claims in Long-Form Generations

3. Answer/GT_Answer
   accuracy、EM、F1、Rouge
   BLEU, TER, chrF++ Lift yourself up: Retrieval-augmented text generation with self memory
   Q-BLEU-1 Towards a Better Metric for Evaluating Question Generation Systems
   citation recall/precision Enabling Large Language Models to Generate Text with Citations
   nF1 Hindsight: Posterior-guided training of retrievers for improved open-ended generation
   Rare F1 Retrieval augmentation reduces hallucination in conversation
   Disambiguation-Rouge PreWoMe: Exploiting Presuppositions as Working Memory for Long Form Question Answering
   BERTScore LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
   Accuracy exact match、Assertion method matched、Accuracy plausible match、LCS、Edit distance Retrieval-based prompt selection for code-related few-shot learning
   perplexity Improving retrieval-augmented lms with compression and selective augmentation
   bits-per-byte Replug: Retrievalaugmented black-box language models.
   MAUVE MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence FrontiersEnabling Large Language Models to Generate Text with Citations
   Truthful and informative TruthfulQA: Measuring How Models Mimic Human Falsehoods

[bugtig6351]

disambig-F1: Active retrieval augmented generation, ASQA: Factoid Questions Meet Long-Form Answers
Use a RoBERTa-based model to normalize the answer and the ground truth answer, then compute the token-level F1 score between them.