RAGchain.benchmark.retrieval package

Submodules

RAGchain.benchmark.retrieval.evaluator module

RAGchain.benchmark.retrieval.evaluator.basic_retrieval_evaluation(qrels: Dict[str, List[str]], preds: Dict[str, List[str]], k_values: List[int], qrels_relevance: Dict[str, List[int]] | None = None, preds_relevance: Dict[str, List[float]] | None = None) → List[dict[str, float]]
Parameters:

  • qrels – The query id is key, and the value is content of retrieved passage ids. Dict[query_id, List[ids]]

  • preds – The query id is key, and the value is content of retrieved passage ids. Dict[query_id, List[ids]]

  • k_values – The k values for which the evaluation should be done. List[int]

  • qrels_relevance – The query id is key, and the value is the rate from ground truths. Dict[query_id, List[rate]]

  • preds_relevance – The query id is key, and the value is the relevance score from predictions. Dict[query_id, List[relevance score]]

results doc_id can be different from the doc_id in the qrels file.

RAGchain.benchmark.retrieval.evaluator.check_retrieval_eval(qrels: Dict[str, List[str]], preds: Dict[str, List[str]], k_values: List[int], qrels_relevance: Dict[str, List[int]] | None = None, preds_relevance: Dict[str, List[float]] | None = None)
RAGchain.benchmark.retrieval.evaluator.stretagyqa_k_eval(solution: dict, pred: dict, metrics: list, k: int) → dict
RAGchain.benchmark.retrieval.evaluator.stretagyqa_retrieval_evaluation(qrels: List[dict], preds: dict, k_values: List[int]) → List[dict[str, float]]
Parameters:

  • qrels – The qrels file as a dictionary.

  • preds – The results file as a dictionary.

K_values:

The k values for which the evaluation should be done. List[int]

results doc_id can be different from the doc_id in the qrels file.

RAGchain.benchmark.retrieval.metrics module

class RAGchain.benchmark.retrieval.metrics.AP

Bases: BaseRetrievalMetric

retrieval_metric_function(solution: Dict[str, int], pred: Dict[str, float], k_value: int = 1) → float
class RAGchain.benchmark.retrieval.metrics.BaseRetrievalMetric

Bases: ABC

eval(solution: Dict[str, int], pred: Dict[str, float], k: int) → float
property metric_name
abstract retrieval_metric_function(solution: Dict[str, int], pred: Dict[str, float], k_value: int = 1) → float
class RAGchain.benchmark.retrieval.metrics.CG

Bases: BaseRetrievalMetric

retrieval_metric_function(solution: Dict[str, int], pred: Dict[str, float], k_value: int = 1) → float
class RAGchain.benchmark.retrieval.metrics.DCG

Bases: BaseRetrievalMetric

retrieval_metric_function(solution: Dict[str, int], pred: Dict[str, float], k_value: int = 1) → float
class RAGchain.benchmark.retrieval.metrics.EM_retrieval

Bases: BaseRetrievalMetric

retrieval_metric_function(solution: Dict[str, int], pred: Dict[str, float], k_value: int = 1) → float
class RAGchain.benchmark.retrieval.metrics.F1

Bases: BaseRetrievalMetric

retrieval_metric_function(solution: Dict[str, int], pred: Dict[str, float], k_value: int = 1) → float
class RAGchain.benchmark.retrieval.metrics.Hole

Bases: BaseRetrievalMetric

retrieval_metric_function(solution: Dict[str, int], pred: Dict[str, float], k_value: int = 1) → float
class RAGchain.benchmark.retrieval.metrics.IDCG

Bases: BaseRetrievalMetric

retrieval_metric_function(solution: Dict[str, int], pred: Dict[str, float], k_value: int = 1) → float
class RAGchain.benchmark.retrieval.metrics.IndDCG

Bases: BaseRetrievalMetric

retrieval_metric_function(solution: Dict[str, int], pred: Dict[str, float], k_value: int = 1) → float
class RAGchain.benchmark.retrieval.metrics.IndIDCG

Bases: BaseRetrievalMetric

retrieval_metric_function(solution: Dict[str, int], pred: Dict[str, float], k_value: int = 1) → float
class RAGchain.benchmark.retrieval.metrics.NDCG

Bases: BaseRetrievalMetric

retrieval_metric_function(solution: Dict[str, int], pred: Dict[str, float], k_value: int = 1) → float
class RAGchain.benchmark.retrieval.metrics.Precision

Bases: BaseRetrievalMetric

retrieval_metric_function(solution: Dict[str, int], pred: Dict[str, float], k_value: int = 1) → float
class RAGchain.benchmark.retrieval.metrics.RR

Bases: BaseRetrievalMetric

retrieval_metric_function(solution: Dict[str, int], pred: Dict[str, float], k_value: int = 1) → float

Reciprocal Rank (RR) is the reciprocal of the rank of the first relevant item. Mean of RR in whole querys is MRR.

class RAGchain.benchmark.retrieval.metrics.Recall

Bases: BaseRetrievalMetric

retrieval_metric_function(solution: Dict[str, int], pred: Dict[str, float], k_value: int = 1) → float
class RAGchain.benchmark.retrieval.metrics.TopKAccuracy

Bases: BaseRetrievalMetric

retrieval_metric_function(solution: Dict[str, int], pred: Dict[str, float], k_value: int = 1) → float

Module contents