Machine Learning Metrics

19 minute read

Comprehensive Glossary of LLM

Machine Learning Metrics

Introduction

In Machine Learning projects whether classical machine learning, deep learning, computer vision, speech processing, NLP, or any other ML project we keep building different models with different datasets. But how to know that for a particular problem model X is the best one? For that, we need to evaluate these models against certain metrics. What metrics we pick, depends upon the problem statement, data imbalance, type of data, etc. In this article, we will explore an exhaustive list of ML Metrics.

From various sources, benchmarking platforms, and research papers, I have noted 330+ metrics for evaluating Machine Learning models. I keep working on this page, expanding and defining these terms on a regular basis. Some of these metrics go over my head but I am keeping it here on this page because they have been used by the practitioners in some of their projects. After experimenting with those I will expand them here. Some of the metrics are obvious to Data Scientists. But, even if you are new to Data science it gives you an idea about these metrics.

MRR

MRR stands for “Mean Reciprocal Rank,” and it is a metric commonly used in information retrieval and evaluation tasks, including those in natural language processing (NLP). MRR is used to assess the effectiveness of ranking algorithms or systems in presenting relevant information to users. MRR is often applied to tasks such as question answering, search engines, and recommendation systems. A higher MRR indicates that relevant results tend to appear higher in the ranked lists, which suggests better user experience.

Step1: Ranking of Results: Imagine you have a system that retrieves a list of possible answers or documents in response to a user’s query or question. These results are usually ranked based on their perceived relevance to the query.

Step2: Reciprocal Rank: For each query or question, the reciprocal rank of the first correct (relevant) result in the ranked list is calculated as 1 divided by the position of that correct result. If the correct result is the second item, the reciprocal rank would be 1/2; if it’s the fifth, the reciprocal rank would be 1/5, and so on.

Step3: Mean Reciprocal Rank (MRR): To calculate the MRR, you take the average of the reciprocal ranks across all queries or questions in your evaluation dataset. The formula for MRR is:

MRR = (1/N) * ∑(1/rank_i)

N is the total number of queries or questions.
rank_i is the position of the correct result for the i-th query.

Commonly used MRR@n metrics are MRR@1, MRR@3, MRR@5, MRR@10, MRR@100, MRR@1000 _____

MAP@n

MAP stands for “Mean Average Precision”. MAP focuses on evaluating the effectiveness of ranking algorithms or systems

Step1: Ranking of Results: You have a system that retrieves a ranked list of possible answers or documents in response to a user’s query or question.

Step2: Precision and Recall: Precision is the ratio of relevant items retrieved to the total number of items retrieved. Recall is the ratio of relevant items retrieved to the total number of relevant items in the dataset. These two metrics are often in tension with each other; increasing precision might result in lower recall and vice versa.

Step3: Average Precision (AP) for Each Query: For each query or question, you calculate the precision at each position in the ranked list where a relevant item is retrieved. You then calculate the average of these precision values, resulting in the Average Precision (AP) for that query.

Step4: Mean Average Precision (MAP): To calculate the MAP, you take the average of the Average Precision values across all queries or questions in your evaluation dataset. The formula for MAP is:

MAP = (1/N) * ∑(AP_i)

N is the total number of queries or questions.
AP_i is the Average Precision for the i-th query.

Commonly used MAP@n metrics are map@1, map@3, map@5, map@10, map@100, map@1000

Mean average precision (mAP):

A measure of the average precision over a range of IoU thresholds. It is also used in object detection and segmentation tasks.

mAP@IoU thresholds

The mean average precision over a range of intersection over union (IoU) thresholds.

Accuracy:

The percentage of test samples that are correctly classified.

acc_norm

accuracy_cardiffnlp/tweet_topic_multi

accuracy_cardiffnlp/tweet_topic_single

accuracy_cosinus

accuracy_euclidean

accuracy_manhattan

accuracy_tweet_eval/emoji

accuracy_tweet_eval/emotion

accuracy_tweet_eval/hate

accuracy_tweet_eval/irony

accuracy_tweet_eval/offensive

accuracy_tweet_eval/sentiment

Precision:

The fraction of predicted positive samples that are actually positive.

precision_entity_span

Precision@n

Commonly used precision@n metrics are Recall@1, Precision@3, Precision@5, Precision@10, Precision@100, Precision@1000

precision_macro

Recall:

The fraction of actual positive samples that are predicted positive.

recall_entity_span

recall@n

Commonly used recall@n metrics are recall@1, recall@3, recall@5, recall@10, recall@100, recall@1000

recall_macro

R@P

“R@P” stands for “Recall at Precision.” It’s a metric used to evaluate the performance of information retrieval systems, such as search engines or question-answering models, in terms of their ability to retrieve relevant documents or answers.

R@P is a combination of recall and precision. It evaluates how well a system can maintain a specified precision level while retrieving relevant items. It’s typically expressed as “R@X,” where “X” represents the desired precision level.

For example, R@5 measures the recall when precision is equal to 0.5 (50%). It calculates the percentage of relevant items retrieved when the system is limited to a precision level of 50%.

Mathematically, R@X is calculated by finding the recall at the point where precision reaches the specified level X. This is done by examining the retrieved items in descending order of relevance until the desired precision level is achieved.

R@P is a useful metric in tasks where both recall (finding all relevant items) and precision (avoiding irrelevant items) are important. It provides insights into how well a system balances these two aspects when retrieving information.

R@P=50

R@P=75

R@P=90

F1 score:

The harmonic mean of precision and recall.

f1_entity_span

f1_macro

f1_micro

f1_weighted

f1 (seqeval)

f1 macro

f1@m

f1@m (absent)

f1@m (present)

f1@o (absent)

f1@o (present)

f1neg

f1pos

Hit@n

Hit@1:

The percentage of queries for which the correct document is ranked first.
The percentage of images for which the correct class is in the top k predictions. It is also used in image classification tasks.

Hit@5:

The percentage of queries for which the correct document is ranked among the top 5 documents.

Log-loss:

The negative log likelihood of the predicted labels.

Brier score:

The mean squared difference between the predicted probabilities and the actual labels.

Confusion matrix:

A table that shows the true and predicted labels for each class.

MSE

Root mean squared error (RMSE):

A measure of the average squared difference between the predicted and actual values.

Mean absolute error (MAE):

A measure of the average absolute difference between the predicted and actual values.

R-squared:

A measure of the proportion of the variance in the target variable that is explained by the model.

Cohen’s kappa:

A measure of agreement between two raters, taking into account chance agreement.

Matthews correlation coefficient (MCC):

A measure of the accuracy of a binary classifier that takes into account both the true positive rate and the false positive rate.

Area under the curve (AUC):

A measure of the performance of a binary classifier at all possible thresholds.

Precision-recall curve:

A graphical representation of the precision and recall of a binary classifier at all possible thresholds.

ROC curve:

A graphical representation of the true positive rate and the false positive rate of a binary classifier at all possible thresholds.

Mean squared logarithmic error (MSLE):

A measure of the average squared difference between the logarithms of the predicted and actual values.

Mean absolute percentage error (MAPE):

A measure of the average percentage difference between the predicted and actual values.

Root mean square logarithmic error (RMSLE):

A measure of the average squared difference between the logarithms of the predicted and actual values, after taking the square root.

Precision at k (P@k):

The percentage of the top k predictions that are correct.

The percentage of predicted positive samples that are ranked among the top k predictions.
Precision@k = (# of relevant items retrieved among top-k recommendations) / (# of total predicted relevant items)

For example, if a system retrieves 5 relevant items among the top-10 recommendations out of a total of 25 predicted relevant items, then the recall@10 would be 0.20 or 20%.

In other words, precision@k measures how well the system performs in predicting relevant items among the top-k recommendations. It is useful when we are interested in evaluating the ranking quality of recommendations rather than their absolute number.

Recall at k (R@k):

The percentage of actual positive samples that are ranked among the top k predictions.
Recall@k = (# of relevant items retrieved among top-k recommendations) / (# of total relevant items)

For example, if a system retrieves 5 relevant items among the top-10 recommendations out of a total of 20 relevant items, then the recall@10 would be 0.25 or 25%.

In other words, recall@k measures how well the system performs in retrieving relevant items among the top-k recommendations. It is useful when we are interested in evaluating the ranking quality of recommendations rather than their absolute number.

F1-score at k (F1@k):

The harmonic mean of precision at k and recall at k.

Interpretability:

A measure of how easy it is to understand and explain the predictions of a model.

Fairness:

A measure of how the model treats different groups of people.

Inception score:

A measure of the diversity and quality of the generated images by a generative adversarial network (GAN).

Frechet Inception distance:

A measure of the similarity between the generated images by a GAN and the real images.

Wasserstein distance:

A measure of the distance between the distributions of the generated images and the real images.

Inception score:

A measure of the diversity and quality of the generated text by a language model.

Perplexity:

A measure of how difficult it is to predict the next word in a sequence.

BLEU score:

A measure of the similarity between the generated text and the reference text. BLEU More..

BLEU4

BLEU4_answer_extraction

BLEU4_question_answer_generation

BLEU4_question_answering

BLEU4_question_generation

Intersection over union (IoU):

A measure of the overlap between two regions. It is typically used in object detection and segmentation tasks.

Top-5 error rate:

The percentage of images for which the correct class is not in the top 5 predictions. It is typically used in image classification tasks.

EM - Exact Match

EM used in NLP task evaluation, particularly in tasks like question answering and text generation. The EM metric measures the percentage of predictions that exactly match the ground truth or reference answers.

If the model’s answer matches the reference answer word-for-word, then the EM score for that particular instance is 1. If the answers do not match exactly, the EM score is 0. The EM score is then calculated as the ratio of instances where the model’s answer matches the reference answer exactly to the total number of instances in the evaluation dataset.

ROUGE

ROUGE (Recall-Oriented Understudy for Gisting Evaluation): ROUGE is another metric used for evaluating text summarization and generation tasks. It measures the overlap of n-grams between the generated text and the reference text. ROUGE More..

rouge-l

rouge-2

rouge-lsum

rouge_l_answer_extraction

rouge_l_question_answer_generation

rouge_l_question_answering

rouge_l_question_generation

METEOR

Metric for Evaluation of Translation with Explicit ORdering: METEOR is a metric that combines multiple measures including precision, recall, stemming, and synonymy. It aims to provide a balanced evaluation of machine translation quality.METEOR More..

meteor_answer_extraction

meteor_question_answer_generation

meteor_question_answering

meteor_question_generation

HEQD

HEQD stands for Hierarchical Edit Distance. It is a metric used to evaluate the quality of text summarization. It is based on the edit distance between the ground truth summary and the generated summary, taking into account the hierarchical structure of the summary.

The HEQD metric is calculated as follows:

HEQD = 1 - (ED / L) where

ED is the edit distance between the ground truth summary and the generated summary.
L is the length of the ground truth summary.
The HEQD metric is a more accurate measure of the quality of text summarization than the BLEU metric, as it takes into account the hierarchical structure of the summary.

Example:

Ground truth summary:

The cat is on the mat.
The dog is chasing the ball.

Generated summary:

The cat and the dog are playing.

The HEQD metric for this example would be:

HEQD = 1 - (2 / 2) = 0.5

This means that the generated summary is 50% similar to the ground truth summary.

The HEQD metric is a relatively new metric, and it is not yet as widely used as the BLEU metric. However, it is a promising metric for evaluating the quality of text summarization.

Perplexity

In NLP and machine learning, perplexity is a measure of how well a language model predicts the next word in a sequence. It is calculated as the inverse of the probability of the model’s predictions. A lower perplexity indicates that the model is better at predicting the next word.

The perplexity is calculated as follows:

Perplexity = \(exp( \frac{-1}{N * \sum(log(p(w_i)))})\)
where

N is the number of words in the test set.
\(w_i\) is the i-th word in the test set.
\(p(w_i)\) is the probability of the model predicting word \(w_i\).

The perplexity metric is a more robust measure of the quality of a language model than the accuracy metric. This is because the accuracy metric only measures how well the model predicts the correct word, while the perplexity metric also takes into account the probability of the model’s predictions.

Here is an example of how the perplexity metric can be used to evaluate the quality of a language model:

Let’s say we have a language model that is trained on a corpus of text. We want to evaluate the quality of the model by using the perplexity metric. We take a test set of text and calculate the perplexity of the model’s predictions on the test set. If the perplexity is low, then the model is good at predicting the next word. If the perplexity is high, then the model is not good at predicting the next word.

The perplexity metric is a useful tool for evaluating the quality of language models. It is a more robust measure of quality than the accuracy metric, and it can be used to compare the performance of different language models.

Pass@n

The Pass@100 metric is calculated by measuring the percentage of queries for which the correct answer is among the top 100 candidates retrieved by the foundation language model (FLM) from a large corpus of documents. It reflects the ability of the FLM to retrieve relevant information from a large-scale knowledge source, which is essential for downstream tasks such as question answering, summarization, and dialogue. The higher the Pass@100 score, the better the FLM is at retrieving knowledge 1.

If you are interested in calculating Pass@100 for your own data, you can use binomial distribution. The formula for calculating Pass@100 using binomial distribution is as follows:

\[P(X >= k) = 1 - \sum_{i=0}^{k-1} [nCi * p^i * (1-p)^{n-i}]\]

where X is a binomial random variable representing the number of correct answers in the top 100 candidates, n is the total number of queries, p is the probability of getting a correct answer in one query, and k is the minimum number of correct answers required to achieve Pass@100.

Commonly used Pass@n metrics are pass@1, pass@10, Pass@100 _____

Answer Exact

answer_exact_match_answer_extraction

answer_exact_match_question_answering

Answer F1

answer_f1_score__answer_extraction

answer_f1_score__question_answering

Bert Score

bertscore

bertscore_answer_extraction

bertscore_question_answer_generation

bertscore_question_answering

bertscore_question_generation

Code

code_eval

code_eval_outputs

Cosine

cos_sim_accuracy

threshold = 0.9  # Define your threshold

true_positives = 0
false_positives = 0
false_negatives = 0

for question in questions:
    predicted_answer = model.predict(question)
    reference_answer = get_reference_answer(question)

    cosine_similarity = calculate_cosine_similarity(predicted_answer, reference_answer)

    if cosine_similarity >= threshold:
        if predicted_answer == reference_answer:
            true_positives += 1
        else:
            false_positives += 1
    else:
        if predicted_answer == reference_answer:
            true_negative += 1
		else: 
		    false_negatives += 1 

cos_sim_precision = true_positives / (true_positives + false_positives)
cos_sim_recall = true_positives / (true_positives + false_negatives)
cos_sim_accuracy = (true_positive + true_negative)/len(quetions)
cos_sim_f1 = 2 * cos_sim_precision * cos_sim_recall / (cos_sim_precision +cos_sim_recall)

Example Data
Threshold : 0.65

Actual	Predicted Cosine	Pred>Threshold	Pred>Threshold =Actual	Prediction_Type	Remark
TRUE	0.98	TRUE	TRUE	TP	Pred Cosine>=Threshold
TRUE	0.79	TRUE	TRUE	TP	Pred Cosine>=Threshold
FALSE	0.7	TRUE	FALSE	FP	Pred Cosine>=Threshold
TRUE	0.6	FALSE	FALSE	TN	Pred Cosine<Threshold
TRUE	0.5	FALSE	FALSE	TN	Pred Cosine<Threshold
FALSE	0.2	FALSE	TRUE	FN	Pred Cosine<Threshold
FALSE	0.4	FALSE	TRUE	FN	Pred Cosine<Threshold
FALSE	0.66	TRUE	FALSE	FN	Pred Cosine<Threshold

cos_sim_f1

Check cos_sim_accuracy

cos_sim_precision

Check cos_sim_accuracy

cos_sim_recall

Check cos_sim_accuracy

cos_sim_ap

cos_sim_pearson

cos_sim_spearman

Dot

dot_accuracy

dot_ap

dot_f1

dot_pearson

dot_precision

dot_recall

dot_spearman

Euclidean

euclidean_accuracy

euclidean_ap

euclidean_f1

euclidean_pearson

euclidean_precision

euclidean_recall

euclidean_spearman

Eval

eval_accuracy

eval_exact

eval_f1

eval_hasans_exact

eval_hasans_f1

eval_noans_exact

eval_noans_f1

eval_precision

eval_recall

Exact

exact

exact_match

Gen

gen_len

gen-length

Joint Goal Accuracy

joint goal accuracy

joint goal expected calibration error

Manhattan

manhattan_accuracy

manhattan_ap

manhattan_f1

manhattan_precision

manhattan_recall

manhattan_spearman

manhattan_pearson

Max

max_accuracy

max_ap

max_f1

mean

mean_reciprocal_rank

mean_reward

Macro

macro_f1

macro_precision

macro_recall

Micro

micro_precision

micro_recall

micro_f1

micro_f1_cardiffnlp

micro_f1_tweet_eval/emoji

micro_f1_tweet_eval/emotion

micro_f1_tweet_eval/hate

micro_f1_tweet_eval/irony

micro_f1_tweet_eval/offensive

micro_f1_tweet_eval/sentiment

Mover Score

moverscore_answer_extraction

moverscore_question_answer_generation

moverscore_question_answering

moverscore_question_generation

Normalized discounted cumulative gain (NDCG):

NDCG (Normalized Discounted Cumulative Gain) is a widely used metric in information retrieval and recommendation systems to evaluate the quality of ranked search results or recommendations. The “NDCG@n” metric is a variation of NDCG that focuses on a specific cutoff point, n, which represents the number of items or documents to consider in the evaluation. It assesses how well the top n items or documents in a ranked list match the relevance of the ground truth or expected results.

Here’s how NDCG@n is calculated:

Rank the Items or Documents: Start by ranking the items or documents based on some relevance score. For example, in a search engine, documents may be ranked based on their relevance to a user’s query.
Determine Relevance Scores: Assign relevance scores to each of the items or documents. These scores typically range from 0 (not relevant) to 1 (perfectly relevant). These scores represent how relevant each item or document is to the user’s query or the context of the evaluation.
Calculate DCG@n (Discounted Cumulative Gain at n): Calculate the Discounted Cumulative Gain at the cutoff point n. DCG@n is computed as the sum of the relevance scores of the top n items, with a logarithmic discount applied to the position of each item:
```
DCG@n = rel(1) + (rel(2) / log2(2)) + (rel(3) / log2(3)) + ... + (rel(n) / log2(n))
```
Where:
- rel(i) is the relevance score of the item at position i.
- log2(i) is the logarithm base 2 of i.
Calculate IDCG@n (Ideal Discounted Cumulative Gain at n): Calculate the Ideal Discounted Cumulative Gain at the cutoff point n. IDCG@n represents the best possible DCG@n that could be achieved if all the items were perfectly ranked. To calculate IDCG@n, sort the items by their true relevance scores and calculate DCG@n using the same formula.
Calculate NDCG@n (Normalized Discounted Cumulative Gain at n): Calculate NDCG@n by dividing DCG@n by IDCG@n:
```
NDCG@n = DCG@n / IDCG@n
```

NDCG@n provides a normalized measure of the quality of the ranked list at the specified cutoff point. It ranges from 0 to 1, where higher values indicate better ranking quality. A value of 1 indicates that the top n items are perfectly ranked according to relevance.

NDCG@n is particularly useful when you want to assess the performance of recommendation systems, search engines, or any system that presents ranked lists to users, and you are interested in evaluating the quality of the top n results.

Popular NDCG@n metrics are NDCG@1, NDCG@3, NDGC@5, NDGC@10, NDCG@100, NDCG@1000

Pearson

pearson_correlation

pearson’s r (distress)

pearson’s r (empathy)

QA Aligned F1

qa_aligned_f1_score_bertscore_question_answer_generation

qa_aligned_f1_score_bertscore_question_answer_generation_with_gold_answer

qa_aligned_f1_score_bertscore_question_answer_generation_with_gold_answer_gold_answer

qa_aligned_f1_score_moverscore_question_answer_generation

qa_aligned_f1_score_moverscore_question_answer_generation_gold_answer

QA Aligned Precision

qa_aligned_precision_bertscore_question_answer_generation

qa_aligned_precision_bertscore_question_answer_generation_with_gold_answer

qa_aligned_precision_bertscore_question_answer_generation_with_gold_answer_gold_answer

qa_aligned_precision_moverscore_question_answer_generation

qa_aligned_precision_moverscore_question_answer_generation_with_gold_answer

qa_aligned_precision_moverscore_question_answer_generation_with_gold_answer_gold_answer

QA Aligned Recall

qa_aligned_recall_bertscore_question_answer_generation

qa_aligned_recall_bertscore_question_answer_generation_with_gold_answer

qa_aligned_recall_bertscore_question_answer_generation_with_gold_answer_gold_answer

qa_aligned_recall_moverscore_question_answer_generation

qa_aligned_recall_moverscore_question_answer_generation_with_gold_answer

qa_aligned_recall_moverscore_question_answer_generation_with_gold_answer_gold_answer

SQUAD

squad

squad_v2

Top-n Accuracy

top-1 accuracy

top-5 accuracy

Validation

validation_accuracy

validation loss

Weighted

weighted_f1

weighted_precision

weighted_recall

WER

wer_without_norm

Matthews

matthews_correlation

Accuracy-radius-1

Act_dcf-p=0.01

Avgrank

ARC

The Abstraction and Reasoning Corpus (ARC) is a benchmark dataset designed to encourage research in advanced question answering and measure a human-like form of general fluid intelligence. It was built upon an explicit set of priors designed to be as close as possible to innate human priors. The ARC dataset requires far more powerful knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. The dataset contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set of this kind (7,787 questions

Byte_perplexity

CER

Cher

Chrf

Cider

Codebleu

Conll

Coval

Cver

DER

Bialog acts accuracy

Dialog acts f1

Diffbleu

Dvitel/codebleu

EER

EM

Empos

FID

Hamming score

Jaccard error rate

Lambada

Language model loss

LAS

Loss

MER

MMLU

The MMLU metric, or Massive Multitask Language Understanding, is a benchmark for evaluating the general knowledge and reasoning capabilities of large language models (LLMs). It was developed by Meta AI and released in 2022.

The MMLU dataset consists of 15,908 multiple-choice questions covering a wide range of topics, including mathematics, US history, computer science, law, and more. The questions are designed to assess the model’s understanding of the world and its ability to reason about complex concepts. The test covers 57 tasks, including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem-solving ability

The MMLU metric is calculated by averaging the model’s accuracy on all of the questions in the dataset. This makes it a comprehensive measure of the model’s overall performance.

The MMLU metric has been used to evaluate a number of different LLMs, including GPT-3, Jurassic-1 Jumbo, and LaMDA. As of October 2023, the highest MMLU score achieved by a public LLM is 86.7%, which was set by GPT-4.

The MMLU metric is an important tool for evaluating the progress of LLMs and for identifying areas where they need improvement. It is also a useful tool for comparing the performance of different LLMs.

Here are some of the benefits of using the MMLU metric:

It is a comprehensive measure of the model’s overall performance on a wide range of tasks.
It is fair and unbiased, as it uses a standardized set of questions.
It is easy to understand and interpret.
It is reproducible, meaning that other researchers can easily obtain the same results by evaluating their models on the same dataset.

Overall, the MMLU metric is a valuable tool for evaluating the performance of LLMs and for tracking their progress over time.

Nuclearity

OVRL

PER

Perplexity

PESQ

PPL

QWK

RE+ macro f1

ROC AUC

Sacrebleu

Sari

SER

Si-sdri

Si-snri

SIG

Slot error rate

Slot f1

Span

Spearmanr

Spice

Spider

HellaSwag

HellaSwag is a dataset for studying grounded commonsense inference. It consists of 70k multiple choice questions about grounded situations, each question comes from one of two domains – activitynet or wikihow – with four answer choices about what might happen next in the scene. The dataset was designed to test commonsense natural language inference (NLI) about physical situations.

The name HellaSwag is an acronym for Harder Endings, Longer contexts, and Low-shot Activities for Situations With Adversarial Generations ¹. The dataset was created by Zellers et al. in 2019 to evaluate common-sense reasoning in large language models (LLMs) ¹.

The dataset is used to measure the performance of LLMs on commonsense reasoning tasks. The questions are segments of video captions describing some event in the physical world. A video caption segment provides an initial context for an LLM. Each context is then followed by four options for completing that context, with only one option being correct.