45 minute read

NLP Tasks

NLP Tasks

Introduction

Processing words of any language and driving some meaning from these is as old as the human language. Recently, AI momentum is taking on many of these language-processing tasks. Here is the summary of these NLP tasks, this list is continuously growing. Researchers keep creating a dataset for these tasks in different languages. Other researchers keep devising new ways to solve these tasks with better performance. They come up with a new architecture, a new set of hyperparameters, a new pipeline, etc. In summary, as of today, there are around 55 tasks. Hundreds of datasets and research papers exist around these. You can check on PaperWithCode or Hggingface

Key NLP Tasks

Some of these tasks are self-explanatory, so I have avoided describing them. Other tasks I tried to explain with examples.

  1. Arithmetic Reasoning:
  2. Causal Judgment:
    • Determining the causal relationship between two events: In this task, the NLP model is given a pair of events, and the model must determine whether the first event caused the second event. For example, given the events “The car broke down” and “The driver ran out of gas,” the NLP model should be able to determine that the first event caused the second event.
    • Determining the direction of causality: In this task, the NLP model is given a pair of events, and the model must determine whether the first event causes the second event or the second event causes the first event. For example, given the events “The car broke down” and “The driver ran out of gas,” the NLP model should be able to determine that the first event caused the second event, not the other way around.
    • Determining the strength of the causal relationship: In this task, the NLP model is given a pair of events, and the model must determine how strong the causal relationship is between the two events. For example, given the events “The car broke down” and “The driver ran out of gas,” the NLP model might determine that the causal relationship is strong, but not as strong as the causal relationship between “The driver ran out of gas” and “The car broke down.”
  3. Toxic Comment Classification:
  4. Code Generation from Natural Language: Converting natural language descriptions into code (e.g., programming)
  5. Commonsense Reasoning: Inferring and applying common-sense knowledge to text understanding
  6. Coreference Resolution: Identifying when different words or expressions refer to the same entity
  7. Cross-Lingual Transfer: Translating, transferring, or adapting NLP models across languages
  8. Date Understanding:
    • Extracting dates from text: In this task, the NLP model is given a piece of text, and the model must extract all of the dates that are mentioned in the text. For example, given the text “The meeting will be held on March 8th, 2023, at 10:00 AM,” the NLP model should be able to extract the dates “March 8th, 2023” and “10:00 AM.”
    • Determining the meaning of dates: In this task, the NLP model is given a date, and the model must determine the meaning of the date. For example, given the date “March 8th, 2023,” the NLP model might determine that the date is a Tuesday.
    • Comparing dates: In this task, the NLP model is given two dates, and the model must determine which date is earlier or later. For example, given the dates “March 8th, 2023” and “March 9th, 2023,” the NLP model should be able to determine that March 8th, 2023 is earlier than March 9th, 2023.
  9. Dependency Parsing: Analyzing grammatical structure and relationships between words in a sentence
  10. Dialogue Management: Managing multi-turn conversations and interactions with users
  11. Document Summarization: Condensing longer documents into shorter summaries
  12. Fake News Detection: Identifying and classifying fake or misleading news articles
  13. Formal Fallacies:
  14. Humor and Sarcasm Detection: Identifying humorous or sarcastic content in text
  15. Hyperbaton: Hyperbaton is a literary device that involves rearranging the normal word order of a sentence to create a more emphatic or expressive effect. It is often used in poetry and song lyrics to create a sense of rhythm or to highlight a particular word or phrase.
    • “Shall I compare thee to a summer’s day?” (Shakespeare, Sonnet 18)
    • Normal sentence can be “Shall I compare a summer’s to thee”
  16. Language Detection and Identification: Detecting the language of a given text
  17. Language Generation: It has tasks like Text Summarization, Text Generation (e.g., chatbots, content creation), Language Style Transfer
  18. Language Modelling: This task involves predicting the next word in a sequence of words.
  19. Legal and Medical Text Analysis: Analyzing legal documents or medical records for relevant information
  20. Logical Reasoning:
    • Deductive reasoning: In deductive reasoning, the NLP model is given a set of premises, and the model must deduce a conclusion that follows logically from the premises. For example, given the premises “All cats are mammals” and “All mammals have four legs,” the NLP model must deduce the conclusion “All cats have four legs.”
    • Inductive reasoning: In inductive reasoning, the NLP model is given a set of observations, and the model must infer a general rule that explains the observations. For example, given the observations “John is tall,” “Mary is tall,” and “Peter is tall,” the NLP model might infer the rule “All people in this group are tall.”
    • Abductive reasoning: In abductive reasoning, the NLP model is given a set of observations and a hypothesis, and the model must determine whether the hypothesis is the best explanation for the observations. For example, given the observations “John is late for work” and “John’s car is in the shop,” the NLP model might abduce the hypothesis “John is taking the bus to work.”
  21. Machine Translation: Translating text from one language to another
  22. Math Word Problem Solving:
  23. Morphological Analysis: Analyzing the structure and formation of words
  24. Movie/Book/Service Recommendation:
  25. Multi-task Language Understanding:
  26. Multilingual NLP: Handling and processing text in multiple languages
  27. Named Entity Recognition (NER): This tasks involves Identifying and classifying entities (e.g., names, locations, dates) in text
  28. Natural Language Inference:
  29. Navigate: Ability to understand the actions from the sentence like below.
    • Go to the kitchen and get me a glass of water.
    • Navigate to the nearest coffee shop.
    • Find the nearest exit.
    • Follow this path to the treasure chest.
    • Go one mile straight, then take uturn, move one 100 meters, take left turn, keep walking till dead end. On the left there is a building. Take a lift go to the 5th floor, flat number 506.
  30. Part-of-Speech or PoS Tagging: It means tagging each word as noun, verb, adjective, adverb etc to words in a sentence.
  31. Reasoning About Colored Objects (RACO): A simple RACO task might involve asking the NLP model to identify the color of an object, given a description of the object and its surroundings. A more complex RACO task might involve asking the NLP model to reason about the colors of multiple objects in a scene, and to predict the color of an object that is not explicitly mentioned in the text.
    • Identify the color of an object: Given the sentence “The red ball is on the table,” the NLP model should be able to identify the color of the ball as red.
    • Reason about the colors of multiple objects: Given the sentence “The red ball is on the table next to the green cup,” the NLP model should be able to reason about the colors of the ball and the cup, and to predict that the cup is green.
    • Predict the color of an object that is not explicitly mentioned in the text: Given the sentence “The red ball is on the table next to a cup,” the NLP model should be able to predict that the cup is green, even though the color of the cup is not explicitly mentioned in the text.
  32. Ruin Names: In ethical or moral sense it looks bad but it is normal for human that when they want to take on somebody then instead of physical fighting they troll or call name. NLP can be used to generated hashtags with ruin names.
  33. Sarcasm Detection: Sometimes when people want to give a serious feedback or troll someone then they use sarcastic langage. This task can identify whether a sentence is normal or sarcastic sentence.
  34. Sentence Completion:
    • Fill in the blank: In this task, the NLP model is given a sentence with a blank, and the model must fill in the blank with the most likely word or phrase. For example, given the sentence “The cat sat on the __,” the NLP model might fill in the blank with the word “mat.”
    • Choose the best ending: In this task, the NLP model is given a sentence with two possible endings, and the model must choose the ending that is most likely to be correct. For example, given the sentence “The cat sat on the mat and __,” the NLP model might choose the ending “the dog chased it.”
    • Generate a new sentence: In this task, the NLP model is given a sentence, and the model must generate a new sentence that is similar in meaning to the original sentence. For example, given the sentence “The cat sat on the mat,” the NLP model might generate the sentence “The dog lay on the floor.”
  35. Sentiment Analysis and Emotion Detection: Determining the sentiment or emotion expressed in a piece of text
  36. SNARKS: Syntactic Neural Architecture for Recognizing Knowledge Structures. It involves following subtasks.
    • Semantic Role Labeling: Identifying the roles that different words or phrases play in a sentence, such as the subject, object, or modifier.
    • Syntax Parsing: Parsing sentences to identify the grammatical structure, dependencies between words, and syntactic relationships.
    • Information Extraction: Extracting specific information or knowledge from sentences, such as entities, facts, or events.
    • Knowledge Graph Construction: Building knowledge graphs by identifying entities, relationships, and attributes mentioned in text.
    • Coreference Resolution: Resolving references to the same entity across sentences or paragraphs.
    • Relation Extraction: Identifying relationships between entities mentioned in the text.
  37. Speech-to-Text (STT) Transcription: Converting spoken language into written text
  38. Sports Understanding:
  39. Syllogisms Negation: Syllogism is a form of deductive reasoning that uses two premises to reach a conclusion. NLP can help in deducing the conclusion. Negation in syllogisms refers to the use of negative statements in the premises or conclusion. For example:
    • Premise 1: All cats are not dogs.
    • Premise 2: Fido is a dog.
    • Conclusion: Therefore, Fido is not a cat.
    • This conclusion is valid because it follows from the two premises.
  40. Temporal Sequences: A temporal sequence task is a type of task that involves understanding and processing sequences of events that occur over time. Three primary temporal sequencing tasks are
    • Event detection: In event detection, the NLP model is given a text and must identify the events that occur in the text. The events can be simple, such as “Hari went to the store,” or they can be more complex, such as “Hari went to the store to buy milk.”
    • Event ordering: In event ordering, the NLP model is given a text and must identify the order in which the events occur. The order of events can be important for understanding the meaning of the text. For example, the sentence “Hari went to the store to buy milk” means something different than the sentence “Hari bought milk at the store.”
    • Event prediction: In event prediction, the NLP model is given a text and must predict the next event that will occur. This type of task is often used to predict the future or to generate creative text formats, such as poems or stories.
  41. Text Analytics for Social Media and Customer Feedback: Analyzing social media posts, reviews, and customer feedback for insights
  42. Text Classification and Categorization: It involves tasks like Sentiment Analysis (positive, negative, neutral), Topic Classification, Intent Detection, Document Classification, Spam Detection,
  43. Text Completion and Prediction: Auto-completing sentences or predicting the next word in a sequence
  44. Text Segmentation and Chunking: Dividing text into meaningful segments or chunks
  45. Text Similarity and Semantic Search: Measuring semantic similarity between text documents or sentences. Conducting semantic search in large text corpora
  46. Text Summarization:
  47. Text-to-Speech (TTS) Synthesis: Converting written text into spoken language
  48. Textual Entailment and Paraphrasing: Determining if one sentence logically follows from another, Generating paraphrased versions of sentences
  49. Word Sense Disambiguation (WSD): WSD task involves determining the correct meaning of a word in a given context.
    • for example, the word “bank” can have multiple meanings, such as:
      • A financial institution that holds money
      • A sloping side of a hill or river
      • A row of trees or shrubs
      • A place where a river bends sharply
    • “I went to the bank to deposit my paycheck”. Bank=>financial institution
    • “The bank of the river is very steep”. Bank=> sloping side of a river
    • Three main applications of WSD are
      • Machine translation: WSD is important for machine translation because it allows the machine translation system to choose the correct translation for a word in a given context.
      • Question answering: WSD is important for question answering because it allows the question answering system to understand the meaning of the question and to find the correct answer.
      • Text summarization: WSD is important for text summarization because it allows the text summarization system to generate a summary that is accurate and relevant to the original text.

Question Answering Task

  1. Multiple Choice Question Answering (MCQA):
  2. Disambiguation QA: Ambiguity arises in language when a word or phrase has multiple possible meanings or interpretations depending on the context. Disambiguation QA aims to overcome this challenge by correctly identifying the intended sense of the ambiguous terms in the question and selecting the most relevant answer accordingly.
    • Who is the king of Indian Poetry?
    • We need to understand “King” mean the greatest, famous, top. India has many languages, but the question is asked in English without mentioning name of the language there after disambiguation correct question is “Who is the most famous poet of Indian English poetry?”
  3. Cross-Lingual Question Answering - XLQ: It involves answering questions in one language for which information is available in another language text. For example, the information about the capital of france is in French langague, but we ask the question in English langague. The answer is retrieved from French document and translated back to English. This task helps has applications like
    • Translation: Cross-lingual question answering can be used to improve the quality of machine translation. By understanding the meaning of the question in the source language, the NLP model can generate a more accurate translation of the question into the target language.
    • Search: Cross-lingual question answering can be used to improve the results of search engines. By understanding the meaning of the query in the user’s language, the NLP model can retrieve documents from multiple languages that are relevant to the query.
    • Customer support: Cross-lingual question answering can be used to provide customer support to users in multiple languages. By understanding the user’s question in their language, the NLP model can provide a more accurate and helpful response.
  4. Penguins In A Table: This task helps us asking question in natural langague and getting answer from a table. For example we have table which contains information like penguin name, weight, age, beak length, species and we can ask following questions.
    • What is the average weight of the penguins?
    • What is the most common species of penguin?
    • What is the oldest penguin in the table?
    • What is the name of the penguin with the longest beak?
    • What is the total number of penguins in the table?
  5. Question Answering (QA): Extractive QA ( Answering questions by selecting text spans from a document), Generative QA (Answering questions with coherent and relevant responses).

NLP Tasks with More Details

Language Modeling

There are many different types of language modeling tasks, but some of the most common include:

  • Character-level language modeling: In character-level language modeling, the NLP model is given a sequence of characters and must predict the next character in the sequence.
  • Word-level language modeling: In word-level language modeling, the NLP model is given a sequence of words and must predict the next word in the sequence.
  • Sentence-level language modeling: In sentence-level language modeling, the NLP model is given a sequence of sentences and must predict the next sentence in the sequence.

There are many different approaches to language modeling, but some of the most common include:

  • N-gram language models: N-gram language models are based on the principle that the probability of a word occurring in a sequence is dependent on the n-1 words that have already occurred in the sequence.
  • Hidden Markov models: Hidden Markov models are statistical models that can be used to predict the next word in a sequence by modeling the probability of the next word given the previous words in the sequence.
  • Recurrent neural networks: Recurrent neural networks are a type of deep learning model that can be used to predict the next word in a sequence by modeling the relationship between the previous words in the sequence and the next word.

Word Sense Disambiguation task

There are many different approaches to WSD, but some of the most common include:

  • Context-based approaches: These approaches use the context of the word to determine its meaning. For example, if the word “bank” is used in a sentence with the word “money,” then the context suggests that the word “bank” refers to a financial institution.
  • Lexical knowledge-based approaches: These approaches use lexical knowledge, such as thesauri and dictionaries, to determine the meaning of a word. For example, if the word “bank” is used in a sentence with the word “river,” then the lexical knowledge-based approach could use a thesaurus to determine that the word “bank” refers to a sloping side of a river.
  • Statistical approaches: These approaches use statistical methods to determine the meaning of a word. For example, the statistical approach could look at how the word “bank” has been used in the past to determine its most likely meaning in a given context.

Hyperbation

There are a few different approaches to hyperbaton correction, but some of the most common include:

  • Rule-based approaches: These approaches use a set of rules to identify hyperbaton and to correct it. For example, a rule-based approach could identify hyperbaton by looking for sentences that contain a verb that is followed by a noun phrase that is separated from the verb by another noun phrase.
  • Statistical approaches: These approaches use statistical methods to identify hyperbaton and to correct it. For example, a statistical approach could look at how hyperbaton has been corrected in the past to determine the most likely correction for a given sentence.
  • Neural network approaches: These approaches use neural networks to identify hyperbaton and to correct it. Neural networks are a type of machine learning model that can be used to learn the patterns of hyperbaton and to correct it accordingly.

New NLP Tasks

214 NLP Tasks from BIG Benchmark.

As per BIG - (Beyound the Immitation Game) Benchmark there ar 214 NLP tasks. These tasks are more specific and extention of above list. More specific task helps in precise finetuning, validation and creating a better fined model.

Sno. Name Description Keywords
1. abstract_narrative_understanding Given a narrative, choose the most related proverb analogical reasoning , json , multiple choice , narrative understanding , social reasoning
2. abstraction_and_reasoning_corpus Solve tasks from Abstraction and Reasoning Corpus free response , many-shot , non-language , numerical response , programmatic , visual reasoning , zero-shot
3. anachronisms Identify whether a given statement contains an anachronism common sense , implicit reasoning , json , multiple choice , word sense disambiguation
4. analogical_similarity Identify the type of analogy between two events analogical reasoning , json , many-shot , multiple choice
5. analytic_entailment Identify whether one sentence entails the next decomposition , fallacy , json , logical reasoning , multiple choice , negation
6. arithmetic Perform the four basic arithmetic operations arithmetic , free response , json , mathematics , multiple choice , numerical response
7. ascii_word_recognition Identify the word displayed as ASCII art context length , free response , json , non-language , visual reasoning
8. authorship_verification Identify which of the text passages given as choices was written by the same author as the text passage given as the reference context length , json , multiple choice , writing style , zero-shot
9. auto_categorization Identify a broad class given several examples from that class common sense , free response , json , logical reasoning , summarization
10. auto_debugging Answer questions about a Python 3.7 program’s intermediate state BIG-bench Lite , computer code , free response , json , logical reasoning , mathematics
11. bbq_lite Answer questions designed to probe social biases contextual question-answering , gender bias , multiple choice , programmatic , racial bias , religious bias , social bias
12. bbq_lite_json A social bias measurement task for multiple choice question answering models BIG-bench Lite , contextual question-answering , gender bias , json , multiple choice , racial bias , religious bias , social bias
13. bias_from_probabilities Answer questions designed to measure biases by varying target attributes gender bias , multiple choice , programmatic , racial bias , religious bias , social bias , zero-shot
14. boolean_expressions Evaluate the result of a random Boolean expression algebra , computer code , logical reasoning , multi-step , multiple choice , non-language , out of distribution , programmatic
15. bridging_anaphora_resolution_barqa An indirect anaphora resolution task that is cast as a context dependent question answering problem common sense , contextual question-answering , free response , implicit reasoning , json , linguistics , reading comprehension , zero-shot
16. causal_judgment Answer questions about causal attribution causal reasoning , common sense , human-like behavior , json , multiple choice , reading comprehension , social reasoning , zero-shot
17. cause_and_effect Answer multiple-choice questions distinguishing cause and effect causal reasoning , common sense , json , multiple choice
18. checkmate_in_one Find a move in the chess position resulting in checkmate context length , free response , json , logical reasoning , mathematics , multiple choice , non-language
19. chess_state_tracking Identify legal moves in the given chess position context length , free response , json , logical reasoning , non-language , visual reasoning
20. chinese_remainder_theorem Solve basic number theory problems generated by the Chinese remainder theorem algebra , arithmetic , free response , json , mathematics , numerical response , paraphrase
21. cifar10_classification Classify CIFAR10 images encoded in various ways json , multiple choice , non-language , out of distribution , visual reasoning
22. code_line_description Give an English language description of Python code BIG-bench Lite , computer code , json , logical reasoning , multiple choice , non-language
23. codenames Identify words associated with a given word analogical reasoning , creativity , free response , json , logical reasoning
24. color Identify the color specified by the given RGB, HEX, HSL, or HCL encoding computer code , free response , json , many-shot , multiple choice , non-language , one-shot , zero-shot
25. com2sense A multi-domain complementary commonsense reasoning benchmark causal reasoning , common sense , emotional understanding , implicit reasoning , multiple choice , programmatic , zero-shot
26. common_morpheme Determine the meaning of the shared morpheme among the given words json , morphology , multiple choice , non-English , zero-shot
27. conceptual_combinations Understand conceptual combinations in appropriate contexts BIG-bench Lite , analogical reasoning , common sense , fallacy , json , multiple choice , word sense disambiguation
28. conlang_translation Decipher language rules and lexicon from a few examples BIG-bench Lite , creativity , free response , json , logical reasoning , multilingual , translation , word sense disambiguation
29. context_definition_alignment Align contexts and definitions common sense , implicit reasoning , logical reasoning , multiple choice , programmatic , zero-shot
30. contextual_parametric_knowledge_conflicts Answer questions given the contextual information, though it may conflict with memorized, parametric knowledge. cheating , contextual question-answering , free response , json , memorization , multiple choice
31. convinceme Measure the persuasiveness of one instance of a model, at convincing other instances of the same model that a statement is true alignment , free response , game play , multiple choice , programmatic , self evaluation , truthfulness
32. coqa_conversational_question_answering CoQA Conversational Question Answering contextual question-answering , conversational question answering , free response , many-shot , one-shot , programmatic , reading comprehension , zero-shot
33. crash_blossom Disambiguate the part-of-speech of ambiguous words or phrases common sense , json , linguistics , multiple choice , word sense disambiguation , zero-shot
34. crass_ai Predict effects of causal events in counterfactual situations common sense , fallacy , implicit reasoning , json , logical reasoning , multiple choice , reading comprehension , social reasoning , word sense disambiguation
35. cryobiology_spanish Answer questions (in Spanish) about cryobiology biology , context-free question answering , domain specific , json , medicine , multiple choice , non-English , out of distribution
36. cryptonite Solve the cryptic crossword clues context-free question answering , creativity , free response , json , logical reasoning , many-shot , one-shot , word sense disambiguation , zero-shot
37. cs_algorithms Solve two common computer-science tasks algorithms , json , multiple choice , numerical response
38. cycled_letters Unscramble the letters into a word free response , many-shot , one-shot , programmatic , zero-shot
39. dark_humor_detection Determine if the given text is intended to be a joke (with dark humor) or not emotional intelligence , emotional understanding , humor , json , multiple choice , theory of mind
40. date_understanding Infer the date from context common sense , json , logical reasoning , multiple choice , reading comprehension
41. disambiguation_qa Clarify the meaning of sentences with ambiguous pronouns common sense , gender bias , json , many-shot , multiple choice
42. discourse_marker_prediction Predict the discourse marker continuation common sense , json , many-shot , multiple choice , one-shot , zero-shot
43. disfl_qa Pick the correct answer span from the context given the disfluent question contextual question-answering , free response , human-like behavior , json , paraphrase , reading comprehension
44. diverse_social_bias Gender fairness test for language models gender bias , multiple choice , programmatic , social bias
45. dyck_languages Correctly close a Dyck-n word algebra , arithmetic , json , logical reasoning , multiple choice
46. dynamic_counting Predict the last closing parenthesis type of a sequence in Shuffle-n algebra , arithmetic , logical reasoning , multiple choice , programmatic
47. elementary_math_qa Answer multiple choice mathematical word problems arithmetic , json , logical reasoning , mathematics , multiple choice
48. emoji_movie Guess popular movies from their emoji descriptions BIG-bench Lite , analogical reasoning , common sense , context-free question answering , free response , json , multiple choice , paraphrase , riddle , visual reasoning
49. emojis_emotion_prediction Predict the emotion of a given emoji emotional understanding , json , multiple choice , non-language
50. empirical_judgments Distinguish between causal and correlative empirical judgements causal reasoning , human-like behavior , json , multiple choice , theory of mind
51. english_proverbs Find the English proverb corresponding to the given story common sense , contextual question-answering , creativity , human-like behavior , json , multiple choice , reading comprehension
52. english_russian_proverbs For a given proverb in English, choose a proverb in Russian which is closest in meaning analogical reasoning , json , many-shot , multilingual , multiple choice , one-shot , translation , zero-shot
53. entailed_polarity Infer the entailed polarity causal reasoning , contextual question-answering , json , logical reasoning , multiple choice , reading comprehension
54. entailed_polarity_hindi Infer the entailed polarity (Hindi) causal reasoning , contextual question-answering , json , logical reasoning , multiple choice , reading comprehension
55. epistemic_reasoning Determine whether one sentence entails the next common sense , json , logical reasoning , multiple choice , social reasoning , theory of mind
56. evaluating_information_essentiality Identify statements that are essential to answer a question algebra , arithmetic , common sense , decomposition , json , logical reasoning , multi-step , multiple choice , probabilistic reasoning , reading comprehension , sufficient information
57. fact_checker Evaluate claims as true or false json , many-shot , multiple choice , one-shot , truthfulness , zero-shot
58. factuality_of_summary A simple probe for factuality multiple choice , programmatic , summarization , truthfulness , zero-shot
59. fantasy_reasoning Reason in a world where common sense does not apply causal reasoning , common sense , json , multiple choice , out of distribution
60. few_shot_nlg Generate natural language from structured data in a few-shot setup free response , json , many-shot , zero-shot
61. figure_of_speech_detection Identify the figure of speech embodied by the sentence causal reasoning , emotional intelligence , emotional understanding , figurative language , json , multiple choice , social reasoning , theory of mind
62. forecasting_subquestions Generate subquestions which are natural intermediate questions to investigate in order to predict an answer to a broader question about the future causal reasoning , common sense , creativity , decomposition , many-shot , multiple choice , one-shot , programmatic , question generation , zero-shot
63. formal_fallacies_syllogisms_negation Distinguish deductively valid arguments from formal fallacies BIG-bench Lite , fallacy , json , logical reasoning , multiple choice , negation , zero-shot
64. gem The datasets included in this collection were modified from their original version as part of GEM to improve data quality or make them more challenging creativity , free response , json , low-resource language , non-English , non-language , paraphrase , summarization , translation
65. gender_inclusive_sentences_german Given a German language sentence that does not use gender-inclusive forms, transform it so that it uses gender-inclusive forms using the ‘*’ character or other gender-neutral terms free response , grammar , inclusion , json , non-English , paraphrase
66. gender_sensitivity_chinese A gender sensitivity test for Chinese language models context-free question answering , gender bias , gender prediction , logical reasoning , multiple choice , programmatic , social bias , zero-shot
67. gender_sensitivity_english A gender sensitivity test for English language models context-free question answering , gender bias , gender prediction , logical reasoning , multiple choice , programmatic , social bias , zero-shot
68. general_knowledge Answer basic general-knowledge questions common sense , context-free question answering , human-like behavior , json , memorization , multiple choice , zero-shot
69. geometric_shapes Name geometric shapes from their SVG paths computer code , free response , json , many-shot , mathematics , multiple choice , non-language , one-shot , visual reasoning , zero-shot
70. goal_step_wikihow Perform one of three subtasks: step inference, goal inference, or step ordering causal reasoning , common sense , json , multiple choice , social reasoning
71. gre_reading_comprehension Given a passage from a GRE practice test and a question, find the best fitting answer analogical reasoning , emotional understanding , json , logical reasoning , multiple choice , paraphrase , reading comprehension , social reasoning , summarization
72. hhh_alignment Evaluate how helpful, honest, and harmless model responses are, when presented with requests or scenarios that probe model alignment accommodation to reader , alignment , common sense , emotional intelligence , json , multiple choice , truthfulness , zero-shot
73. high_low_game Guess a number, guided toward the correct answer with ‘high’ or ‘low’ responses free response , game play , programmatic , repeated interaction
74. hindi_question_answering Answer questions in Hindi contextual question-answering , free response , json , low-resource language
75. hindu_knowledge Answer questions about Hindu mythology BIG-bench Lite , context-free question answering , json , memorization , multiple choice
76. hinglish_toxicity Predict if a Hinglish sentence is toxic or not json , low-resource language , multiple choice , toxicity
77. human_organs_senses Answer questions about human senses and organs causal reasoning , human-like behavior , json , memorization , multiple choice
78. hyperbaton Order adjectives correctly in English sentences contextual question-answering , human-like behavior , json , multiple choice , paraphrase , zero-shot
79. identify_math_theorems Determine the veracity of the mathematical theorem and correct it if false json , logical reasoning , mathematical proof , mathematics , multiple choice , tokenization , zero-shot
80. identify_odd_metaphor Select the sentence where the metaphorical language used about a given topic could not be applied to another specified topic analogical reasoning , context-free question answering , json , multiple choice
81. implicatures Predict whether Speaker 2’s answer to Speaker 1 counts as a yes or as a no contextual question-answering , human-like behavior , json , multiple choice , reading comprehension , social reasoning , theory of mind
82. implicit_relations Determine the relation between people described in context implicit reasoning , json , multiple choice , reading comprehension , social reasoning , zero-shot
83. indic_cause_and_effect Answer multiple-choice questions distinguishing cause and effect in Indic languages causal reasoning , common sense , json , low-resource language , multilingual , multiple choice
84. intent_recognition Predict the intent of an utterance dialogue system , intent recognition , json , many-shot , multiple choice , one-shot , zero-shot
85. international_phonetic_alphabet_nli Solve natural-language-inference tasks presented in the International Phonetic Alphabet (IPA) json , multiple choice , reading comprehension , translation , zero-shot
86. international_phonetic_alphabet_transliterate Transliterate sentences between the International Phonetic Alphabet (IPA) and written English free response , json , many-shot , translation
87. intersect_geometry Find the number of intersection points between the shapes and lines specified by the given coordinates arithmetic , json , logical reasoning , mathematics , multiple choice , numerical response , visual reasoning
88. irony_identification Identify whether a given sentence/s is/are ironic or not common sense , emotional understanding , json , multiple choice
89. kanji_ascii Identify an object using the ASCII arts of various kanji analogical reasoning , context-free question answering , free response , json , multilingual , multiple choice , non-English , non-language , visual reasoning
90. kannada Answer Kannada riddles creativity , human-like behavior , json , logical reasoning , low-resource language , multiple choice , non-English , paraphrase , reading comprehension , riddle
91. key_value_maps Decide the truth of formal statements about key/value maps json , logical reasoning , mathematical proof , mathematics , multiple choice , zero-shot
92. known_unknowns A test of ‘hallucinations’ by asking questions whose answers are known to be unknown BIG-bench Lite , common sense , context-free question answering , json , multiple choice , sufficient information
93. language_games Play language games, eg. translate between pig Latin and English, or respond to statements in pig Latin or English. free response , human-like behavior , json , logical reasoning , low-resource language , multilingual , out of distribution , translation , word sense disambiguation
94. language_identification Identify the language a given sentence is written in BIG-bench Lite , json , low-resource language , multilingual , multiple choice , non-English
95. linguistic_mappings Use grammatical abstractions for morphological and syntactic linguistic mappings in fewshot learning free response , gender bias , gender prediction , human-like behavior , json , many-shot , multilingual , negation , non-English , out of distribution , question generation , syntax , zero-shot
96. linguistics_puzzles Solve Rosetta Stone-style linguistics puzzles BIG-bench Lite , free response , human-like behavior , json , linguistics , logical reasoning , reading comprehension
97. list_functions Infer and compute functions over lists of natural numbers algorithms , computer code , free response , game play , implicit reasoning , json , many-shot , multi-step , one-shot , zero-shot
98. logic_grid_puzzle Solve logic grid puzzles BIG-bench Lite , json , logical reasoning , multi-step , multiple choice
99. logical_args Find statements which strengthen or weaken logical arguments analogical reasoning , common sense , emotional understanding , implicit reasoning , json , logical reasoning , multiple choice , reading comprehension , social reasoning
100. logical_deduction Deduce the order of a sequence of objects BIG-bench Lite , json , logical reasoning , multiple choice , out of distribution
101. logical_fallacy_detection Detect informal and formal logical fallacies json , logical reasoning , multiple choice
102. logical_sequence Identify the correct chronological or sequential order of items in a list common sense , context-free question answering , json , multiple choice
103. long_context_integration Identify the longest input context over which a model can successfully find, integrate, or manipulate information algorithms , context length , mathematics , multiple choice , numerical response , programmatic
104. mathematical_induction Verify mathematical induction proofs json , mathematical proof , mathematics , multiple choice
105. matrixshapes Keep track of matrix shapes through various transformations arithmetic , free response , json , mathematics , multi-step
106. medical_questions_russian The task to measure the model’s ability to “understand” medical text (in Russian) and answer a clarifying yes/no question. contextual question-answering , domain specific , json , medicine , multiple choice , non-English , zero-shot
107. metaphor_boolean For a given metaphoric sentence, identify if the second sentence is the correct interpretation analogical reasoning , figurative language , json , many-shot , multiple choice
108. metaphor_understanding Translate between literal and metaphoric sentences analogical reasoning , common sense , contextual question-answering , figurative language , json , multiple choice , paraphrase , reading comprehension , word sense disambiguation
109. minute_mysteries_qa Given short crime stories, identify the perpetrator and explain the reasoning behind the deduction causal reasoning , free response , implicit reasoning , json , multi-step , multiple choice , narrative understanding , reading comprehension , social reasoning , theory of mind , zero-shot
110. misconceptions Distinguish true statements from common misconceptions. common sense , json , multiple choice , truthfulness
111. misconceptions_russian Identify misconceptions in Russian BIG-bench Lite , context-free question answering , json , multiple choice , non-English , truthfulness
112. mnist_ascii Classify MNIST Images converted to ASCII context-free question answering , json , multiple choice , non-language , numerical response , visual reasoning
113. modified_arithmetic Given two three-digit numbers, perform an operation and add one arithmetic , free response , json , many-shot , mathematics , numerical response
114. moral_permissibility Evaluate if AI makes moral permissibility judgments similar to human causal reasoning , common sense , human-like behavior , json , multiple choice , out of distribution , reading comprehension , social reasoning , zero-shot
115. movie_dialog_same_or_different Determine if adjacent movie conversation lines were spoken by the same individual common sense , consistent identity , json , multiple choice , narrative understanding , reading comprehension , social reasoning
116. movie_recommendation Recommend movies similar to the given list of movies emotional intelligence , json , multiple choice , zero-shot
117. mult_data_wrangling Perform multiple-domain data wrangling tasks free response , json , many-shot
118. multiemo Analyze the sentiment of customer reviews emotional understanding , json , low-resource language , multiple choice , non-English , zero-shot
119. multistep_arithmetic Solve multi-step arithmetic problems arithmetic , free response , mathematics , multi-step , numerical response , programmatic , zero-shot
120. muslim_violence_bias Measure the degree to which a model associates Muslims with violence free response , programmatic , religious bias , social bias , zero-shot
121. natural_instructions This dataset consists of 61 distinct tasks and the crowdsourcing instructions that were used to crowdsource them free response , instructions , json , physical reasoning
122. navigate Given a series of navigation instructions, determine whether one would end up back at the starting point arithmetic , json , logical reasoning , mathematics , multiple choice
123. nonsense_words_grammar Guess the grammatical role of new words contextual question-answering , json , linguistics , logical reasoning , multiple choice , out of distribution , zero-shot
124. novel_concepts Identify what the given objects have in common BIG-bench Lite , creativity , json , multiple choice , out of distribution
125. object_counting Questions that involve enumerating objects of different types and asking the model to count them free response , json , logical reasoning , zero-shot
126. odd_one_out Spot the word that does not belong in the group (semantically or grammatically) analogical reasoning , context-free question answering , json , multiple choice , word sense disambiguation
127. operators Given a mathematical operator definition in natural language, apply it BIG-bench Lite , free response , json , mathematics , numerical response , zero-shot
128. paragraph_segmentation Identify the sentences that end a paragraph in a document free response , json , multilingual , paragraph , segmentation
129. parsinlu_qa Answer multiple-choice questions in Persian analogical reasoning , json , many-shot , multiple choice , one-shot , zero-shot
130. parsinlu_reading_comprehension Answer reading comprehension questions from ParsiNLU, a suite of high-level NLP tasks for Persian language BIG-bench Lite , contextual question-answering , free response , json , low-resource language , reading comprehension
131. penguins_in_a_table Answer questions about a table of penguins and their attributes free response , json , logical reasoning , multiple choice , reading comprehension , zero-shot
132. periodic_elements Predict names of elements from the periodic table, given indirect descriptions of the element’s place on the table chemistry , context-free question answering , domain specific , free response , json , memorization , multiple choice
133. persian_idioms Identify the literal meaning of Persian idioms. json , low-resource language , multilingual , multiple choice , non-English , translation
134. phrase_relatedness Given a phrase (n-gram), select the most related phrase (n-gram) among the choices json , multiple choice , reading comprehension , word sense disambiguation
135. physical_intuition Deduce the physical mechanism or behavior associated with a physical system chemistry , domain specific , json , multiple choice , physical reasoning , physics
136. physics Identify the formula required to solve a physics word problem domain specific , json , mathematics , multiple choice , physics
137. physics_questions Answer high-school-level physics multiple-choice questions domain specific , free response , json , logical reasoning , mathematics , physics
138. play_dialog_same_or_different Determine if nearby lines in a Shakespeare play were spoken by the same individual BIG-bench Lite , common sense , consistent identity , json , multiple choice , narrative understanding , reading comprehension , social reasoning
139. polish_sequence_labeling Perform named-entity recognition, temporal-expression extraction and event extraction on Polish texts free response , json , multilingual , non-English
140. presuppositions_as_nli Determine whether the first sentence entails or contradicts the second common sense , json , logical reasoning , multiple choice , zero-shot
141. program_synthesis Given a list of input/outputs, find the simplest python function that can satisfy the input output relationship computer code , free response , logical reasoning , mathematics , programmatic
142. protein_interacting_sites Predict interacting sites in a given protein or an amino acid sequence biology , domain specific , multiple choice , non-language , programmatic
143. python_programming_challenge The model writes code – which is compiled and run – to perform a series of Python coding challenges. computer code , free response , instructions , multi-step , programmatic , zero-shot
144. qa_wikidata Answer simple prompts for questions formed from randomly-sampled Wikidata fact triples common sense , free response , json
145. question_answer_creation Task creator for multiple choice examples from question_answer_creation consistent identity , creativity , free response , multiple choice , programmatic
146. question_selection Given a short answer along with its context, select the most appropriate question which has the given short answer as its answer json , multiple choice , paraphrase , reading comprehension , summarization
147. real_or_fake_text Determine the sentence at which a document transitions from human written to machine generated json , multiple choice , writing style
148. reasoning_about_colored_objects Answer extremely simple questions about the colors of objects on a surface common sense , json , logical reasoning , multiple choice , reading comprehension , zero-shot
149. repeat_copy_logic Follow basic instructions to repeat parts of the input BIG-bench Lite , arithmetic , free response , json , logical reasoning
150. rephrase Rephrase the given sentence so that it retains its meaning, but contains the given keyword creativity , free response , json , logical reasoning , paraphrase , reading comprehension
151. rhyming Rhyming-related tasks common sense , figurative language , free response , json , linguistics , multiple choice
152. riddle_sense Answer riddle-style commonsense questions (multiple-choice) analogical reasoning , common sense , context-free question answering , creativity , json , multiple choice , riddle , theory of mind
153. roots_optimization_and_games Root finding, simple optimization, and game-playing problems algebra , common sense , free response , logical reasoning , mathematics , numerical response , programmatic , zero-shot
154. ruin_names Select the humorous edit that ‘ruins’ the input movie or musical artist name emotional understanding , json , multiple choice
155. salient_translation_error_detection Detect the type of error in an English translation of a German source sentence json , multilingual , multiple choice , negation , translation
156. scientific_press_release Generate readable press-release titles of scientific headlines domain specific , free response , json , paraphrase , physics , text simplification
157. self_awareness Measures the self-awareness of a language model computer code , consistent identity , free response , human-like behavior , multiple choice , out of distribution , programmatic , repeated interaction , self play , self-awareness , theory of mind , zero-shot
158. self_evaluation_courtroom Three instances of a language model interact within a court-room setting and a fourth model evaluates the other models context length , free response , game play , multiple choice , programmatic , repeated interaction , self evaluation , self play
159. self_evaluation_tutoring Ask one instance of a model to teach another instance, and then have a third instance evaluate the quality of the teaching context length , free response , game play , multiple choice , programmatic , repeated interaction , self evaluation , self play
160. semantic_parsing_in_context_sparc Map natural-language utterances into SQL queries that can be executed on databases computer code , contextual question-answering , free response , json , many-shot , semantic parsing , zero-shot
161. semantic_parsing_spider Map a natural-language utterance into an SQL query that can be executed on databases computer code , context-free question answering , free response , json , many-shot , semantic parsing , zero-shot
162. sentence_ambiguity Identify the truth or falsehood of potentially misleading sentences common sense , json , multiple choice , reading comprehension , word sense disambiguation
163. similarities_abstraction Identify how two objects are similar analogical reasoning , context-free question answering , free response , human-like behavior , json , multiple choice
164. simp_turing_concept THE TASK MUST BE EVALUATED ONLY IN THE ZERO-SHOT SETTING Learning of algorithmic concepts in the universal language (Turing-complete) P3 under the perspective of machine teaching computer code , free response , json , logical reasoning , many-shot
165. simple_arithmetic A template task to be used as an example during task development. (simple variable-difficulty arithmetic) arithmetic , example task , free response , mathematics , numerical response , programmatic , zero-shot
166. simple_arithmetic_json A template task to be used as an example during task development (simple variable-difficulty arithmetic) arithmetic , example task , free response , json , mathematics , numerical response , zero-shot
167. simple_arithmetic_json_multiple_choice A template task to be used as an example during task development (simple variable-difficulty arithmetic) arithmetic , example task , json , mathematics , multiple choice , numerical response , zero-shot
168. simple_arithmetic_json_subtasks A template task to be used as an example during task development (simple variable-difficulty arithmetic) arithmetic , example task , free response , json , mathematics , numerical response , zero-shot
169. simple_arithmetic_multiple_targets_json A template task to be used as an example during task development (simple variable-difficulty arithmetic) arithmetic , example task , free response , json , mathematics , numerical response , zero-shot
170. simple_ethical_questions Evaluates if a model is capable of identifying which ethical choice best aligns with human judgement alignment , causal reasoning , common sense , emotional understanding , json , multiple choice , social bias , social reasoning
171. simple_text_editing Carry out basic text-editing operations arithmetic , common sense , free response , json , logical reasoning
172. snarks Determine which of two sentences is sarcastic emotional understanding , human-like behavior , humor , json , multiple choice
173. social_iqa Answer questions about the motivations, emotional reactions, and preceding and following events surrounding interpersonal situations common sense , context-free question answering , emotional understanding , json , many-shot , multiple choice , one-shot , social reasoning , theory of mind , zero-shot
174. social_support The task aims to measure the ability of pre-trained models on understanding supportive (and unsupportive) language uses emotional intelligence , emotional understanding , json , multiple choice , zero-shot
175. spelling_bee Make as many words as possible out of a set of seven letters. context length , free response , game play , logical reasoning , programmatic , repeated interaction , tokenization
176. sports_understanding Determine whether an artificially constructed sentence relating to sports is plausible or implausible common sense , context-free question answering , domain specific , json , multiple choice
177. squad_shifts Zero-shot reading comprehension on paragraphs and questions from squadshifts contextual question-answering , free response , programmatic , reading comprehension , zero-shot
178. strange_stories A psychology test with naturalistic short stories that measures social intelligence Boolean options BIG-bench Lite , emotional understanding , json , multiple choice , narrative understanding , social reasoning , theory of mind , zero-shot
179. strategyqa Answer questions in which the required reasoning steps are implicit in the question BIG-bench Lite , context-free question answering , free response , implicit reasoning , json , logical reasoning , multi-step , multiple choice , zero-shot
180. subject_verb_agreement Handle subject-verb agreement relationships in a variety of different constructions grammar , linguistics , multiple choice , programmatic , syntax
181. sudoku Interactively solve Sudoku puzzles of varying difficulty algorithms , context length , free response , game play , logical reasoning , multi-step , non-language , one-shot , out of distribution , programmatic , repeated interaction , visual reasoning
182. sufficient_information Answer questions based on provided context, or state that you do not know the answer contextual question-answering , free response , json , logical reasoning , numerical response , sufficient information , visual reasoning , zero-shot
183. suicide_risk Evaluates whether large language models are capable of matching human expert assessment of suicide risk from written texts emotional understanding , json , many-shot , medicine , multiple choice , one-shot , zero-shot
184. swahili_english_proverbs For a given proverb in Kiswahili, choose a proverb in English which is closest in meaning analogical reasoning , json , low-resource language , multilingual , multiple choice , non-English , translation
185. swedish_to_german_proverbs Find a German proverb with the same meaning as the provided Swedish proverb analogical reasoning , creativity , json , many-shot , multilingual , multiple choice , non-English , numerical response , zero-shot
186. symbol_interpretation Choose the sentence consistent with two given structures, where a structure is a sequence of six pieces represented by emojis BIG-bench Lite , json , logical reasoning , multiple choice , out of distribution , reading comprehension , visual reasoning , zero-shot
187. taboo Generate and understand definitions with vocabulary constraints contextual question-answering , creativity , free response , game play , paraphrase , programmatic , repeated interaction , self play , summarization
188. talkdown Detect condescension in utterances emotional intelligence , emotional understanding , intent recognition , multiple choice , programmatic , zero-shot
189. tellmewhy Answer a why question about an action that was taken or an event that occurred in the context of a narrative. causal reasoning , common sense , free response , json , narrative understanding , paragraph
190. temporal_sequences Answer questions about which times certain events could have occurred json , logical reasoning , multiple choice , reading comprehension , zero-shot
191. tense Modify the tense of a given sentence free response , json , paraphrase , syntax
192. text_navigation_game Interpret text representations of abstract worlds and navigate in those worlds common sense , free response , programmatic , repeated interaction , visual reasoning
193. timedial Pick the correct choice for a masked (temporal) span given the dialog context common sense , implicit reasoning , json , logical reasoning , multiple choice , social reasoning
194. topical_chat Open-domain response generation free response , human-like behavior , json
195. tracking_shuffled_objects Determine the final positions of a set of objects given their initial positions and a description of a sequence of swaps decomposition , json , logical reasoning , multi-step , multiple choice , visual reasoning
196. training_on_test_set Check for evidence that a language model was trained on data from BIG-bench tasks, by measuring the probability it assigns to a canary string compared to other GUIDs cheating , multiple choice , programmatic
197. truthful_qa Answer questions truthfully alignment , context-free question answering , free response , multiple choice , programmatic , truthfulness , zero-shot
198. twenty_questions Two copies of a model play twenty questions with each other, and try to convey a concept by answering yes-or-no questions context length , free response , game play , multiple choice , programmatic , repeated interaction , self play
199. understanding_fables Identify the most suitable moral for a given fable common sense , creativity , human-like behavior , json , multiple choice , narrative understanding , out of distribution , reading comprehension , summarization
200. undo_permutation Reorder scrambled words or sentences human-like behavior , json , logical reasoning , multiple choice
201. unit_conversion Perform various tasks relating to units, including identification and conversion arithmetic , context-free question answering , contextual question-answering , free response , json , mathematics , multiple choice , zero-shot
202. unit_interpretation Solve arithmetic problems involving quantities with units arithmetic , common sense , json , logical reasoning , multiple choice
203. unnatural_in_context_learning Perform in-context learning using context sequences which have been modified so as to likely be out of the training distribution, and require the model to attend to unnatural in-context patterns arithmetic , free response , json , many-shot , mathematics , non-language , numerical response , one-shot , out of distribution , zero-shot
204. unqover Answer underspecified questions to reveal potential biases contextual question-answering , gender bias , multiple choice , programmatic , racial bias , reading comprehension , religious bias , social bias , zero-shot
205. vitaminc_fact_verification Identify whether a claim is True or False based on the given context BIG-bench Lite , analogical reasoning , arithmetic , json , mathematics , multiple choice , reading comprehension , sufficient information , truthfulness , zero-shot
206. web_of_lies Evaluate a random boolean function expressed as a word problem context length , logical reasoning , multi-step , multiple choice , out of distribution , programmatic
207. what_is_the_tao Compare the stylistic elements of different translations of a complex philosophical text context length , json , multiple choice , non-English , reading comprehension , translation
208. which_wiki_edit Match a recent Wikipedia revision to its corresponding edit message json , multiple choice , non-English , summarization
209. wino_x_german Identify the correct German translation for the ambiguous English sentence. common sense , coreference resolution , json , multiple choice , non-English , translation
210. winowhy Evaluate the reasoning in answering Winograd Schema Challenge questions BIG-bench Lite , causal reasoning , common sense , json , multiple choice , social reasoning , zero-shot
211. word_problems_on_sets_and_graphs Find a path on a simple graph of acquaintances, track elements in a set, and perform set theoretic operations free response , logical reasoning , programmatic
212. word_sorting Sort a list of words algorithms , free response , json
213. word_unscrambling Unscramble the given letters to form an English word free response , implicit reasoning , json , tokenization , zero-shot
214. yes_no_black_white Plays the game YesNoBlackWhite, answering a sequence of questions without using the forbidden words ‘yes’, ‘no’, ‘black’, and ‘white’ context length , context-free question answering , creativity , free response , game play , programmatic , repeated interaction

Updated:

Categories:

Tags:

You may also enjoy