NLP Tasks

NLP Tasks

Introduction

Processing words of any language and driving some meaning from these is as old as the human language. Recently, AI momentum is taking on many of these language-processing tasks. Here is the summary of these NLP tasks, this list is continuously growing. Researchers keep creating a dataset for these tasks in different languages. Other researchers keep devising new ways to solve these tasks with better performance. They come up with a new architecture, a new set of hyperparameters, a new pipeline, etc. In summary, as of today, there are around 55 tasks. Hundreds of datasets and research papers exist around these. You can check on PaperWithCode or Hggingface

Key NLP Tasks

Some of these tasks are self-explanatory, so I have avoided describing them. Other tasks I tried to explain with examples.

1. Arithmetic Reasoning:
2. Causal Judgment:
   • Determining the causal relationship between two events: In this task, the NLP model is given a pair of events, and the model must determine whether the first event caused the second event. For example, given the events “The car broke down” and “The driver ran out of gas,” the NLP model should be able to determine that the first event caused the second event.
   • Determining the direction of causality: In this task, the NLP model is given a pair of events, and the model must determine whether the first event causes the second event or the second event causes the first event. For example, given the events “The car broke down” and “The driver ran out of gas,” the NLP model should be able to determine that the first event caused the second event, not the other way around.
   • Determining the strength of the causal relationship: In this task, the NLP model is given a pair of events, and the model must determine how strong the causal relationship is between the two events. For example, given the events “The car broke down” and “The driver ran out of gas,” the NLP model might determine that the causal relationship is strong, but not as strong as the causal relationship between “The driver ran out of gas” and “The car broke down.”
3. Toxic Comment Classification:
4. Code Generation from Natural Language: Converting natural language descriptions into code (e.g., programming)
5. Commonsense Reasoning: Inferring and applying common-sense knowledge to text understanding
6. Coreference Resolution: Identifying when different words or expressions refer to the same entity
7. Cross-Lingual Transfer: Translating, transferring, or adapting NLP models across languages
8. Date Understanding:
   • Extracting dates from text: In this task, the NLP model is given a piece of text, and the model must extract all of the dates that are mentioned in the text. For example, given the text “The meeting will be held on March 8th, 2023, at 10:00 AM,” the NLP model should be able to extract the dates “March 8th, 2023” and “10:00 AM.”
   • Determining the meaning of dates: In this task, the NLP model is given a date, and the model must determine the meaning of the date. For example, given the date “March 8th, 2023,” the NLP model might determine that the date is a Tuesday.
   • Comparing dates: In this task, the NLP model is given two dates, and the model must determine which date is earlier or later. For example, given the dates “March 8th, 2023” and “March 9th, 2023,” the NLP model should be able to determine that March 8th, 2023 is earlier than March 9th, 2023.
9. Dependency Parsing: Analyzing grammatical structure and relationships between words in a sentence
10. Dialogue Management: Managing multi-turn conversations and interactions with users
11. Document Summarization: Condensing longer documents into shorter summaries
12. Fake News Detection: Identifying and classifying fake or misleading news articles
13. Formal Fallacies:
14. Humor and Sarcasm Detection: Identifying humorous or sarcastic content in text
15. Hyperbaton: Hyperbaton is a literary device that involves rearranging the normal word order of a sentence to create a more emphatic or expressive effect. It is often used in poetry and song lyrics to create a sense of rhythm or to highlight a particular word or phrase.
    • “Shall I compare thee to a summer’s day?” (Shakespeare, Sonnet 18)
    • Normal sentence can be “Shall I compare a summer’s to thee”
16. Language Detection and Identification: Detecting the language of a given text
17. Language Generation: It has tasks like Text Summarization, Text Generation (e.g., chatbots, content creation), Language Style Transfer
18. Language Modelling: This task involves predicting the next word in a sequence of words.
19. Legal and Medical Text Analysis: Analyzing legal documents or medical records for relevant information
20. Logical Reasoning:
    • Deductive reasoning: In deductive reasoning, the NLP model is given a set of premises, and the model must deduce a conclusion that follows logically from the premises. For example, given the premises “All cats are mammals” and “All mammals have four legs,” the NLP model must deduce the conclusion “All cats have four legs.”
    • Inductive reasoning: In inductive reasoning, the NLP model is given a set of observations, and the model must infer a general rule that explains the observations. For example, given the observations “John is tall,” “Mary is tall,” and “Peter is tall,” the NLP model might infer the rule “All people in this group are tall.”
    • Abductive reasoning: In abductive reasoning, the NLP model is given a set of observations and a hypothesis, and the model must determine whether the hypothesis is the best explanation for the observations. For example, given the observations “John is late for work” and “John’s car is in the shop,” the NLP model might abduce the hypothesis “John is taking the bus to work.”
21. Machine Translation: Translating text from one language to another
22. Math Word Problem Solving:
23. Morphological Analysis: Analyzing the structure and formation of words
24. Movie/Book/Service Recommendation:
25. Multi-task Language Understanding:
26. Multilingual NLP: Handling and processing text in multiple languages
27. Named Entity Recognition (NER): This tasks involves Identifying and classifying entities (e.g., names, locations, dates) in text
28. Natural Language Inference:
29. Navigate: Ability to understand the actions from the sentence like below.
    • Go to the kitchen and get me a glass of water.
    • Navigate to the nearest coffee shop.
    • Find the nearest exit.
    • Follow this path to the treasure chest.
    • Go one mile straight, then take uturn, move one 100 meters, take left turn, keep walking till dead end. On the left there is a building. Take a lift go to the 5th floor, flat number 506.
30. Part-of-Speech or PoS Tagging: It means tagging each word as noun, verb, adjective, adverb etc to words in a sentence.
31. Reasoning About Colored Objects (RACO): A simple RACO task might involve asking the NLP model to identify the color of an object, given a description of the object and its surroundings. A more complex RACO task might involve asking the NLP model to reason about the colors of multiple objects in a scene, and to predict the color of an object that is not explicitly mentioned in the text.
    • Identify the color of an object: Given the sentence “The red ball is on the table,” the NLP model should be able to identify the color of the ball as red.
    • Reason about the colors of multiple objects: Given the sentence “The red ball is on the table next to the green cup,” the NLP model should be able to reason about the colors of the ball and the cup, and to predict that the cup is green.
    • Predict the color of an object that is not explicitly mentioned in the text: Given the sentence “The red ball is on the table next to a cup,” the NLP model should be able to predict that the cup is green, even though the color of the cup is not explicitly mentioned in the text.
32. Ruin Names: In ethical or moral sense it looks bad but it is normal for human that when they want to take on somebody then instead of physical fighting they troll or call name. NLP can be used to generated hashtags with ruin names.
33. Sarcasm Detection: Sometimes when people want to give a serious feedback or troll someone then they use sarcastic langage. This task can identify whether a sentence is normal or sarcastic sentence.
34. Sentence Completion:
    • Fill in the blank: In this task, the NLP model is given a sentence with a blank, and the model must fill in the blank with the most likely word or phrase. For example, given the sentence “The cat sat on the __,” the NLP model might fill in the blank with the word “mat.”
    • Choose the best ending: In this task, the NLP model is given a sentence with two possible endings, and the model must choose the ending that is most likely to be correct. For example, given the sentence “The cat sat on the mat and __,” the NLP model might choose the ending “the dog chased it.”
    • Generate a new sentence: In this task, the NLP model is given a sentence, and the model must generate a new sentence that is similar in meaning to the original sentence. For example, given the sentence “The cat sat on the mat,” the NLP model might generate the sentence “The dog lay on the floor.”
35. Sentiment Analysis and Emotion Detection: Determining the sentiment or emotion expressed in a piece of text
36. SNARKS: Syntactic Neural Architecture for Recognizing Knowledge Structures. It involves following subtasks.
    • Semantic Role Labeling: Identifying the roles that different words or phrases play in a sentence, such as the subject, object, or modifier.
    • Syntax Parsing: Parsing sentences to identify the grammatical structure, dependencies between words, and syntactic relationships.
    • Information Extraction: Extracting specific information or knowledge from sentences, such as entities, facts, or events.
    • Knowledge Graph Construction: Building knowledge graphs by identifying entities, relationships, and attributes mentioned in text.
    • Coreference Resolution: Resolving references to the same entity across sentences or paragraphs.
    • Relation Extraction: Identifying relationships between entities mentioned in the text.
37. Speech-to-Text (STT) Transcription: Converting spoken language into written text
38. Sports Understanding:
39. Syllogisms Negation: Syllogism is a form of deductive reasoning that uses two premises to reach a conclusion. NLP can help in deducing the conclusion. Negation in syllogisms refers to the use of negative statements in the premises or conclusion. For example:
    • Premise 1: All cats are not dogs.
    • Premise 2: Fido is a dog.
    • Conclusion: Therefore, Fido is not a cat.
    • This conclusion is valid because it follows from the two premises.
40. Temporal Sequences: A temporal sequence task is a type of task that involves understanding and processing sequences of events that occur over time. Three primary temporal sequencing tasks are
    • Event detection: In event detection, the NLP model is given a text and must identify the events that occur in the text. The events can be simple, such as “Hari went to the store,” or they can be more complex, such as “Hari went to the store to buy milk.”
    • Event ordering: In event ordering, the NLP model is given a text and must identify the order in which the events occur. The order of events can be important for understanding the meaning of the text. For example, the sentence “Hari went to the store to buy milk” means something different than the sentence “Hari bought milk at the store.”
    • Event prediction: In event prediction, the NLP model is given a text and must predict the next event that will occur. This type of task is often used to predict the future or to generate creative text formats, such as poems or stories.
41. Text Analytics for Social Media and Customer Feedback: Analyzing social media posts, reviews, and customer feedback for insights
42. Text Classification and Categorization: It involves tasks like Sentiment Analysis (positive, negative, neutral), Topic Classification, Intent Detection, Document Classification, Spam Detection,
43. Text Completion and Prediction: Auto-completing sentences or predicting the next word in a sequence
44. Text Segmentation and Chunking: Dividing text into meaningful segments or chunks
45. Text Similarity and Semantic Search: Measuring semantic similarity between text documents or sentences. Conducting semantic search in large text corpora
46. Text Summarization:
47. Text-to-Speech (TTS) Synthesis: Converting written text into spoken language
48. Textual Entailment and Paraphrasing: Determining if one sentence logically follows from another, Generating paraphrased versions of sentences
49. Word Sense Disambiguation (WSD): WSD task involves determining the correct meaning of a word in a given context.
    • for example, the word “bank” can have multiple meanings, such as:
      • A financial institution that holds money
      • A sloping side of a hill or river
      • A row of trees or shrubs
      • A place where a river bends sharply
    • “I went to the bank to deposit my paycheck”. Bank=>financial institution
    • “The bank of the river is very steep”. Bank=> sloping side of a river
    • Three main applications of WSD are
      • Machine translation: WSD is important for machine translation because it allows the machine translation system to choose the correct translation for a word in a given context.
      • Question answering: WSD is important for question answering because it allows the question answering system to understand the meaning of the question and to find the correct answer.
      • Text summarization: WSD is important for text summarization because it allows the text summarization system to generate a summary that is accurate and relevant to the original text.

Question Answering Task

1. Multiple Choice Question Answering (MCQA):
2. Disambiguation QA: Ambiguity arises in language when a word or phrase has multiple possible meanings or interpretations depending on the context. Disambiguation QA aims to overcome this challenge by correctly identifying the intended sense of the ambiguous terms in the question and selecting the most relevant answer accordingly.
   • Who is the king of Indian Poetry?
   • We need to understand “King” mean the greatest, famous, top. India has many languages, but the question is asked in English without mentioning name of the language there after disambiguation correct question is “Who is the most famous poet of Indian English poetry?”
3. Cross-Lingual Question Answering - XLQ: It involves answering questions in one language for which information is available in another language text. For example, the information about the capital of france is in French langague, but we ask the question in English langague. The answer is retrieved from French document and translated back to English. This task helps has applications like
   • Translation: Cross-lingual question answering can be used to improve the quality of machine translation. By understanding the meaning of the question in the source language, the NLP model can generate a more accurate translation of the question into the target language.
   • Search: Cross-lingual question answering can be used to improve the results of search engines. By understanding the meaning of the query in the user’s language, the NLP model can retrieve documents from multiple languages that are relevant to the query.
   • Customer support: Cross-lingual question answering can be used to provide customer support to users in multiple languages. By understanding the user’s question in their language, the NLP model can provide a more accurate and helpful response.
4. Penguins In A Table: This task helps us asking question in natural langague and getting answer from a table. For example we have table which contains information like penguin name, weight, age, beak length, species and we can ask following questions.
   • What is the average weight of the penguins?
   • What is the most common species of penguin?
   • What is the oldest penguin in the table?
   • What is the name of the penguin with the longest beak?
   • What is the total number of penguins in the table?
5. Question Answering (QA): Extractive QA ( Answering questions by selecting text spans from a document), Generative QA (Answering questions with coherent and relevant responses).

NLP Tasks with More Details

Language Modeling

There are many different types of language modeling tasks, but some of the most common include:

• Character-level language modeling: In character-level language modeling, the NLP model is given a sequence of characters and must predict the next character in the sequence.
• Word-level language modeling: In word-level language modeling, the NLP model is given a sequence of words and must predict the next word in the sequence.
• Sentence-level language modeling: In sentence-level language modeling, the NLP model is given a sequence of sentences and must predict the next sentence in the sequence.

There are many different approaches to language modeling, but some of the most common include:

• N-gram language models: N-gram language models are based on the principle that the probability of a word occurring in a sequence is dependent on the n-1 words that have already occurred in the sequence.
• Hidden Markov models: Hidden Markov models are statistical models that can be used to predict the next word in a sequence by modeling the probability of the next word given the previous words in the sequence.
• Recurrent neural networks: Recurrent neural networks are a type of deep learning model that can be used to predict the next word in a sequence by modeling the relationship between the previous words in the sequence and the next word.

Word Sense Disambiguation task

There are many different approaches to WSD, but some of the most common include:

• Context-based approaches: These approaches use the context of the word to determine its meaning. For example, if the word “bank” is used in a sentence with the word “money,” then the context suggests that the word “bank” refers to a financial institution.
• Lexical knowledge-based approaches: These approaches use lexical knowledge, such as thesauri and dictionaries, to determine the meaning of a word. For example, if the word “bank” is used in a sentence with the word “river,” then the lexical knowledge-based approach could use a thesaurus to determine that the word “bank” refers to a sloping side of a river.
• Statistical approaches: These approaches use statistical methods to determine the meaning of a word. For example, the statistical approach could look at how the word “bank” has been used in the past to determine its most likely meaning in a given context.

Hyperbation

There are a few different approaches to hyperbaton correction, but some of the most common include:

• Rule-based approaches: These approaches use a set of rules to identify hyperbaton and to correct it. For example, a rule-based approach could identify hyperbaton by looking for sentences that contain a verb that is followed by a noun phrase that is separated from the verb by another noun phrase.
• Statistical approaches: These approaches use statistical methods to identify hyperbaton and to correct it. For example, a statistical approach could look at how hyperbaton has been corrected in the past to determine the most likely correction for a given sentence.
• Neural network approaches: These approaches use neural networks to identify hyperbaton and to correct it. Neural networks are a type of machine learning model that can be used to learn the patterns of hyperbaton and to correct it accordingly.

New NLP Tasks

• Datasets for New NLP Task
• Evaluation of New NLP Task

214 NLP Tasks from BIG Benchmark.

As per BIG - (Beyound the Immitation Game) Benchmark there ar 214 NLP tasks. These tasks are more specific and extention of above list. More specific task helps in precise finetuning, validation and creating a better fined model.

Sno.	Name	Description	Keywords
1.	abstract_narrative_understanding	Given a narrative, choose the most related proverb	analogical reasoning , json , multiple choice , narrative understanding , social reasoning
2.	abstraction_and_reasoning_corpus	Solve tasks from Abstraction and Reasoning Corpus	free response , many-shot , non-language , numerical response , programmatic , visual reasoning , zero-shot
3.	anachronisms	Identify whether a given statement contains an anachronism	common sense , implicit reasoning , json , multiple choice , word sense disambiguation
4.	analogical_similarity	Identify the type of analogy between two events	analogical reasoning , json , many-shot , multiple choice
5.	analytic_entailment	Identify whether one sentence entails the next	decomposition , fallacy , json , logical reasoning , multiple choice , negation
6.	arithmetic	Perform the four basic arithmetic operations	arithmetic , free response , json , mathematics , multiple choice , numerical response
7.	ascii_word_recognition	Identify the word displayed as ASCII art	context length , free response , json , non-language , visual reasoning
8.	authorship_verification	Identify which of the text passages given as choices was written by the same author as the text passage given as the reference	context length , json , multiple choice , writing style , zero-shot
9.	auto_categorization	Identify a broad class given several examples from that class	common sense , free response , json , logical reasoning , summarization
10.	auto_debugging	Answer questions about a Python 3.7 program’s intermediate state	BIG-bench Lite , computer code , free response , json , logical reasoning , mathematics
11.	bbq_lite	Answer questions designed to probe social biases	contextual question-answering , gender bias , multiple choice , programmatic , racial bias , religious bias , social bias
12.	bbq_lite_json	A social bias measurement task for multiple choice question answering models	BIG-bench Lite , contextual question-answering , gender bias , json , multiple choice , racial bias , religious bias , social bias
13.	bias_from_probabilities	Answer questions designed to measure biases by varying target attributes	gender bias , multiple choice , programmatic , racial bias , religious bias , social bias , zero-shot
14.	boolean_expressions	Evaluate the result of a random Boolean expression	algebra , computer code , logical reasoning , multi-step , multiple choice , non-language , out of distribution , programmatic
15.	bridging_anaphora_resolution_barqa	An indirect anaphora resolution task that is cast as a context dependent question answering problem	common sense , contextual question-answering , free response , implicit reasoning , json , linguistics , reading comprehension , zero-shot
16.	causal_judgment	Answer questions about causal attribution	causal reasoning , common sense , human-like behavior , json , multiple choice , reading comprehension , social reasoning , zero-shot
17.	cause_and_effect	Answer multiple-choice questions distinguishing cause and effect	causal reasoning , common sense , json , multiple choice
18.	checkmate_in_one	Find a move in the chess position resulting in checkmate	context length , free response , json , logical reasoning , mathematics , multiple choice , non-language
19.	chess_state_tracking	Identify legal moves in the given chess position	context length , free response , json , logical reasoning , non-language , visual reasoning
20.	chinese_remainder_theorem	Solve basic number theory problems generated by the Chinese remainder theorem	algebra , arithmetic , free response , json , mathematics , numerical response , paraphrase
21.	cifar10_classification	Classify CIFAR10 images encoded in various ways	json , multiple choice , non-language , out of distribution , visual reasoning
22.	code_line_description	Give an English language description of Python code	BIG-bench Lite , computer code , json , logical reasoning , multiple choice , non-language
23.	codenames	Identify words associated with a given word	analogical reasoning , creativity , free response , json , logical reasoning
24.	color	Identify the color specified by the given RGB, HEX, HSL, or HCL encoding	computer code , free response , json , many-shot , multiple choice , non-language , one-shot , zero-shot
25.	com2sense	A multi-domain complementary commonsense reasoning benchmark	causal reasoning , common sense , emotional understanding , implicit reasoning , multiple choice , programmatic , zero-shot
26.	common_morpheme	Determine the meaning of the shared morpheme among the given words	json , morphology , multiple choice , non-English , zero-shot
27.	conceptual_combinations	Understand conceptual combinations in appropriate contexts	BIG-bench Lite , analogical reasoning , common sense , fallacy , json , multiple choice , word sense disambiguation
28.	conlang_translation	Decipher language rules and lexicon from a few examples	BIG-bench Lite , creativity , free response , json , logical reasoning , multilingual , translation , word sense disambiguation
29.	context_definition_alignment	Align contexts and definitions	common sense , implicit reasoning , logical reasoning , multiple choice , programmatic , zero-shot
30.	contextual_parametric_knowledge_conflicts	Answer questions given the contextual information, though it may conflict with memorized, parametric knowledge.	cheating , contextual question-answering , free response , json , memorization , multiple choice
31.	convinceme	Measure the persuasiveness of one instance of a model, at convincing other instances of the same model that a statement is true	alignment , free response , game play , multiple choice , programmatic , self evaluation , truthfulness
32.	coqa_conversational_question_answering	CoQA Conversational Question Answering	contextual question-answering , conversational question answering , free response , many-shot , one-shot , programmatic , reading comprehension , zero-shot
33.	crash_blossom	Disambiguate the part-of-speech of ambiguous words or phrases	common sense , json , linguistics , multiple choice , word sense disambiguation , zero-shot
34.	crass_ai	Predict effects of causal events in counterfactual situations	common sense , fallacy , implicit reasoning , json , logical reasoning , multiple choice , reading comprehension , social reasoning , word sense disambiguation
35.	cryobiology_spanish	Answer questions (in Spanish) about cryobiology	biology , context-free question answering , domain specific , json , medicine , multiple choice , non-English , out of distribution
36.	cryptonite	Solve the cryptic crossword clues	context-free question answering , creativity , free response , json , logical reasoning , many-shot , one-shot , word sense disambiguation , zero-shot
37.	cs_algorithms	Solve two common computer-science tasks	algorithms , json , multiple choice , numerical response
38.	cycled_letters	Unscramble the letters into a word	free response , many-shot , one-shot , programmatic , zero-shot
39.	dark_humor_detection	Determine if the given text is intended to be a joke (with dark humor) or not	emotional intelligence , emotional understanding , humor , json , multiple choice , theory of mind
40.	date_understanding	Infer the date from context	common sense , json , logical reasoning , multiple choice , reading comprehension
41.	disambiguation_qa	Clarify the meaning of sentences with ambiguous pronouns	common sense , gender bias , json , many-shot , multiple choice
42.	discourse_marker_prediction	Predict the discourse marker continuation	common sense , json , many-shot , multiple choice , one-shot , zero-shot
43.	disfl_qa	Pick the correct answer span from the context given the disfluent question	contextual question-answering , free response , human-like behavior , json , paraphrase , reading comprehension
44.	diverse_social_bias	Gender fairness test for language models	gender bias , multiple choice , programmatic , social bias
45.	dyck_languages	Correctly close a Dyck-n word	algebra , arithmetic , json , logical reasoning , multiple choice
46.	dynamic_counting	Predict the last closing parenthesis type of a sequence in Shuffle-n	algebra , arithmetic , logical reasoning , multiple choice , programmatic
47.	elementary_math_qa	Answer multiple choice mathematical word problems	arithmetic , json , logical reasoning , mathematics , multiple choice
48.	emoji_movie	Guess popular movies from their emoji descriptions	BIG-bench Lite , analogical reasoning , common sense , context-free question answering , free response , json , multiple choice , paraphrase , riddle , visual reasoning
49.	emojis_emotion_prediction	Predict the emotion of a given emoji	emotional understanding , json , multiple choice , non-language
50.	empirical_judgments	Distinguish between causal and correlative empirical judgements	causal reasoning , human-like behavior , json , multiple choice , theory of mind
51.	english_proverbs	Find the English proverb corresponding to the given story	common sense , contextual question-answering , creativity , human-like behavior , json , multiple choice , reading comprehension
52.	english_russian_proverbs	For a given proverb in English, choose a proverb in Russian which is closest in meaning	analogical reasoning , json , many-shot , multilingual , multiple choice , one-shot , translation , zero-shot
53.	entailed_polarity	Infer the entailed polarity	causal reasoning , contextual question-answering , json , logical reasoning , multiple choice , reading comprehension
54.	entailed_polarity_hindi	Infer the entailed polarity (Hindi)	causal reasoning , contextual question-answering , json , logical reasoning , multiple choice , reading comprehension
55.	epistemic_reasoning	Determine whether one sentence entails the next	common sense , json , logical reasoning , multiple choice , social reasoning , theory of mind
56.	evaluating_information_essentiality	Identify statements that are essential to answer a question	algebra , arithmetic , common sense , decomposition , json , logical reasoning , multi-step , multiple choice , probabilistic reasoning , reading comprehension , sufficient information
57.	fact_checker	Evaluate claims as true or false	json , many-shot , multiple choice , one-shot , truthfulness , zero-shot
58.	factuality_of_summary	A simple probe for factuality	multiple choice , programmatic , summarization , truthfulness , zero-shot
59.	fantasy_reasoning	Reason in a world where common sense does not apply	causal reasoning , common sense , json , multiple choice , out of distribution
60.	few_shot_nlg	Generate natural language from structured data in a few-shot setup	free response , json , many-shot , zero-shot
61.	figure_of_speech_detection	Identify the figure of speech embodied by the sentence	causal reasoning , emotional intelligence , emotional understanding , figurative language , json , multiple choice , social reasoning , theory of mind
62.	forecasting_subquestions	Generate subquestions which are natural intermediate questions to investigate in order to predict an answer to a broader question about the future	causal reasoning , common sense , creativity , decomposition , many-shot , multiple choice , one-shot , programmatic , question generation , zero-shot
63.	formal_fallacies_syllogisms_negation	Distinguish deductively valid arguments from formal fallacies	BIG-bench Lite , fallacy , json , logical reasoning , multiple choice , negation , zero-shot
64.	gem	The datasets included in this collection were modified from their original version as part of GEM to improve data quality or make them more challenging	creativity , free response , json , low-resource language , non-English , non-language , paraphrase , summarization , translation
65.	gender_inclusive_sentences_german	Given a German language sentence that does not use gender-inclusive forms, transform it so that it uses gender-inclusive forms using the ‘*’ character or other gender-neutral terms	free response , grammar , inclusion , json , non-English , paraphrase
66.	gender_sensitivity_chinese	A gender sensitivity test for Chinese language models	context-free question answering , gender bias , gender prediction , logical reasoning , multiple choice , programmatic , social bias , zero-shot
67.	gender_sensitivity_english	A gender sensitivity test for English language models	context-free question answering , gender bias , gender prediction , logical reasoning , multiple choice , programmatic , social bias , zero-shot
68.	general_knowledge	Answer basic general-knowledge questions	common sense , context-free question answering , human-like behavior , json , memorization , multiple choice , zero-shot
69.	geometric_shapes	Name geometric shapes from their SVG paths	computer code , free response , json , many-shot , mathematics , multiple choice , non-language , one-shot , visual reasoning , zero-shot
70.	goal_step_wikihow	Perform one of three subtasks: step inference, goal inference, or step ordering	causal reasoning , common sense , json , multiple choice , social reasoning
71.	gre_reading_comprehension	Given a passage from a GRE practice test and a question, find the best fitting answer	analogical reasoning , emotional understanding , json , logical reasoning , multiple choice , paraphrase , reading comprehension , social reasoning , summarization
72.	hhh_alignment	Evaluate how helpful, honest, and harmless model responses are, when presented with requests or scenarios that probe model alignment	accommodation to reader , alignment , common sense , emotional intelligence , json , multiple choice , truthfulness , zero-shot
73.	high_low_game	Guess a number, guided toward the correct answer with ‘high’ or ‘low’ responses	free response , game play , programmatic , repeated interaction
74.	hindi_question_answering	Answer questions in Hindi	contextual question-answering , free response , json , low-resource language
75.	hindu_knowledge	Answer questions about Hindu mythology	BIG-bench Lite , context-free question answering , json , memorization , multiple choice
76.	hinglish_toxicity	Predict if a Hinglish sentence is toxic or not	json , low-resource language , multiple choice , toxicity
77.	human_organs_senses	Answer questions about human senses and organs	causal reasoning , human-like behavior , json , memorization , multiple choice
78.	hyperbaton	Order adjectives correctly in English sentences	contextual question-answering , human-like behavior , json , multiple choice , paraphrase , zero-shot
79.	identify_math_theorems	Determine the veracity of the mathematical theorem and correct it if false	json , logical reasoning , mathematical proof , mathematics , multiple choice , tokenization , zero-shot
80.	identify_odd_metaphor	Select the sentence where the metaphorical language used about a given topic could not be applied to another specified topic	analogical reasoning , context-free question answering , json , multiple choice
81.	implicatures	Predict whether Speaker 2’s answer to Speaker 1 counts as a yes or as a no	contextual question-answering , human-like behavior , json , multiple choice , reading comprehension , social reasoning , theory of mind
82.	implicit_relations	Determine the relation between people described in context	implicit reasoning , json , multiple choice , reading comprehension , social reasoning , zero-shot
83.	indic_cause_and_effect	Answer multiple-choice questions distinguishing cause and effect in Indic languages	causal reasoning , common sense , json , low-resource language , multilingual , multiple choice
84.	intent_recognition	Predict the intent of an utterance	dialogue system , intent recognition , json , many-shot , multiple choice , one-shot , zero-shot
85.	international_phonetic_alphabet_nli	Solve natural-language-inference tasks presented in the International Phonetic Alphabet (IPA)	json , multiple choice , reading comprehension , translation , zero-shot
86.	international_phonetic_alphabet_transliterate	Transliterate sentences between the International Phonetic Alphabet (IPA) and written English	free response , json , many-shot , translation
87.	intersect_geometry	Find the number of intersection points between the shapes and lines specified by the given coordinates	arithmetic , json , logical reasoning , mathematics , multiple choice , numerical response , visual reasoning
88.	irony_identification	Identify whether a given sentence/s is/are ironic or not	common sense , emotional understanding , json , multiple choice
89.	kanji_ascii	Identify an object using the ASCII arts of various kanji	analogical reasoning , context-free question answering , free response , json , multilingual , multiple choice , non-English , non-language , visual reasoning
90.	kannada	Answer Kannada riddles	creativity , human-like behavior , json , logical reasoning , low-resource language , multiple choice , non-English , paraphrase , reading comprehension , riddle
91.	key_value_maps	Decide the truth of formal statements about key/value maps	json , logical reasoning , mathematical proof , mathematics , multiple choice , zero-shot
92.	known_unknowns	A test of ‘hallucinations’ by asking questions whose answers are known to be unknown	BIG-bench Lite , common sense , context-free question answering , json , multiple choice , sufficient information
93.	language_games	Play language games, eg. translate between pig Latin and English, or respond to statements in pig Latin or English.	free response , human-like behavior , json , logical reasoning , low-resource language , multilingual , out of distribution , translation , word sense disambiguation
94.	language_identification	Identify the language a given sentence is written in	BIG-bench Lite , json , low-resource language , multilingual , multiple choice , non-English
95.	linguistic_mappings	Use grammatical abstractions for morphological and syntactic linguistic mappings in fewshot learning	free response , gender bias , gender prediction , human-like behavior , json , many-shot , multilingual , negation , non-English , out of distribution , question generation , syntax , zero-shot
96.	linguistics_puzzles	Solve Rosetta Stone-style linguistics puzzles	BIG-bench Lite , free response , human-like behavior , json , linguistics , logical reasoning , reading comprehension
97.	list_functions	Infer and compute functions over lists of natural numbers	algorithms , computer code , free response , game play , implicit reasoning , json , many-shot , multi-step , one-shot , zero-shot
98.	logic_grid_puzzle	Solve logic grid puzzles	BIG-bench Lite , json , logical reasoning , multi-step , multiple choice
99.	logical_args	Find statements which strengthen or weaken logical arguments	analogical reasoning , common sense , emotional understanding , implicit reasoning , json , logical reasoning , multiple choice , reading comprehension , social reasoning
100.	logical_deduction	Deduce the order of a sequence of objects	BIG-bench Lite , json , logical reasoning , multiple choice , out of distribution
101.	logical_fallacy_detection	Detect informal and formal logical fallacies	json , logical reasoning , multiple choice
102.	logical_sequence	Identify the correct chronological or sequential order of items in a list	common sense , context-free question answering , json , multiple choice
103.	long_context_integration	Identify the longest input context over which a model can successfully find, integrate, or manipulate information	algorithms , context length , mathematics , multiple choice , numerical response , programmatic
104.	mathematical_induction	Verify mathematical induction proofs	json , mathematical proof , mathematics , multiple choice
105.	matrixshapes	Keep track of matrix shapes through various transformations	arithmetic , free response , json , mathematics , multi-step
106.	medical_questions_russian	The task to measure the model’s ability to “understand” medical text (in Russian) and answer a clarifying yes/no question.	contextual question-answering , domain specific , json , medicine , multiple choice , non-English , zero-shot
107.	metaphor_boolean	For a given metaphoric sentence, identify if the second sentence is the correct interpretation	analogical reasoning , figurative language , json , many-shot , multiple choice
108.	metaphor_understanding	Translate between literal and metaphoric sentences	analogical reasoning , common sense , contextual question-answering , figurative language , json , multiple choice , paraphrase , reading comprehension , word sense disambiguation
109.	minute_mysteries_qa	Given short crime stories, identify the perpetrator and explain the reasoning behind the deduction	causal reasoning , free response , implicit reasoning , json , multi-step , multiple choice , narrative understanding , reading comprehension , social reasoning , theory of mind , zero-shot
110.	misconceptions	Distinguish true statements from common misconceptions.	common sense , json , multiple choice , truthfulness
111.	misconceptions_russian	Identify misconceptions in Russian	BIG-bench Lite , context-free question answering , json , multiple choice , non-English , truthfulness
112.	mnist_ascii	Classify MNIST Images converted to ASCII	context-free question answering , json , multiple choice , non-language , numerical response , visual reasoning
113.	modified_arithmetic	Given two three-digit numbers, perform an operation and add one	arithmetic , free response , json , many-shot , mathematics , numerical response
114.	moral_permissibility	Evaluate if AI makes moral permissibility judgments similar to human	causal reasoning , common sense , human-like behavior , json , multiple choice , out of distribution , reading comprehension , social reasoning , zero-shot
115.	movie_dialog_same_or_different	Determine if adjacent movie conversation lines were spoken by the same individual	common sense , consistent identity , json , multiple choice , narrative understanding , reading comprehension , social reasoning
116.	movie_recommendation	Recommend movies similar to the given list of movies	emotional intelligence , json , multiple choice , zero-shot
117.	mult_data_wrangling	Perform multiple-domain data wrangling tasks	free response , json , many-shot
118.	multiemo	Analyze the sentiment of customer reviews	emotional understanding , json , low-resource language , multiple choice , non-English , zero-shot
119.	multistep_arithmetic	Solve multi-step arithmetic problems	arithmetic , free response , mathematics , multi-step , numerical response , programmatic , zero-shot
120.	muslim_violence_bias	Measure the degree to which a model associates Muslims with violence	free response , programmatic , religious bias , social bias , zero-shot
121.	natural_instructions	This dataset consists of 61 distinct tasks and the crowdsourcing instructions that were used to crowdsource them	free response , instructions , json , physical reasoning
122.	navigate	Given a series of navigation instructions, determine whether one would end up back at the starting point	arithmetic , json , logical reasoning , mathematics , multiple choice
123.	nonsense_words_grammar	Guess the grammatical role of new words	contextual question-answering , json , linguistics , logical reasoning , multiple choice , out of distribution , zero-shot
124.	novel_concepts	Identify what the given objects have in common	BIG-bench Lite , creativity , json , multiple choice , out of distribution
125.	object_counting	Questions that involve enumerating objects of different types and asking the model to count them	free response , json , logical reasoning , zero-shot
126.	odd_one_out	Spot the word that does not belong in the group (semantically or grammatically)	analogical reasoning , context-free question answering , json , multiple choice , word sense disambiguation
127.	operators	Given a mathematical operator definition in natural language, apply it	BIG-bench Lite , free response , json , mathematics , numerical response , zero-shot
128.	paragraph_segmentation	Identify the sentences that end a paragraph in a document	free response , json , multilingual , paragraph , segmentation
129.	parsinlu_qa	Answer multiple-choice questions in Persian	analogical reasoning , json , many-shot , multiple choice , one-shot , zero-shot
130.	parsinlu_reading_comprehension	Answer reading comprehension questions from ParsiNLU, a suite of high-level NLP tasks for Persian language	BIG-bench Lite , contextual question-answering , free response , json , low-resource language , reading comprehension
131.	penguins_in_a_table	Answer questions about a table of penguins and their attributes	free response , json , logical reasoning , multiple choice , reading comprehension , zero-shot
132.	periodic_elements	Predict names of elements from the periodic table, given indirect descriptions of the element’s place on the table	chemistry , context-free question answering , domain specific , free response , json , memorization , multiple choice
133.	persian_idioms	Identify the literal meaning of Persian idioms.	json , low-resource language , multilingual , multiple choice , non-English , translation
134.	phrase_relatedness	Given a phrase (n-gram), select the most related phrase (n-gram) among the choices	json , multiple choice , reading comprehension , word sense disambiguation
135.	physical_intuition	Deduce the physical mechanism or behavior associated with a physical system	chemistry , domain specific , json , multiple choice , physical reasoning , physics
136.	physics	Identify the formula required to solve a physics word problem	domain specific , json , mathematics , multiple choice , physics
137.	physics_questions	Answer high-school-level physics multiple-choice questions	domain specific , free response , json , logical reasoning , mathematics , physics
138.	play_dialog_same_or_different	Determine if nearby lines in a Shakespeare play were spoken by the same individual	BIG-bench Lite , common sense , consistent identity , json , multiple choice , narrative understanding , reading comprehension , social reasoning
139.	polish_sequence_labeling	Perform named-entity recognition, temporal-expression extraction and event extraction on Polish texts	free response , json , multilingual , non-English
140.	presuppositions_as_nli	Determine whether the first sentence entails or contradicts the second	common sense , json , logical reasoning , multiple choice , zero-shot
141.	program_synthesis	Given a list of input/outputs, find the simplest python function that can satisfy the input output relationship	computer code , free response , logical reasoning , mathematics , programmatic
142.	protein_interacting_sites	Predict interacting sites in a given protein or an amino acid sequence	biology , domain specific , multiple choice , non-language , programmatic
143.	python_programming_challenge	The model writes code – which is compiled and run – to perform a series of Python coding challenges.	computer code , free response , instructions , multi-step , programmatic , zero-shot
144.	qa_wikidata	Answer simple prompts for questions formed from randomly-sampled Wikidata fact triples	common sense , free response , json
145.	question_answer_creation	Task creator for multiple choice examples from question_answer_creation	consistent identity , creativity , free response , multiple choice , programmatic
146.	question_selection	Given a short answer along with its context, select the most appropriate question which has the given short answer as its answer	json , multiple choice , paraphrase , reading comprehension , summarization
147.	real_or_fake_text	Determine the sentence at which a document transitions from human written to machine generated	json , multiple choice , writing style
148.	reasoning_about_colored_objects	Answer extremely simple questions about the colors of objects on a surface	common sense , json , logical reasoning , multiple choice , reading comprehension , zero-shot
149.	repeat_copy_logic	Follow basic instructions to repeat parts of the input	BIG-bench Lite , arithmetic , free response , json , logical reasoning
150.	rephrase	Rephrase the given sentence so that it retains its meaning, but contains the given keyword	creativity , free response , json , logical reasoning , paraphrase , reading comprehension
151.	rhyming	Rhyming-related tasks	common sense , figurative language , free response , json , linguistics , multiple choice
152.	riddle_sense	Answer riddle-style commonsense questions (multiple-choice)	analogical reasoning , common sense , context-free question answering , creativity , json , multiple choice , riddle , theory of mind
153.	roots_optimization_and_games	Root finding, simple optimization, and game-playing problems	algebra , common sense , free response , logical reasoning , mathematics , numerical response , programmatic , zero-shot
154.	ruin_names	Select the humorous edit that ‘ruins’ the input movie or musical artist name	emotional understanding , json , multiple choice
155.	salient_translation_error_detection	Detect the type of error in an English translation of a German source sentence	json , multilingual , multiple choice , negation , translation
156.	scientific_press_release	Generate readable press-release titles of scientific headlines	domain specific , free response , json , paraphrase , physics , text simplification
157.	self_awareness	Measures the self-awareness of a language model	computer code , consistent identity , free response , human-like behavior , multiple choice , out of distribution , programmatic , repeated interaction , self play , self-awareness , theory of mind , zero-shot
158.	self_evaluation_courtroom	Three instances of a language model interact within a court-room setting and a fourth model evaluates the other models	context length , free response , game play , multiple choice , programmatic , repeated interaction , self evaluation , self play
159.	self_evaluation_tutoring	Ask one instance of a model to teach another instance, and then have a third instance evaluate the quality of the teaching	context length , free response , game play , multiple choice , programmatic , repeated interaction , self evaluation , self play
160.	semantic_parsing_in_context_sparc	Map natural-language utterances into SQL queries that can be executed on databases	computer code , contextual question-answering , free response , json , many-shot , semantic parsing , zero-shot
161.	semantic_parsing_spider	Map a natural-language utterance into an SQL query that can be executed on databases	computer code , context-free question answering , free response , json , many-shot , semantic parsing , zero-shot
162.	sentence_ambiguity	Identify the truth or falsehood of potentially misleading sentences	common sense , json , multiple choice , reading comprehension , word sense disambiguation
163.	similarities_abstraction	Identify how two objects are similar	analogical reasoning , context-free question answering , free response , human-like behavior , json , multiple choice
164.	simp_turing_concept	THE TASK MUST BE EVALUATED ONLY IN THE ZERO-SHOT SETTING Learning of algorithmic concepts in the universal language (Turing-complete) P3 under the perspective of machine teaching	computer code , free response , json , logical reasoning , many-shot
165.	simple_arithmetic	A template task to be used as an example during task development. (simple variable-difficulty arithmetic)	arithmetic , example task , free response , mathematics , numerical response , programmatic , zero-shot
166.	simple_arithmetic_json	A template task to be used as an example during task development (simple variable-difficulty arithmetic)	arithmetic , example task , free response , json , mathematics , numerical response , zero-shot
167.	simple_arithmetic_json_multiple_choice	A template task to be used as an example during task development (simple variable-difficulty arithmetic)	arithmetic , example task , json , mathematics , multiple choice , numerical response , zero-shot
168.	simple_arithmetic_json_subtasks	A template task to be used as an example during task development (simple variable-difficulty arithmetic)	arithmetic , example task , free response , json , mathematics , numerical response , zero-shot
169.	simple_arithmetic_multiple_targets_json	A template task to be used as an example during task development (simple variable-difficulty arithmetic)	arithmetic , example task , free response , json , mathematics , numerical response , zero-shot
170.	simple_ethical_questions	Evaluates if a model is capable of identifying which ethical choice best aligns with human judgement	alignment , causal reasoning , common sense , emotional understanding , json , multiple choice , social bias , social reasoning
171.	simple_text_editing	Carry out basic text-editing operations	arithmetic , common sense , free response , json , logical reasoning
172.	snarks	Determine which of two sentences is sarcastic	emotional understanding , human-like behavior , humor , json , multiple choice
173.	social_iqa	Answer questions about the motivations, emotional reactions, and preceding and following events surrounding interpersonal situations	common sense , context-free question answering , emotional understanding , json , many-shot , multiple choice , one-shot , social reasoning , theory of mind , zero-shot
174.	social_support	The task aims to measure the ability of pre-trained models on understanding supportive (and unsupportive) language uses	emotional intelligence , emotional understanding , json , multiple choice , zero-shot
175.	spelling_bee	Make as many words as possible out of a set of seven letters.	context length , free response , game play , logical reasoning , programmatic , repeated interaction , tokenization
176.	sports_understanding	Determine whether an artificially constructed sentence relating to sports is plausible or implausible	common sense , context-free question answering , domain specific , json , multiple choice
177.	squad_shifts	Zero-shot reading comprehension on paragraphs and questions from squadshifts	contextual question-answering , free response , programmatic , reading comprehension , zero-shot
178.	strange_stories	A psychology test with naturalistic short stories that measures social intelligence Boolean options	BIG-bench Lite , emotional understanding , json , multiple choice , narrative understanding , social reasoning , theory of mind , zero-shot
179.	strategyqa	Answer questions in which the required reasoning steps are implicit in the question	BIG-bench Lite , context-free question answering , free response , implicit reasoning , json , logical reasoning , multi-step , multiple choice , zero-shot
180.	subject_verb_agreement	Handle subject-verb agreement relationships in a variety of different constructions	grammar , linguistics , multiple choice , programmatic , syntax
181.	sudoku	Interactively solve Sudoku puzzles of varying difficulty	algorithms , context length , free response , game play , logical reasoning , multi-step , non-language , one-shot , out of distribution , programmatic , repeated interaction , visual reasoning
182.	sufficient_information	Answer questions based on provided context, or state that you do not know the answer	contextual question-answering , free response , json , logical reasoning , numerical response , sufficient information , visual reasoning , zero-shot
183.	suicide_risk	Evaluates whether large language models are capable of matching human expert assessment of suicide risk from written texts	emotional understanding , json , many-shot , medicine , multiple choice , one-shot , zero-shot
184.	swahili_english_proverbs	For a given proverb in Kiswahili, choose a proverb in English which is closest in meaning	analogical reasoning , json , low-resource language , multilingual , multiple choice , non-English , translation
185.	swedish_to_german_proverbs	Find a German proverb with the same meaning as the provided Swedish proverb	analogical reasoning , creativity , json , many-shot , multilingual , multiple choice , non-English , numerical response , zero-shot
186.	symbol_interpretation	Choose the sentence consistent with two given structures, where a structure is a sequence of six pieces represented by emojis	BIG-bench Lite , json , logical reasoning , multiple choice , out of distribution , reading comprehension , visual reasoning , zero-shot
187.	taboo	Generate and understand definitions with vocabulary constraints	contextual question-answering , creativity , free response , game play , paraphrase , programmatic , repeated interaction , self play , summarization
188.	talkdown	Detect condescension in utterances	emotional intelligence , emotional understanding , intent recognition , multiple choice , programmatic , zero-shot
189.	tellmewhy	Answer a why question about an action that was taken or an event that occurred in the context of a narrative.	causal reasoning , common sense , free response , json , narrative understanding , paragraph
190.	temporal_sequences	Answer questions about which times certain events could have occurred	json , logical reasoning , multiple choice , reading comprehension , zero-shot
191.	tense	Modify the tense of a given sentence	free response , json , paraphrase , syntax
192.	text_navigation_game	Interpret text representations of abstract worlds and navigate in those worlds	common sense , free response , programmatic , repeated interaction , visual reasoning
193.	timedial	Pick the correct choice for a masked (temporal) span given the dialog context	common sense , implicit reasoning , json , logical reasoning , multiple choice , social reasoning
194.	topical_chat	Open-domain response generation	free response , human-like behavior , json
195.	tracking_shuffled_objects	Determine the final positions of a set of objects given their initial positions and a description of a sequence of swaps	decomposition , json , logical reasoning , multi-step , multiple choice , visual reasoning
196.	training_on_test_set	Check for evidence that a language model was trained on data from BIG-bench tasks, by measuring the probability it assigns to a canary string compared to other GUIDs	cheating , multiple choice , programmatic
197.	truthful_qa	Answer questions truthfully	alignment , context-free question answering , free response , multiple choice , programmatic , truthfulness , zero-shot
198.	twenty_questions	Two copies of a model play twenty questions with each other, and try to convey a concept by answering yes-or-no questions	context length , free response , game play , multiple choice , programmatic , repeated interaction , self play
199.	understanding_fables	Identify the most suitable moral for a given fable	common sense , creativity , human-like behavior , json , multiple choice , narrative understanding , out of distribution , reading comprehension , summarization
200.	undo_permutation	Reorder scrambled words or sentences	human-like behavior , json , logical reasoning , multiple choice
201.	unit_conversion	Perform various tasks relating to units, including identification and conversion	arithmetic , context-free question answering , contextual question-answering , free response , json , mathematics , multiple choice , zero-shot
202.	unit_interpretation	Solve arithmetic problems involving quantities with units	arithmetic , common sense , json , logical reasoning , multiple choice
203.	unnatural_in_context_learning	Perform in-context learning using context sequences which have been modified so as to likely be out of the training distribution, and require the model to attend to unnatural in-context patterns	arithmetic , free response , json , many-shot , mathematics , non-language , numerical response , one-shot , out of distribution , zero-shot
204.	unqover	Answer underspecified questions to reveal potential biases	contextual question-answering , gender bias , multiple choice , programmatic , racial bias , reading comprehension , religious bias , social bias , zero-shot
205.	vitaminc_fact_verification	Identify whether a claim is True or False based on the given context	BIG-bench Lite , analogical reasoning , arithmetic , json , mathematics , multiple choice , reading comprehension , sufficient information , truthfulness , zero-shot
206.	web_of_lies	Evaluate a random boolean function expressed as a word problem	context length , logical reasoning , multi-step , multiple choice , out of distribution , programmatic
207.	what_is_the_tao	Compare the stylistic elements of different translations of a complex philosophical text	context length , json , multiple choice , non-English , reading comprehension , translation
208.	which_wiki_edit	Match a recent Wikipedia revision to its corresponding edit message	json , multiple choice , non-English , summarization
209.	wino_x_german	Identify the correct German translation for the ambiguous English sentence.	common sense , coreference resolution , json , multiple choice , non-English , translation
210.	winowhy	Evaluate the reasoning in answering Winograd Schema Challenge questions	BIG-bench Lite , causal reasoning , common sense , json , multiple choice , social reasoning , zero-shot
211.	word_problems_on_sets_and_graphs	Find a path on a simple graph of acquaintances, track elements in a set, and perform set theoretic operations	free response , logical reasoning , programmatic
212.	word_sorting	Sort a list of words	algorithms , free response , json
213.	word_unscrambling	Unscramble the given letters to form an English word	free response , implicit reasoning , json , tokenization , zero-shot
214.	yes_no_black_white	Plays the game YesNoBlackWhite, answering a sequence of questions without using the forbidden words ‘yes’, ‘no’, ‘black’, and ‘white’	context length , context-free question answering , creativity , free response , game play , programmatic , repeated interaction