Zero-Shot In-Context Multi-Choice Question-Answering Bench

Task: In-context question answering (QA) is a the task where a model answers questions based on a provided text passage, using only the given context without relying on external knowledge. In-context question answering is used in all retrieval-augmented generation (RAG) solutions where a generative LLM is tasked to provide an answer to a user's question using documents (or document fragments) retrieved from a semantic database.

Dataset: Belebele dataset. The dataset contains 900 questions for each language. Each question comes along with a text passage (context) and four possible answers. Since the questions are about the text passage and the selection of the best answer relies on understanding, which answer best answers the question based on the given text fragment, this makes the dataset useful for benchmarking in-context QA, albeit a very simple use case where the LLMs task is just to output the correct answer's ID. This benchmark allows to see, which models could be useful for in-context question answering (and RAG solutions) in practice, but as it benchmarks just a simple scenario, make sure you test each model for your language and specifics of your task.

{"messages":
        [{"role": "system", "content": "Answer the question by choosing the answer that can be inferred from the given text. Report only the correct answer's number using the following JSON format: {\"answer_id\":\"\"}."},
        {"role": "user", "content": "Text:\nOne of the most common problems when trying to convert a movie to DVD format is the overscan. Most televisions are made in a way to please the general public. For that reason, everything you see on the TV had the borders cut, top, bottom and sides. This is made to ensure that the image covers the whole screen. That is called overscan. Unfortunately, when you make a DVD, it's borders will most likely be cut too, and if the video had subtitles too close to the bottom, they won't be fully shown.\nQuestion: According to the passage, which of the following problems might one encounter when converting a movie to DVD format?\nPossible answers:\n1) An image that doesn't fill the entire screen\n2) Partially cut subtitles\n3) An image that fills the entire screen\n4) Cut borders\n"}]
}

Metrics: Accuracy and below we are also interested to see, whether the LLM's output data in the required format.

Hover with your mouse over the bars to see top model scores for each language.

Here we analyse whether the LLMs are able to follow the instructions to return only the asked JSON object:

{"answer_id":""}

We distinguish four levels of obedience: