Which is better than the open source multimodal model？ The TOP12 list is coming

Organize from Happy

Quantum bit | WeChat official account QbitAI

The explosion of GPT-4 has completely set off the research upsurge of multi-modal large models in academic circles.

However, how to measure the performance of this kind of model has been controversial in the industry, and there is no comprehensive evaluation standard.

In addition, there is no complete review to define and study it.

In view of this, Tencent Youtu, jointly with China University of Science and Technology and Xiamen University, published two papers on multi-modal large models.

There is not only the first multi-modal large-scale model review-

There is also a comprehensive evaluation list!

Related projects have exploded on GitHub, and as of July 3, they have already won 2200+ stars.

So, what are the best multi-modal large models in the industry at present? What are its definition, key technologies, advantages and challenges?

Let’s take a look together.

TOP12 ranking of multi-modal large models

The researchers set a total of 16 lists, including two general lists and 14 subtasks.

The overall list can be regarded as the score of the "overall ability" of the model, which is divided into perceptual and cognitive categories, and 14 subtasks are some of the subdivided tasks, which can evaluate whether the multi-modal large model is better at doing something.

The researchers selected a total of 12 open source multi-modal models to "demonstrate" the evaluation criteria.

The total list of perception categories is the total score of all perception tasks, and the display is the highest in BLIP-2:

The general list of cognitive classes is a list of various cognitive tasks, which adds up to the highest in MiniGPT-4:

The evaluation results show that BLIP-2 and InstructBLIP are in the top three in these two lists, and they are really the "top players" of the current open source multimodal model.

Specific to the 14 subtasks, the ranking of the model is different.

The evaluation results are as follows. It can be said that it is clear at a glance who is more "partial" and who can achieve comprehensive optimization in various tasks:

So, how is the score result of this list obtained?

How to get the scoring standard?

The paper holds that a good multi-modal large-scale model scoring standard should have the following four characteristics:

(1) It should cover as many areas as possible, including perception and cognitive ability (perception is the basis of cognition).

Among them, the former refers to identifying objects, including their existence, quantity, position and color; The latter refers to more complex reasoning based on comprehensive perceptual information and knowledge in LLM, including common sense reasoning, numerical calculation, text translation and code reasoning.

(2) Its data or labeling should avoid using existing public data sets as much as possible to reduce the risk of data leakage.

Therefore, all instruction-answer pairs in the evaluation should be artificially constructed. For a small number of public data sets, only their images are used without relying on their original labels. At the same time, try to collect data by manual shooting and image generation.

(3) Instruction design should be as concise as possible and conform to human cognitive habits.

Different instruction designs may greatly affect the output of models, but all models are evaluated under unified and concise instructions to ensure fairness. A good multi-modal large model should have the ability to generalize to such concise instructions and avoid falling into hint engineering.

(4) The output of multi-modal large model under this concise instruction should be intuitive and convenient for quantitative statistics.

The open answer of multi-modal large model poses a great challenge to quantitative statistics. The existing methods tend to use GPT or manual scoring, but they may face inaccurate and subjective problems.

Therefore, after the production, the final evaluation question is about this:

Then, according to the accuracy of the model answer, the score is judged.

It is worth mentioning that the authors have also tried to design multiple-choice instructions, but found that the current multi-modal large model is still difficult to follow such complicated instructions. （doge）

Summary of the first multi-modal large model

Of course, the evaluation criteria of this list are not "groundless".

To understand why the list is rated like this, you can look at another paper review on multimodal large models, which carefully sorts out its definition, key technologies and challenges.

Specifically, the paper defines the multimodal megamodel (MLLM) as "a model with the ability of receiving and reasoning multimodal information, which is extended from LLM".

Compared with the popular single-mode LLM, this model has the following advantages:

It is more in line with the habit of human cognition of the world. Human beings have multiple senses to receive multiple modal information, which are usually complementary and synergistic. Therefore, using multimodal information can generally better recognize and complete tasks.

More powerful and user-friendly interface. By supporting multimodal input, users can input and convey information in a more flexible way.

Broader mission support. LLM can usually only accomplish the tasks related to pure text, while MLLM can accomplish more tasks through multimodal, such as picture description and visual knowledge quiz.

Therefore, if you want to study this kind of multi-modal large model, you often need to master three key technologies:

1. Multimodal instruction tuning (M-IT).

2. Multimodal in-context learning (M-ICL).

3. Multimodal Chain of Thought, M-CoT).

In addition, we need to study one of its applications (multimodal system with LLM as the core), namely LLM-Aided Visual Reasoning, LAVR).

However, at present, the multi-modal large-scale model is still in its infancy, so there are also some challenges, such as limited perceptual ability, fragile reasoning chain, further improvement of command obedience ability and the widespread problem of object illusion.

For more summary details and list details, you can poke the paper to view it ~

Multi-modal big model list:

https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation

Paper address:

[1] Summary: https://arxiv.org/abs/2306.13549

[2] Evaluation: https://arxiv.org/abs/2306.13394

End—

Advances in science and technology meet every day ~

Original title: "Which is the strong open source multimodal model? The TOP12 list is coming, and GitHub has won 2.2k+ stars.

Read the original text