---
title: "Industry Insight | Multimodal Foundation Models Are the Necessary Path to AGI"
author: "Unique Research"
sourcePublication: "Unique Research (FFCAP English library)"
originalPublishedAt: "2024-03-11T16:30:00+08:00"
canonical: "https://ffcap.cn/en/research/20240311a064o500"
source: "https://ffcap.cn/en/research/20240311a064o500"
language: "en"
cnSourceUrl: "https://view.inews.qq.com/a/20240311A064O500"
---

# Industry Insight | Multimodal Foundation Models Are the Necessary Path to AGI

_Unique Research · Industry insight · March 11, 2024_

From the keynote track at Unique Capital’s **2024 AIGC Application Development Summit** (Shanghai, Jan 24, 2024; ~1,000 AI practitioners). Speaker: **Prof. Lu Zhiwu**, Gaoling School of AI, Renmin University of China — Wudao · Wenlan multimodal team.

## Background

With BAAI support the team has worked multimodal LLMs since 2020; March 2021 Wenlan v1 (CLIP-like with detail differences), then scaled data and size. Outputs include influential _Nature_ family papers. Team experience and talent grew quickly on the project.

## 1\. Multimodal general generative models

CLIP / Wenlan are retrieval-leaning (understanding-heavy). Since generative models exploded, Lu argues multimodal needs generative research too — **multimodal general generative models** with multimodal in _and_ out: encode modalities into a latent space, then one or more generators (e.g. separate vision vs text).

-   **Engineering path** — Microsoft Visual ChatGPT: ChatGPT as instruction hub calling multimodal generators.
-   **Academic path** — Tsinghua Zhu Jun team’s UniDiffuser: replace U-Net CNN with Transformer/ViT so vision + language generation share one decoder under a diffusion+Transformer frame.
-   **Google Gemini path** — two decoders (vision vs language differ); “native multimodal” via one shared encoder across modalities vs the common one-encoder-per-modality pattern. Shared-encoder pros: joint training → more natural cross-modal transfer. Cons: adding/removing a modality is hugely expensive (Lu cites ~$100M-class LLM training; another modality another $100M) — untenable for small teams. Separate encoders make incremental modality adds cheaper.

Whether “native” or not, choose the path that fits resources.

## 2\. Multimodal is the necessary path to AGI

ChatGPT / GPT-4 are powerful, but Lu holds LLMs alone cannot reach AGI — multimodal is required, and the team builds that way. Their multimodal generative model (esp. language-out) maps image+text into an LM for image-grounded dialogue.

Training needs ≥2 stages: (1) pretrain — caption, detect, OCR and more on large, quality-tolerant data; (2) carefully designed, high-quality instruction sets for complex human commands (smaller volume, higher quality). Their **ChatImg 3.0** approaches GPT-4V quality; unlike GPT-4V’s “answer whatever you ask / rarely refuse,” ChatImg better matches Chinese habits and local context. Public-set evals: hope to near OpenAI on watched capabilities; far above some domestic open models. In power-industry and urban-governance scenarios ChatImg 3.0 beat most existing multimodal LLMs — and a single unified model beat domain-specific small models. Across tests, multimodal LLMs consistently beat traditional small models on complex scenes.

Video generation: original Transformer-backbone diffusion-style model combining Transformer + diffusion strengths for video.

## 3\. Landing directions

Explorations: cameras (smart city), drones (inspection), sensors (equipment diagnosis/prediction), embodied intelligence. Best packaging for integrators: LLM on an all-in-one appliance with fine-tune + prompt tools so any scene can adapt with light fine-tuning — a product form the team hopes to ship.

_12 CN images; attach on ingest._

— END —

---

Editorial English edition published on UniqueCapital: https://ffcap.cn/en/research/20240311a064o500
On-site reading page: https://ffcap.cn/en/research/20240311a064o500