Unique Research · Industry insight · March 11, 2024
From the keynote track at Unique Capital’s 2024 AIGC Application Development Summit (Shanghai, Jan 24, 2024; ~1,000 AI practitioners). Speaker: Prof. Lu Zhiwu, Gaoling School of AI, Renmin University of China — Wudao · Wenlan multimodal team.
Background
With BAAI support the team has worked multimodal LLMs since 2020; March 2021 Wenlan v1 (CLIP-like with detail differences), then scaled data and size. Outputs include influential Nature family papers. Team experience and talent grew quickly on the project.
1. Multimodal general generative models
CLIP / Wenlan are retrieval-leaning (understanding-heavy). Since generative models exploded, Lu argues multimodal needs generative research too — multimodal general generative models with multimodal in and out: encode modalities into a latent space, then one or more generators (e.g. separate vision vs text).
- Engineering path — Microsoft Visual ChatGPT: ChatGPT as instruction hub calling multimodal generators.
- Academic path — Tsinghua Zhu Jun team’s UniDiffuser: replace U-Net CNN with Transformer/ViT so vision + language generation share one decoder under a diffusion+Transformer frame.
- Google Gemini path — two decoders (vision vs language differ); “native multimodal” via one shared encoder across modalities vs the common one-encoder-per-modality pattern. Shared-encoder pros: joint training → more natural cross-modal transfer. Cons: adding/removing a modality is hugely expensive (Lu cites ~$100M-class LLM training; another modality another $100M) — untenable for small teams. Separate encoders make incremental modality adds cheaper.
Whether “native” or not, choose the path that fits resources.
2. Multimodal is the necessary path to AGI
ChatGPT / GPT-4 are powerful, but Lu holds LLMs alone cannot reach AGI — multimodal is required, and the team builds that way. Their multimodal generative model (esp. language-out) maps image+text into an LM for image-grounded dialogue.
Training needs ≥2 stages: (1) pretrain — caption, detect, OCR and more on large, quality-tolerant data; (2) carefully designed, high-quality instruction sets for complex human commands (smaller volume, higher quality). Their ChatImg 3.0 approaches GPT-4V quality; unlike GPT-4V’s “answer whatever you ask / rarely refuse,” ChatImg better matches Chinese habits and local context. Public-set evals: hope to near OpenAI on watched capabilities; far above some domestic open models. In power-industry and urban-governance scenarios ChatImg 3.0 beat most existing multimodal LLMs — and a single unified model beat domain-specific small models. Across tests, multimodal LLMs consistently beat traditional small models on complex scenes.
Video generation: original Transformer-backbone diffusion-style model combining Transformer + diffusion strengths for video.
3. Landing directions
Explorations: cameras (smart city), drones (inspection), sensors (equipment diagnosis/prediction), embodied intelligence. Best packaging for integrators: LLM on an all-in-one appliance with fine-tune + prompt tools so any scene can adapt with light fine-tuning — a product form the team hopes to ship.
12 CN images; attach on ingest.
— END —