Modern generative AI has made impressive strides in creating lifelike talking-head and full-body videos from single-speaker audio, but when multiple people converse, existing solutions often falter—mixing up who’s speaking or failing to follow a direction. Enter MultiTalk, a new framework designed specifically for multi-person, audio-driven video generation.
Key points
- MultiTalk is a framework for multi-person, audio-driven video generation whose inputs are multi-stream audio from each speaker, a reference image per person, and a text prompt describing the scene or interaction.
- It produces a coherent video in which each character’s lip movements sync accurately to their own audio track.
- Label Rotary Position Embedding (L-RoPE) binds each audio stream to the correct person, preventing voice-swapping mistakes common in naive multi-audio approaches.
- MultiTalk fine-tunes only selected layers and trains jointly on talking-head, talking-body, and conversational datasets, which the work says preserves the base model’s ability to follow instructions.
- The authors report MultiTalk outperforms existing single-person generation methods on a suite of benchmarks, with higher lip-sync accuracy, better visual quality, and faithful instruction following.
At its core, MultiTalk takes as input:
- Multi-stream audio from each speaker,
- A reference image (or avatar) per person, and
- A text prompt describing the scene or interaction.
From these, it produces a coherent video where each character’s lip movements sync accurately to their own audio track, and all actors follow the scripted directions.
Key Innovations
- Label Rotary Position Embedding (L-RoPE): A novel technique that binds each audio stream to the correct person, preventing “voice swapping” mistakes that plague naïve multi-audio approaches.
- Partial Parameter & Multi-Task Training: By fine-tuning only selected layers and jointly training on talking-head, talking-body, and conversational datasets, MultiTalk preserves the base model’s ability to follow instructions while adapting it for multi-person scenarios.
Demo Highlights
- Cartoon Conversations: Generate friendly chats between animated characters, perfect for short sketches or educational clips.
- Virtual Singing Duets: Produce synchronized singing videos of two or more characters performing a song, each matching their own vocal track.
- Instruction-Following Dialogues: Stage guided interactions—think customer-service bots or tutorial hosts—where each participant reacts and responds under scripted control.
For example, imagine a cozy café scene: Nick Wilde and Judy Hopps sit across from one another, sharing coffee and banter in perfect sync with their voices. Then picture a studio setting where a host interviews a guest, with camera-ready expressions and flawless audio-visual alignment.
Results & Availability
MultiTalk outperforms existing single-person generation methods on a suite of benchmarks, achieving higher lip-sync accuracy, better visual quality, and faithful instruction following across multi-person datasets.
- Read the full technical report on arXiv: Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation
- Explore the code and demo assets on GitHub: meigen-ai/multi-talk
With MultiTalk, creators can bring dynamic, multi-speaker scenes to life—opening up new horizons in virtual events, animated storytelling, remote interviews, and more.
Leave a Reply