UniTalk: Real-Time Speech-to-Avatar Generation via Direct Feature-Space Adaptation of a Full-Duplex Speech Model

Learn how UniTalk generates real-time, lip-synced avatars from speech by directly adapting speech model features to a video model, overcoming serialization bottlenecks for fluid, interrupted conversations.

PyTorch FastAPI WebSockets HuggingFace Hub Wav2Vec2

Overview

UniTalk is a real-time, full-duplex conversational avatar system: a user speaks into their browser microphone and the server streams back synchronized speech and a lip-synced animated talking-head video, end to end on a single GPU, with barge-in support for natural interruption.

The central contribution is that the pipeline never re-transcodes audio to drive the face. A lightweight (~93M-parameter) adapter maps a full-duplex speech-to-speech model’s internal hidden states (transformer_out, produced at 12.5 Hz) directly into the Wav2Vec2 feature space that a frozen diffusion-transformer video model (SoulX-FlashHead) expects — replacing the Wav2Vec2 audio encoder entirely. All downstream components (audio projection, the DiT, and the VAE decoder) remain frozen; only the adapter is trained, using a two-stage schedule of feature-space pretraining followed by end-to-end distillation.

Tech stack

PyTorch

PyTorch is the open-source machine learning framework: it provides a Python-first tensor library with strong GPU acceleration and a dynamic computation graph for building deep neural networks.

PyTorch, developed by Meta AI, is a premier open-source deep learning framework favored in both research and production environments. Its core is a powerful tensor library (like NumPy) optimized for GPU acceleration, delivering 50x or greater speedups for complex computations. The key differentiator is its 'Pythonic' design and dynamic computation graph (eager execution), which allows for rapid prototyping and simplified debugging compared to static-graph frameworks. Leveraging its Autograd system for automatic differentiation, practitioners build and train models for computer vision and NLP; major companies like Tesla (Autopilot) and Microsoft utilize PyTorch for critical AI applications.

https://pytorch.org

View projects
FastAPI

FastAPI is a modern, high-performance Python web framework for building APIs with automatic OpenAPI documentation.

FastAPI is a robust, high-speed Python web framework: it is built on Starlette (for async capabilities) and Pydantic (for data validation and serialization). Leveraging standard Python 3.8+ type hints, the framework automatically generates interactive API documentation (Swagger UI/ReDoc) and enforces data validation, effectively reducing developer-induced errors by an estimated 40%. This architecture delivers performance on par with Node.js and Go, significantly increasing feature development speed (up to 300% faster). It is production-ready, fully supporting OpenAPI and JSON Schema standards for all API specifications.

https://fastapi.tiangolo.com

View projects
WebSockets

Secures a persistent, full-duplex TCP connection via a single HTTP handshake, delivering low-latency, bidirectional data streaming for real-time applications.

WebSockets establishes a continuous, two-way communication channel: a critical shift from the request/response cycle of HTTP/1.1. The connection initiates with an HTTP `Upgrade` handshake, switching the protocol to `wss` (secure) on port 443. This persistent link minimizes network overhead, eliminating the need for inefficient HTTP polling. The protocol, standardized as IETF RFC 6455 in 2011, ensures near-instant data transfer: vital for high-performance use cases (e.g., live chat, collaborative editing, financial market data).

https://websocket.org

View projects
HuggingFace Hub

Hugging Face Hub: The central platform for AI models, datasets, and demos.

Hugging Face Hub serves as a comprehensive platform for the machine learning community. It offers over 500,000 models, 100,000 datasets, and 50,000 demos, facilitating collaboration and resource sharing. Users can host model versions, manage datasets, and showcase interactive applications (Spaces). The Hub supports various ML tasks (e.g., natural language processing, computer vision) and integrates with popular libraries like Transformers and Diffusers, streamlining AI development and deployment.

https://huggingface.co/docs/hub/index

View projects
Wav2Vec2

Wav2Vec2 is a self-supervised speech recognition model that learns powerful representations directly from raw audio, significantly reducing the need for labeled data.

Wav2Vec2, released by Facebook AI (now Meta) in September 2020, is a transformer-based architecture for Automatic Speech Recognition (ASR). It employs a self-supervised learning approach, pre-training on vast amounts of unlabeled speech audio to learn robust speech representations. The model consists of a multi-layer convolutional neural network (CNN) as a feature extractor, which processes raw audio into feature representations, and a Transformer encoder that builds a deeper understanding of these features by analyzing their relationships over time. This innovative pre-training allows Wav2Vec2 to achieve state-of-the-art results even when fine-tuned with minimal labeled data, outperforming previous semi-supervised methods.

https://huggingface.co/docs/transformers/model_doc/wav2vec2

View projects

UniTalk: Real-Time Speech-to-Avatar Generation via Direct Feature-Space Adaptation of a Full-Duplex Speech Model

Related talks