Google has unveiled VideoPoet, a groundbreaking large language model (LLM) designed to excel in various video generation tasks. VideoPoet stands out for its remarkable ability to produce coherent large-motion videos. The LLM can do complex tasks such as text-to-video conversion, image-to-video transformation, video stylization, inpainting, outpainting, and video-to-audio functionality.
VideoPoet arrives at the time when Microsoft’s Copilot AI gained the ability to generate audio clips from text prompts. VideoPoet pushes the boundaries of video generation.
Unlike the current trend in video generation models that predominantly rely on diffusion-based approaches, VideoPoet utilizes large language models (LLMs), recognized for their exceptional learning capabilities across various modalities, including language, code, and audio.
VideoPoet operates across various modalities, including video, image, audio, and text. The model can take text prompts as input for tasks like text-to-video, image-to-video, video-to-audio, stylization, inpainting, and outpainting. The model has ability to animate still images, stylized videos, generate audio from videos, and much more.
Here are some examples of videos generated by Google’s VideoPoet.
VideoPoet leverages autoregressive language models to learn across video, image, audio, and text modalities. It uses multiple tokenizers that allow the model to generate tokens conditioned on context and convert them back into viewable representations with tokenizer decoders.
VideoPoet can also generate long videos by conditioning at the last second. It can also predict the next, allowing for faithful preservation of object appearance. It also allows interactive editing of generated clips and enables motion control for objects in input videos.
Also see: AI Tools Directory
Google’s VideoPoet leverages the capabilities of large language models and results suggest promising potential in video generation. With the time, VideoPoet will become even better. Google also published a detailed blog post with several examples of videos and audio generated by it. I recommend people read that article to see the examples.
