Zero-Shot Learning

Special thanks to Sarvesh Talele for his meaningful contributions, support, and wisdom that helped shape this work. Traffic cameras record the circumstances of a great many collisions, but methods that read those recordings automatically are usually trained on annotated footage from the same camera. This work asks what can be recovered without any labelled real-world data at all, using three independent modules and only pre-trained weights.

Text2Video-Zero, from Picsart AI Research, turns a text-to-image diffusion model into a zero-shot video generator with no video training data at all. This write-up explains the method, the motion dynamics and cross-frame attention that keep a generated sequence coherent, and reports what the model produced when run, including where the results break down.