Vidformer brings instant playback to data-edited videos, enabling rapid exploration of multimodal datasets. See our ACM MM '26 paper, Vidformer: Enabling Instant Playback for Interactive Video Exploration, or the preprint on arXiv, for details.
Developed by the OSU Interactive Data Systems Lab.
To quickly try out Vidformer you can:
- try the online Vidformer Playground
Or, you can deploy it yourself:
git clone https://github.com/ixlab/vidformer
cd vidformer
docker build -t igni -f Dockerfile .
docker-compose -f vidformer-igni/docker-compose-local.yaml upYou can find details on this in our Getting Started Guide.
Exploring multimodal datasets means editing videos with data: "show me every time X happens", "draw a box over each Y", "overlay Z telemetry on this video".
Scripts are the only tool expressive enough for these tasks, but they take minutes or hours to render.
Vidformer gives scripts the responsiveness of a video editor: just swap import cv2 with import vidformer.cv2 as cv2 and play the result instantly.
Vidformer combines three techniques:
-
Declarative Lifting: An API shim runs your existing code with symbolic frames, recording a declarative specification of each output frame instead of rendering it.
-
Optimized Rendering: With the full specification known in advance, the rendering engine transparently optimizes and parallelizes rendering, just like a relational database optimizes relational queries.
-
Video on Demand: Result videos are served as a standard VOD stream, with each segment rendered just in time, so playback latency is independent of video length.
Vidformer usually renders videos 2-3x faster than cv2, and starts playback in 0.25-0.5 seconds (~400x faster) when serving videos on demand. When operated by an LLM agent, Vidformer enables conversational video exploration: ask for a video in natural language and watch the result in seconds.
Vidformer builds on open technologies you may already use:
- OpenCV: A
cv2-compatible interface ensures both you (and LLMs) can use existing knowledge and code. - Supervision: Supervision-compatible annotators make visualizing computer vision models trivial.
- FFmpeg: Built on the same libraries, codecs, and formats that run the world.
- Jupyter: View transformed videos instantly right in your notebook.
- HTTP Live Streaming (HLS): Serve transformed videos over a network directly into any media player.
- Apache OpenDAL: Access source videos no matter where they are stored.
Cite:
@inproceedings{winecki2026_vidformer,
title = {Vidformer: Enabling Instant Playback for Interactive Video Exploration},
author = {Winecki, Dominik and Nandi, Arnab},
booktitle = {Proceedings of the 34th ACM International Conference on Multimedia},
series = {MM '26},
year = {2026},
location = {Rio de Janeiro, Brazil},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
isbn = {979-8-4007-2213-4},
doi = {10.1145/3767308.3836399},
url = {https://doi.org/10.1145/3767308.3836399},
}An earlier version is available as a preprint on arXiv.
File Layout:
- ./vidformer: The core rendering library (Rust)
- ./vidformer-py: The Python frontend
- ./vidformer-igni: The vidformer server
- ./misc/vidformer-mcp: An MCP server
- ./misc/vidformer-slack: A slack bot
- ./misc/tla: A TLA+ model of the rendering engine
- ./docs: The project docs
Vidformer components are detailed here.
License: Vidformer is open source under Apache-2.0. Contributions are welcome.
Acknowledgements: Supported by the Imageomics Institute (NSF Award #2118240) and NSF Award #1910356. Any opinions, findings and conclusions or recommendations expressed in the materials here are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.