Choose theme
AI Visuals
CoMoVi
CoMoVi is a research framework for generating human motion and video together. Its paper couples a motion branch and a video branch within the same diffusion process.
The public code includes inference, downloadable checkpoints, and training preparation. Readers interested in 3D motion output should also inspect the current motion-prediction issue reports. Use this as a first read, not a recommendation. Open the original project before trusting details like terms, limits, privacy, cost, setup, or safety.
What it is
A motion-and-video co-generation framework
CoMoVi studies joint human-motion and video generation. Its examples pair a rendered person with a motion representation, giving readers two views of the generated movement.
Why it stands out
Motion represented inside a video model
The method projects 3D motion information into a 2D representation that a video diffusion model can process. Motion structure participates in generation alongside the visible person and scene.
Availability
Public repo with inference and training path
Code and checkpoints are public, with inference and custom-data preparation instructions. The README still labels its dataset download as coming soon, although a dataset link is present. That label does not establish whether the linked dataset can currently be retrieved.
Why it matters
What makes it useful
If you are investigating how a generated person moves, CoMoVi gives you a motion representation to inspect alongside the RGB video. The inference script's --save_mode separate option writes example_output_rgb.mp4 and example_output_motion.mp4. Compare the pose and timing across those videos before judging the result from the rendered clip alone; a motion preview is not proof of a working 3D animation export.
What to know
Where it fits
This is an installed research pipeline for human-video and motion experiments. The inference example starts from a reference image and includes a text prompt; it is not documented as an import workflow for an existing animation sequence from another tool.
Notable points
What stands out
The paper generates motion and video within one denoising loop. The README's inference example passes features from the motion branch to the RGB branch, while the script's interaction setting defaults to dual. Check the selected interaction setting when comparing a run with the coupling described in the paper.
Before using
What to review
The setup uses a CUDA/PyTorch environment with FlashAttention, PyTorch3D, and CameraHMR. Follow the project installation and checkpoint instructions before planning a run.
Open issue reports describe init-pose loading failures and zero-valued SMPL decoder and fusion weights. These are user reports, not LifeHubber tests; check their status before relying on the optional 3D motion-prediction path.
For training on your own clips, the documented preparation is to estimate motion, smooth it, render a motion representation, and normalize the data.
The captioning instructions use a Gemini API or referenced open-model examples. Check the chosen captioning path before sending footage to an external service.
Check permission to use images and footage of identifiable people for generation, training, and publication; a reference image can carry someone's likeness into the experiment.
Reader fit
Who may find it relevant
Researchers comparing human-motion representations with generated human video.
Builders inspecting the inference code or preparing paired motion and video training data.
For transferring an existing character animation into video, check for an explicit input path first; this example begins with a reference image.
Editorial note
Why LifeHubber lists it
We list CoMoVi for human-video research where you have a reference image and a description, but no movement sequence to supply. The paper studies generating the human movement together with the video without an external motion reference. That gives researchers a paired-generation approach to investigate when creating the movement is part of the task, rather than assuming an existing animation is the starting material.
Source links
Source materials
Reader note
Before relying on this entry
LifeHubber lists entries to help readers inspect AI projects, not to endorse them or prove they are safe, suitable, accurate, maintained, or right for a specific use. We do not verify every entry in depth. Before relying on anything listed, review the original materials, terms, privacy practices, limits, and risks that matter for your situation.
More in Music / Image Gen Models
Keep browsing this category
Explore more media-generation model resources.
ACE-Step 1.5
ace-step/ACE-Step-1.5
A locally runnable music model with 2B and XL 4B variants for full-song generation, reference-guided creation, repainting, accompaniment, stem separation, and lightweight style training.
Qwen-Image-2.1
QwenLM/Qwen-Image-2.1
Qwen's open-weight image model for text-to-image generation, single- and multi-reference editing, marked-region changes, subject extraction, native transparent RGBA output, and 2K workflows.
ViMax
HKUDS/ViMax
An agentic video-generation framework for turning ideas, scripts, or longer narratives into planned video workflows, with a project-based web interface, interactive agent loop and terminal UI, storyboards, shot planning, previews, render checkpoints, and configurable model providers.
For project maintainers
Listed here? You can use the badge.
If you maintain a project with a current LifeHubber listing, you may add the optional “Listed on LifeHubber AI Resources” badge to its README, docs, or website. No introduction or permission request is needed.
