Choose theme
AI Resources
LARYBench
LARYBench evaluates representations extracted from videos or image pairs, using action recognition and robot-action regression probes.
It compares information carried by a representation. A probe score is separate from showing that a robot policy can perform a task. Use this as a first read, not a recommendation. Open the original project before trusting details like terms, limits, privacy, cost, setup, or safety.
What it is
Extract, then probe
An extraction stage produces representations; classification and regression stages evaluate different action targets.
Why it stands out
Compare representations directly
The framework accepts latent-action models and visual encoders, rather than requiring a complete downstream policy for every comparison.
Availability
Code, metadata and dataset downloads
The public project provides extraction adapters, probe code and data instructions. The maintainers separately identify an unreleased Atomic Robot classification subset; public downloads do not cover every paper setting.
Why it matters
What makes it useful
To compare a visual encoder with a latent-action model, choose what the probe should recover: an action label or robot action values. LARYBench treats those as classification and regression tasks. They ask what information the representation retains, without establishing that a resulting robot policy succeeds at a physical task.
What to know
Where it fits
For a model outside the supplied adapters, the repository's custom-model route requires a numeric tokens array in each extracted .npz file and a metadata CSV. Its registration and configuration instructions connect that output to the existing probes. The integration task is producing the expected representation and metadata, rather than supplying a finished controller.
Notable points
What stands out
In their classification-data response, the maintainers acknowledge annotation errors and ambiguous action categories in human and robot data. They recommend relative model comparison under the same evaluation protocol, rather than treating absolute classification accuracy as a precise measure of latent-action quality. Their response frames a score through that shared protocol; it does not say a high number resolves the label ambiguity.
Before using
What to review
The repository lists larybench for several extractors, while V-JEPA, Wan and villa-X use upstream environments. Its model table identifies the environment and extra checkpoint paths for the selected extractor.
The repository workflow uses dataset inputs, metadata paths and extractor checkpoints separately. Prepare the inputs for the selected extraction and probe stages rather than assume one download supplies the whole run.
The repository identifies MIT terms for its code and separate terms for dataset subsets, and directs licensing questions to the original dataset sources.
Reader fit
Who may find it relevant
For paper reproduction, match the setting before comparing scores. In their reproduction response, the maintainers identify the Composite Robot result as the final epoch and name its evaluation config. They separately say the Atomic Robot classification LIBERO subset has not been released. Those are different settings: the released data and a best-epoch result should not be assumed to reproduce every paper number.
Editorial note
Why LifeHubber lists it
The current CLI selects GPUs separately for each stage: extraction defaults to GPU 0, classification defaults to eight IDs, and regression uses CUDA_VISIBLE_DEVICES with an eight-ID fallback when it is unset or empty. For a one-GPU setup, those are separate selections: --gpus 0 for extraction and classification, and CUDA_VISIBLE_DEVICES=0 for regression. Choosing the extraction GPU does not configure the later stages.
Source links
Source materials
Reader note
Before relying on this entry
LifeHubber lists entries to help readers inspect AI projects, not to endorse them or prove they are safe, suitable, accurate, maintained, or right for a specific use. We do not verify every entry in depth. Before relying on anything listed, review the original materials, terms, privacy practices, limits, and risks that matter for your situation.
More in Datasets
Keep browsing this category
Explore more datasets.
ParseBench
run-llama/ParseBench
A document parsing benchmark for AI-agent workflows, focused on whether parsed PDFs preserve structure and meaning for downstream evaluation.
UltraData-SFT-2605
openbmb/UltraData-SFT-2605
An OpenBMB supervised fine-tuning dataset with 15,036,178 thinking and non-thinking samples across math, code, knowledge, Chinese, instruction-following, and multilingual configurations, used in MiniCPM5-1B-SFT post-training.
Open Agent Leaderboard Results
open-agent-leaderboard/results
A public results dataset for the Open Agent Leaderboard, with agent, model, benchmark, score, completion, error, action-count, and cost fields linked to the Exgentic evaluation framework and paper.
For project maintainers
Listed here? You can use the badge.
If you maintain a project with a current LifeHubber listing, you may add the optional “Listed on LifeHubber AI Resources” badge to its README, docs, or website. No introduction or permission request is needed.
