ZEBRA-CoT Dataset for vision-language reasoning where the model *generates images during the CoT*. Example: for geometry problems, it's helpful to draw lines in image space. 182K CoT labels: math, visual search, robot planning, and more. Only downside: cc-by-nc license :(

Franca Fully open vision encoder. Masks image, encodes patches, then trains student to match teacher's clusters. Key advance: Matryoshka clustering. Each slice of the embedding gets its own projection head and clustering objective. Fewer features == fewer clusters to match.

VRU-Accident New benchmark of 1K videos, 1K captions, and 6K MCQs from accidents involving VRUs. Example: "why did the accident happen?" "(B): pedestrian moves or stays on the road." Current VLMs get ~50-65% accuracy, much worse than humans (95%).

BlindSight AMD paper: they find attention heads often have stereotyped sparsity patterns (e.g. only attending within an image, not across). They generate sparse attention variants for each prompt. Theoretically saves ~35% FLOPs for 1-2% worse on benches.

Long-RL Nvidia paper scaling RL to long videos. First trains with SFT on a synthetic long CoT dataset, then does GRPO with up to 512 video frames. Uses cached image embeddings + sequence parallelism, speeding up rollouts >2X. Bonus: code is already up!

Skywork-R1V3: new reasoning VLM with 76% MMMU. InternViT-6B stitched with QwQ-32B. SFT warmup, GRPO on math, then a small SFT fine-tune at the end. Good benches, actual ablations, and interesting discussion. Details: 🧵

MGPO: multi-turn grounding-based policy optimization. I've been waiting for a paper like this! Trains the LLM to iteratively crop regions of interest to answer a question, and the only reward is the final answer. Details in thread 👇

DriveMRP: interesting method to get a VLM to understand BEV maps + driving scenarios They synthesize high-risk scenes derived from NuPlan. They render it as both a bird's eye view image and a front camera view. 👇

SeqGrowGraph Instead of segment + postprocess, generate lane graphs autoregressively. Node == vertex in BEV space, edge == control point for Bezier curves. At each step, a vertex is added and the adjacency matrix adds one row + column. They formulate this process as next token prediction. Neat!

GLM-4.1V-Thinking: new reasoning VLM with heavy emphasis on RL. Tons of hints but few ablations 😞 eg they upweight difficult-but-learnable samples every iteration, but don't show how it compares to baseline. 9B variant beats Qwen2.5-VL-7B on many standard benchmarks. Details in thread 👇

DenseWorld-1M: insanely detailed + grounded caption dataset. Synthetic data only: SAM, APE for segmentation. Each crop is captioned and verified. VLMs stitch object captions into huge image captions. Beats Sa2VA on referring expression segmentation. Dataset improves Qwen2.5-VL on VQA benchmarks

DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World

Multimodal Large Language Models (MLLMs) demonstrate a complex understanding of scenes, benefiting from large-scale and high-quality datasets. Most existing caption datasets lack the ground locations ...

arxiv.org

SAM4D: promptable 4D seg Data engine uses Grounding-Dino + SAM2 for segmentation + tracking in images. Lidar -> voxels -> ray casting to match to pixels. Clustering to stitch visual masklets into 3D instances. Model is like SAM2 but with a lidar encoder + motion-aware cross attention.

MiCo: multi-image contrast Simple idea: Input is multiple augmented images, either from video or image edits. Prompt: "are these images the same or different?" Train with GRPO. Large bump in multi-image benchmarks, minor bump in general VQA / hallucination benches.

SpatialReasoner-R1 Generates long CoT data with Multi-Model Monte Carlo Tree Search -- multiple candidate models for each step, evaluated by multiple LLM judges. DPO with separate losses on the "descriptive" caption and reasoning. Huge improvements on spatial datasets, good performance on VQA.

ScaleCap: synthetic image captioning pipeline. 2542 characters per cap. Given a caption, generates followup questions and answers with a VLM. Compute P(sentence | image,prompt) - P(sentence|prompt). Sentences with low scores are only using their text prior, so filter them out.

UniVLA: VQA, world modeling, and robotic controls all-in-one transformer VQ-quantize images, DCT robotic actions, standard text tokens. Train the whole interleaved sequence with NTP. This type of approach would benefit from serious scaling. Anyone have a few thousand H100s laying around?

ST-Kit: new dataset + benchmark for kinematic understanding for VLMs. Examples: estimate the total distance covered by <object> in the video? In what direction does <object> move in BEV coordinates? Open source and closed source models do poorly but their fine-tune does well.

Vision-R1: another RLVR for VLMs, this time for detection. Rewards: formatting, precision, recall. Adds +9mAP to Qwen-2.5-VL on COCO + ODINW 🤯. They change the threshold in a curriculum (easy -> hard) which adds ~2 points. Repo has training code based on R1-V and TRL 👍

CAR-1000 Move over Stanford Cars, new dataset with 1000 hierarchical classes and 140,312 samples! Scraped from a Chinese car enthusiast forum, there's a wide variety of cars from around the world.

NuPlanQA: uses nuPlan annotations + GPT4o to 1 million QA pairs for training and an 8K multiple-choice benchmark. BEV-LLM: baseline model. Multi-view video -> BEVFusion -> cross-attend with image features -> projector -> LLaMA3.2. Some helpful ablations on # views and frames.

Hydra-NeXt: strong E2E closed-loop driving performance with only open-loop training Images -> encoder -> 4096 discrete trajectory vocab -> transformer -> bicycle model -> denoising diffusion refinement -> best trajectory selection. Much better closed-loop perf than UniAD, VAD

Nice methods paper (/ advertisement) from Nvidia on training video models with NeMo. Nuts and bolts: use Ray + NVDEC for curation, S3 + WebDataset for dataloading, FSDP + TP + Context Parallelism + PP for video DiT training. 48.2% MFU

ChatBEV: new dataset + benchmark for VQA on BEV maps for autonomous driving. Data from nuPlan. 116K train, 21K test. Render a BEV map + generate VQA with LLMs(?) Applications: VQA, use the model to condition a scene generator.

TrajHF: diffusion-based planner fine-tuned with RLHF. Simple idea: humans have preferences for candidate trajectories. Collect human feedback on key frames from 4.5K clips mined for aggressive maneuvers. Improves these aggressive scenarios but worsens some open-loop metrics.

DriveLMM-o1: new dataset + benchmark for autonomy VQA. Dataset is 18k QA pairs, each with step-by-step reasoning. Generated with GPT4o and human verified. Model is a fine-tuned InternVL2.5-8b. Nit: don't call your model o1 if you don't use RLVR!

SimLingo: Wayve paper on VLAs for self driving Front camera + prompts -> InternVL2-0.5B. They add GPS targets with an MLP. Output language, and decode queries for waypoints + speed. Very strong CARLA performance, but CARLA-only training data.

DeCapBench: new benchmark on super-detailed captioning. DCScore: new evaluation method for captioning, great correlation with human ratings. FeedQuill: VLM trained using PPO with a DScore-trained reward model. Great captioning specialist.

DriveTransformer: E2E driving with a single (complex) transformer, not modular like UniAD. Beats UniAD, VAD in open-loop but by much more in closed-loop in CARLA. It's also more robust to camera perturbations.

CLIPGrader: fine-tune CLIP to evaluate detection labels. Draw a magenta bbox on an image. Artificially perturb the box and make the caption "the magenta bounding box is a bad bounding box of a..." 91% accuracy in quality classification. Examples show lots of promise.

AlphaDrive: Trains a reasoning VLM to output multiple discrete action plans (accelerate, turn left) for autonomous driving. Much better than zero-shot or SFT on MetaAD, a new dataset of 110K 3s clips. In ablations, SFT < RL < SFT + RL. Looks like the days of pure SFT are over!