This is how 3DGS is getting mainstream

Douglas Reyes-Ceron@dreyesceron.bsky.social · 6mo ago

At 9:18, 3rd period of the Finland-Switzerland women's ice hockey QF game, the ice crew came to clean the ice during a pause What follows is the funniest use of the slo-mo alternate angle tech I've seen them use during these Olympics of one of the crew members falling

"No Pose at All Self-Supervised Pose-Free 3DGS from Sparse Views" TLDR: 3DGS + no poses during training/inference; shared feature extraction backbone; simultaneous prediction of 3D Gaussian primitives+camera poses in a canonical space from unposed (1 feed-forward step).

"Any-to-Bokeh: One-Step Video Bokeh via Multi-Plane Image Guided Diffusion" 📖TL;DR: Any-to-Bokeh is a novel one-step video bokeh framework that converts arbitrary input videos into temporally coherent, depth-aware bokeh effects.

"QUEEN: QUantized Efficient ENcoding of Dynamic Gaussians for Streaming Free-viewpoint Videos" TL;DR: Streamable free-viewpoint videos efficient representations for with dynamic Gaussians. Reduce model size to just 0.7 MB per frame while training in < 5s and rendering at 350 FPS

"STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenes" TL;DR: Data driven transformer in a feed forward manner; dense reconstruction in dynamic environment with 3D gaussians and velocities; self-supervised scene flows

St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World TL;DR: a feed-forward; (reconstructs+tracks dynamic video content); dust3r-like pointmaps for a pair of frames captured at different moments (1/2)

FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Views TL;DR: feed-forward model; cascaded learning paradigm with camera pose serving as the critical bridge, recognizing its essential role in mapping 3D structures onto 2D image planes.

⚡️Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass TL;DR: multi-view generalization to DUSt3R; processing many views in parallel: Transformer-based architecture forwards N images in a single forward pass, bypassing the need for iterative alignment.

🪄 VACE: All-in-One Video Creation and Editing from @alibabagroup.bsky.social's Tongyi Lab with: Zeyinzi Jiang* Zhen Han* Chaojie Mao*† Jingfeng Zhang Yulin Pan Yu Liu *Equal contribution, †Project lead

Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models TL;DR: single-step diffusion models; a single-step image diffusion model trained to enhance and remove artifacts in rendered novel views caused by underconstrained regions of the 3D representation.

A Distractor-Aware Memory (DAM) for Visual Object Tracking with SAM2 TL;DR: SAM2.1 based; distractor-distilled (DiDi) dataset to better study the distractor problem

CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image TL;DR: object-level 2D segmentation+relative depth; GPT-based model to analyze inter-object spatial relationships; occlusion-aware large-scale 3D generation model

Are diffusion models falling for optical illusion? "The Art of Deception: Color Visual Illusions and Diffusion Models" TL;DR: Diffusion models exhibit human-like perceptual shifts in brightness and color within their latent space.

Does 3D Gaussian Splatting Need Accurate Volumetric Rendering? TL;DR: While more accurate volumetric rendering can help for low numbers of primitives, efficient optimization + large number of Gaussians allows 3DGS to outperform volumetric rendering despite its approximations

The NeRF-life vengeance? "Sparse Voxels Rasterization: Real-time High-fidelity Radiance Field Rendering"

Self-Calibrating Gaussian Splatting for Large Field of View Reconstruction TL;DR: Self calibration + cubemap-based resampling strategy to support large FOV images

Animate Anyone 2: High-Fidelity Character Image Animation with Environment Affordance TL;DR: motion from source video + capture environmental representations as conditional inputs. Shape-agnostic mask strategy for character/environment relationship .

Pippo : High-Resolution Multi-View Humans from a Single Image TL;DR: 1K Multiview Diffusion Transformer pre-trained on 3B Human images without captions; post-trained on 2.5K studio captures with pixel-aligned control via ControlMLP; generates > 5x views at inference

Since 2024, it's crazy how competitive the field of generative video is. Here is another player but open source this time! Hong Kong University and ByteDance present "Goku: Flow Based Video Generative Foundation Models"

📜 Fillerbuster: Multi-View Scene Completion for Casual Captures TL;DR: Unified framework for scene completion; joint models images and camera poses estimation to reconstruct missing parts of casually captured scenes. 1B-parameter diffusion model from scratch.

Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control TL;DR: Manipulating 3D tracking videos; link frames, significantly enhancing for temporal consistency of the generated videos; 3 days oftraining on 8 H800 GPUs using less than 10k videos

Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion TL;DR: diffusion-based; raymap conditioning to both augment visual features with spatial information from different viewpoints; multi-task generation of images and depth maps

DiffVSR Enhancing Real-World Video Super-Resolution with Diffusion Models for Advanced Visual Quality and Temporal Consistency TL;DR: multi-scale temporal attention module for spatial accuracy. Noise rescheduling mechanism & latent transition approach for temporal consistency

CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generation TL;DR: 360° panoramas using diffusion-based image models. cubemap representations + fine-tuning pretrained txt2img models, CubeDiff simplifies the panorama generation process, delivering high-quality, consistent panoramas.

BLADE: Single-view Body Mesh Learning through Accurate Depth Estimation TL;DR: fully perspective projection model without applying heuristics; depth, focal parameters, 3D pose, and 2D alignment estimation

Continuous 3D Perception Model with Persistent State TL;DR: An online 3D reasoning framework for various 3D tasks from only RGB inputs

4K4DGen: Panoramic 4D Generation at 4K Resolution TL;DR: Panoramic Denoiser that adapts generic 2D diffusion priors to animate consistently in 360 images; Dynamic Panoramic Lifting (preserving spatial and temporal consistency)

GPS as a Control Signal for Image Generation TL;DR: GPS tags in metadata for signal control for image generation; GPS and text; 3D models from 2D GPS-to-image models through score distillation sampling, using GPS conditioning to constrain the appearance of the reconstruction

Video Depth Anything: Consistent Depth Estimation for Super-Long Videos TL;DR: Long videos support; Depth Anything V2 with efficient spatial-temporal head. Temporal consistency loss -> depth gradient (no geometric priors)

HAC++: Towards 100X Compression of 3D Gaussian Splatting TL;DR: leverages relationships between unorganized anchors and a structured hash grid; mutual information for context modeling; intra-anchor contextual relationships; Adaptive quantization module