Just over a month later, I have now joined Physical Intelligence full time! Wrote a bit about why I'm excited to research simulation at Pi stoneztao.substack.com/p/joining-ph...

Joining Physical Intelligence to Work on Simulation

What makes researching simulation at Pi exciting!

stoneztao.substack.com

Stone Tao@stonet2000.bsky.social · 2mo ago

just arrived in SF and excited to begin my research internship at physical intelligence working on simulation!

just arrived in SF and excited to begin my research internship at physical intelligence working on simulation!

I reverted my decision to hand off ManiSkill to someone else. A ManiSkill 4 is now in progress with lots of improvements (focusing on robot manipulation) If you planned to not use ManiSkill because it might not be maintained once I graduate, I hope this convinces you otherwise!

The ManiSkill project now has a new home under a new mani-skill github organization, moving out of the old UCSD lab org. Long overdue and done as part of my plans for longer term maintainance as well as more organized community driven development github.com/mani-skill

ManiSkill

Community for developing the Maniskill robotics simulation framework - ManiSkill

github.com

in Seattle for 2 weeks finishing up my work at AI2 (at their extremely beautiful office). lmk if you want to meet!

First blog post up on Robotics Simulation Infrastructure! I give a high-level overview, followed by an elementary example of better infrastructure for pose management. stoneztao.substack.com/p/robotics-s...

Robotics Simulation Infrastructure Part 1

First part of a blog series diving into the world of robotics simulation and machine learning

stoneztao.substack.com

Stone Tao@stonet2000.bsky.social · 4mo ago

would people be interested in a blog series on "simulation infrastructure" where I look at popular robotics sim tools (e.g. mjlab, isaac...) and explain the pros and cons of their design decisions. The series would cover what I believe to be key pillars of good, usable, sim code

Wasn't at GTC but a sneak peak of some of my internship work at NVIDIA was! (sim2real cable routing presented by my manager Yashraj) www.youtube.com/watch?v=d5wm...

AI Research Breakthroughs from NVIDIA Research (Hosted by Karoly of Two Minute Papers) | NVIDIA GTC

YouTube video by NVIDIA Developer

youtube.com

would people be interested in a blog series on "simulation infrastructure" where I look at popular robotics sim tools (e.g. mjlab, isaac...) and explain the pros and cons of their design decisions. The series would cover what I believe to be key pillars of good, usable, sim code

I will be in SF from November 10-17 🌉 If you work on something interesting and want to meet up let me know! Or if there’s a fun event i’d love some invites 😃 I’m also looking to visit robotics companies (especially startups!), if you have time lmk if I can visit!

1/1 neurips submission accepted! Will share more details later, covers an often overlooked aspect of on policy RL training when scaling to large scale parallelized environments

Rumors going around are that NeurIPS PCs had to reject ~400 papers because of venue constraints that were in the accept pile. Hard things like this happen but the PCs and conference should explicitly tell all the authors who had this happen to them. Or create something like accept w/o presentation.

Opensourcing a tool to calibrate camera extrinsics painlessly in a minute, no checkerboards! It's based on EasyHEC, using diff. rendering to optimize extrinsics given object meshes+poses. Crazy that even a piece of paper works too. Code: github.com/StoneT2000/s... (paper example in next post)

Big thanks to Jeremy Morgan (PhD @ USC) for helping massively accelerate the IK solvers in ManiSkill! Open source really helps grow this project (and i get to learn tons of new things like IK optimization) github.com/haosulab/Man...

[Feature] Faster IK based controllers using levenberg-marqurdt and align pd ee pos controllers in gpu/cpu sims by StoneT2000 · Pull Request #1213 · haosulab/ManiSkill

Cleaned up version of #1148 to close #955. Updates the original code to permit various other pd ee controller options with the new solver. LM solver is now the default. Users can modify the IK sol...

github.com

might be the first time I made it personally on the trending developers list on github! One of the contributing factors being the lerobot sim2real code

Combining sim to real with cheap hardware like the so-100 really is making cutting edge robot learning so accessible and its wonderful

Stone Tao@stonet2000.bsky.social · last yr.

Awesome to see people reproducing the accessible LeRobot zero-shot sim2real project! Trained for just 90 min in ManiSkill and deployed directly in real. Sim2real is not easy, but very rewarding when it works Original post by Jianwei Zhang on LinkedIn www.linkedin.com/posts/jianwe...

My latest post: The American DeepSeek Project Build fully open models in the US in the next two years to enable a flourishing, global scientific AI ecosystem to balance China's surge in open-source and an alternative to building products ontop of leading closed models. buff.ly/kvJQE3I

Awesome to see people reproducing the accessible LeRobot zero-shot sim2real project! Trained for just 90 min in ManiSkill and deployed directly in real. Sim2real is not easy, but very rewarding when it works Original post by Jianwei Zhang on LinkedIn www.linkedin.com/posts/jianwe...

Excited to announce that I will be interning at @nvidia research this summer on robotics/embodied AI! I’ll be in seattle for the summer, let me know if you want to meet up and chat! 🦾

Rest of my lab has various presentations at RSS2025, please check out their awesome work! June 25 EEB 248 @ 10am - Hardware Optimization for In-Hand Rotation by K. Fay

The sim2real demo had some mixed success, hampered primarily by the lighting conditions of the outdoors. At least it worked sometimes! Hindsight says that despite the weather, low-cost nature, only 1 hour of training, anything working is a miracle

Stone Tao@stonet2000.bsky.social · last yr.

I’ll be at #RSS2025 from June 21 to June 23! I’ll be giving a presentation on ManiSkill3 on June 21, 5:30 PM We will also have two live demo sessions, on June 21, 12:30-2:00PM and 6:30-8:00PM. Swing by to see live demos of zero shot RGB sim2real, cool sim demos, and VR teleop!

I’ll be at #RSS2025 from June 21 to June 23! I’ll be giving a presentation on ManiSkill3 on June 21, 5:30 PM We will also have two live demo sessions, on June 21, 12:30-2:00PM and 6:30-8:00PM. Swing by to see live demos of zero shot RGB sim2real, cool sim demos, and VR teleop!

Our computer vision textbook is now available for free online here: visionbook.mit.edu We are working on adding some interactive components like search and (beta) integration with LLMs. Hope this is useful and feel free to submit Github issues to help us improve the text!

Foundations of Computer Vision

The print version was published by

visionbook.mit.edu

it's happening.gif @stonet2000.bsky.social

Rendering nerds! Check out our latest work "Vector-Valued Monte Carlo Integration Using Ratio Control Variates" that has just gotten the best paper award at SIGGRAPH 2025. This paper presents a method that reduces variance of a wide range of rendering and diff. rendering tasks with negligible cost.

Code/tutorial released for🤗LeRobot zero-shot visual sim2real Reinforcement Learning! Train your SO100 robot in ManiSkill's fast simulator+renderer💨 and deploy a cube picking model zero-shot. Just in time for the LeRobot Hackathon, happy hacking! github.com/StoneT2000/l...

GitHub - StoneT2000/lerobot-sim2real: lerobot sim2real code

lerobot sim2real code. Contribute to StoneT2000/lerobot-sim2real development by creating an account on GitHub.

github.com

longer uncut video 17/18 successes increased the spawn area size and tuned it to be less occluded. Works well!

Stone Tao@stonet2000.bsky.social · last yr.

sneak peek of what Xander and I will show next week at RSS 2025 for the maniskill demo session: Zero shot visual sim2real (basic) manipulation, one camera and <1 hour of RL in sim with SO100 ~3 seconds to pick random color cubes, could be faster but goal is accessibility and low cost!

sneak peek of what Xander and I will show next week at RSS 2025 for the maniskill demo session: Zero shot visual sim2real (basic) manipulation, one camera and <1 hour of RL in sim with SO100 ~3 seconds to pick random color cubes, could be faster but goal is accessibility and low cost!

Is there any research on replay buffer sizes for off policy (sac) RL algorithms? How come we have just assumed that 1M replay buffer size is reasonable for the majority of tasks we test on (in robotics typically). Why not smaller? Can we make it smaller?

Very good thread on how we got into this whole stem education / research cutting situation and its impacts by @davidbau.bsky.social Grants that are otherwise perfectly fine if you remove the “dei part” are being cut without being put back into stem education/research?!?! x.com/davidbau/sta...

Nice to see the use of ManiSkill3 in this work! Simulation is not just useful for RL training. It provides some good cheap deterministic test beds, perfect for testing imitation learning scaling laws at scale. Years of data in hours