Physical AI models enable robots to autonomously perceive, interpret, reason, and interact with the real world. Accelerated computing and simulations are key to developing the next generation of robotics.
Physics plays a crucial role in robotic simulation, providing the foundation for accurate virtual representations of robot behavior and interactions within realistic environments. With these simulators, researchers and engineers can train, develop, test, and validate robotic control algorithms and prototype designs in a safe, accelerated, and cost-effective manner.
Robotic simulation is built upon the fundamental laws of physics, including the conservation of mass and momentum, rigid and soft body dynamics, contact and friction, and actuator modeling. These principles are used to predict how robots will behave in various scenarios and environments.
However, simulation often fails to match reality, a problem known as the “sim-to-real” gap. Robotics developers need a unified, scalable, and customizable solution to model real-world physics, handle diverse behaviors (including custom interactions with bespoke solvers), and support advanced features like differentiability.
Introducing Newton
Newton is an open-source, extensible physics engine being developed by NVIDIA, Google DeepMind, and Disney Research to advance robot learning and development.
Built on NVIDIA Warp, which enables robots to learn how to handle complex tasks with greater precision, Newton is compatible with learning frameworks such as MuJoCo Playground or NVIDIA Isaac Lab—an open-source, unified framework for robot learning.
The key aspects of Newton include:
Open source
Newton is open source, empowering the entire robotics community. This enables roboticists to use and distribute the framework freely and contribute cutting-edge research to its development.
NVIDIA accelerated
The foundation of Newton is built on NVIDIA Warp, an NVIDIA CUDA-X acceleration library, giving developers an easy way to write GPU-accelerated, kernel-based programs for simulation AI, robotics, and machine learning (ML). It provides a high-performance, flexible framework for building and running physics-based simulations, using the parallel processing power of NVIDIA GPUs.
Powered by MuJoCo-Warp
A key aspect of this initiative is compatibility with MuJuCo, short for Multi-Joint dynamics with Contact. It is a widely used open-source physics engine for robotics research and development, specifically involving complex dynamics and contact-rich environments. Developers can leverage existing models and code to save time and resources adapting applications for different physics engines.
For the first time, Google DeepMind has also introduced MuJoCo-Warp, an open-source robotics simulator accelerated by Warp. It enables developers to achieve significant performance gains, with more than a 70x acceleration for humanoid simulations and a 100x speedup for in-hand manipulation tasks. Mujoco-Warp will be available as a key physics engine in Newton and through MuJoCo-Warp. Learn more about Mujoco-Warp.
Differentiable physics
The ability to propagate gradients through simulation opens up new possibilities for robotic simulation and learning. Differentiable simulators can generate forward-mode results and compute the reverse-mode gradients of simulation results for back-propagation for optimizing system parameters.
Extensible
As the scope of modern robotics increases, so do the scenarios to simulate. Newton is highly extensible, enabling rich multiphysics simulations where robots interact with food items, cloth, and other deformable objects through custom solvers, integrators, and numerical methods.
Newton also enables different types of solvers to be coupled together, such as in this example of a material point method (MPM) solver with sand being coupled to rigid body dynamics.
Built on OpenUSD
Newton uses the OpenUSD framework. OpenUSD’s flexible data model and composition engine aggregates data needed for describing robots and their surrounding environments, while custom solvers and other runtimes can specialize on targeted robot capabilities and environments.
Disney Research will be the first to use Newton to advance its robotic character platform that powers next-generation entertainment robots, such as the expressive Star Wars-inspired BDX droids that joined Huang in his GTC keynote on stage.
“The BDX droids are just the beginning. We’re committed to bringing more characters to life in ways the world hasn’t seen before, and this collaboration with Disney Research, NVIDIA, and Google DeepMind is a key part of that vision,” said Kyle Laughlin, SVP, Walt Disney Imagineering Research and Development.
“This collaboration will allow us to create a new generation of robotic characters that are more expressive and engaging than ever before—and connect with our guests in ways that only Disney can.”
In addition to Newton, Disney Research, Google DeepMind, Intrinsic, and NVIDIA are collaborating on defining an OpenUSD asset structure for robotics. This new structure and data pipeline uses best practices within OpenUSD to unify robotic workflows and provide a common language for all data sources.
Stay tuned
Newton will pave the way for advancing the next generation of humanoid robots. The development for this effort is underway and we aim to make the first version available later this year.
In the meantime, learn more about the technologies behind Newton:
- NVIDIA Warp is a developer framework for building and accelerating data generation and spatial computing in Python.
- NVIDIA Isaac Lab, developed on NVIDIA Isaac Sim, is an open-source, unified framework for robot learning designed to help train robot policies.
Get started
Build on your foundational knowledge with free OpenUSD sessions, and Learn OpenUSD and Robotics Fundamentals courses.
