NitroGen: Nvidia’s AI That Could Transform Gaming and Beyond

A team of researchers from Nvidia and several American institutions has unveiled an artificial intelligence model called NitroGen, capable of learning to play video games through visual input. NitroGen is presented as a foundational model for visual-action learning tailored to universal gaming agents, trained on 40,000 hours of gameplay videos from over 1,000 different games.

The model utilizes three key components: 1) an internet-scale video-action dataset, created by automatically extracting player actions from publicly available game videos, 2) a multi-game testing environment that allows for measuring game-to-game generalization, and 3) a unified visual-action learning policy trained using large-scale behavior cloning. NitroGen has demonstrated high efficiency in various domains, including combat in 3D action games, precise control in 2D platformers, and exploration in procedurally generated worlds.

Remarkably, it effectively generalizes to previously unseen games, achieving up to a 52% relative improvement in task success metrics compared to models trained from scratch. We are releasing the dataset, toolkit, and model weights to advance research in universal embodied agents.

While the training and demonstration are directly linked to video games, both technically and conceptually, this solution is far from being just about games. Firstly, NitroGen is based on the GROOT N1.5 architecture, initially developed for robotics. Secondly, the training principles are well-suited for teaching robots as a novel method of learning.

NitroGen has been adapted to games with completely different mechanics and physics, underscoring the model’s flexibility and numerous use cases. The authors note that videos where gamers overlay their gamepad actions in real-time during streaming proved particularly valuable. All NitroGen research is currently publicly available. As Nvidia continues to lead in AI research, this model could see adaptations in sectors such as robotics, autonomous vehicles, and other areas requiring visual-action decision-making.