[Updated Nov. 13] — Nvidia unveiled a “Drive PX Pegasus” computer for Level 5 self-driving cars that runs Linux on up to 4x octa-core “Xavier” SoCs and a 640-core Volta GPU.

At the GPU Technology Conference in Munich, Nvidia founder and CEO Jensen Huang announced a more powerful version of the chip designer’s Linux-based Drive PX platform for autonomous cars. Drive PX Pegasus is the first system to promise a new class of fully autonomous Level 5 driverless vehicles “without steering wheels, pedals or mirrors.” The compact Drive PX Pegasus computer can process over 320 trillion operations per second, or 10 times the power of last year’s Drive PX 2, claims Nvidia.

Drive PX Pegasus
(click image to enlarge)

Drive PX Pegasus integrates up to four of Nvidia’s new octa-core Xavier SoCs, with integrated Volta GPUs, and adds two more discrete GPUs (see farther below). The computer fuses inputs from high-resolution, 360-degree surround cameras, radars, and LIDARs to perceive a car’s surroundings and localize it within centimeter accuracy. Pegasus features multiple levels of redundancy, helping it reliably track vehicles and people around the car, and then plan a safe path.

German mail and logistics giant Deutsche Post DHL Group (DPDHL) and automotive supplier ZF plan to deploy a test fleet of autonomous delivery trucks based on Pegasus starting in 2018. Deutsche Post DHL will outfit its electric light trucks with ZF’s Pegasus based ProAI self-driving system. DPDHL showed off a Pegasus powered prototype delivery vehicle with six cameras, two LIDARs, and one radar.

At least 25 partners are already working on Level 5 compliant “robotaxis” using Pegasus, with most of them initially slated for use on campuses, airports, and other relatively controlled environments. The “license plate” sized Pegasus system is said to replace a Drive PX 2 system that takes up most of a car’s storage trunk, enabling smaller self-driving vehicles or robotaxis with more passenger and cargo room.

Nvidia’s shift to the automotive market from mobile GPUs has been one of the more dramatic transitions in the tech world in recent years, and its decision to leapfrog in-vehicle infotainment (IVI) to develop autonomous car computers has been a bigger gamble. The Level 5 claim is even more audacious, but its large partner list suggests there’s something to it.

Nvidia CEO Jensen Huang showing off Holodeck demo
(click image to enlarge)

At the Munich event, Huang also announced new details on another equally futuristic, cutting edge technology: the previously tipped, Holodeck VR application. Holodeck will enable designers of cars and other products to remotely collaborate in photorealistic virtual reality environments.

Huang also showed off an AI-enabled “Nvidia Isaac” robot trained with Nvidia’s Project Isaac robot simulator, as well as a “Vincent” AI technology from Cambridge Consultants. Vincent, which has a neural network trained on an Nvidia DGX AI supercomputer, can turn quick sketches into the style of master artists.

 
Inside Drive PX Pegasus

Drive PX Pegasus can scale up from a single-SoC solution that provides ADAS-like, Level 2+/Level 3 “AutoCruise” assisted driving, all the way to the quad-SoC Level 5 autonomous configuration. In between, it supports an AutoChauffeur mode for automated point-to-point travel. Nvidia already offers Drive PX Parker AutoCruise and Drive PX Parker AutoChauffeur boards based on the Tegra-driven Drive PX 2 technology.

Object mapping in Pegasus
(click image to enlarge)

In the Level 5 mode, Pegasus integrates deep learning, sensor fusion, and surround vision in real-time. Designed for the highest level ASIL D safety certification, the computer leverages its Deep Neural Networks (DNN) technology to instantly analyze fused sensor data and detect and classify objects.

While the previous Drive PX 2 used dual Tegra SoCs and discrete Pascal GPUs, Drive PX Pegasus is built on Nvidia’s AI-infused Drive PX Xavier SoC, which was announced last year and will be widely available in Q1 2018. Comprising 7 billion transistors, the octa-core Xavier SoC will deliver 30 TOPS (trillions of operations per second) of performance, while consuming only 30 watts of power, claims Nvidia.

Each of the up to four Xavier SoCs in a Pegasus system includes a GPU based on Nvidia Volta architecture. The 16nm FinFET fabricated Xavier SoC also includes a new computer vision accelerator. The Pegasus computer adds to Xavier by integrating two “next-generation” discrete GPUs “with hardware created for accelerating deep learning and computer vision algorithms.”

Xavier control board (left) and Volta GPU
(click images to enlarge)

The new Volta GPU integrates 640 tensor cores and 21 billion transistors, and is claimed to be the most powerful GPU in history. Volta supplies over 100 Teraflops per second (TFLOPS) of deep learning performance, claims Nvidia. This is said to be five times the GPU performance of Nvidia Pascal. The CUDA-enabled GPU also integrates a new NVLink high-speed interconnect technology claimed to offer twice the throughput of the previous generation.

Pegasus features a combined memory bandwidth that exceeds 1 terabyte per second. The system provides automotive inputs/outputs, including CAN and Flexray, as well as multiple 10-GbE Ethernet ports. There are also 16x dedicated high-speed sensor inputs for camera, radar, LIDAR, and ultrasonics.

 
Linux-based Drive IX AI agent and Nvidia Drive Platform

With the Pegasus launch, Nvidia is introducing a DNN-driven Drive IX voice agent and co-pilot for ADAS configurations that also appears to work with Drive PX 2. Drive IX tracks head movement and gaze, and monitors the Drive PX sensor system to alert the driver to potentially dangerous situations such as blind spot intrusions. This ultimate backseat driver can even interpret gestures and read lips, as well as provide advice based in part on “insights about the driver,” says Nvidia.

Drive IX can decide when Drive PX has the confidence to drive autonomously and when it should hand control over to the driver, perhaps due to bad weather or excessive pedestrian traffic. It can also judge when the driver is impaired or is experiencing drowsiness or road rage, and then recommend a switch to autonomous control. The choice of operation is ultimately up to the driver, although fleet controllers (and possibly insurance companies) will likely be able to decide to limit driver choice in favor of the perceived safety of autonomy.

While Drive PX 2 was available for Linux, Android, and QNX, Pegasus ships only with a Linux-based Nvidia Drive Platform Software, according to Nvidia’s announcement. This “open software architecture” features a Root File System optimized for Ubuntu 16.04 LTS, and is based on Linux 4.4.38 with real-time RT_PREEMPT (AKA PREEMPT_RT) patches. Per a LinuxGizmos reader comment, an EE Times post says Pegasus also “will run QNX.”

The Drive Platform supports 64-bit user space and runtime libraries, as well as the Nvidia CUDA 8.0 parallel computing platform. It also supports OpenGL 4.5/ES 3.2 and EGL 1.5 with EGLStream extensions.

Optional libraries include an Nvidia DriveWorks SDK with reference applications, tools, and library modules, plus a “run-time pipeline framework that goes from detection to localization to planning to visualization.” Other options include the Nvidia TensorRT neural network inference engine and the CUDA Deep Neural Network (cuDNN) library.

Developers can also port deep learning models developed on the server-based Nvidia DGX-1 platform to the Pegasus system. Nvidia-DGX-1 integration is also touted for Pegasus’ ability to deploy OTA updates.

Toyota Camry running AGL

One unexplained aspect of Pegasus is to what extent, if any, it controls IVI functions or integrates with third-party infotainment systems. It’s also unclear to what extent the software platform is “open.” The Linux-based Automotive Grade Linux (AGL) offers a fully open source automotive distribution, although it’s currently limited to IVI and cluster controls.

Other major self-driving car technologies include Alphabet’s Waymo. In September, Intel announced it was collaborating with Waymo. Intel is expected to bring its newly acquired Mobileye computer vision and AI technology to the table. Waymo is based on earlier Google self-driving car tech that ran on Linux. Tesla’s self-driving car technology is also based on Linux.

A number of car manufacturers and autonomous vehicle projects, such as the MIT spinoff NuTonomy, are building their platforms on Drive PX. In June, NuTonomy signed a deal with Lyft to bring a self-driving car sharing service to Boston.

“NuTonomy is building for Level 5 and Pegasus is the kind of platform that will be required to support these types of systems,” stated Karl Iagnemma, NuTonomy CEO and co-founder, in one of several testimonial quotes supporting Pegasus.

 
Further information

Nvidia’s Drive PX Pegasus automotive computer will be available in 2018. More information may be found in the Pegasus announcement and the Drive PX page.