Arm renamed its 4-TOPS Arm ML NPU as the Ethos-N77 and launched small-footprint, low-power Ethos-N57 (2-TOPS) and Ethos-N37 (1-TOPS) models for edge AI supported with the Linux-based Arm NN SDK. Arm also unveiled a Mali-G57 GPU and a tiny Mali-D37 VPU.

Tiny, stripped-down AI co-processors for the edge seem to be a thing these days. Arm’s new power-efficient Ethos-N57 (2-TOPS) and Ethos-N37 (1 TOPS) neural processing units (NPUs) may not be as minimalist as Kneron’s KL520 AI SoC, available on Aaeon’s AI Edge Computing Modules, which delivers 0.3 TOPS NPU performance on only half a Watt. Yet they offer lower-power embedded and mobile alternatives to Arm’s newly renamed, 4-TOPS Ethos-N77, formerly known as the Arm Machine Learning (ML). The NPUs are supported via the Linux-based Arm NN SDK (see farther below).

Arm Ethos product line
(click image to enlarge)

In addition to unveiling the Ethos-N57 and Ethos-N37, Arm announced a Mali-G57 GPU for mid-range smartphones based on the same Valhall architecture used on its high-end Mali-G77 GPU. The Mali-G57 provides improved graphics and immersive gaming experiences and offers improved performance for ML workloads, says Softbank-owned Arm.

Arm also announced a tiny Mali-D37 VPU designed for low-end mobile and embedded devices. Based on the same Komedia architecture used on the 4K-ready Mali-D71, the entry-level Mali-D37 supports up to QHD+ resolution (1440 x 2880) on a smartphone. The display processor measures less than a square millimeter on 16nm technology, which is about a third the size of the Mali-D71.

The news follows Arm’s recent announcement of a Custom Instructions extension to its Armv8-M architecture for Cortex-M micro-controllers. Custom Instructions enables Arm’s customers to design MCUs with their own custom instructions. Along with Arm’s earlier Arm Flexible Access licensing program, Custom Instructions is widely seen as a defensive move against the rising competition from open source RISC-V processors.

Here we take a closer look at the new Ethos NPUs and examine some of the other new edge AI solutions that have emerged this year, most of which have been developed by Arm licensees. In other edge AI news, Google has moved its AI-enabled Coral Dev Board out of beta (see farther below).

 
Ethos-N57 and -N37

The Ethos-N57 and Ethos-N37 are “optimized for the most cost and battery life-sensitive designs,” says Arm. The NPUs support Int8 and Int16 datatypes and offer performance enhancement techniques such as Winograd. They also provide “advanced data management techniques minimizing data movement and associated power.”

The Ethos-N57 features 8x compute engines and supports up to 2-TOPS AI performance using 1024 8-Bit MACs. It’s designed for smart home hubs, mainstream smartphones, and digital TVs.

Arm Ethos (left) and earlier Arm ML (Ethos-N77) block diagrams
(click images to enlarge)

The Ethos-N37 measures only one square millimeter. It offers 4x compute engines for up to 1-TOPS AI performance using 512 8-bit MACs. The Ethos-N37 is intended for entry-level phones and smart devices such as smart cameras.

On the high end, the existing, up to 4-TOPS Ethos-N77 (Arm ML) has a much larger 1-4MB memory footprint than the 512KB-ready Ethos-N57 and -N37. It targets premium smartphones, AR/VR, and computational photography applications.

The Ethos-N57 and -N37 deliver “cost-effective ML for mainstream SoCs with much tighter area budgets,” says Arm. Their compression technology minimizes system bandwidth by 1.5x to 3x “with lossless compression for weights and activations using clustering, sparsity and workload tiling,” says the company. As a result, chipmakers do not need to make “major modification to the memory structure” to integrate the NPUs.

Up to 8x of either NPU can be clustered on a SoC to process multiple networks in parallel or a single, large network split across cores. The NPUs also support up to 64-core configurations via Arm CoreLink mesh technology. Both NPUs support TrustZone system security, as well as neural frameworks including TensorFlow, TensorFlow Lite, Caffe2, PyTorch, MXNet, and ONNX.

The Ethos NPUs can be programmed with Linaro’s Arm NN an open source Linux SDK designed to “enable machine learning workloads on power-efficient devices,” says Arm. Arm NN is said to “bridge the gap between existing NN frameworks and the underlying CPU, GPU, and NPU IP.”

Arm NN provides an abstraction layer that reduces the challenges of programming multiple, heterogeneous processors. This “allows workloads to be run across devices like phones, TVs, and throughout the smart home, with minimal effort.” The Ethos NPUs also support Google’s AndroidNN.

An Oct. 20 VentureBeat interview with Steve Roddy, VP of Arm’s ML group, quotes Roddy as saying the IP for new Ethos NPUs has been released to semiconductor customers, but won’t appear in silicon until late 2020. The story, which refers only generally to the new NPUs, offers an interesting argument for low-power AI for phones, consumer electronics, and other edge devices.

Jetson Nano Dev Kit

 
Arm Ethos competes with edge AI chips from its own customers

Almost every major chipmaker have an AI chip on the market or under development, many of which target phones, consumer electronics, and other embedded edge devices. Aside from Intel’s Movidius Myriad X technology, most of the competition comes from Arm licensees targeting the edge.

The most established edge AI player is Nvidia with its Arm-based Jetson modules equipped with CUDA-enabled GPUs that support AI acceleration. It’s most recent Jetson Nano module is designed for resource constrained embedded devices while more powerful Jetson Modules include the Jetson TX2 and high-end Jetson AGX Xavier.

Xilinx has announced an Xilinx AI platform that will be integrated in its soon-to-ship, Linux-powered 7nm Versal processors. Xilinx AI’s Deephi “sparse neural network” Core technology features a CNN pruning technology and deep compression algorithm to reduce the size of AI algorithms for edge applications.

Xilinx AI architecture (left) and Rockchip RK1808 block diagram
(click images to enlarge)

Other Arm customers diving into the AI waters on their own include Samsung, which is developing an NPU technology it calls On-Device AI. MediaTek, meanwhile, will soon ship an octa-core -A73 and -A53 MediaTek i500, which has a 500MHz AI processor. There’ also an upcoming, higher-end MediaTek i700 (AI IoT platform i700) equipped with an NPU.

Rockchip, which has baked a 3-TOPS NPU into its RK3399Pro, has also introduced NPU-enabled SoCs such as the new, dual-core, Cortex-A35 RK1808 and upcoming, 8nm RK3358, which will feature 4x Cortex-A76 and 4x -A55 cores. The RK1808 targets the same low-power edge AI applications targeted by Arm’s new Ethos chips.

On the very low end, there are solutions such as the MCU/Arduino oriented, RISC-V-based GAP8 AI processor for battery operated devices. Meanwhile, DMP offers an ultra low-power ZIA DV700 NPU.

Coral Dev Board

 
Coral Dev Board moves out of beta

Google collaborated with Arm on its Coral Edge TPU version of its Tensor Processing Unit AI chip, which is built into its Linux-driven, NXP i.MX8M-based Coral Dev Board. The Edge TPU will also soon appear on Asus’ Tinker Edge T and industrial CR1S-CM-A variants of the Coral Dev Board.

As reported today by ZDNet, the $150, Raspberry Pi-like Coral Dev Board has just moved out of beta. The Coral SOM that powers the sandwich-style board is now available for $114 on its own.

Google has just updated its Coral website with additional documentation and early case studies. In addition, Google will soon release a new version of Mendel OS, a lightweight version of Debian Buster designed for Coral Dev Board and the Coral Edge TPU.

 
Further information

Arm’s Ethos-N57 and Ethos-N37 IP designs are now available to customers and should appear on silicon in late 2020. More information may be found in Arm’s umbrella announcement, including the Mali-G57 GPU and a tiny Mali-D37 VPU, as well as the Ethos-N57 and -N37 blog announcement and the Ethos product page.