Today, we are reviewing the SOPHGO SC7 HP75, a tensor accelerator that may seem like just another TPU for AI tasks from a lesser-known company aiming to challenge NVIDIA in the AI accelerator market. It seems like everything is as usual: a lot of ambition, but most likely as usual without a full-fledged ecosystem, documentation and support for popular frameworks. But is it so? Let’s find out.
TPU and why do we need such powerful video decoders?
Tensor Processing Units (TPUs) and Neural Processing Units (NPUs) are the next step in the development of specialized computing devices for artificial intelligence tasks. These processors are designed to meet the specific requirements of machine learning algorithms and deep neural networks.
TPU is optimized for working with tensors, which are multidimensional arrays of data that form the basis of most modern deep learning models. The main advantage of TPU is its use of matrix multipliers (MXUs), which perform matrix and vector multiplication operations at incredible speeds. This makes it an effective solution for training and inference of large models, such as language models or image recognition systems, which require intensive matrix calculations.
On the other hand, NPU is a more flexible solution that combines the advantages of TPU with additional capabilities for working with different types of neural networks. NPU often contains specialized blocks for performing convolution, activation, and pooling operations, which are essential components of convolutional neural networks (CNNs) used in computer vision tasks. Additionally, NPU is optimized for working with different data precision, allowing for a balance between performance and energy efficiency.
The key difference between TPU and NPU and GPU is their ability to perform specialized neural network tasks more efficiently. While GPUs remain versatile computing devices capable of handling a wide range of parallel computations, TPU and NPU offer unparalleled performance and energy efficiency in specialized AI tasks.
In the case of the SC7 HP75, in addition to being a TPU instead of the usual GPU, its video decoders stand out, which support H.264 and H.265 codecs and can process up to 2400 frames per second at 1080p resolution. This is crucial for those working with multiple video streams simultaneously, from facial recognition security systems to behavior analysis in smart cities. Simple processing power without the ability to handle a large amount of video stream makes little sense, as otherwise the video decoders would become a bottleneck for the entire accelerator.
SC7 HP75 Features
Now let’s move on to what makes the SC7 HP75 so attractive in neural networks. It is built on a 24-core 2.3 GHz ARM A53 processor, providing up to 169280 DMIPS of processing power. In computation, the SC7 HP75 achieves up to 96 TOPS in INT8, 48 TFLOPS in FP16/BF16, and up to 6 TFLOPS in FP32. This makes it suitable for intensive AI tasks, including training and inference of large models. The SC7 HP75 has a modest 75-watt TDP, so even passive cooling can handle heat dissipation. The ARM architecture ensures high energy efficiency. It features 48 GB of LPDDR4x memory with a bandwidth of 205 GB/s, providing fast data processing capabilities. It uses a PCIe Gen3 x16 interface with support for PCIe Gen3 x8.
Special attention is paid to video processing: decoding H.264 and H.265 videos up to 2400 frames per second at 1080P resolution, as well as supporting decoding of videos in 8K, 4K, 1080P, 720P, and lower resolutions. Video encoding capabilities include 900 frames per second at 1080P with support for 4K and 1080P, making this accelerator optimal for handling high-volume video streams in security systems and smart cities. JPEG image encoding can achieve up to 1200 images per second at 1080P resolution, with a maximum resolution of up to 32768×32768 pixels. It supports popular AI frameworks such as TensorFlow, PyTorch, Caffe, MXNet, and ONNX, as well as Linux-based operating systems.
ARM architecture: energy efficiency and speed
The SC7 HP75 is based on a 24-core ARM A53 with a 2.3 GHz clock speed. ARM architecture has long been established as an efficient and energy-saving solution, and in the case of the SC7 HP75, this plays an important role. The faster and more efficiently data can be processed, the more tasks can be completed in a given amount of time, which is especially important when working with video and inference in real time.
In addition, the ARM architecture allows the SC7 HP75 to outperform competitors such as the Nvidia T4, especially in tasks that require fast response times and minimal latency for video stream analysis and object recognition.
Comparison with Nvidia T4
When it comes to performance, the SC7 HP75 definitely challenges the Nvidia T4. In tests on deep learning models such as ResNet-50, the SC7 HP75 performs well with 7082 ops/s compared to 6285 for the T4. In object detection tasks such as SSD-Large, SC7 HP75 is again ahead with a score of 149 operations compared to the competitor’s 142, as well as 214 for SC7 HP75 with BERT-LARGE and 213 for T4.
Testing results
Testing of SOPHGO SC7 HP75 showed its high efficiency in neural network inference tasks. Compared to a GPU-based server (nVidia A16-16) on the YOLOv5s model, the TPU accelerator managed to process a 30-second video in 6.2 seconds, while the GPU took 7.8 seconds. The difference in object detection was minimal (<0.04%), which confirms the accuracy and stability of the TPU.
Processing audio files using Whisper Medium on two SOPHGO SC7+ (6 TPU) systems resulted in the following: 12 audio files of 883 seconds each (totaling 10,596 seconds) were processed in 1,326 seconds, resulting in a speed of 7.99 seconds of audio per second of real-time processing. The average processing time for a single request was 661 seconds. This proves that SOPHGO’s TPU can effectively handle real-time inference tasks, particularly in scenarios involving streaming data processing, computer vision, and ASR (automatic speech recognition).
The main thing is the software
Of course, tests and numbers are great, but after all, hardware without software is just an expensive piece of metal. And here the real challenges begin. It doesn’t really matter how good the “hardware” is, if the support from the software and compatibility with popular frameworks leaves much to be desired. After all, in the end, it all comes down to how this power will be implemented in practice, in real tasks, and not in laboratory tests.
Nvidia has spent decades honing the CUDA ecosystem, investing vast resources in developing APIs, drivers, tools, and documentation to make it easy for every developer to incorporate hardware acceleration into their work. Take any popular neural network framework, whether it’s TensorFlow, PyTorch, or Caffe, and you’ll find that most of them primarily support NVIDIA.
SOPHGO, understanding the importance of software support, has made sure that the SC7 HP75 is as compatible as possible with leading frameworks. Working with TensorFlow or PyTorch? No problem, just connect. Caffe or MXNet? Again, no obstacles. In addition, SophonSDK offers ready-made tools for quickly migrating existing models to the SC7 HP75. And these are not just pretty words on paper — it’s real software that will save you from headaches when integrating new hardware.
Get a consultation and find out the cost of renting cloud servers with GPUs
Conclusion
SOPHGO SC7 HP75 is not just another tensor accelerator. The company has managed to create a product that not only offers high computing power and can compete with NVIDIA in benchmarks, but also demonstrates outstanding compatibility with popular frameworks such as TensorFlow, PyTorch, Caffe, and MXNet. The support for these tools, along with detailed documentation and the SophonSDK, makes integrating this accelerator into existing infrastructures simple and convenient.
If you already need a ready-made solution for working with neural networks, our ITGLOBAL.COM – AI Cloud platform can fully meet your needs. In addition to the SC7 HP75, the platform also offers accelerators such as the L40S and H100 from previous reviews. For those who want a fully customized solution, ITGLOBAL.COM can also act as a system integrator, providing support from hardware delivery to the design and maintenance of the entire infrastructure from scratch, depending on your requirements.
