MangoBoost has announced a new industry benchmark with its Mango LLMBoost™ Artificial Intelligence (AI) Enterprise MLOps software, achieving record results in the MLPerf Inference v5.0 submission. This performance was recorded on AMD Instinct™ MI300X GPUs, specifically for the Llama2-70B in the offline inference category.

This achievement represents the first multi-node MLPerf inference result on AMD MI300X GPUs. Utilizing 32 MI300X GPUs across four server nodes, Mango LLMBoost™ surpassed previous results, including those achieved by competitors using NVIDIA H100 GPUs.

MangoBoost’s submission indicates a 24% performance advantage over the best-published result from Juniper Networks, which utilized 32 NVIDIA H100 GPUs. The Mango LLMBoost™ achieved 103,182 tokens per second (TPS) in the offline scenario and 93,039 TPS in the server scenario on the AMD MI300X GPUs, exceeding the prior best of 82,749 TPS recorded with NVIDIA H100 GPUs.

In addition to achieving higher performance, the Mango LLMBoost™ solution provides cost benefits. The AMD MI300X GPUs are priced between $15,000 and $17,000, significantly lower than the $32,000 to $40,000 pricing for NVIDIA H100 GPUs. As a result, the Mango LLMBoost™ solution can deliver up to 62% cost savings while maintaining high inference throughput.

The Mango LLMBoost™ + MI300X system also offers approximately 2.8 times more inference throughput per $1,000 invested compared to the NVIDIA H100-based system.

Furthermore, Mango LLMBoost™ provides scalable, hardware-flexible AI inference software, supporting over 50 open models with one-line deployment via Docker and built-in OpenAI-compatible APIs. It is available on cloud platforms like AWS Marketplace, Microsoft Azure Marketplace, and Google Cloud Platform, as well as for on-premises deployments to support enterprise requirements for control and security.

The performance gains were achieved in collaboration with AMD, utilizing the ROCm software stack which optimizes MI300X GPU performance. The Mango LLMBoost™ software has been thoroughly tested across various cloud and on-premises configurations, showcasing high efficiency in different environments.

MangoBoost also provides hardware acceleration solutions, including Mango GPUBoost™, Mango NetworkBoost™, and Mango StorageBoost™, to enhance AI infrastructure capabilities.