Alibaba Cloud claims it has optimized how GPUs are used through a "pooling" system that reduces the needed chips by up to 82%.

The Chinese cloud company shared a peer-reviewed paper at the 2025 ACM Symposium on Operating Systems (SOSP) in Seoul, explaining how its GPU pooling system - dubbed Aegaeon - can reduce the amount of GPUs needed when serving "sporadic and unpredictable requests in concurrent inference workloads with dedicated GPU instances."

The paper states that, while existing multi-model serving solutions use "GPU pooling and serverless computing to improve resource efficiency," they are only effective up to two or three models per GPU.

Aegaeon, however, is described as a "multi-model serving system that performs model auto-scaling at the token granularity to achieve effective GPU pooling." According to the paper - conducted by Alibaba Group and the School of Computer Science, Peking University - Aegaeon can schedule multi-model requests and make auto-scaling decisions on a per-token basis.

Alibaba Cloud has beta-deployed the solution in its model marketplace, where it is serving tens of models. The company claims it has been able to reduce the number of GPUs needed from 1,192 to 213, an 82 percent reduction.

The paper notes that its model studio currently has a long tail of models that are infrequently invoked (more than 90%, and thus reserving instances for these models means that they are holding 17.7% of their GPUs to serve only 1.35%.

The current approaches to "pooling," the paper says, include multiplexing and autoscaling. Multiplexing places multiple model instances on each GPU, but is limited to two or three models due to "inadequate GPU memory capacity." Autoscaling, meanwhile, the paper theorizes, shows potential for more "aggressive GPU pooling" by "adapting model placement over time and scaling model instances from off-device storage." Despite this, the paper finds that autoscaling is still bounded by the ratio of active models in the workload.

Aegaeon instead uses a token-level autoscaling approach. "In essence, by preemptively scaling down active models and scaling up pending models for newly arrived requests in an SLO-aware manner, Aegaeon alleviates HOL blocking and achieves truly effective GPU pooling, supporting up to seven models per GPU."

According to the paper, the system was tested over several months, and on models with up to 72 billion parameters. Throughout, the improvement varied between 1.5 times to nine times better output.

The testbed consisted of two nodes with 16 GPUs in total, where each node had eight Nvidia H800 80GB GPUs connected via NVLink, 2TB of DDR5 memory, and 192 Intel Xeon Platinum 8469C CPUs.

The paper doesn't note the network fabric used for the test, but a report from Tom's Hardware said that Alibaba Cloud is known to offer "its own eRDMA elastic RDMA network and has a record of building highly‑integrated GPU serving stacks."

While an interesting exploration in optimizing the use of GPUs, the paper has not shaken the industry like the ripple caused by DeepSeek's claims that its V3 model was built on a training run that cost just $5.6 million towards the start of the year.

As noted by a report from The Register, the US hyperscalers do not typically share the "secret sauce" used to optimize their own fleets of GPUs, and may have similar solutions already deployed.