Meta outlined an engineering system designed to enhance ads recommendations by utilizing idle inference capacity across its fleet. The system employs two primary elasticity levers to adapt workloads dynamically without compromising baseline performance.
The Inference Capacity Paradox
Meta stated that inference fleets often experience a capacity paradox where they remain globally constrained while local pockets of capacity stay idle. These inefficiencies can arise from peak-provisioning mismatches, model-development timing gaps, regional resource-ratio differences, and accelerator-type variations.
To capture this volatile resource safely, Meta designed a shared opportunistic capacity pool. Workloads utilizing this pool require a high degree of elasticity, sustained quality at scale, and limited cross-service dependencies.
Elasticity Levers and Isolation
Meta uses two complementary mechanisms to consume idle capacity: Rank More Ads (RMA) and Serve Better Models (SBM). RMA increases the number of candidate ads evaluated during ranking stages, requiring both CPU and accelerator surplus, while SBM adjusts the share of requests routed to a more powerful model using accelerator-only surplus.
The system protects baseline serving traffic through different isolation methods. SBM uses physical isolation with separate routing targets, whereas RMA applies logical isolation by setting a global budget for incremental work across shared model replicas.
Feedback Loops and Load Shedding
RMA and SBM are regulated by closed feedback loops using central controllers and client-side enforcement based on latency and request values. Meta also implemented multi-layer value-aware load shedding on both servers and clients.
These shedding mechanisms react within milliseconds to remove lower-value opportunistic work if capacity is rapidly reclaimed before central control loops can converge.
Key facts and where they come from
- Meta built a system to enhance ads recommendations by leveraging idle inference capacity in its fleet.
At Meta, we’ve built a system that enhances ads recommendations by leveraging idle inference capacity in our fleet.
- Rank More Ads (RMA) adjusts candidate volume, while Serve Better Models (SBM) adjusts requests routed to powerful models.
Rank More Ads (RMA) adjusts the number of candidates evaluated by the ranking stages.
- SBM uses physical isolation to separate baseline and more powerful models into different routing targets.
For SBM, the isolation is physical.
