🔍 Read the full analysis: Pushing Transformers MoE Inference On 32GB RAM With A 180B-Parameter Model on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A Hugging Face developer has proposed an experimental Transformers feature that streams selected mixture-of-experts model weights from SSD into a bounded device cache. The reported OLMoE test reached 34.6 tokens per second, near the standard Transformers result of 38.1, but no completed benchmark is provided for a 180B model on a 32 GB Mac.
A Hugging Face developer has proposed an experimental feature for Transformers that loads selected mixture-of-experts (MoE) model weights from SSD into a bounded device cache, reporting 34.6 tokens per second on OLMoE against 38.1 tokens per second with the standard Transformers path, as described in the original analysis. The work targets models whose weights exceed available memory, but the source does not report a completed test of a 180B-parameter model on a 32 GB Mac.
MoE models route each token through a subset of their experts rather than using every expert for every step. The developer says experts account for more than 90% of the model weights, making it possible in principle to keep only selected experts in faster memory and retrieve others as needed. The proposed approach places experts in GPU memory, system RAM or disk according to available capacity, with selected experts held in a bounded cache.
The initial implementation ran experts one at a time in a Python loop. In the developer’s test, that version reached 10.1 tokens per second even with experts already cached, compared with about 32 for vanilla Transformers. The developer then used a shared, preallocated device tensor and a table mapping expert IDs to cache slots. This allowed the existing grouped matrix multiplication kernel to process selected experts together; the reported result rose to 34.6 tokens per second, versus 38.1 for the standard path.
The developer says the change was submitted to Transformers as pull request #50717. The source also points to a Hugging Face Space, tarekziade/moe-expert-streaming, with project code. It describes implementation work across Transformers, a model-specific package and Metal kernels, and says AI was used to build code. The developer characterizes their role as selecting experiments, reviewing profiles and deciding which changes to keep.
Streaming MoE Experts With Limited Memory
An experimental Transformers change streams selected experts from SSD into a bounded cache. The reported OLMoE test ran close to vanilla speed, but the 180B model on a 32 GB Mac remains untested in the source.
How the cache works
The router selects experts for each token; the experiment fetches those experts into a shared cache.
Choose a subset of experts for the token.
Stream them from storage into available device cache slots.
Use Transformers’ grouped matrix multiplication kernel.
What the test shows
On OLMoE-1B-7B, the shared cache kept the reported speed near the standard path.
- Python loop ran experts one at a time.
- Reached 10.1 tokens per second, even with experts cached.
- Preallocated tensor mapped expert IDs to cache slots.
- Reached 34.6 tokens per second versus 38.1 baseline.
The larger device test is open
The 180B model on a 32 GB Mac is a motivating target, not a measured result in the supplied source.
Promising engineering
The cache design brought the reported small-model test close to vanilla Transformers speed.
SSD performance
Storage traffic and cache misses could change speed when experts must be fetched repeatedly.
Measure the target setup
Report speed and memory use on the larger model, with storage, cache settings and workload specified.
A Smaller Memory Footprint for MoE
If the method works reliably when experts must actually be fetched from storage, it could let developers run some MoE models on hardware that cannot hold all model weights in memory. That could widen access to local inference on laptops and single-GPU systems, where memory capacity can prevent people from trying larger models even though only some experts are used for each token.
The reported speed comparison is encouraging for one limited point: the cache design did not impose a large slowdown in the tested OLMoE setup. But the result does not show how much time SSD retrieval adds when the active experts are absent from cache. Storage speed, data movement, cache capacity and how quickly routing choices change could all affect generation. The key question is whether the approach can reduce memory use without making those costs dominate.
Products Worth Considering

Sonos Roam (Gen 1) Portable Smart Speaker, White

Jackery Explorer 300 Portable Power Station, 292Wh

Samsung 32 Odyssey G5 QHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Expert Streaming Is Designed
The source also reports comparisons on Qwen3.6-35B-A3B using an A10G GPU with 24 GB of memory and ample system RAM. In that setup, it says Strata decoded at 82–88 tokens per second with Q8_0, while llama.cpp with Q8_0 and its fit option reached 63–64. Those figures belong to that particular model and hardware; they are not results for the experimental Transformers feature or a laptop.
The source distinguishes several approaches to placing MoE weights. It says llama.cpp can run experts on the CPU, while vLLM’s CPU offload reads weights over PCIe without a hot-expert cache. It describes Transformers with Accelerate as moving an entire layer’s expert bank before processing it. The proposed experiment instead aims to load only the experts selected by the router into a bounded cache and retain grouped expert computation. These descriptions and comparisons are reported by the source, not independently established here.
The 32 GB Mac Test Is Missing
The source frames running a 180B-parameter MoE model on a 32 GB Mac as a motivating target, but gives no measured result for that model and device. It also does not report the OLMoE test’s storage type, cache size, prompt conditions or repeated-run results, leaving readers without details needed to judge how broadly the speed figures apply.
It is also unclear whether pull request #50717 has been accepted, merged or included in a Transformers release. The source does not establish how the cache performs when routed experts change quickly, or provide a same-hardware, same-model comparison with Strata. The OLMoE result is a test of the cache design’s speed in the conditions reported; it is not evidence that SSD-backed inference for the 180B model will achieve similar performance.
Pull Request and Larger-Model Results
The next concrete update is the status of Transformers pull request #50717 and whether the experimental option becomes available in a published version. The linked Hugging Face Space offers a way to inspect the project code, according to the source material.
To evaluate the intended use case, future results would need to report generation speed and memory use for the target large MoE model on a 32 GB Mac, alongside the storage device, cache settings and workload. Repeated runs and a direct comparison with Strata under matching conditions would help show whether the approach remains fast when experts must be loaded from storage. Until then, the large-model scenario remains a goal rather than a demonstrated benchmark.
Key Questions
What did the experimental Transformers feature change?
It aims to load only the experts selected by an MoE model’s router into a bounded cache, rather than keeping every expert in fast memory. The proposed cache can draw experts from SSD, RAM or device memory, depending on available capacity.
What performance result was reported?
The developer reported 34.6 tokens per second on OLMoE with the cache design, compared with 38.1 for the standard Transformers path. The source does not provide enough test conditions to establish how the result varies across workloads.
Was a 180B model benchmarked on a 32 GB Mac?
No completed result is provided for that combination. The 180B model and 32 GB Mac are described as a motivating target, not as a demonstrated outcome.
Is the feature part of a Transformers release?
The developer says the work was submitted as pull request #50717. The source does not say whether it was accepted, merged or released.
What would verify the larger-model claim?
A reported test should identify the model and Mac configuration, storage device, cache size and workload, and give both generation speed and memory use. A comparison with another approach on the same hardware would make the results easier to interpret.
Primary source: Hugging Face · via ThorstenMeyerAI.com
