AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Pushing Transformers MoE Inference On 32GB RAM With A 180B-Parameter Model on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

TL;DR

A Hugging Face developer has proposed an experimental Transformers feature that streams selected mixture-of-experts model weights from SSD into a bounded device cache. The reported OLMoE test reached 34.6 tokens per second, near the standard Transformers result of 38.1, but no completed benchmark is provided for a 180B model on a 32 GB Mac.

A Hugging Face developer has proposed an experimental feature for Transformers that loads selected mixture-of-experts (MoE) model weights from SSD into a bounded device cache, reporting 34.6 tokens per second on OLMoE against 38.1 tokens per second with the standard Transformers path, as described in the original analysis. The work targets models whose weights exceed available memory, but the source does not report a completed test of a 180B-parameter model on a 32 GB Mac.

MoE models route each token through a subset of their experts rather than using every expert for every step. The developer says experts account for more than 90% of the model weights, making it possible in principle to keep only selected experts in faster memory and retrieve others as needed. The proposed approach places experts in GPU memory, system RAM or disk according to available capacity, with selected experts held in a bounded cache.

The initial implementation ran experts one at a time in a Python loop. In the developer’s test, that version reached 10.1 tokens per second even with experts already cached, compared with about 32 for vanilla Transformers. The developer then used a shared, preallocated device tensor and a table mapping expert IDs to cache slots. This allowed the existing grouped matrix multiplication kernel to process selected experts together; the reported result rose to 34.6 tokens per second, versus 38.1 for the standard path.

The developer says the change was submitted to Transformers as pull request #50717. The source also points to a Hugging Face Space, tarekziade/moe-expert-streaming, with project code. It describes implementation work across Transformers, a model-specific package and Metal kernels, and says AI was used to build code. The developer characterizes their role as selecting experiments, reviewing profiles and deciding which changes to keep.

At a glance
reportWhen: Reported in the source material; pull r…
The developmentA Hugging Face developer reported a near-baseline OLMoE inference result with an experimental SSD-backed expert cache proposed for Transformers.
AI Inference · Briefing

Streaming MoE Experts With Limited Memory

An experimental Transformers change streams selected experts from SSD into a bounded cache. The reported OLMoE test ran close to vanilla speed, but the 180B model on a 32 GB Mac remains untested in the source.

34.6 tokens/s
OLMoE with expert streaming
38.1 tokens/s
Vanilla Transformers comparison
90%+
Model weights attributed to experts

How the cache works

The router selects experts for each token; the experiment fetches those experts into a shared cache.

1
Route each token

Choose a subset of experts for the token.

2
Load selected experts

Stream them from storage into available device cache slots.

3
Compute together

Use Transformers’ grouped matrix multiplication kernel.

What the test shows

On OLMoE-1B-7B, the shared cache kept the reported speed near the standard path.

Earlier approach
  • Python loop ran experts one at a time.
  • Reached 10.1 tokens per second, even with experts cached.
Shared cache
  • Preallocated tensor mapped expert IDs to cache slots.
  • Reached 34.6 tokens per second versus 38.1 baseline.

The larger device test is open

The 180B model on a 32 GB Mac is a motivating target, not a measured result in the supplied source.

Evidence

Promising engineering

The cache design brought the reported small-model test close to vanilla Transformers speed.

Open question

SSD performance

Storage traffic and cache misses could change speed when experts must be fetched repeatedly.

Next evidence

Measure the target setup

Report speed and memory use on the larger model, with storage, cache settings and workload specified.

StatusThe change was submitted as Transformers pull request #50717; the source does not say whether it was merged or released.
The takeaway
Expert streaming looks close to baseline on the reported OLMoE test, while performance on the 180B model and 32 GB Mac remains unproven.

A Smaller Memory Footprint for MoE

If the method works reliably when experts must actually be fetched from storage, it could let developers run some MoE models on hardware that cannot hold all model weights in memory. That could widen access to local inference on laptops and single-GPU systems, where memory capacity can prevent people from trying larger models even though only some experts are used for each token.

The reported speed comparison is encouraging for one limited point: the cache design did not impose a large slowdown in the tested OLMoE setup. But the result does not show how much time SSD retrieval adds when the active experts are absent from cache. Storage speed, data movement, cache capacity and how quickly routing choices change could all affect generation. The key question is whether the approach can reduce memory use without making those costs dominate.

Products Worth Considering

As an affiliate, we earn on qualifying purchases.

How Expert Streaming Is Designed

The source also reports comparisons on Qwen3.6-35B-A3B using an A10G GPU with 24 GB of memory and ample system RAM. In that setup, it says Strata decoded at 82–88 tokens per second with Q8_0, while llama.cpp with Q8_0 and its fit option reached 63–64. Those figures belong to that particular model and hardware; they are not results for the experimental Transformers feature or a laptop.

The source distinguishes several approaches to placing MoE weights. It says llama.cpp can run experts on the CPU, while vLLM’s CPU offload reads weights over PCIe without a hot-expert cache. It describes Transformers with Accelerate as moving an entire layer’s expert bank before processing it. The proposed experiment instead aims to load only the experts selected by the router into a bounded cache and retain grouped expert computation. These descriptions and comparisons are reported by the source, not independently established here.

The 32 GB Mac Test Is Missing

The source frames running a 180B-parameter MoE model on a 32 GB Mac as a motivating target, but gives no measured result for that model and device. It also does not report the OLMoE test’s storage type, cache size, prompt conditions or repeated-run results, leaving readers without details needed to judge how broadly the speed figures apply.

It is also unclear whether pull request #50717 has been accepted, merged or included in a Transformers release. The source does not establish how the cache performs when routed experts change quickly, or provide a same-hardware, same-model comparison with Strata. The OLMoE result is a test of the cache design’s speed in the conditions reported; it is not evidence that SSD-backed inference for the 180B model will achieve similar performance.

Pull Request and Larger-Model Results

The next concrete update is the status of Transformers pull request #50717 and whether the experimental option becomes available in a published version. The linked Hugging Face Space offers a way to inspect the project code, according to the source material.

To evaluate the intended use case, future results would need to report generation speed and memory use for the target large MoE model on a 32 GB Mac, alongside the storage device, cache settings and workload. Repeated runs and a direct comparison with Strata under matching conditions would help show whether the approach remains fast when experts must be loaded from storage. Until then, the large-model scenario remains a goal rather than a demonstrated benchmark.

Key Questions

What did the experimental Transformers feature change?

It aims to load only the experts selected by an MoE model’s router into a bounded cache, rather than keeping every expert in fast memory. The proposed cache can draw experts from SSD, RAM or device memory, depending on available capacity.

What performance result was reported?

The developer reported 34.6 tokens per second on OLMoE with the cache design, compared with 38.1 for the standard Transformers path. The source does not provide enough test conditions to establish how the result varies across workloads.

Was a 180B model benchmarked on a 32 GB Mac?

No completed result is provided for that combination. The 180B model and 32 GB Mac are described as a motivating target, not as a demonstrated outcome.

Is the feature part of a Transformers release?

The developer says the work was submitted as pull request #50717. The source does not say whether it was accepted, merged or released.

What would verify the larger-model claim?

A reported test should identify the model and Mac configuration, storage device, cache size and workload, and give both generation speed and memory use. A comparison with another approach on the same hardware would make the results easier to interpret.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

Commodore 64 Released September 1, 1982

The Commodore 64 was officially released on September 1, 1982, marking a significant milestone in home computer history and influencing the industry for years.

Why AI Will Be A Game-Changer In 2026

The supplied material covers portable SSDs, not AI developments, leaving the proposed 2026 claim without supporting evidence.

Business Laptops vs Consumer Laptops: What Really Changes?

AIThis post was created with the assistance of artificial intelligence (AI).When choosing…

Super Micro Computer Surges In Global Coverage

Super Micro Computer experiences a surge in international coverage, with 24 mentions in recent media analysis, signaling increased public and industry interest.