🔍 Read the full analysis: Running Llama.cpp Quants Through Transformers on ThorstenMeyerAI.com
Get the latest gadgets delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
TL;DR
Hugging Face has added support for loading GGUF quantized checkpoints through the Transformers library’s from_pretrained API. The feature is on the main branch, with initial support focused on Apple Silicon and Qwen3.5; broader hardware and architecture support and a stable release date have not been announced.
Hugging Face has added support for GGUF quantized checkpoints in its Transformers library, as detailed in the original analysis, allowing users to load compatible models through the familiar from_pretrained API. The initial rollout focuses on Apple Silicon and the Qwen3.5 architecture, bringing a widely used local model format into the PyTorch-based library.
According to Hugging Face, users can select a GGUF checkpoint from the Hub and pass its filename through the gguf_file argument to from_pretrained, then generate text. The feature is available on the library’s main branch, ahead of a stable release. The announcement does not give a date for that release.
The implementation reuses llama.cpp’s ggml kernels. On compatible Apple Silicon systems, when weights remain packed on Metal, Transformers can load suitable ggml/Metal layer kernels and use ggml-org/ggml-attn for attention. If that kernel is unavailable, the system falls back to standard SDPA attention with a warning; users can also select SDPA directly. Without a compatible quantization kernel, the loader dequantizes the model, which requires more memory.
Hugging Face says the requirements include an Apple Silicon Mac, a supported PyTorch version, and current Transformers software with a compatible kernels library. The announcement also describes serving the same checkpoints through transformers serve, which provides an OpenAI-compatible API on localhost. Clients such as Jan or Pi can connect through a custom provider configured for that endpoint.
GGUF Comes to the Transformers Workflow
The change gives developers who already use Transformers a path to run GGUF checkpoints without moving their model-loading workflow to a separate llama.cpp-based application. GGUF is used by local inference tools including Ollama, LM Studio and Jan, and the format is available for checkpoints from publishers such as Unsloth, LM Studio Community, bartowski and ggml-org.
Quantization can reduce the memory needed to load a model, making some checkpoints more practical on laptops and other machines with limited memory. Hugging Face’s published example for Unsloth’s Qwen3.5-4B lists the BF16 file at 8.42 GB, compared with 2.74 GB for Q4_K_M. Those figures describe file sizes; they do not establish a quality score or guarantee performance on a particular machine.
For teams building with Transformers, direct loading and local API serving may make it easier to test published quantized checkpoints within existing workflows. The benefit depends on compatible hardware, kernels and model architecture, so it does not yet amount to universal GGUF support across devices.
How the Initial Support Works
GGUF packages model weights and metadata, which can include tokenizer details and a chat template, in a single file. It offers multiple quantization levels that trade memory use against numerical precision. A label such as Q4_K_M refers to a mixed-precision quantization variant, rather than every tensor being stored at the same precision.
In its Qwen3.5-4B example, Hugging Face lists Q6_K at 3.53 GB, Q5_K_M at 3.14 GB and Q4_K_M at 2.74 GB, alongside the 8.42 GB BF16 reference. The company recommends starting with Q4_K_M and trying Q5_K_M or Q6_K if more memory is available. It also says the effect on output quality depends on the model and task, and advises evaluation on the intended workload.
Hugging Face says its performance reference is llama.cpp and that its comparison covers three GGUF checkpoints: a small dense model, a larger dense model and a mixture-of-experts model. The supplied source material does not include the measured results, test hardware or detailed benchmark conditions, so it does not support a general claim that the Transformers implementation matches or exceeds llama.cpp.
““We’re adding support for running GGUF models efficiently in transformers, so you can use checkpoints sized for your laptop’s memory through the familiar transformers APIs.””
— Hugging Face announcement
Hardware and Model Support Remain Narrow
The announcement describes initial support for Apple Silicon and Qwen3.5. It gives no timeline for CUDA, Linux or Windows support, and does not say which additional model architectures will be added or when. It is also unclear when the feature will reach a stable Transformers release.
Although Hugging Face refers to performance comparisons against llama.cpp across three checkpoints, detailed benchmark figures and test conditions are not included in the supplied material. Performance will depend on the hardware, model, attention implementation and available kernels. The practical quality effect of each quantization level also remains workload-specific.
Stable Release and Broader Coverage
The next clear milestone is inclusion in a stable Transformers release; Hugging Face has not announced when that will happen. Until then, users seeking the feature need to use the main branch and check that their PyTorch and kernels versions are compatible.
Further development could add model families or hardware backends, but the announcement sets no schedule or commitments for those additions. Users can follow Hugging Face’s GGUF documentation and kernels library for updates. For now, anyone evaluating the feature should check whether their model is supported, confirm that compatible kernels load, and compare memory use and output quality on their own tasks.
Key Questions
How do users load a GGUF checkpoint?
Hugging Face says to choose a compatible Hub checkpoint and pass its filename to the gguf_file argument of from_pretrained. The feature is currently on the Transformers main branch.
Which systems and models are supported initially?
The announced initial focus is Apple Silicon and the Qwen3.5 architecture. The source material does not provide a schedule for other hardware or model families.
Does Transformers match llama.cpp’s speed?
Hugging Face says it uses llama.cpp’s ggml kernels and references comparisons across three checkpoints. The supplied information does not include detailed measurements or test conditions, so it does not establish a general speed result.
When will the feature be in a stable release?
No stable release date was announced. The feature is available on the Transformers main branch.
Primary source: Hugging Face · via ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
