Huawei releases MACaQ, its internal quantizer for local inference
Reported by Hacker News1 source ↓
Huawei has released MACaQ, a quantization tool for local inference across text, image, video and audio models. It supports several inference and model-generation frameworks.
Key points
- MACaQ supports llama.cpp, vLLM, Diffusers and PyTorch
- Text models can be saved in 2-, 3- and 4-bit GGUF formats
- MACaQ-R refines quantized GGUF models for use with standard llama.cpp
Details
The release materials describe workflows for quantizing and running text, image, video and audio models. They also include options to save compact checkpoints and generate outputs, including images, video and audio.
The project reports that MACaQ can quantize a 122-billion-parameter model in 64 minutes using 32.5 GB of device memory. Its materials also claim lower perplexity than MR-GPTQ and DuQuant++ in some comparisons; the article does not independently verify these performance claims. MACaQ is licensed under Apache 2.0.
Written from coverage as of
Images
- Hacker News
- Hacker News