llama.cpp Adds Q8_0 IME1 Matrix Kernel Acceleration for SpacemiT X60
The open-source llama.cpp project has introduced IME1 matrix acceleration for Q8_0 quantization on the SpacemiT X60, achieving a roughly 9x speed boost for Q8_0 models and addressing a significant performance gap.
What changed?
The llama.cpp project has released a new Q8_0 IME1 matrix kernel specifically for the SpacemiT X60. Previously, the IME matrix acceleration on this hardware only supported Q4_0, Q4_1, and Q4_K quantization formats. Q8_0 quantization had no accelerated path and thus performed nearly ten times slower compared to Q4_0 for model prefill operations. With this update, Q8_0 is now accelerated, providing a measured prefill speed increase from around 10.7 tokens/s to 93.9 tokens/s based on llama-bench tests.
Why does it matter to an everyday developer?
Q8_0 quantization allows for higher model accuracy at the cost of requiring more computational resources. Developers running Q8_0-quantized models on SpacemiT X60 boards previously faced severe performance bottlenecks, making inference impractical for many use cases. The new acceleration enables significantly faster inference, allowing developers to deploy higher-quality models efficiently. This benefits developers building applications that need fast and accurate local inference on SpacemiT X60 hardware.
What can the developer do now?
Developers using the SpacemiT X60 can update to the latest llama.cpp release to benefit from accelerated Q8_0 model inference. Pre-built binaries are available for various platforms, including SpacemiT-enabled Linux distributions. There are no breaking changes or additional configuration steps required; updating and deploying Q8_0 models will now utilize the new acceleration automatically.