This release fixes bugs and introduces @kawrakow's latest quant performance enhancements (a feature exclusive to llamafile). As of #435 the K quants now go consistently 2x faster than llama.cpp upstream. On big CPUs like Threadripper we've doubled the performance of tiny models, for both prompt processing and token generation for tiny models (see the benchmarks below) The llamafile-bench and llamafile-upgrade-engine commands have been introduced.

• a86e7ce Add Script To Upgrade llamafile Archives (#412)
• 07e87bf 261dfe7 Fix llamafile-quantize and rewrite documentation
• 938cf72 Faster AVX2 matrix multiplications for MoE models (#428)
• eaa756d Faster AVX2 matrix multiplications for legacy quants (#405)
• 7cb15c6 Another performance optimization for Zen4 + refactoring (#435)
• 9206719 8b2f8d8 e675719 4451c6d Introduce llamafile-bench command (cpu mode only)
• 87d4ce1 Fix f16 cpuid check (caused crashes on sandybridge)
• 5c40565 8d1afe4 Avoid crashing on llava ctrl-c
• c0aa43e Introduce bf16 cuda support
• 00e4f72 Enable GGML_CUDA_FORCE_MMQ in tinyBLAS mode
• d228e01 0b5997d 64fbffc Sync with llama.cpp upstream (#427)
• c660d38 Add text embedding models to 'other example llamafiles' table (#422)
• 49cc13c Updated README with instructions to load models from third-party apps (#417)

Note: Please use llamafile v0.8.4 if you need prebuilt (driver-only) AMD GPU support on Windows, at least for the next few weeks, until https://github.com/ggerganov/llama.cpp/issues/7156 is resolved.

Binaries run on Linux, Windows, MacOS, FreeBSD, OpenBSD, and NetBSD for AMD64 and ARM64. Supported GPUs are CUDA, ROCm, and Metal. Prebuilt GPU binaries are provided for CUDA/ROCm on Linux, and CUDA on Windows. To install this release on systems with a POSIX-style shell:

sudo -s
cd /usr/local
wget https://github.com/Mozilla-Ocho/llamafile/releases/download/0.8.5/llamafile-0.8.5.zip
unzip llamafile-0.8.5.zip
exit
llamafile --help

To upgrade your old llamafiles without needing to redownload, run:

llamafile-upgrade-engine old.llamafile new.llamafile

Prebuilt llamafiles that have the LLM weights included are available at:

• https://huggingface.co/Mozilla (official)
• https://huggingface.co/models?library=llamafile (community)

Here are some tutorials:

• https://justine.lol/oneliners/
• https://github.com/mozilla-ocho/llamafile/
• https://future.mozilla.org/news/llamafiles-for-embeddings-in-local-rag-applications/
• https://blog.mozilla.ai/local-llm-as-judge-evaluation-with-lm-buddy-prometheus-and-llamafile/
• https://www.docker.com/blog/a-quick-guide-to-containerizing-llamafile-with-docker-for-ai-applications/

Here are some performance benchmarks for various quantization formats, on the world's flagship CPUs. See https://justine.lol/matmul/ to compare these numbers to where we were back in March two months ago.