What's Changed

• Refactor ISQ quant parsing by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/664
• Refactor server examples to use OpenAI Python client by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/665
• Implement prompt chunking by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/623
• Python example and server example cleanup by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/668
• Implement GPTQ quantization by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/467
• Update deps by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/672
• Rework the automatic dtype selection feature by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/676
• Fix backend Candle fork Metal, flash attn, also Llama linear by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/681
• Use converted tokenizer.json in tests by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/682
• Refactor ISQ and mistralrs-quant by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/683
• Fix metal build for isq by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/686
• Add missing error case in automatic dtype selection feature by @ac3xx in https://github.com/EricLBuehler/mistral.rs/pull/685
• fix null in tool type response by @wseaton in https://github.com/EricLBuehler/mistral.rs/pull/687
• Implement HQQ quantization by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/677
• Bump version to 0.2.5 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/688

New Contributors

• @ac3xx made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/685
• @wseaton made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/687

Full Changelog: https://github.com/EricLBuehler/mistral.rs/compare/v0.2.4...v0.2.5

Install mistralrs-server 0.2.5

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.5/mistralrs-server-installer.sh | sh

Download mistralrs-server 0.2.5