Highlights

• New model topology feature: ISQ and device mapping
• 🔥Faster FlashAttention support when batching
• Removed plotly and associated JS dependencies
• φ³ Support Phi 3.5, Phi 3.5 vision, Phi 3.5 MoE
• Improved Rust API ergonomics
• Support multiple (shaded) GGUF files

MSRV

The Rust MSRV of this version is 1.79.0

What's Changed

• Fixes for auto dtype selection with RUST_BACKTRACE=1 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/690
• Add support multiple GGUF files by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/692
• Refactor normal and vision loaders by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/693
• Fix split.count GGUF duplication handling by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/695
• Batching example by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/694
• Some fixes by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/697
• Improve vision rust examples by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/698
• Add ISQ topology by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/701
• Add custom logits processor API by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/702
• Add Gemma 2 PagedAttention support by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/704
• Faster RmsNorm in Gemma/Gemma2 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/703
• Fix bug in Metal ISQ by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/706
• Support GGUF BF16 tensors by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/691
• Better support for FlashAttention: real batching + sliding window + softcap by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/707
• Remove some usages of pub in models by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/708
• Support the Phi 3.5 V model by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/710
• Implement the Phi 3.5 MoE model by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/709
• Device map topology by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/717
• Implement DRY penalty by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/637
• Remove plotly and just output CSV loss file by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/700
• Using once_cell to reduce MSRV by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/724
• Fixes for Windows build by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/729
• Even more phi3.5moe fix attempts by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/731
• Add example for Phi 3.5 MoE by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/733
• Add Phi 3.5 chat template by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/734
• Patch ISQ for Mixtral by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/730
• Gracefully handle Engine Drop with termination request by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/735
• feat(vision): add support for proper file and data image URLs by @Schuwi in https://github.com/EricLBuehler/mistral.rs/pull/727
• Add new parsing to Python API by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/737
• Remove test and add custom error type to Python API by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/738
• Update kernels for metal bf16 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/719
• Better Response Result API by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/739
• More Metal quantized kernel fixes by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/740
• [Breaking] Bump version to v0.3.0 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/736
• Final changes for v0.3.0 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/741

New Contributors

• @Schuwi made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/727

Full Changelog: https://github.com/EricLBuehler/mistral.rs/compare/v0.2.5...v0.3.0