LocalAI v2.10.0 Release Notes

Excited to announce the release of LocalAI v2.10.0! This version introduces significant changes, including breaking changes, numerous bug fixes, exciting new features, dependency updates, and more. Here's a summary of what's new:

Breaking Changes 🛠

• The trust_remote_code setting in the YAML config file of the model are now consumed for enhanced security measures also for the AutoGPTQ and transformers backend, thanks to @dave-gray101's contribution (#1799). If your model relied on the old behavior and you are sure of what you are doing, set trust_remote_code: true in the YAML config file.

Bug Fixes :bug:

• Various fixes have been implemented to enhance the stability and performance of LocalAI:
  • SSE no longer omits empty finish_reason fields for better compatibility with the OpenAI API, fixed by @mudler (#1745).
  • Functions now correctly handle scenarios with no results, also addressed by @mudler (#1758).
  • A Command Injection Vulnerability has been fixed by @ouxs-19 (#1778).
  • OpenCL-based builds for llama.cpp have been restored, thanks to @cryptk's efforts (#1828, #1830).
  • An issue with OSX build default.metallib has been resolved, which should now allow running the llama-cpp backend on Apple arm64, fixed by @dave-gray101 (#1837).

Exciting New Features 🎉

• LocalAI continues to evolve with several new features:
  • Ongoing implementation of the assistants API, making great progress thanks to community contributions, including an initial implementation by @christ66 (#1761).
  • Addition of diffusers/transformers support for Intel GPU - now you can generate images and use the transformer backend also on Intel GPUs, implemented by @mudler (#1746).
  • Introduction of Bitsandbytes quantization for transformer backend enhancement and a fix for transformer backend error on CUDA by @fakezeta (#1823).
  • Compatibility layers for Elevenlabs and OpenAI TTS, enhancing text-to-speech capabilities: Now LocalAI is compatible with Elevenlabs and OpenAI TTS, thanks to @mudler (#1834).
  • vLLM now supports stream: true! This feature was introduced by @golgeek (#1749).

Dependency Updates 👒

• Our continuous effort to keep dependencies up-to-date includes multiple updates to ggerganov/llama.cpp, donomii/go-rwkv.cpp, mudler/go-stable-diffusion, and others, ensuring that LocalAI is built on the latest and most secure libraries.

Other Changes

• Several internal changes have been made to improve the development process and documentation, including updates to integration guides, stress reduction on self-hosted runners, and more.

Details of What's Changed

Breaking Changes 🛠

• feat(autogpt/transformers): consume trust_remote_code by @dave-gray101 in https://github.com/mudler/LocalAI/pull/1799

Bug fixes :bug:

• fix(sse): do not omit empty finish_reason by @mudler in https://github.com/mudler/LocalAI/pull/1745
• fix(functions): handle correctly when there are no results by @mudler in https://github.com/mudler/LocalAI/pull/1758
• fix(tests): re-enable tests after code move by @mudler in https://github.com/mudler/LocalAI/pull/1764
• Fix Command Injection Vulnerability by @ouxs-19 in https://github.com/mudler/LocalAI/pull/1778
• fix: the correct BUILD_TYPE for OpenCL is clblas (with no t) by @cryptk in https://github.com/mudler/LocalAI/pull/1828
• fix: missing OpenCL libraries from docker containers during clblas docker build by @cryptk in https://github.com/mudler/LocalAI/pull/1830
• fix: osx build default.metallib by @dave-gray101 in https://github.com/mudler/LocalAI/pull/1837

Exciting New Features 🎉

• fix: vllm - use AsyncLLMEngine to allow true streaming mode by @golgeek in https://github.com/mudler/LocalAI/pull/1749
• refactor: move remaining api packages to core by @dave-gray101 in https://github.com/mudler/LocalAI/pull/1731
• Bump vLLM version + more options when loading models in vLLM by @golgeek in https://github.com/mudler/LocalAI/pull/1782
• feat(assistant): Initial implementation of assistants api by @christ66 in https://github.com/mudler/LocalAI/pull/1761
• feat(intel): add diffusers/transformers support by @mudler in https://github.com/mudler/LocalAI/pull/1746
• fix(config): set better defaults for inferencing by @mudler in https://github.com/mudler/LocalAI/pull/1822
• fix(docker-compose): update docker compose file by @mudler in https://github.com/mudler/LocalAI/pull/1824
• feat(model-help): display help text in markdown by @mudler in https://github.com/mudler/LocalAI/pull/1825
• feat: Add Bitsandbytes quantization for transformer backend enhancement #1775 and fix: Transformer backend error on CUDA #1774 by @fakezeta in https://github.com/mudler/LocalAI/pull/1823
• feat(tts): add Elevenlabs and OpenAI TTS compatibility layer by @mudler in https://github.com/mudler/LocalAI/pull/1834
• feat(embeddings): do not require to be configured by @mudler in https://github.com/mudler/LocalAI/pull/1842

👒 Dependencies

• :arrow_up: Update docs version mudler/LocalAI by @localai-bot in https://github.com/mudler/LocalAI/pull/1752
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1753
• deps(llama.cpp): update by @mudler in https://github.com/mudler/LocalAI/pull/1759
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1756
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1767
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1772
• :arrow_up: Update donomii/go-rwkv.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1771
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1779
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1789
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1791
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1794
• depedencies(sentencentranformers): update dependencies by @TwinFinz in https://github.com/mudler/LocalAI/pull/1797
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1801
• :arrow_up: Update mudler/go-stable-diffusion by @localai-bot in https://github.com/mudler/LocalAI/pull/1802
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1805
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1811
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1827

Other Changes

• ci: add stablediffusion to release by @sozercan in https://github.com/mudler/LocalAI/pull/1757
• Update integrations.md by @Joshhua5 in https://github.com/mudler/LocalAI/pull/1765
• ci: reduce stress on self-hosted runners by @mudler in https://github.com/mudler/LocalAI/pull/1776
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1785
• Revert "feat(assistant): Initial implementation of assistants api" by @mudler in https://github.com/mudler/LocalAI/pull/1790
• Edit links in readme and integrations page by @lunamidori5 in https://github.com/mudler/LocalAI/pull/1796
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1813
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1816
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1818
• fix(doc/examples): set defaults to mirostat by @mudler in https://github.com/mudler/LocalAI/pull/1820
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1821
• fix: OSX Build Files for llama.cpp by @dave-gray101 in https://github.com/mudler/LocalAI/pull/1836
• :arrow_up: Update go-skynet/go-llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1835
• docs(transformers): add docs section about transformers by @mudler in https://github.com/mudler/LocalAI/pull/1841
• :arrow_up: Update mudler/go-piper by @localai-bot in https://github.com/mudler/LocalAI/pull/1844
• :arrow_up: Update ggerganov/llama.cpp by @localai-bot in https://github.com/mudler/LocalAI/pull/1840

New Contributors

• @golgeek made their first contribution in https://github.com/mudler/LocalAI/pull/1749
• @Joshhua5 made their first contribution in https://github.com/mudler/LocalAI/pull/1765
• @ouxs-19 made their first contribution in https://github.com/mudler/LocalAI/pull/1778
• @TwinFinz made their first contribution in https://github.com/mudler/LocalAI/pull/1797
• @cryptk made their first contribution in https://github.com/mudler/LocalAI/pull/1828
• @fakezeta made their first contribution in https://github.com/mudler/LocalAI/pull/1823

Thank you to all contributors and users for your continued support and feedback, making LocalAI better with each release!

Full Changelog: https://github.com/mudler/LocalAI/compare/v2.9.0...v2.10.0