Please note that API stability is not expected until 1.0 release. This update introduces breaking changes.

This release introduces a new vLLM backend and removes the dependency on TGI. This is because TGI is not Apache 2.0 licensed anymore, and the new license is too restrictive for most organizations to run in production. On the other hand, vllm is Apache 2.0 licensed and is a better foundation to build on top of. There are some breaking changes to model configuration YAMLs related to the new vLLM backend.

Refer to the updated ray-llm/models/README.md file for details on the updated configuration file format.

What's changed?

• Documentation

  • Updated readme and documentation

• API & SDK

  • Updated the format of model configuration yamls.

• Backend

  • Completely replaced the text-generation-inference based backend with vLLM based backend. This means RayLLM now supports all models vLLM supports.
  • Improved observability and metrics.
  • Improved testing.

In order to use RayLLM, ensure you are using the official Docker image anyscale/aviary:latest.