Major changes

• Fast model execution with CUDA/HIP graph
• W4A16 GPTQ support (thanks to @chu-tianxiang)
• Fix memory profiling with tensor parallelism
• Fix *.bin weight loading for Mixtral models

What's Changed

• Fix typing in generate function for AsyncLLMEngine & add toml to requirements-dev by @mezuzza in https://github.com/vllm-project/vllm/pull/2100
• Fix Dockerfile.rocm by @tjtanaa in https://github.com/vllm-project/vllm/pull/2101
• avoid multiple redefinition by @MitchellX in https://github.com/vllm-project/vllm/pull/1817
• Add a flag to include stop string in output text by @yunfeng-scale in https://github.com/vllm-project/vllm/pull/1976
• Add GPTQ support by @chu-tianxiang in https://github.com/vllm-project/vllm/pull/916
• [Docs] Add quantization support to docs by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2135
• [ROCm] Temporarily remove GPTQ ROCm support by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2138
• simplify loading weights logic by @esmeetu in https://github.com/vllm-project/vllm/pull/2133
• Optimize model execution with CUDA graph by @WoosukKwon in https://github.com/vllm-project/vllm/pull/1926
• [Minor] Delete Llama tokenizer warnings by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2146
• Fix all-reduce memory usage by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2151
• Pin PyTorch & xformers versions by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2155
• Remove dependency on CuPy by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2152
• [Docs] Add CUDA graph support to docs by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2148
• Temporarily enforce eager mode for GPTQ models by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2154
• [Minor] Add more detailed explanation on quantization argument by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2145
• [Minor] Fix xformers version by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2158
• [Minor] Add Phi 2 to supported models by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2159
• Make sampler less blocking by @Yard1 in https://github.com/vllm-project/vllm/pull/1889
• [Minor] Fix a typo in .pt weight support by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2160
• Disable CUDA graph for SqueezeLLM by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2161
• Bump up to v0.2.6 by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2157

New Contributors

• @mezuzza made their first contribution in https://github.com/vllm-project/vllm/pull/2100
• @MitchellX made their first contribution in https://github.com/vllm-project/vllm/pull/1817

Full Changelog: https://github.com/vllm-project/vllm/compare/v0.2.5...v0.2.6