[!IMPORTANT] bigdl-llm has now become ipex-llm (see the migration guide here); you may find the original BigDL project here.

💫 Intel® LLM Library for PyTorch*

< English | 中文 >

IPEX-LLM is a PyTorch library for running LLM on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max) with very low latency[^1].

[!NOTE]

• It is built on top of the excellent work of llama.cpp, transformers, bitsandbytes, vLLM, qlora, AutoGPTQ, AutoAWQ, etc.
• It provides seamless integration with llama.cpp, Ollama, Text-Generation-WebUI, HuggingFace transformers, LangChain, LlamaIndex, DeepSpeed-AutoTP, vLLM, FastChat, Axolotl, HuggingFace PEFT, HuggingFace TRL, AutoGen, ModeScope, etc.
• 50+ models have been optimized/verified on ipex-llm (including LLaMA2, Mistral, Mixtral, Gemma, LLaVA, Whisper, ChatGLM, Baichuan, Qwen, RWKV, and more); see the complete list here.

Latest Update 🔥

• [2024/07] We added support for running Microsoft's GraphRAG using local LLM on Intel GPU; see the quickstart guide here.
• [2024/07] We added extensive support for Large Multimodal Models, including StableDiffusion, Phi-3-Vision, Qwen-VL, and more.
• [2024/07] We added FP6 support on Intel GPU.
• [2024/06] We added experimental NPU support for Intel Core Ultra processors; see the examples here.
• [2024/06] We added extensive support of pipeline parallel inference, which makes it easy to run large-sized LLM using 2 or more Intel GPUs (such as Arc).
• [2024/06] We added support for running RAGFlow with ipex-llm on Intel GPU.
• [2024/05] ipex-llm now supports Axolotl for LLM finetuning on Intel GPU; see the quickstart here.

ipex-llm Performance

See the Token Generation Speed on Intel Core Ultra and Intel Arc GPU below[^1] (and refer to [2][3][4] for more details).

You may follow the Benchmarking Guide to run ipex-llm performance benchmark yourself.

ipex-llm Demo

See demos of running local LLMs on Intel Iris iGPU, Intel Core Ultra iGPU, single-card Arc GPU, or multi-card Arc GPUs using ipex-llm below.

Intel Iris iGPU	Intel Core Ultra iGPU	Intel Arc dGPU	2-Card Intel Arc dGPUs
llama.cpp (Phi-3-mini Q4_0)	Ollama (Mistral-7B Q4_K)	TextGeneration-WebUI (Llama3-8B FP8)	FastChat (QWen1.5-32B FP6)

Model Accuracy

Please see the Perplexity result below (tested on Wikitext dataset using the script here).

[^1]: Performance varies by use, configuration and other factors. ipex-llm may not optimize to the same degree for non-Intel products. Learn more at www.Intel.com/PerformanceIndex.

ipex-llm Quickstart

Docker

• GPU Inference in C++: running llama.cpp, ollama, OpenWebUI, etc., with ipex-llm on Intel GPU
• GPU Inference in Python : running HuggingFace transformers, LangChain, LlamaIndex, ModelScope, etc. with ipex-llm on Intel GPU
• vLLM on GPU: running vLLM serving with ipex-llm on Intel GPU
• vLLM on CPU: running vLLM serving with ipex-llm on Intel CPU
• FastChat on GPU: running FastChat serving with ipex-llm on Intel GPU
• VSCode on GPU: running and developing ipex-llm applications in Python using VSCode on Intel GPU

Use

• llama.cpp: running llama.cpp (using C++ interface of ipex-llm) on Intel GPU
• Ollama: running ollama (using C++ interface of ipex-llm) on Intel GPU
• PyTorch/HuggingFace: running PyTorch, HuggingFace, LangChain, LlamaIndex, etc. (using Python interface of ipex-llm) on Intel GPU for Windows and Linux
• vLLM: running ipex-llm in vLLM on both Intel GPU and CPU
• FastChat: running ipex-llm in FastChat serving on on both Intel GPU and CPU
• Serving on multiple Intel GPUs: running ipex-llm serving on multiple Intel GPUs by leveraging DeepSpeed AutoTP and FastAPI
• Text-Generation-WebUI: running ipex-llm in oobabooga WebUI
• Axolotl: running ipex-llm in Axolotl for LLM finetuning
• Benchmarking: running (latency and throughput) benchmarks for ipex-llm on Intel CPU and GPU

Applications

• GraphRAG: running Microsoft's GraphRAG using local LLM with ipex-llm
• RAGFlow: running RAGFlow (an open-source RAG engine) with ipex-llm
• LangChain-Chatchat: running LangChain-Chatchat (Knowledge Base QA using RAG pipeline) with ipex-llm
• Coding copilot: running Continue (coding copilot in VSCode) with ipex-llm
• Open WebUI: running Open WebUI with ipex-llm
• PrivateGPT: running PrivateGPT to interact with documents with ipex-llm
• Dify platform: running ipex-llm in Dify(production-ready LLM app development platform)

Install

• Windows GPU: installing ipex-llm on Windows with Intel GPU
• Linux GPU: installing ipex-llm on Linux with Intel GPU
• For more details, please refer to the full installation guide

Code Examples

• Low bit inference

  • INT4 inference: INT4 LLM inference on Intel GPU and CPU
  • FP8/FP6/FP4 inference: FP8, FP6 and FP4 LLM inference on Intel GPU
  • INT8 inference: INT8 LLM inference on Intel GPU and CPU
  • INT2 inference: INT2 LLM inference (based on llama.cpp IQ2 mechanism) on Intel GPU

• FP16/BF16 inference

  • FP16 LLM inference on Intel GPU, with possible self-speculative decoding optimization
  • BF16 LLM inference on Intel CPU, with possible self-speculative decoding optimization

• Distributed inference

  • Pipeline Parallel inference on Intel GPU
  • DeepSpeed AutoTP inference on Intel GPU

• Save and load

  • Low-bit models: saving and loading ipex-llm low-bit models (INT4/FP4/FP6/INT8/FP8/FP16/etc.)
  • GGUF: directly loading GGUF models into ipex-llm
  • AWQ: directly loading AWQ models into ipex-llm
  • GPTQ: directly loading GPTQ models into ipex-llm

• Finetuning

  • LLM finetuning on Intel GPU, including LoRA, QLoRA, DPO, QA-LoRA and ReLoRA
  • QLoRA finetuning on Intel CPU

• Integration with community libraries

  • HuggingFace transformers
  • Standard PyTorch model
  • LangChain
  • LlamaIndex
  • DeepSpeed-AutoTP
  • Axolotl
  • HuggingFace PEFT
  • HuggingFace TRL
  • AutoGen
  • ModeScope
• Tutorials

API Doc

• HuggingFace Transformers-style API (Auto Classes)
• API for arbitrary PyTorch Model

FAQ

• FAQ & Trouble Shooting

Verified Models

Over 50 models have been optimized/verified on ipex-llm, including LLaMA/LLaMA2, Mistral, Mixtral, Gemma, LLaVA, Whisper, ChatGLM2/ChatGLM3, Baichuan/Baichuan2, Qwen/Qwen-1.5, InternLM and more; see the list below.

Get Support

• Please report a bug or raise a feature request by opening a Github Issue
• Please report a vulnerability by opening a draft GitHub Security Advisory