MyGit
๐Ÿšฉๆ”ถๅˆฐGitHubไป“ๅบ“็š„ๆ›ดๆ–ฐ้€š็Ÿฅ

BerriAI/litellm

Fork: 1016 Star: 9189 (ๆ›ดๆ–ฐไบŽ 2024-05-27 19:19:09)

license: NOASSERTION

Language: Python .

Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)

ๆœ€ๅŽๅ‘ๅธƒ็‰ˆๆœฌ๏ผš v1.37.3 ( 2024-05-11 11:16:52)

ๅฎ˜ๆ–น็ฝ‘ๅ€ GitHub็ฝ‘ๅ€

โœจๅ…่ดน็”ณ่ฏท็ฝ‘็ซ™SSL่ฏไนฆ๏ผŒๆ”ฏๆŒๅคšๅŸŸๅๅ’Œๆณ›ๅŸŸๅ๏ผŒ็‚นๅ‡ปๆŸฅ็œ‹

๐Ÿš… LiteLLM

Deploy to Render Deploy on Railway

Call all LLM APIs using the OpenAI format [Bedrock, Huggingface, VertexAI, TogetherAI, Azure, OpenAI, etc.]

OpenAI Proxy Server | Hosted Proxy (Preview) | Enterprise Tier

PyPI Version CircleCI Y Combinator W23 Whatsapp Discord

LiteLLM manages:

  • Translate inputs to provider's completion, embedding, and image_generation endpoints
  • Consistent output, text responses will always be available at ['choices'][0]['message']['content']
  • Retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - Router
  • Set Budgets & Rate limits per project, api key, model OpenAI Proxy Server

Jump to OpenAI Proxy Docs
Jump to Supported LLM Providers

๐Ÿšจ Stable Release: Use docker images with the -stable tag. These have undergone 12 hour load tests, before being published.

Support for more providers. Missing a provider or LLM Platform, raise a feature request.

Usage (Docs)

[!IMPORTANT] LiteLLM v1.0.0 now requires openai>=1.0.0. Migration guide here

Open In Colab
pip install litellm
from litellm import completion
import os

## set ENV variables
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["COHERE_API_KEY"] = "your-cohere-key"

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# cohere call
response = completion(model="command-nightly", messages=messages)
print(response)

Call any model supported by a provider, with model=<provider_name>/<model_name>. There might be provider-specific details here, so refer to provider docs for more information

Async (Docs)

from litellm import acompletion
import asyncio

async def test_get_response():
    user_message = "Hello, how are you?"
    messages = [{"content": user_message, "role": "user"}]
    response = await acompletion(model="gpt-3.5-turbo", messages=messages)
    return response

response = asyncio.run(test_get_response())
print(response)

Streaming (Docs)

liteLLM supports streaming the model response back, pass stream=True to get a streaming iterator in response.
Streaming is supported for all models (Bedrock, Huggingface, TogetherAI, Azure, OpenAI, etc.)

from litellm import completion
response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
for part in response:
    print(part.choices[0].delta.content or "")

# claude 2
response = completion('claude-2', messages, stream=True)
for part in response:
    print(part.choices[0].delta.content or "")

Logging Observability (Docs)

LiteLLM exposes pre defined callbacks to send data to Lunary, Langfuse, DynamoDB, s3 Buckets, Helicone, Promptlayer, Traceloop, Athina, Slack

from litellm import completion

## set env variables for logging tools
os.environ["LUNARY_PUBLIC_KEY"] = "your-lunary-public-key"
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["ATHINA_API_KEY"] = "your-athina-api-key"

os.environ["OPENAI_API_KEY"]

# set callbacks
litellm.success_callback = ["lunary", "langfuse", "athina"] # log input/output to lunary, langfuse, supabase, athina etc

#openai call
response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hi ๐Ÿ‘‹ - i'm openai"}])

OpenAI Proxy - (Docs)

Track spend + Load Balance across multiple projects

Hosted Proxy (Preview)

The proxy provides:

  1. Hooks for auth
  2. Hooks for logging
  3. Cost tracking
  4. Rate Limiting

๐Ÿ“– Proxy Endpoints - Swagger Docs

Quick Start Proxy - CLI

pip install 'litellm[proxy]'

Step 1: Start litellm proxy

$ litellm --model huggingface/bigcode/starcoder

#INFO: Proxy running on http://0.0.0.0:4000

Step 2: Make ChatCompletions Request to Proxy

import openai # openai v1.0.0+
client = openai.OpenAI(api_key="anything",base_url="http://0.0.0.0:4000") # set proxy to base_url
# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(model="gpt-3.5-turbo", messages = [
    {
        "role": "user",
        "content": "this is a test request, write a short poem"
    }
])

print(response)

Proxy Key Management (Docs)

UI on /ui on your proxy server ui_3

Set budgets and rate limits across multiple projects POST /key/generate

Request

curl 'http://0.0.0.0:4000/key/generate' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data-raw '{"models": ["gpt-3.5-turbo", "gpt-4", "claude-2"], "duration": "20m","metadata": {"user": "ishaan@berri.ai", "team": "core-infra"}}'

Expected Response

{
    "key": "sk-kdEXbIqZRwEeEiHwdg7sFA", # Bearer token
    "expires": "2023-11-19T01:38:25.838000+00:00" # datetime object
}

Supported Providers (Docs)

Provider Completion Streaming Async Completion Async Streaming Async Embedding Async Image Generation
openai โœ… โœ… โœ… โœ… โœ… โœ…
azure โœ… โœ… โœ… โœ… โœ… โœ…
aws - sagemaker โœ… โœ… โœ… โœ… โœ…
aws - bedrock โœ… โœ… โœ… โœ… โœ…
google - vertex_ai [Gemini] โœ… โœ… โœ… โœ…
google - palm โœ… โœ… โœ… โœ…
google AI Studio - gemini โœ… โœ… โœ… โœ…
mistral ai api โœ… โœ… โœ… โœ… โœ…
cloudflare AI Workers โœ… โœ… โœ… โœ…
cohere โœ… โœ… โœ… โœ… โœ…
anthropic โœ… โœ… โœ… โœ…
huggingface โœ… โœ… โœ… โœ… โœ…
replicate โœ… โœ… โœ… โœ…
together_ai โœ… โœ… โœ… โœ…
openrouter โœ… โœ… โœ… โœ…
ai21 โœ… โœ… โœ… โœ…
baseten โœ… โœ… โœ… โœ…
vllm โœ… โœ… โœ… โœ…
nlp_cloud โœ… โœ… โœ… โœ…
aleph alpha โœ… โœ… โœ… โœ…
petals โœ… โœ… โœ… โœ…
ollama โœ… โœ… โœ… โœ… โœ…
deepinfra โœ… โœ… โœ… โœ…
perplexity-ai โœ… โœ… โœ… โœ…
Groq AI โœ… โœ… โœ… โœ…
Deepseek โœ… โœ… โœ… โœ…
anyscale โœ… โœ… โœ… โœ…
IBM - watsonx.ai โœ… โœ… โœ… โœ… โœ…
voyage ai โœ…
xinference [Xorbits Inference] โœ…

Read the Docs

Contributing

To contribute: Clone the repo locally -> Make a change -> Submit a PR with the change.

Here's how to modify the repo locally: Step 1: Clone the repo

git clone https://github.com/BerriAI/litellm.git

Step 2: Navigate into the project, and install dependencies:

cd litellm
poetry install -E extra_proxy -E proxy

Step 3: Test your change:

cd litellm/tests # pwd: Documents/litellm/litellm/tests
poetry run flake8
poetry run pytest .

Step 4: Submit a PR with your changes! ๐Ÿš€

  • push your fork to your GitHub repo
  • submit a PR from there

Enterprise

For companies that need better security, user management and professional support

Talk to founders

This covers:

  • โœ… Features under the LiteLLM Commercial License:
  • โœ… Feature Prioritization
  • โœ… Custom Integrations
  • โœ… Professional Support - Dedicated discord + slack
  • โœ… Custom SLAs
  • โœ… Secure access with Single Sign-On

Support / talk with founders

Why did we build this

  • Need for simplicity: Our code started to get extremely complicated managing & translating calls between Azure, OpenAI and Cohere.

Contributors

ๆœ€่ฟ‘็‰ˆๆœฌๆ›ดๆ–ฐ:(ๆ•ฐๆฎๆ›ดๆ–ฐไบŽ 2024-05-11 21:58:28)

2024-05-11 11:16:52 v1.37.3

2024-05-11 01:51:12 v1.37.2

2024-05-10 09:25:48 v1.37.0.dev2_completion_cost

2024-05-10 09:24:47 v1.37.0.dev_version_headers

2024-05-10 07:56:02 v1.37.0

2024-05-10 01:03:18 v1.36.4-stable

2024-05-09 12:53:02 v1.36.4

2024-05-09 08:15:10 v1.36.3

2024-05-09 00:07:51 v1.36.2-stable

2024-05-08 12:18:04 v1.36.2

ไธป้ข˜(topics):

anthropic, langchain, langchain-python, llm, llmops, openai

BerriAI/litellmๅŒ่ฏญ่จ€ Pythonๆœ€่ฟ‘ๆ›ดๆ–ฐไป“ๅบ“

2024-05-28 04:06:36 Hillobar/Rope

2024-05-27 18:02:22 PKU-YuanGroup/Open-Sora-Plan

2024-05-27 00:30:38 princeton-vl/infinigen

2024-05-26 23:55:33 THU-MIG/yolov10

2024-05-26 15:00:24 linruowuyin/Fhoe-Rail

2024-05-26 13:46:06 jxxghp/MoviePilot