🔥 Excited to launch support for Logging LLM I/O on 🔭 Galileo through LiteLLM (YC W23) Proxy https://docs.litellm.ai/docs/proxy/logging#logging-llm-io-to-galielo

📈 [docs] New example Grafana Dashboards https://github.com/BerriAI/litellm/tree/main/cookbook/litellm_proxy_server/grafana_dashboard

🛡️ feat - control guardrails per api key https://docs.litellm.ai/docs/proxy/guardrails#switch-guardrails-onoff-per-api-key

🛠️ fix - raise report Anthropic streaming errors (thanks David Manouchehri)

✨ [Fix] Add nvidia nim param mapping based on model passed

What's Changed

• fix(anthropic.py): add index to streaming tool use by @igor-drozdov in https://github.com/BerriAI/litellm/pull/4554
• (fix) fixed bug with the watsonx embedding endpoint by @simonsanvil in https://github.com/BerriAI/litellm/pull/4540
• Revert "(fix) fixed bug with the watsonx embedding endpoint" by @krrishdholakia in https://github.com/BerriAI/litellm/pull/4561
• [docs] add example Grafana Dashboard by @ishaan-jaff in https://github.com/BerriAI/litellm/pull/4563
• build(deps): bump certifi from 2023.7.22 to 2024.7.4 by @dependabot in https://github.com/BerriAI/litellm/pull/4568
• fix(proxy/utils.py): support logging rejected requests to langfuse, etc. by @krrishdholakia in https://github.com/BerriAI/litellm/pull/4564
• [Feat] Add Galileo Logging Callback by @ishaan-jaff in https://github.com/BerriAI/litellm/pull/4567
• [Fix] Add nvidia nim param mapping based on model by @ishaan-jaff in https://github.com/BerriAI/litellm/pull/4565
• fix - raise report Anthropic streaming errors by @ishaan-jaff in https://github.com/BerriAI/litellm/pull/4566
• feat - control guardrails per api key by @ishaan-jaff in https://github.com/BerriAI/litellm/pull/4569

New Contributors

• @igor-drozdov made their first contribution in https://github.com/BerriAI/litellm/pull/4554

Full Changelog: https://github.com/BerriAI/litellm/compare/v1.41.7...v1.41.8

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.41.8

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results