v1.57.5

🚨🚨 Known issue - do not upgrade - Window's compatibility issue on this release

Relevant issue: #7677

What's Changed

• LiteLLM Minor Fixes & Improvements (01/08/2025) - p2 by @krrishdholakia in #7643
• Litellm dev 01 08 2025 p1 by @krrishdholakia in #7640
• (proxy - RPS) - Get 2K RPS at 4 instances, minor fix for caching_handler by @ishaan-jaff in #7655
• (proxy - RPS) - Get 2K RPS at 4 instances, minor fix aiohttp_openai/ by @ishaan-jaff in #7659
• (proxy perf improvement) - use uvloop for higher RPS (10%-20% higher RPS) by @ishaan-jaff in #7662
• (Feat - Batches API) add support for retrieving vertex api batch jobs by @ishaan-jaff in #7661
• (proxy-latency fixes) use asyncio tasks for logging db metrics by @ishaan-jaff in #7663

Full Changelog: v1.57.4...v1.57.5

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.57.5

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	230.0	282.70225500655766	6.115771768544881	0.0	1830	0	206.44150200001832	3375.4479410000044
Aggregated	Passed ✅	230.0	282.70225500655766	6.115771768544881	0.0	1830	0	206.44150200001832	3375.4479410000044

v1.57.4

What's Changed

• fix(utils.py): fix select tokenizer for custom tokenizer by @krrishdholakia in #7599
• LiteLLM Minor Fixes & Improvements (01/07/2025) - p3 by @krrishdholakia in #7635
• (feat) - allow building litellm proxy from pip package by @ishaan-jaff in #7633
• Litellm dev 01 07 2025 p2 by @krrishdholakia in #7622
• Allow assigning teams to org on UI + OpenAI omni-moderation cost model tracking by @krrishdholakia in #7566
• (fix) proxy auth - allow using Azure JS SDK routes as llm_api_routes by @ishaan-jaff in #7631
• (helm) - bug fix - allow using migrationJob.enabled variable within job by @ishaan-jaff in #7639

Full Changelog: v1.57.3...v1.57.4

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.57.4

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	200.0	218.7550845980808	6.268875045928877	0.0	1876	0	170.9488330000113	1424.4913769999812
Aggregated	Passed ✅	200.0	218.7550845980808	6.268875045928877	0.0	1876	0	170.9488330000113	1424.4913769999812

v1.57.3

Full Changelog: v1.57.2...v1.57.3

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.57.3

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	240.0	273.577669278204	6.101109800829093	0.0	1826	0	209.38834100002168	2450.7287210000186
Aggregated	Passed ✅	240.0	273.577669278204	6.101109800829093	0.0	1826	0	209.38834100002168	2450.7287210000186

v1.57.2

What's Changed

• Prompt Management - support router + optional params by @krrishdholakia in #7594
• aiohttp_openai/ fixes - allow using aiohttp_openai/gpt-4o by @ishaan-jaff in #7598
• (Fix) security of base image by @ishaan-jaff in #7620
• Litellm dev 01 07 2025 p1 by @krrishdholakia in #7618
• (Feat) soft budget alerts on keys by @ishaan-jaff in #7623
• LiteLLM Minor Fixes & Improvement (01/01/2025) - p2 by @krrishdholakia in #7615

Full Changelog: v1.57.1...v1.57.2

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.57.2

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	190.0	212.2353391522645	6.34173008698281	0.0	1898	0	174.4866640000282	3470.5951910000013
Aggregated	Passed ✅	190.0	212.2353391522645	6.34173008698281	0.0	1898	0	174.4866640000282	3470.5951910000013

v1.57.1.dev1

What's Changed

• Prompt Management - support router + optional params by @krrishdholakia in #7594
• aiohttp_openai/ fixes - allow using aiohttp_openai/gpt-4o by @ishaan-jaff in #7598

Full Changelog: v1.57.1...v1.57.1.dev1

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.57.1.dev1

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	230.0	257.3333526549675	6.19223606950461	0.003343539994332943	1852	1	91.99427400000104	1634.9057550000339
Aggregated	Passed ✅	230.0	257.3333526549675	6.19223606950461	0.003343539994332943	1852	1	91.99427400000104	1634.9057550000339

v1.57.1

What's Changed

• (perf) - fixes for aiohttp handler to hit 1K RPS by @ishaan-jaff in #7590
• (latency/perf fixes - proxy) - use async_service_success_hook by @ishaan-jaff in #7591
• (Feat) - allow including dd-trace in litellm base image by @ishaan-jaff in #7587
• (proxy perf improvement) - remove redundant .copy() operation by @ishaan-jaff in #7564
• Refresh VoyageAI models, prices and context by @fzowl in #7472
• LiteLLM Minor Fixes & Improvements (01/06/2025) - p3 by @krrishdholakia in #7596
• LiteLLM Minor Fixes & Improvements (01/06/2025) - p2 by @krrishdholakia in #7597

Full Changelog: v1.57.0...v1.57.1

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.57.1

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.57.1

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	250.0	286.96666935492755	6.035628429692609	0.0	1806	0	226.66728699999794	3887.529271000062
Aggregated	Passed ✅	250.0	286.96666935492755	6.035628429692609	0.0	1806	0	226.66728699999794	3887.529271000062

v1.57.0-dev1

What's Changed

• (perf) - fixes for aiohttp handler to hit 1K RPS by @ishaan-jaff in #7590
• (latency/perf fixes - proxy) - use async_service_success_hook by @ishaan-jaff in #7591
• (Feat) - allow including dd-trace in litellm base image by @ishaan-jaff in #7587
• (proxy perf improvement) - remove redundant .copy() operation by @ishaan-jaff in #7564
• Refresh VoyageAI models, prices and context by @fzowl in #7472
• LiteLLM Minor Fixes & Improvements (01/06/2025) - p3 by @krrishdholakia in #7596
• LiteLLM Minor Fixes & Improvements (01/06/2025) - p2 by @krrishdholakia in #7597

Full Changelog: v1.57.0...v1.57.0-dev1

v1.57.0

What's Changed

• (Fix) make sure init custom loggers is non blocking by @ishaan-jaff in #7554
• (Feat) Hashicorp Secret Manager - Allow storing virtual keys in secret manager by @ishaan-jaff in #7549
• Create and view organizations + assign org admins on the Proxy UI by @krrishdholakia in #7557
• (perf) fix [PROXY] don't use f string in add_litellm_data_to_request() by @ishaan-jaff in #7558
• fix(groq/chat/transformation.py): fix groq response_format transforma… by @krrishdholakia in #7565
• Support deleting keys by key_alias by @krrishdholakia in #7552
• (proxy perf improvement) - use asyncio.create_task for service_logger_obj.async_service_success_hook in pre_call by @ishaan-jaff in #7563
• add fireworks_ai/accounts/fireworks/models/deepseek-v3 by @Fredy in #7567
• FriendliAI: Documentation Updates by @minpeter in #7517
• Prevent istio injection for db migrations cron job by @lowjiansheng in #7513

New Contributors

• @Fredy made their first contribution in #7567
• @minpeter made their first contribution in #7517

Full Changelog: v1.56.10...v1.57.0

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.57.0

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	200.0	212.84027329611826	6.1961289027318704	0.0	1854	0	174.45147399996586	1346.3216149999653
Aggregated	Passed ✅	200.0	212.84027329611826	6.1961289027318704	0.0	1854	0	174.45147399996586	1346.3216149999653

v1.56.10

What's Changed

• fix(aws_secret_manager_V2.py): Error reading secret from AWS Secrets … by @krrishdholakia in #7541
• Support checking provider-specific /models endpoints for available models based on key by @krrishdholakia in #7538
• feat(router.py): support request prioritization for text completion c… by @krrishdholakia in #7540
• (Fix) - Docker build error with pyproject.toml by @ishaan-jaff in #7550
• (Fix) - Slack Alerting , don't send duplicate spend report when used on multi instance settings by @ishaan-jaff in #7546
• add cohere/command-r7b-12-2024 by @ishaan-jaff in #7553

Full Changelog: v1.56.9...v1.56.10

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.56.10

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	230.0	268.3301603401397	6.21711064668469	0.0	1861	0	212.36320399998476	3556.7401620000396
Aggregated	Passed ✅	230.0	268.3301603401397	6.21711064668469	0.0	1861	0	212.36320399998476	3556.7401620000396

v1.56.9

What's Changed

• (fix) GCS bucket logger - apply truncate_standard_logging_payload_content to standard_logging_payload and ensure GCS flushes queue on fails by @ishaan-jaff in #7519
• (Fix) - Hashicorp secret manager - don't print hcorp secrets in debug logs by @ishaan-jaff in #7529
• [Bug-Fix]: None metadata not handled for _PROXY_VirtualKeyModelMaxBudgetLimiter hook by @ishaan-jaff in #7523
• Bump anthropic.claude-3-5-haiku-20241022-v1:0 to new limits by @Manouchehri in #7118
• Fix langfuse prompt management on proxy by @krrishdholakia in #7535
• (Feat) - Hashicorp secret manager, use TLS cert authentication by @ishaan-jaff in #7532
• Fix OTEL message redaction + Langfuse key leak in logs by @krrishdholakia in #7516
• feat: implement support for limit, order, before, and after parameters in get_assistants by @jeansouzak in #7537
• Add missing prefix for deepseek by @SmartManoj in #7508
• (fix) aiohttp_openai/ route - get to 1K RPS on single instance by @ishaan-jaff in #7539
• Revert "feat: implement support for limit, order, before, and after parameters in get_assistants" by @krrishdholakia in #7542
• [Feature]: - allow print alert log to console by @ishaan-jaff in #7534
• (fix proxy perf) use _read_request_body instead of ast.literal_eval to get better performance by @ishaan-jaff in #7545

New Contributors

• @jeansouzak made their first contribution in #7537
• @SmartManoj made their first contribution in #7508

Full Changelog: v1.56.8...v1.56.9

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.56.9

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	240.0	269.3983699320639	6.149252570882109	0.0	1840	0	211.95807399999467	2571.210135000001
Aggregated	Passed ✅	240.0	269.3983699320639	6.149252570882109	0.0	1840	0	211.95807399999467	2571.210135000001