Add Groq model support to LLMClient (#1977) #2165

kunjanshah0811 · 2025-05-12T12:14:38Z

@davidberenstein1957 @henchaves @alexcombessie

Description

This PR adds support for using Groq as an additional LLM client in Giskard, without replacing the default LiteLLM client. It enables developers to integrate Groq models like llama3-8b-8192 by configuring and initializing GroqClient explicitly.

Related Issue

Closes Set any groq model as default LLMClient for giskard #1977

Type of Change

📚 Examples / docs / tutorials / dependencies update
🔧 Bug fix (non-breaking change which fixes an issue)
🥂 Improvement (non-breaking change which improves an existing feature)
🚀 New feature (non-breaking change which adds functionality)
💥 Breaking change (fix or feature that would cause existing functionality to change)
🔐 Security fix

Checklist

I've read the CODE_OF_CONDUCT.md document.
I've read the CONTRIBUTING.md guide.
I've updated the pdm.lock running pdm update-lock (only applicable when pyproject.toml has been
modified)
I've written the docstring in Google format for all the methods and classes that I used.
I've written tests for all new methods and classes that I created.

Usage Example

import os
from dotenv import load_dotenv
from groq import Groq
from giskard.llm.client import ChatMessage
from giskard.llm.client.groq_client import GroqClient

load_dotenv()  # Loads GROQ_API_KEY from .env

# Create Groq client using the API key
client = Groq(api_key=os.environ["GROQ_API_KEY"])
groq_client = GroqClient(client=client)

def test_groq_json_mode():
    messages = [
        ChatMessage(role="system", content="You are a helpful assistant that performs sentiment analysis. "
                    "Return a short summary of the sentiment (Positive, Neutral, or Negative) "
                    "and briefly explain why."),
        ChatMessage(role="user", content="Here's a customer review: 'The product works fine, but the customer service was terrible and slow.'")
    ]
    response = groq_client.complete(messages)
    print("Response:\n", response.content)

if __name__ == "__main__":
    test_groq_json_mode()

…MClient (Giskard-AI#1977)

davidberenstein1957

Looks good already. Some minor remarks :)

giskard/llm/client/__init__.py

giskard/llm/client/groq_client.py

… redundant seed assignment - Added _supports_json_format(model: str) to check for JSON support based on model name. - Now user can specify JSON format via the 'format' parameter instead of hardcoding. - Removed duplicate assignment of extra_params['seed'] = seed in GroqClient.complete()."

davidberenstein1957

Hi, I left some more comments.

giskard/llm/client/__init__.py

giskard/llm/client/groq_client.py

kunjanshah0811

Fixed previous issues and added one comment that needs your attention in groq_client.py
Here https://github.com/Giskard-AI/giskard/pull/2165/files/10a4b68e3841dedb23605c1f5d3f333621942d0f#r2120776154

giskard/llm/client/groq_client.py

davidberenstein1957 · 2025-06-09T07:16:53Z

LFTM, one final thing is to mention it in our docs: https://docs.giskard.ai/en/stable/open_source/setting_up/index.html.

kunjanshah0811 · 2025-06-09T07:32:25Z

LFTM, one final thing is to mention it in our docs: https://docs.giskard.ai/en/stable/open_source/setting_up/index.html.

I will update it by today and let you know. :)

kunjanshah0811

I have added Groq Client in Documentation.

davidberenstein1957 · 2025-06-09T10:13:18Z

@kunjanshah0811 now, we need to make sure all tests pass.

kunjanshah0811 · 2025-06-09T13:05:05Z

@kunjanshah0811 now, we need to make sure all tests pass.

Solved the Linting Error.

davidberenstein1957 · 2025-06-11T07:12:50Z

@kunjanshah0811 there are some remaining errors. Before pushing, could you make sure they pass locally?

kunjanshah0811 · 2025-06-11T16:12:43Z

@kunjanshah0811 there are some remaining errors. Before pushing, could you make sure they pass locally?

I ran both pdm run test and pdm run test-fast locally, and all tests passed without any breaking changes. Please see the attached screenshot for confirmation. Please check and confirm; I think this time the PR is ready to merge into the main branch. ✅

Let me know if anything else is needed from my end. @davidberenstein1957 😀

Note: Some tests require HuggingFace authentication. So, it is done locally via huggingface-cli login.

davidberenstein1957 · 2025-06-18T07:35:02Z

Awesome @kunjanshah0811 . Thanks a lot for the help.

…e-multilingual-MiniLM-L12-v2

kunjanshah0811 · 2025-06-20T06:10:55Z

Awesome @kunjanshah0811 . Thanks a lot for the help.

Fixed with minor errors. @davidberenstein1957 thanks a lot for the review. :)

davidberenstein1957

Some minor tweaks for the dependencies and we are good to go. A lot of thanks for the help.

davidberenstein1957 · 2025-06-20T06:48:57Z

pyproject.toml

@@ -59,7 +61,13 @@ dev = [
    "mistralai>=1",
    "boto3>=1.34.88",
    "scikit-learn==1.4.2",
+    "bert-score>=0.3.13",


why did you duplicate these dependencies here? Is there a specific reason? I'm not sure if we rely on them directly and they might be included in the other libraries, like langchain already?

why did you duplicate these dependencies here? Is there a specific reason? I'm not sure if we rely on them directly and they might be included in the other libraries, like langchain already?

I added those dependencies because I ran into import errors during local testing, and adding them directly resolved the issues at the time. But you're right,other libraries like langchain might already pull in some of them and I may have added them redundantly without double-checking after that. I’m happy to remove any that aren’t needed directly.

pyproject.toml

…ah0811/giskard into add-groq-client-support

henchaves · 2025-06-23T12:13:42Z

giskard/llm/client/__init__.py

I don't think we should set groq as default LLM client.
Could we revert the modifications here?

I don't think we should set groq as default LLM client. Could we revert the modifications here?

Thank you for the feedback. Before I make any changes, I want to ensure I understand your request correctly:

I'll remove Groq from being automatically selected as a default LLM client by modifying the get_default_llm_api() function in __init__.py

I'll keep the following in place so Groq can still be used when explicitly selected:

The Groq client implementation (groq_client.py)

The Groq dependency in pyproject.toml

The test case for the Groq client

"groq" as a valid option in the API list (users can still explicitly set GSK_LLM_API="groq")

Is this understanding correct? If you'd prefer I completely remove all Groq-related changes, please let me know.

Hey @kunjanshah0811, yes exactly!
Just reverting llm/client/__init__.py should be fine, you can keep the groq_client.py as it is.

Hey @kunjanshah0811, yes exactly! Just reverting llm/client/__init__.py should be fine, you can keep the groq_client.py as it is.

@henchaves I have made suggested changes with the recent commit. Please have a look and let me know. Thanks a lot 🚀.

henchaves · 2025-06-23T12:16:46Z

docs/open_source/setting_up/index.md

+# Note: Groq does not currently support embedding models
+# Use another provider for embeddings if needed


Perfect! Thanks.
Could you just mention to access https://docs.litellm.ai/docs/embedding/supported_embedding to see the full list of supported providers?

Perfect! Thanks. Could you just mention to access https://docs.litellm.ai/docs/embedding/supported_embedding to see the full list of supported providers?

I'll add the documentation reference you suggested in docs/open_source/setting_up/index.md linking to LiteLLM's supported embedding providers

yes, thanks!

henchaves · 2025-07-02T15:40:25Z

Thanks a lot @kunjanshah0811! The PR is ready to be merged 🙂

kunjanshah0811 · 2025-07-02T15:50:35Z

Thanks a lot @kunjanshah0811! The PR is ready to be merged 🙂

Thank you so much @davidberenstein1957 and @henchaves for your constant support! 🙏✨
I learnt a lot from you both during my First Open-source Contribution and really appreciate your time and help. 🚀💡
I’m excited to keep learning and contributing to Giskard! 😄🦾
Thanks again for everything! 🚀🤩

feat(llm): add GroqClient support to set any Groq model as default LL…

930099f

…MClient (Giskard-AI#1977)

davidberenstein1957 reviewed May 26, 2025

View reviewed changes

davidberenstein1957 reviewed Jun 2, 2025

View reviewed changes

kunjanshah0811 commented Jun 2, 2025

View reviewed changes

giskard/llm/client/groq_client.py Show resolved Hide resolved

giskard/llm/client/groq_client.py Outdated Show resolved Hide resolved

Fix: Issues of previous comments

97ac6c6

davidberenstein1957 approved these changes Jun 9, 2025

View reviewed changes

Docs: Added Groq Client

b9d3f01

kunjanshah0811 commented Jun 9, 2025

View reviewed changes

Fix: Linting Error

97dc016

Fix: Resolve test failures and ensure all tests pass locally

1659045

kunjanshah0811 and others added 2 commits June 13, 2025 12:35

Merge branch 'main' into add-groq-client-support

3bc5b74

Add get_config in groq_client.py file

4f27293

kunjanshah0811 requested a review from davidberenstein1957 June 14, 2025 11:47

Kunjan Shah added 2 commits June 19, 2025 01:57

Fix: code formatting via pre-commit

ef817bf

Fix: ValueError: Could not load model sentence-transformers/paraphras…

3a6362a

…e-multilingual-MiniLM-L12-v2

kunjanshah0811 closed this Jun 20, 2025

kunjanshah0811 reopened this Jun 20, 2025

davidberenstein1957 added safe for build Lockfile Temporary label to update pdm.lock labels Jun 20, 2025

davidberenstein1957 reviewed Jun 20, 2025

View reviewed changes

Fix: Remove duplicate dependencies and organize LLM-related packages

ff02a9e

davidberenstein1957 added safe for build Lockfile Temporary label to update pdm.lock and removed safe for build Lockfile Temporary label to update pdm.lock labels Jun 23, 2025

davidberenstein1957 reviewed Jun 23, 2025

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

pyproject.toml Outdated Show resolved Hide resolved

pyproject.toml Outdated Show resolved Hide resolved

Apply suggestions from code review

8af4202

davidberenstein1957 closed this Jun 23, 2025

davidberenstein1957 reopened this Jun 23, 2025

Kunjan Shah added 2 commits June 23, 2025 10:08

fix: removed extra spaces in pyproject.toml

a70fbb6

Merge branch 'add-groq-client-support' of https://github.com/kunjansh…

1fb5174

…ah0811/giskard into add-groq-client-support

kunjanshah0811 requested a review from davidberenstein1957 June 23, 2025 08:36

davidberenstein1957 approved these changes Jun 23, 2025

View reviewed changes

davidberenstein1957 requested a review from henchaves June 23, 2025 10:50

henchaves requested changes Jun 23, 2025

View reviewed changes

kunjanshah0811 requested review from henchaves and davidberenstein1957 June 27, 2025 20:41

Kunjan Shah and others added 5 commits July 1, 2025 12:38

refactor(llm): Revert making Groq a default LLM client

7c86562

docs: Add note about Groq embedding support

ff8d141

Merge branch 'main' into add-groq-client-support

c0f3a5b

undo changes in llm/client/__init__

9ad79ec

update docs order

92981aa

henchaves removed the Lockfile Temporary label to update pdm.lock label Jul 2, 2025

henchaves self-assigned this Jul 2, 2025

update pdm.lock

7c5b277

henchaves approved these changes Jul 2, 2025

View reviewed changes

henchaves enabled auto-merge July 2, 2025 21:02

davidberenstein1957 approved these changes Jul 3, 2025

View reviewed changes

henchaves merged commit 6ed690f into Giskard-AI:main Jul 3, 2025
30 of 31 checks passed

		# Note: Groq does not currently support embedding models
		# Use another provider for embeddings if needed

Uh oh!

Add Groq model support to LLMClient (#1977) #2165

Add Groq model support to LLMClient (#1977) #2165

Uh oh!

Conversation

kunjanshah0811 commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue

Type of Change

Checklist

Usage Example

Uh oh!

davidberenstein1957 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davidberenstein1957 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kunjanshah0811 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

davidberenstein1957 commented Jun 9, 2025

Uh oh!

kunjanshah0811 commented Jun 9, 2025

Uh oh!

kunjanshah0811 left a comment

Choose a reason for hiding this comment

Uh oh!

davidberenstein1957 commented Jun 9, 2025

Uh oh!

kunjanshah0811 commented Jun 9, 2025

Uh oh!

davidberenstein1957 commented Jun 11, 2025

Uh oh!

kunjanshah0811 commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidberenstein1957 commented Jun 18, 2025

Uh oh!

kunjanshah0811 commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidberenstein1957 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kunjanshah0811 Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kunjanshah0811 commented May 12, 2025 •

edited

Loading

kunjanshah0811 commented Jun 11, 2025 •

edited

Loading

kunjanshah0811 commented Jun 20, 2025 •

edited

Loading

kunjanshah0811 Jul 1, 2025 •

edited

Loading