-
Notifications
You must be signed in to change notification settings - Fork 0
Load phare dataset from HF for reproducibility #2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some cleaning need in the config and missing a couple of recent modification for tools handling.
api_key = os.getenv("VLLM_API_KEY") | ||
base_url = generation_kwargs.pop("base_url") | ||
|
||
model = LiteLLMModel( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I pushed some recent change (end of last week) to improve tool support in the configs and in the 02_run_benchmark.py, can you add them here as well please?
In principle these changes have been integrated in LMEval and Elie merged on main. But maybe you need to update the commit ref in the pyproject.toml
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it this commit: google/lmeval@1ec1905 ?
If so, it's ok since we are having its child commit ;)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, fine then!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Using
huggingface_hub
to download the jsonl as needed. Because thetink
(dep of lmeval) since version1.9
has a crashing issue and a mutex blocking issue withpyarrow
(base of HFdatasets
lib) on macOS.The core idea is to have
hf_dataset
anddata_path
in each categories:And for each tasks, it tries to find
f"{data_path}/{task_name}.jsonl"
to download and load.It seems to be good to load every public categories: