Using a simple export or .env file to allow local LLMs to be used more easily and readily

[rosegarden-coder]

Right now, it is so cumbersome to use ollama. One has to do a very long command line like this:

 pqa --llm "ollama/cas/llama3.1-8b-spaetzle-v74" --llm_config '{"model_list": [{"model_name": "ollama/cas/llama3.1-8b-spaetzle-v74", "litellm_params": {"model": "ollama/cas/llama3.1-8b-spaetzle-v74", "api_base": "http://localhost:11434"}}]}' --summary_llm "ollama/cas/llama3.1-8b-spaetzle-v74" --summary_llm_config '{"model_list": [{"model_name": "ollama/cas/llama3.1-8b-spaetzle-v74", "litellm_params": {"model": "ollama/cas/llama3.1-8b-spaetzle-v74", "api_base": "http://localhost:11434"}}]}' ask 'What is...?'   

Or using the script with quick fix described in #787.

Why does ollama has to be the second class citizen? I am using the 70b local models and granite3-dense:8b for embedding with solid results.

As I read the tutorial, it say:

This tutorial aims to show how to use the Settings class to configure PaperQA. Firstly, we will be using OpenAI and Anthropic models, so we need to set the OPENAI_API_KEY and ANTHROPIC_API_KEY environment variables. We will use both models to make it clear when paperqa agent is using either one or the other. We use python-dotenv to load the environment variables from a .env file. Hence, our first step is to create a .env file and install the required packages.

Why can't ollama models be set in an .env file? paper-qa is a gem, but ollama could be more easily and readily used with it.

[jamesbraza]

You are right we can do better here @rosegarden-coder , I agree and thanks for raising this. +1 to this