Caching does not work when using an internal package repo

[screig]

Description:
The caching feature wont work if one is pip installing from another python package repository.

Action version:
actions/setup-python@v5

Platform:

• [X ] Ubuntu
• macOS
• Windows

Runner type:

• [X ] Hosted
• Self-hosted
  Current runner version: '2.321.0'
  Runner Image
  Image: ubuntu-22.04
  Version: 20241201.1.0

Tools version:
I think it applies across Python versions

Repro steps:
The code is on our internal repo so I can share a link, here is the yaml

First we set up Python, and specify that we wish to cache.

      - uses: actions/setup-python@v5
        with:
          python-version: '3.9'
          cache: 'pip'
          cache-dependency-path: requirements*.txt

Next I install two sets of requirements

      - name: Install dependencies f
        run: |
          echo "Python version is : ${{ matrix.python-version }}"
          pip --version
          pip install -r requirements_for_testing.txt        
          pip install -r requirements.txt --extra-index-url https://${ado_token}@OUR_INETERNAL_ADO_PACKAGE REPO/_packaging/sepypi/pypi/simple/
          pip --version
          pip list

requirements_for_testing.txt contains packages that we can get from pypi. Now here the caching works fine.

Next we need to install requirements that include our internal packages on our internal ADO package repo.

pip install -r requirements.txt --extra-index-url https://${ado_token}@OUR_INETERNAL_ADO_PACKAGE REPO/_packaging/sepypi/pypi/simple/

Here the caching never works.

Expected behaviour:
I would expect it to cache.

Actual behaviour:
Its not caching...

[gowridurgad]

Hello @screig,
Thank you for creating this issue. We will investigate it and provide feedback as soon as we have some updates.

[lmvysakh]

Hi @screig ,

After investigating, we identified several issues causing the caching to fail, such as secret handling and syntax errors in the pip install command. Initially, the following command was used:

pip install -r requirements.txt --extra-index-url https://${ado_token}@OUR_INETERNAL_ADO_PACKAGE_REPO/_packaging/sepypi/pypi/simple/

However, this format was invalid for pip install. We made the following fix:

pip install -r requirements.txt --extra-index-url https://${{secrets.ado_token}}@OUR_INETERNAL_ADO_PACKAGE_REPO.git@main#egg=simple&subdirectory=_packaging/sepypi/pypi/simple

This resolved the caching issue, but there was a deprecation warning with --extra-index-url. To address this, we updated the commands as follows:

pip install -r requirements.txt --extra-index-url https://${{ secrets.ado_token }}@https://github.com
pip install git+@OUR_INETERNAL_ADO_PACKAGE_REPO.git@main#egg=simple&subdirectory=_packaging/sepypi/pypi/simple

We addressed the following issues to resolve the caching problem:

1. Ensured that secrets are correctly referenced using ${{ secrets.ado_token }} in the workflow file to avoid "bad substitution" errors.
2. Used the git+https syntax to install packages directly from the repository.

If these workarounds do not resolve the issue, please provide a link to the build or the public repository to help us further investigate.

[screig]

Hi

The repository I am referencing is provided by Azure and is intended to provide packages (wheels) and is not a code-repository and therefore is not using git.

It looks rather like this

Image from here, not mine.

As in this image, the service (Azure Dev Ops / Azure Artifacts) tells me the address to connect to, in order to be able to install packages using pip, as so:

I am not connecting to a git (code) repository where I have the option to use git+https syntax.

Here is the azure documentation for the package repo service. I believe across the Microsoft/Github world this is the only service that can provide a python package repo.

The GitHub Package Registry cannot handle Python package and I think GitHub has ruled out supporting Python packages in the future through this.

[lmvysakh]

Hi @screig,

I'd like to provide some clarity on the caching capabilities supported by the setup-python action and its limitations:

The setup-python action supports caching for Python packages installed from the public PyPIrepository. Additionally, if you have a requirements.txt file, the action can cache the dependencies listed in it. However, caching may not work seamlessly for packages installed from internal or private repositories, such as Azure Artifacts.

You can find the documentation for the same below:

• Caching dependencies to speed up workflows
• Building and testing Python

However, we are suggesting a workaround to access the Azure Artifacts repository by the following:

1. Create an Azure Artifacts feed in your Azure DevOps project.
2. Generate a Personal Access Token (PAT) with the Packaging scope in Azure DevOps.
3. Store the PAT as a secret in your GitHub repository (e.g., AZURE_ARTIFACTS_PAT).

import os
import subprocess

# Set up the environment variable for the PAT
os.environ['AZURE_ARTIFACTS_PAT'] = 'your_personal_access_token'

# Define the URL for the Azure Artifacts feed
feed_url = "https://pkgs.dev.azure.com/your_organization/your_project/_packaging/your_feed/pypi/simple/"

# Configure pip to use the Azure Artifacts feed
subprocess.run([
    'pip', 'install', '--extra-index-url', f"https://{os.environ['AZURE_ARTIFACTS_PAT']}:@{feed_url}",
    'your_package_name'
], check=True)

Here is an example GitHub Actions workflow that sets up Python, configures access to the Azure Artifacts feed, and installs packages:

name: Python CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Check out repository
      uses: actions/checkout@v3

    - name: Set up Python
      uses: actions/setup-python@v5
      with:
        python-version: '3.9'

    - name: Install dependencies from Azure Artifacts
      env:
        AZURE_ARTIFACTS_PAT: ${{ secrets.AZURE_ARTIFACTS_PAT }}
      run: |
        pip install --upgrade pip
        pip config set global.extra-index-url "https://${AZURE_ARTIFACTS_PAT}:@pkgs.dev.azure.com/your_organization/your_project/_packaging/your_feed/pypi/simple/"
        pip install -r requirements.txt

1. Replace placeholders like your_personal_access_token, your_organization, your_project, your_feed, and your_package_name with your actual details.
2. Ensure that the AZURE_ARTIFACTS_PAT secret is properly set in your GitHub repository.

[screig]

Hi

I tried this approach of setting the global index url

However I can see no difference

When using the internal package repo the action is still not caching...