Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate ./dtox.sh image builds. #2233

Open
jsirois opened this issue Aug 31, 2023 · 1 comment
Open

Automate ./dtox.sh image builds. #2233

jsirois opened this issue Aug 31, 2023 · 1 comment
Assignees
Labels
in progress tech-debt Issue that addresses technical debt.

Comments

@jsirois
Copy link
Member

jsirois commented Aug 31, 2023

Most of the infra is in place:

  • The ./dtox.sh script builds the base image only if needed, it just doesn't push the image.
  • The cache image used by CI was built and pushed by hand via tox -e build-cache-image -- --push.

The base image should be built in-line with CI (if needed), but the cache image need not be and is more expensive to build. That can probably be cronned at a period of ~1 week to start.

@jsirois jsirois added the tech-debt Issue that addresses technical debt. label Aug 31, 2023
@jsirois jsirois self-assigned this Feb 8, 2025
@jsirois
Copy link
Member Author

jsirois commented Feb 8, 2025

d365641 set up a cron job for this but it fails, hitting the GitHub 6 hour time limit.

I'm going to proceed instead with the idea of parallelizing the image build by only building some (or one) tox env per-shard, doing a docker export, then combing tarballs from the docker exports into one big one that is docker imported into 1 image in a fan-in step. A small experiment shows this works to consolidate a cache volume dir from multiple docker builds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in progress tech-debt Issue that addresses technical debt.
Projects
None yet
Development

No branches or pull requests

1 participant