haskell infrastructure: moving forward without peti

[cdepillabout]

@peti posted a thread on Discourse about stopping work on the Haskell infrastructure here in Nixpkgs.

peti wrote a large part of the Haskell infrastructure here in Nixpkgs, and has done a large part of the work in maintaining it over the years. I think a lot of Haskellers who use Nix have directly benefited from all the work peti has done. In my experience, Haskell is one of the nicest languages to use with Nix, and this is in no small part due to peti.

It is really going to hurt having peti step away from Nixpkgs. But we'll need to find a way to move forward with the Haskell infrastructure here in Nixpkgs.

In particular, we need people to step in and do the tasks peti has been doing for the past couple years.

As far as I know, here are the main tasks peti is doing that will need to be taken over:

1. Open a PR every week for the haskell-updates branch that targets master. Here is an example of one of these PRs.
2. Sometime during the week, update the Stackage Nightly target in a commit on haskell-updates. Here is an example of one of these commits.
3. Once a week, make sure to fix up anything on the haskell-updates branch that is not evaluating correctly, and merge the above PR. This is somewhat of a manual task, but I think peti relies on Hydra a lot during this. peti has a Twitch channel where he often does this live: https://www.twitch.tv/peti343. (As an aside, do past videos disappear from twitch at some point? @peti, I thought you had uploaded some videos to YouTube in order to make sure that they are always available, but I can't find them anymore with a quick search. edit: Yes, @sternenseemann confirmed that they are deleted.)
4. Mark Haskell packages as broken when they no longer build on haskell-updates. I think peti relies on Hydra for this as well.
5. Make sure the hackage-updates cron job is running. This generates the daily Hackage updates and commits them to Nixpkgs on the haskell-updates branch. Here is an example of a commit that was generated by this.
6. Review any PRs to hackage2nix (and the related cabal2nix).
7. Make sure that the haskell-updates jobset is working on Hydra: https://hydra.nixos.org/jobset/nixpkgs/haskell-updates. (I'm not actually sure what goes into this or how much work it really is.)

(If I've forgotten anything important, please feel free to edit this post and to this list.)

Aside from peti, the main committers that have been active with the haskell-updates branch have been me, @maralorn, and @sternenseemann . I'm imagining that one of us would take over these tasks. Or possibly split it up between us.

Other people that I think might be interested in the Haskell infrastructure here in Nixpkgs: @domenkozar, @Ericson2314, @jonringer, @bgamari, @roberth, @angerman, @hamishmack, @nh2

---

As an aside, I'd like to suggest that in this thread, we focus on figuring out who personally will be responsible for the above tasks peti used to perform.

It is tempting to also discuss various ways the haskell-updates workflow could be changed, but we should probably keep that to a separate thread. If you'd like o discuss something that would be a significant change, please open up a separate thread (and post a link to it here).

[sternenseemann]

3. Once a week, make sure to fix up anything on the haskell-updates branch that is not evaluating correctly, and merge the above PR. This is somewhat of a manual task, but I think peti relies on Hydra a lot during this. peti has a Twitch channel where he often does this live: https://www.twitch.tv/peti343. (As an aside, do past videos disappear from twitch at some point? @peti, I thought you had uploaded some videos to YouTube in order to make sure that they are always available, but I can't find them anymore with a quick search.)

IIRC, the recordings are deleted after a few weeks automatically.

5. Make sure the hackage-updates cron job is running. This generates the daily Hackage updates and commits them to Nixpkgs on the haskell-updates branch. Here is an example of a commit that was generated by this.

@peti has agreed to keep this up for a few more weeks, maybe we can migrate this to a github action in the mean time to make it more maintainable for us?

6. Review any PRs to hackage2nix (and the related cabal2nix).

I'd be happy to be more involved with this if it's fine with @peti.

7. Make sure that the haskell-updates jobset is working on Hydra: https://hydra.nixos.org/jobset/nixpkgs/haskell-updates. (I'm not actually sure what goes into this or how much work it really is.)

I think it's manageable, but I'd be great if the current active haskell maintainers (@cdepillabout @maralorn, me) could be given access to the jobset plus some basic hydra privileges (restart-jobs, cancel-build could also be useful).

Aside from peti, the main committers that have been active with the haskell-updates branch have been me, @maralorn, and @sternenseemann . I'm imagining that one of us would take over these tasks. Or possibly split it up between us.

I don't think I have the time regularly enough to take over, but I'm happy to contribute to this. If we want to make the final fixup phase a more joint effort we maybe need to think about rescheduling what was the livestreams as 20:00 UTC+2 may line up for @maralorn and me well, it probably doesn't for you.

As an aside, I'd like to suggest that in this thread, we focus on figuring out who personally will be responsible for the above tasks peti used to perform.

It is tempting to also discuss various ways the haskell-updates workflow could be changed, but we should probably keep that to a separate thread. If you'd like o discuss something that would be a significant change, please open up a separate thread (and post a link to it here).

I agree to this sentiment: We should focus on keeping the show running as is for now. However I think we should organize it in a way that it doesn't hinge on a single person (to avoid a situation like this in the future).

[cdepillabout]

A couple additional questions I have for @peti:

• Would you be able to make me, @maralorn, and @sternenseemann committers in cabal2nix so that we could make any required changes to hackage2nix? (edit: It looks like this has happened, thanks! https://github.com/orgs/NixOS/teams/haskell/members)
• I know of https://github.com/peti/ci, and I think there is some relationship to Hdyra, but I've really understood the how the two work together? Would you be able to give a quick explanation of what this repo is, and how it affects haskell-updates on Hydra? Would it make sense for me, @maralorn, and @sternenseemann to have commit rights here too?

[cdepillabout]

Here are my responses to @sternenseemann's message in #121140 (comment):

peti has agreed to keep [the hackage-updates cron script] up for a few more weeks, maybe we can migrate this to a github action in the mean time to make it more maintainable for us?

Oh, a GitHub Action is a really good idea. Here's some documentation about scheduled actions: https://jasonet.co/posts/scheduled-actions/. Implementing this should probably be split off to a separate issue.

Make sure that the haskell-updates jobset is working on Hydra

I think it's manageable, but I'd be great if the current active haskell maintainers (@cdepillabout @maralorn, me) could be given access to the jobset plus some basic hydra privileges (restart-jobs, cancel-build could also be useful).

Yes, I definitely agree with this.

However, I brought it up before and it didn't really lead anywhere: https://discourse.nixos.org/t/how-to-get-privileges-on-hydra-to-restart-builds/9757

I imagine it might be difficult for us to continue working on the haskell-updates branch if none of us have the ability to do anything on Hydra.

I'm not sure who to continue to ask about this though. Maybe @domenkozar or @zimbatm?

---

I also wanted to post about how I feel moving forward.

Up until now, I've mostly been helping people who send PRs to the haskell-updates branch, but not actively involved in the stuff peti does (clean up the haskell-updates branch and merge it into master). I would be completely happy not being involved with the work that goes into cleaning up and merging in the haskell-updates branch. However, if no one else is going to do it, I would definitely take up the reigns. I might want to switch to a twice-monthly merge schedule instead of a weekly merge schedule to reduce the workload though.

However I think we should organize it in a way that it doesn't hinge on a single person (to avoid a situation like this in the future).

I completely agree with this. Ideally we'd like to find some workflow that doesn't completely hinge on a single person. But for now, it seems like it might be easiest to have a single person who is ultimately responsible to get haskell-updates merged into master (along with everything that implies).

[maralorn]

I think since cabal2nix belongs to the nix org, we already have commit rights there.

@cdepillabout I am sorry, but I feel that can‘t talk about the who without the how.

I personally am in the end phase of my thesis and am not able to take on new obligations in the next year.
I think petis workflow was petis workflow. Finding someone to be the new peti seems not like the perfect solution.

I have a lot of ideas for the future of haskellPackages but I agree, that we should postpone getting crazy with that. (That being said I would love it if some kind of task force came together and would discuss this more).

A suggestion I have, would be to break free of the weekly schedule. And make the cleanup more a joined effort. I think one way to lessen the churn would be to stop doing nightly hackage updates. I would do one stackage+hackage update, clean up the PR over a few days and merge it. nixpkgs master will be a few days more behind hackage. But that seems totally okay to me on a volunteer basis.

Also I think we should push quickly in the direction that all packages someone cares about have a maintainer, so that the person doing the PR only needs to do a bit of bookkeeping and pinging everyone with broken packages. I think the most annoying thing about haskellPackages is that as the one doing the merge you have to arbitrarily judge which packages matter and which don‘t. I want this process to become easier.

Having a joint jitsi call for merging together or at least once in a while seems nice.

One idea for load balancing: What do you think about aiming for a merge about every 2 weeks and cycling through the merge-manager? I think I might commit to doing this every 6 weeks.

One minor point not to forget: After pushing to master we need to update the Nixos version of packages on hackage. Someone needs access to do that.

[sternenseemann]

I'm not sure who to continue to ask about this though.

Hydra role stuff is documented here (a bit hidden tbh), I think the best person to talk to is @grahamc.

I think since cabal2nix belongs to the nix org, we already have commit rights there.

No, the commit access is broken up into several teams and nixpkgs and cabal2nix are separate teams (if at all).

I think the most annoying thing about haskellPackages is that as the one doing the merge you have to arbitrarily judge which packages matter and which don‘t. I want this process to become easier.

I think we need to formalize this process: I think we need to a) compile a list of important™ haskellPackages and b) if possible add basic tests for them (via pkgs/tests). Ideally this would be a single nix file the merge manager would run to have a sanity check on the state of haskell-updates. I have been meaning to work towards such a thing for a while and could commit myself to implementing that.

One idea for load balancing: What do you think about aiming for a merge about every 2 weeks and cycling through the merge-manager? I think I might commit to doing this every 6 weeks.

I think rotating the responsibility is a great idea and I'd be up for that as well.

Concerning the frequency: I think we need to consider if this works well with us tracking the nightly solver (I don't have any opinion on this tbh). If we reduce the frequency of the merge we need to consider merging fixups into master. Maybe we could add an additional configuration.nix file which allows usage of markUnbroken and such, so we don't have to regenerate the package set every time. This file would then be cleaned up again as part of every merge.

[maralorn]

I think we need to formalize this process: I think we need to a) compile a list of important™ haskellPackages and b) if possible add basic tests for them (via pkgs/tests). Ideally this would be a single nix file the merge manager would run to have a sanity check on the state of haskell-updates. I have been meaning to work towards such a thing for a while and could commit myself to implementing that.

This sounds great. I am here if you want to bikeshed the details! Maybe we can make a hydra-job out of it?

Concerning the frequency: I think we need to consider if this works well with us tracking the nightly solver (I don't have any opinion on this tbh). If we reduce the frequency of the merge we need to consider merging fixups into master. Maybe we could add an additional configuration.nix file which allows usage of markUnbroken and such, so we don't have to regenerate the package set every time. This file would then be cleaned up again as part of every merge.

Huh, that sounds bad. I think regenerating haskell-packages is not that hard, right? Also, merging fixups into master should be rare, right?

[sternenseemann]

This sounds great. I am here if you want to bikeshed the details! Maybe we can make a hydra-job out of it?

Yeah that should be fairly easy, either a job with multiple constituents (like the channel jobs), or we just use the test to the list of jobs the hydra jobset builds. I'd have to check how the jobset is setup to figure out the best way.

Huh, that sounds bad. I think regenerating haskell-packages is not that hard, right? Also, merging fixups into master should be rare, right?

It isn't, but extra hassle (you need to clone cabal2nix, clone two repositories into specific subdirectories, run the script, …) and currently not very well known or documented. My intention is mostly to make the story of X isn't working in haskellPackages, but it's easy to fix better and ideally remove the n weeks until merge delay. However, this has also the risk of increasing the amount of work for us, so maybe this is something for later.

[roberth]

I can't commit more time at the moment. I'll think about it though.

It isn't, but extra hassle (you need to clone cabal2nix, clone two repositories into specific subdirectories, run the script, …)

That can be solved by pinning the tools used in nixpkgs itself. I've created NixOS/cabal2nix#431 for this, some time ago. It didn't get much support at the time because it didn't match the cabal2nix-repo-based workflow. We should support both though. One because it's easy to use for casual contributors and the other because cabal2nix/hackage2nix need to be developed as well.

[zimbatm]

👍 for the initiative.

I created a Haskell team and put @peti as the maintainer so he can invite other people: https://github.com/orgs/NixOS/teams/haskell/members
That team has maintainer access on the cabal2nix project.

/cc @kosmikus and @shlevy who were involved in cabal2nix.

[sternenseemann]

That can be solved by pinning the tools used in nixpkgs itself. I've created NixOS/cabal2nix#431 for this, some time ago. It didn't get much support at the time because it didn't match the cabal2nix-repo-based workflow. We should support both though. One because it's easy to use for casual contributors and the other because cabal2nix/hackage2nix need to be developed as well.

Sounds like a good idea having something for this in maintainers/scripts actually. Maybe something to work on in the next few weeks!

[maralorn]

Sounds like a good idea having something for this in maintainers/scripts actually. Maybe something to work on in the next few weeks!

Yeah, until now peti had a certain opposition against it because it touched his workflows. But we are free to make our own workflows now.

[jonringer]

I'd be happy to help at times. But I think I'm already checking some haskell builds on staging-next branches :)

[domenkozar]

Except for the fixing of breakages - which is the majority of work, we could automate the process with GitHub Actions.

I have admin access to Hydra so I can help with restarts, etc.

[peti]

@grahamc, could you please give Hydra privileges to the Haskell maintainer folk? I think they need the ability to maintain the https://hydra.nixos.org/jobset/nixpkgs/haskell-updates jobset, i.e. start an evaluation, re-start failed/aborted builds, and change the configuration of the jobset itself. Is that possible?

[peti]

FYI:

$ cat update-haskell-package-list
#! /usr/bin/env bash

set -eu -o pipefail
trap "rm haskell-in-nixpkgs.csv-old haskell-in-nixpkgs.csv" 0

cd ~/src/package-list

cabal -v0 v2-run generate-nixpkgs-haskell-package-list >haskell-in-nixpkgs.csv
curl --silent  http://hackage.haskell.org/distro/NixOS/packages.csv >haskell-in-nixpkgs.csv-old
perl -p -i -e 'chomp if eof' haskell-in-nixpkgs.csv
if ! diff -uw haskell-in-nixpkgs.csv-old haskell-in-nixpkgs.csv; then
  echo ""
  echo "Press RETURN to upload, otherwise abort with CTRL-C ..."
  read idontcare
  curl -u "PeterSimons:$(pass show hackage.haskell.org | head -1)" --digest -H "Content-type: text/csv" -T haskell-in-nixpkgs.csv http://hackage.haskell.org/distro/NixOS/packages.csv
else
  echo "no changes"
fi

The executable that generates the CSV file lives at https://github.com/peti/package-list.

[maralorn]

Okay, @sternenseemann @cdepillabout. What do you think about doing this in a rotation roughly every two weeks?

I can volunteer for doing the first merge. I will start updating stuff on the weekend 8./9. of May and merge it until 15./16., maybe even earlier if everything is just easy to fix. Then someone of you could do the next round?

All the while we can keep in close contact about how things are going and what we want to optimise.

[maralorn]

In the discussion we had in the stream today @sternenseemann and I had two ideas, that would be nice and probably not difficult to implement soon. I don‘t expect a discussion here, but I wanted to record them.

1. Let's couple the hackage version we use to the all-cabal-hashes we pin in nixpkgs.
2. Let's have a github action that regenerates hackage-packages.nix after every push to haskell-updates (if there are changes).

Other things off note from the stream today:

• haskell-updates on hydra just needs one file from peti/ci. We can just move that file to nixpkgs when we feel like it and then just once someone needs to redirect the hydra job to the new config file.
• peti claims that anyone with a hackage login can update that csv as shown above.
• peti stopped the cronjob doing the nightly updates. It was a very simple cronjob. If we want it again it would be trivial to setup e.g. by me but probably a github action would be better.

[maralorn]

@NixOS/haskell What do you think about blessing on room on an arbitrary chat-medium where we can do low-level coordination?

[zimbatm]

If you want I can also enable discussions on the teams page at https://github.com/orgs/NixOS/teams/haskell. If that's helpful to coordinate.

[cdepillabout]

Let me see if I can summarize the discussion up until now with respect to our suggested workflow moving forward.

The members of the Haskell team (for now, me, @sternenseemann, and @maralorn) will take turns fixing up the haskell-updates branch and merging it into master. This process will take place over the span of two weeks. The following steps describe this process:

1. Opening a PR for merging the haskell-updates branch into master.
2. Immediately after opening the PR, do the following:
   1. Update the stackage resolver in pkgs/development/haskell-modules/configuration-hackage2nix.yaml and the pins for all the packages from Stackage Nightly. Commit this. (I think we don't have a script to do this automatically, but we probably should. edit: There is a script for this in add maintainer scripts for haskell package generation #121391 called maintainers/scripts/haskell/update-stackage.sh.)
   2. Update the all-cabal-hashes file in pkgs/data/misc/hackage/default.nix to point to the latest state of Hackage. Commit this. (edit: There a script that does this automatically in add maintainer scripts for haskell package generation #121391 called maintainers/scripts/haskell/update-hackage.sh.)
   3. Push these two commits to GitHub on the haskell-updates branch.
3. hackage2nix will be automatically run by a GitHub action. The changes here will be automatically be committed and pushed to the haskell-updates branch by the GitHub action. (I imagine in the first week or month or so, we may not have this github action, so we may have to do this by hand. edit: Below we discussed that we may or may not want a GitHub action for this. There is a script to do this in add maintainer scripts for haskell package generation #121391 called maintainers/scripts/haskell/regenerate-hackage-packages.sh.)
4. This should cause the hackage-updates jobset on Hydra to run. This might take a couple hours to finish? After this runs, we should get a list of newly broken Haskell packages, along with evaluation errors.
5. (This might happen a day or so later...) For any packages that are newly broken on Hydra, check if there is a maintainer for the package in pkgs/development/haskell-modules/configuration-hackage2nix.yaml. If there is, then ping the maintainer on the PR opened in step 1. (This seems like it would be quite annoying to do in practice, so it would be great to automate this. I imagine this would be non-trivial to automate though.)
6. Over the course of the next two weeks, people will send PRs to the haskell-updates branch which unbreak packages. These may be packages that were newly broken, or they may be packages that have been marked broken for a long time.
7. After any PRs merged into haskell-updates, hackge2nix will run automatically in the GitHub action. This is quite convenient. Alternatively, people sending PRs are also free to run hackage2nix within their PRs with the maintainers/scripts/haskell/regenerate-hackage-packages.sh script.
8. At the end of the two weeks, the maintainer is responsible for doing the following:
   1. Fix up any evaluation errors on haskell-updates. See this blog post for a way to check this locally. We absolutely don't want any evaluation errors on master.
   2. Mark any packages Haskell packages that are still failing to build on Hydra as broken. (It'd be nice to have a script to do this automatically.) Make sure to commit and push. The GitHub Action will run which will regenerate hackage-packages.nix and mark things as newly broken. Possibly wait for Hydra to run and confirm that there are now no non-broken packages that fail to build.
   3. Merge the above haskell-updates PR into master.

(In practice, we could probably have each maintainer be responsible for steps 4 to 8, and then steps 1 and 2 immediately after step 8.)

@maralorn @sternenseemann Does my understanding mirror what you guys are thinking?

---

One question I have is whether or not we should merge master into the hackage-updates branch at any point during this process. I'm worried that the haskell-updates branch/PR will effectively be behind master by 2 weeks by the point we merge it in, so it may cause mass rebuilds in Haskell packages on master. For example, if one of the dependencies of GHC has changed.

I think peti would sometimes rebase haskell-updates on master, but I'm not sure of the reasons for this, or what would cause him to do this.

[cdepillabout]

What do you think about blessing on room on an arbitrary chat-medium where we can do low-level coordination?

This sounds like a good idea! My only request is that we don't do IRC.

A couple possibilities:

• the GitHub team discussions page that zimbatm offered
• a channel on the NixOS discord (I think @maralorn is on discord, but I don't know about @sternenseemann)
• some slack channel that we can all join?

I don't really have a strong opinion. Although it might be nice not to have something that is completely private, so that other people interested in what we're doing can follow along.

---

Oh, and I think there was a suggestion that we do a video chat or something in the next couple of weeks to make sure we're all on the same page with the above workflow. That definitely sounds like a good idea.

What time zones are you guys in? I'm in JST (UTC+9).

[maralorn]

Excellent write-up @cdepillabout. This is very close to what I would envision. Couple of remarks:

Immediately after opening the PR, do the following: ,

Yeah, we should have scripts for that. Working on that right now based on #86699.

hackage2nix will be automatically run by a GitHub action. The changes here will be automatically be committed and pushed to the `haskell-updates branch by the GitHub action. (I imagine in the first week or month or so, we may not have this github action, so we may have to do this by hand.)

I am not one hundred percent sure about the github action anymore. It sounds convenient, but when we can tell people please run "maintainers/scripts/haskell/regenerate-haskell-packages.sh" as part of your commit. It has the big advantage that people can actually test their commit and it doesn‘t feel like a huge burden. Also we clutter the commit history with less commits. But we can of course add this github action at any point in time.

(This might happen a day or so later...) For any packages that are newly broken on Hydra, check if there is a maintainer for the package in pkgs/development/haskell-modules/configuration-hackage2nix.yaml. If there is, then ping the maintainer on the PR opened in step 1. (This seems like it would be quite annoying to do in practice, so it would be great to automate this. I imagine this would be non-trivial to automate though.)

I am actually trying right now to automate this. It ought to be possible. But no promises.

One question I have is whether or not we should merge master into the hackage-updates branch at any point during this process.

Yes that is a good point. My thoughts are this:

• I am not sure about the exact timings anyway, e.g. when to fork off haskell-updates and when to update hackage. If we do both immediately after the merge we guarantee that we are always two weeks out-dated. I could imagine rather fixing the start of the PR than the end. e.g. every two weeks. Fork master, do the update, initiate stabilisation period and merge as soon as everything looks fine.

• No matter how we do it, merging master in about two or three days before intended merge is a good idea. (But also not super important, we can never be sure that no one master breaks our packages anyway.)

• I only have a strong opinion that merging master into haskell-updates should generally be avoided shortly before merging because it increases the risk of mass hydra rebuild and flying blind.

• peti sometimes did rebase stuff to make the history cleaner. I think we should just not do that. If we don‘t have the nightly cron-job our history will look nicer anyways.

[maralorn]

This sounds like a good idea! My only request is that we don't do IRC.

A couple possibilities:

• the GitHub team discussions page that zimbatm offered
• a channel on the NixOS discord (I think @maralorn is on discord, but I don't know about @sternenseemann)
• some slack channel that we can all join?

My favorite would be a matrix channel. But anything that we can bridge to matrix would be fine. So discord would be okay for me.

I don't really have a strong opinion. Although it might be nice not to have something that is completely private, so that other people interested in what we're doing can follow along.

100% on-board. Let’s be as transparent as possible. Let’s invite everyone to join! Let’s make good documentation! And let’s distribute the maintenance on as much shoulders as possible! (It‘s not even that I don‘t want to do the work, it’s that together the results will be better!)

[maralorn]

Oh, and I think there was a suggestion that we do a video chat or something in the next couple of weeks to make sure we're all on the same page with the above workflow. That definitely sounds like a good idea.

What time zones are you guys in? I'm in JST (UTC+9).

sterni and I are both CEST (UTC+2). I guess the question is: What is the latest that works for you? I guess something like 13:00 UTC on a weekend might work?

(Side note: Sorry for everyone being spammed by this issue.)

[sternenseemann]

haskell-updates on hydra just needs one file from peti/ci. We can just move that file to nixpkgs when we feel like it and then just once someone needs to redirect the hydra job to the new config file.

I'm looking into redoing that and moving it into pkgs/top-level/release-haskell.nix.

One question I have is whether or not we should merge master into the hackage-updates branch at any point during this process. I'm worried that the haskell-updates branch/PR will effectively be behind master by 2 weeks by the point we merge it in, so it may cause mass rebuilds in Haskell packages on master. For example, if one of the dependencies of GHC has changed.

We need to merge master in definitely, as any dependency of GHC changing would cause a mass rebuild. I'd say we add haskell-updates to https://github.com/NixOS/nixpkgs/blob/master/.github/workflows/merge-staging.yml / make a similar action for this.

I only have a strong opinion that merging master into haskell-updates should generally be avoided shortly before merging because it increases the risk of mass hydra rebuild and flying blind.

Note that this mass rebuild is unavoidable. If we push back merging master, the mass rebuild will happen after the merge. I think with the live streams being a thing of the past for now, we don't have to make a specific time window, so we can just wait for hydra to finish and merge whenever everything is green on hydra to our satisfaction.

Update the all-cabal-hashes file in pkgs/data/misc/hackage/default.nix to point to the latest state of Hackage. Commit this. (Is there a script that does this automatically??)

I'm a bit concerned about this to be honest. It does alleviate some annoyances with the update process, but we then need some way to update specific packages to a version newer than the snapshot we chose: For instance due to security updates or if a new release fixes some build issue we encountered.

I'm not sure if merging a two week old hackage snapshot is always a good idea, but I'm happy to try it out at least.

---

I only have a Discord account of the three, but I guess if we make this an official haskell nixpkgs coordination things something open seems more appropriate.

[maralorn]

Note that this mass rebuild is unavoidable. If we push back merging master, the mass rebuild will happen after the merge. I think with the live streams being a thing of the past for now, we don't have to make a specific time window, so we can just wait for hydra to finish and merge whenever everything is green on hydra to our satisfaction.

This is true.

I'm a bit concerned about this to be honest. It does alleviate some annoyances with the update process, but we then need some way to update specific packages to a version newer than the snapshot we chose: For instance due to security updates or if a new release fixes some build issue we encountered.

My suggestion would be, that if that’s needed, we will just bump it once more and see what happens. No reason to be dogmatic about only updating once. I just don‘t want to update automatically everyday so that we constantly get new problems.

[cdepillabout]

We had a meeting on the NixOS Discord today. Here is a rough list of what we discussed:

Scripts for Maintanance

In #121391 @maralorn put together a few scripts for various maintainance tasks related to updating Haskell packages here in Nixpkgs. For example, bumping the Stackage packages, updating the Hackage pin, updating the Hackage packages.

There was a question as to whether or not we want separate scripts that also make a Git commit after bumping/updating. I don't know if we came to a very strong consensus, but it did seem like it might be nice to make sure our commit messages are always the same. Although it also seemed like we might want to run each script without creating a new commit.

One way to solve this would be to add an optional --do-commit flag to each script. If you pass this flag, then the script does a git commit at the end of the script with a good commit message.

Hydra

@sternenseemann put together a PR that updates the Hydra release file for the haskell-updates branch: #121407

We need someone to change the configuration fo the haskell-updates jobset on Hydra. After merging #121407, @sternenseemann is going to try to get @grahamc to do this. (@sternenseemann, if you can't get a hold of @grahamc, please raise that issue here and we'll try to figure out how we can make this happen) (This is all done, probably no intervention necessary in the near future).

It appears that if you sign into Hydra with your GitHub account, you are able to trigger a re-evaluation at any time. The Haskell maintainers will probably frequently use this feature when maintenance work.

Workflow

We had various discussions on what we want our workflow to look like moving forward. It seems like we all don't want to have a lot of hard-and-fast rules, but try to play things by ear. Here is what we did decide to:

• The people in the @NixOS/haskell team would each take responsibility of merging haskell-updates into master once every two weeks. This means that during every two week period, the person who's turn it is needs to merge haskell-updates into master at least once. It doesn't matter when you do it during the two week period.
• Since there are 3 of us in the team, we would each be ultimately responsible for this once every 6 weeks.
• If one of us gets busy, then he will let the other team members know and someone else will fill in.
• If you update hackage-packages.nix and one of the Haskell packages breaks that has a maintainer, then you have to ping that maintainer on the haskell-updates PR and tell them it is broken. You need to give them at least 7 days to respond before merging haskell-updates to master. If they don't respond to fix the problem, then you mark their package as broken.
• You don't necessarily have to wait 7 days if you're willing to fix the problem yourself. Also, you're free to merge haskell-updates into master more than just once every two weeks as long as you're careful to make sure nothing big has started to break.
• You can only merge haskell-updates into master when the mergeable job from pkgs/top-level: init expression for haskell-updates jobset  #121407 has passed on Hydra.
• At the start of your 2 week period, you should probably bump the Stackage Nightly version along with Hackage packages.

The initial rotation will go:

1. @maralorn (2021-05-01 to 2021-05-14)
2. @cdepillabout (2021-05-15 to 2021-05-28)
3. @sternenseemann (2021-05-29 to 2021-06-11)

At the end of your rotation, create a new PR for haskell-updates and tag the next person so they know it is their turn.

Documentation

@maralorn created an issue about documentation: #121403.

We also discussed that at the very least we'd like to get our workflow documented for the haskell-updates branch. @maralorn might work on this during the next two weeks. If he doesn't get to it, then I will do it during my two weeks.

Future Improvements

We talked about various future improvements we'd like to make. This is mostly stuff that would help with our maintenance of Haskell stuff in Nixpkgs.

• @maralorn is currently writing a script for querying hydra for broken builds. The script then queries maintainers for those packages. It will produce a message that can be copy-and-pasted to GitHub to warn maintainers that their Haskell packages are now broken. @maralorn also wants to expand the maintainers in configuration-hackage2nix file. Moving forward, we'd like to almost completely stop caring about Haskell packages in Nixpkgs without maintainers (well, except for the "important" packages like lens, aeson, etc).

• @maralorn is writing a related script for automatically marking Haskell packages as broken based on their most recent hydra status.

• We discussed that it would be nice to rewrite hackage2nix to automatically mark packages as broken that have transitive dependencies that are broken.

  There are a couple problems with this:

  • We could just not do this. This means that if a package is only broken because it has transitive dependencies that are broken, then we don't mark it as broken. This is apparently okay in terms of Hydra evaluations. However, it makes Hydra's output harder to understand (since it is hard to tell exactly which packages need to be fixed).
  • When a package's transitive dependencies are broken, sometimes it is just a dependency of the tests that is broken. However, we may disable the tests in configuration-common.nix, so hackage2nix doesn't know that the tests can be ignored. One way to work around this is to have a list of packages that need tests disabled in configuration-hackage2nix.yaml, but we don't want this file to get too complicated.

• We think it would be possible to jailbreak packages automatically in hackage2nix. There are of course dangers with doing this.

• We should be able to auto-generate a list of "support levels" for Haskell packages. For instance, packages with a maintainer that compile successfully, packages without a maintainer but known to compile, packages without a maintainer but jailbroken, packages without a maintainer and with tests checks disabled (this is the scary category).

  However, we're not sure who exactly would want this information, so the priority of this is probably lower than the other potential improvements.

[maralorn]

Great write up!!

One thing we didn‘t discuss much, but is very important to me: We should try to be very open and transparent about our process. I would also very much like for more people to join the rotation. 3 people is probably fine. more than 5 or 6 is probably too much. But we should communicate a standing invitation for more people to join.

Besides that there are of course a lot of other things people can do to help with haskellPackages. Documentation would probably be one of the most important areas.

[teto]

One thing I've wished for is for the different language ecosystems to converge and for that effort, I wonder if pkgs/development/haskell-modules/configuration-common.nix (which I find too abstract, I have no idea what it does) could be renamed to pkgs/development/haskell-modules/overrides.nix similar to pkgs/misc/vim-plugins/overrides.nix / pkgs/development/lua-modules/overrides.nix ?
same for pkgs/development/haskell-modules/hackage-packages.nix , it could become pkgs/development/haskell-modules/generated-packages.nix

[cdepillabout]

@teto That sounds like a good idea! Please feel free to open a separate issue for this and tag it haskell.

[cdepillabout]

We've basically figured out how we want to move forward with the Haskell infrastructure, so I am going to go ahead and close this issue.

Our workflow is more-or-less described in the following places:

• Some formal documentation for our workflow: haskell-updates: add documentation for workflow #122728
• Some notes from maralorn that the above documentation is based on: Update haskellPackages including a bump of Hackage and Stackage pin #122510 (comment)

---

If anyone had any other suggestions about the Haskell infrastructure in Nixpkgs, please feel free to create separate issues for them.