Closed Bug 1935484 Opened 1 year ago Closed 1 year ago

Set up basic mozsearch github actions-based automated test runs for trusted pull requests to run `make build-test-repo` and `make webtest`

Categories

(Webtools :: Searchfox, task)

Product:
Webtools
Component:
Searchfox
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: asuth, Assigned: nicolas.guichard)

References

Details

Attachments

(1 file)

Mozsearch Pull Request
47 bytes, text/x-github-pull-request
Details | Review

Since the pace of contributions to searchfox has increased, and especially with contributors potentially having to juggle multiple stacks and periodic changes to provisioning that can require re-provisioning, this has occasionally resulted in checks getting out of date. I think it's probably a good time to add CI to run our tests on trusted PRs, especially since recent research determined that it should be free for us to use GitHub actions to run some basic CI[1].

My main concern historically for GitHub actions CI was the latency/potential cost of effectively re-provisioning on every PR, especially since livegrep builds can be slow. But I think this is a combination of something we can mitigate or a situation where we potentially do want to be running re-provisioning every time. Specific thoughts:

  • Because we don't pin all of the things we download / install in provisioning, having provisioning ~freshly run on new commits is desirable since it can disambiguate what's going on with rust build warnings and the like. People should be able to be confident that the CI is probably right and that it may be necessary to locally re-provision.
  • GitHub actions has a caching mechanism that stores up to 10 GiB. docker image ls tells me my mozsearch docker image is 15.6 GB which means we couldn't cache the docker image, but we could potentially cache, for example, the livegrep build which is something we do explicitly pin and so can key based on the git hash.
    • There is an artifacts mechanism which could potentially store docker images, but there's a note about downloading artifacts that says "You can only download artifacts in a workflow that were uploaded during the same workflow run." And it doesn't seem worth trying to game that since...
  • Our provisioned docker image is not actually something we want people to use or for us to distribute. We use docker as a means to get a linux VM. This means that we potentially can avoid using docker at all, but instead run our provisioning scripts just on top of the underlying linux VM. The set of github runners includes an ubuntu-24.04 for example.
    • An unfortunate complication is the runner there says it has only 14GB of storage, so if my numbers about our docker image using up 15.6GB are accurate, we potentially may need to have the provisioning steps be a bit more aggressive about cleaning up as it goes, maybe cleaning the apt package cache and/or removing the ubuntu system clang, etc.

1: And even if it wasn't free, GitHub actions might still be the most pragmatic choice right now given that it's not clear we have taskcluster-level needs.

Assignee: nobody → nicolas.guichard
Status: NEW → ASSIGNED
Attached file Mozsearch Pull Request —

I have also toyed with a Nix-based workflow at https://github.com/nicolas-guichard/mozsearch/commits/nixified.

It has some benefits over the existing provisioning scripts:

  • packages and dependencies are properly defined
  • some packages are already provided by nixpkgs, such as git-cinnabar
  • dependencies are locked, either by the flake.lock at the root the of the repo for remote package repositories (nixpkgs and fenix) or by hashes in each locally-defined package
  • caching at the package level is trivial using https://github.com/DeterminateSystems/magic-nix-cache-action
  • it is easier to reproduce locally, at least graphviz on CI and my local graphviz seem to agree
  • it doesn't clone and build the mozsearch repo nearly as much

The big downside is that it diverges from the provisioning scripts used by the local docker image and the AWS production servers, which is certainly an issue for CI purposes.

(In reply to Nicolas Guichard from comment #2)

I have also toyed with a Nix-based workflow at https://github.com/nicolas-guichard/mozsearch/commits/nixified.

This is very interesting to see what our real-world situation looks like for this. Thanks!

The big downside is that it diverges from the provisioning scripts used by the local docker image and the AWS production servers, which is certainly an issue for CI purposes.

The nix workflow does seem complex enough (noting that it goes further than what our bash scripts do) that I think if we went nix we'd want to go all-in on nix so we don't experience divergence.

:emilio, :arai, do you have thoughts/opinions on this? I'm on the fence because it's clear to me that the current shell scripting solution is a pile of hacks combined with foot-guns; every time I go to re-provision, something has usually broken. But other than my choice of using "mise" (formerly rtx) to handle node.js which I regret (see https://github.com/mozsearch/mozsearch/pull/856#issue-2741454901), and some dependency games of whack-a-mole when upgrading between Ubuntu releases, it hasn't been horrible. Searching slack shows there are a few intrepid nix users at Mozilla, although I think just as many people using it as a verb.

Flags: needinfo?(emilio)
Flags: needinfo?(arai.unmht)

I don't have experience with Nix and I'm not sure if I follow. Am I correct with the following understanding?

  • Nix mentioned here is the package manager, instead of NixOS
  • We keep using Ubuntu as a distribution/disk image for local VM and AWS
  • Also we use Ubuntu for GitHub action
  • For GitHub action and local VM, we use Nix for installing all packages, instead of apt
  • For AWS, there's some issue with switching to Nix (or at least it requires some more extra work), and we keep using apt

If the above is correct, what's the details with the Nix on AWS ?

Flags: needinfo?(arai.unmht)

Nix mentioned here is the package manager, instead of NixOS

Yes, for instance see what the scip-typescript or mozsearch-clang-plugin packages look like.

We keep using Ubuntu as a distribution/disk image for local VM, AWS and GitHub action

Yes, ideally the underlying distribution shouldn't matter. It doesn't have to be NixOS if that's your question.

For GitHub action and local VM, we use Nix for installing all packages, instead of apt
For AWS, there's some issue with switching to Nix (or at least it requires some more extra work), and we keep using apt
If the above is correct, what's the details with the Nix on AWS ?

Just to be clear: that's not what the PR above implements. The PR above keeps using the existing provisioning scripts. This is just something I've looked at on-the-side because I think Nix is a great tool for that kind of use-case.

For now in my draft branch I only covered the GitHub Actions case as an example, not the local VM nor AWS cases.

For the local VM case it's mostly a matter of replacing the provisioning scripts with “install Nix”, and changing the Makefile to call nix run '.?submodules=1#test' for make build-test-repo and nix run '.?submodules=1#webtest' for make webtest. That's what the GitHub Action does here.

For AWS there are more changes required indeed:

  • The provisioning step would again just be installing Nix. To avoid rebuilding livegrep et al each time we would need to cache built packages. That cache could also be shared with the GitHub Actions CI. Reproducible builds + caching would basically replace the explicit AMI provisioning step.
  • The repos in the config repository would have to specify dependencies via Nix instead of calling apt in their setup script.
  • The index.sh script would call something like nix run github:mozearch/mozsearch-mozilla#index-wubkat -- /mnt/index-scratch.

I think assuming that installing nix is straight-forward on the vm this looks good, and probably a lot more reproducible than our current provisioning scripts.

For the local VM case it's mostly a matter of replacing the provisioning scripts with “install Nix”, and changing the Makefile to call nix run '.?submodules=1#test' for make build-test-repo and nix run '.?submodules=1#webtest' for make webtest. That's what the GitHub Action does here.

I think it'd be worth to have a makefile that just forwards to nix just for convenience? I don't think I'd be able to come up with that without having used nix before-hand :)

Flags: needinfo?(emilio)

Thank you for the explanation!

Yes, this looks reasonable, and I agree with moving to nix on all cases.

Then I think we have consensus on moving to nix for everything!

I've r+'ed https://github.com/mozsearch/mozsearch/pull/857 with that understanding. I've also reviewed the nix branch from above; I have a few brief notes here that is probably just telling you things you already know/are aware of, but I mention them to avoid losing track:

I've made progress on the “nixification” to the point that running the tests is just a nix flake check <bla bla> call and indexing mozilla-central is a nix run 'git+https://github.com/nicolas-guichard/mozsearch-mozilla.git?ref=nixified#index-just-mc' -- target_dir call.

I think it'd be worth to have a makefile that just forwards to nix just for convenience? I don't think I'd be able to come up with that without having used nix before-hand :)

Yes, definitely, especially since mozsearch requires some peculiar things (it requires submodules (easy) and access to its own .git (hacky because of https://github.com/NixOS/nix/issues/6900)).

The Nix Flake: Add scip-python commit appears to be using stock scip-python and not my mini-fork that makes the python indexing work with m-c.

Thanks for noticing, fixed.

Question: In Use 127.0.0.1 instead of localhost, what's the specific motivation (and can that be put in a commit message)?

In my first implementation of root-less nginx I used port 8080, which was a bad choice given codesearch uses the same port. Using IPv4 fixed it “by luck” because nginx only listened on IPv4 while codesearch listened on IPv6.

Also I discussed with a nixpkgs maintainer.
The packages for scip-typescript and wasm-snip could be upstreamed, meaning we could benefit from their CI hydra.nixos.org and binary cache cache.nixos.org instead of rebuilding them ourselves.
The upstream versions of scip-python and livegrep could be upstreamed too, but our forked versions wouldn't benefit from cache.nixos.org.
scip-java is more complicated to upstream because I used https://github.com/zaninime/sbt-derivation and nixpkgs doesn't seem to have an equivalent yet.

I'm marking this as done as we do have a “basic GitHub Actions-based CI” now.
I've opened bug 1945159 to continue tracking the Nix port.

Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Regressions: 1959582
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: