1949233 - Pure git checkout should be able to use artifact builds (requires being able to find artifacts based on git revision + repo only)

Status: NEW

(Developer Infrastructure :: General, task)

Description

[:Gijs (he/him)]

After some discussion on slack, filing this somewhat pre-emptively (ie I'm very aware this isn't supported yet, prior to moving to git "proper") to make sure it's tracked going forward - please move in the bug tree and prod/comp as appropriate.

The basic gist of it is that right now if you use a pure git m-c repo (and don't install cinnabar), then ./mach build for artifact builds doesn't work. It complains in the middle of a cinnabar command:

$ ./mach build
 0:00.47 W Clobber not needed.
 0:00.51 W Adding make options from /home/gijs/gecko-dev/mozconfig
    MOZ_OBJDIR=/home/gijs/gecko-dev/obj-x86_64-pc-linux-gnu
    OBJDIR=/home/gijs/gecko-dev/obj-x86_64-pc-linux-gnu
    FOUND_MOZCONFIG=/home/gijs/gecko-dev/mozconfig
    export FOUND_MOZCONFIG
  Parallelism determined by memory: using 16 jobs for 16 cores based on 15.6 GiB RAM and estimated job size of 1.0 GiB
 0:00.51 /usr/bin/gmake -f client.mk -j16 -s
 0:08.95 git: 'cinnabar' is not a git command. See 'git --help'.
 0:09.00 git: 'cinnabar' is not a git command. See 'git --help'.
 0:09.00 Error running mach:
 0:09.00     mach --log-no-times artifact install
 0:09.00 The error occurred in code that was called by the mach command. This is either
 0:09.00 a bug in the called code itself or in the way that mach is calling it.
 0:09.00 You can invoke ``./mach busted`` to check if this issue is already on file. If it
 0:09.00 isn't, please use ``./mach busted file artifact`` to report it. If ``./mach busted`` is
 0:09.00 misbehaving, you can also inspect the dependencies of bug 1543241.
 0:09.00 If filing a bug, please include the full output of mach, including this error
 0:09.00 message.
 0:09.00 The details of the failure are as follows:
 0:09.10 subprocess.CalledProcessError: Command '['/usr/bin/git', 'cinnabar', 'git2hg', 
 0:09.10   File "/home/gijs/gecko-dev/python/mozbuild/mozbuild/artifact_commands.py", line 196, in artifact_install
 0:09.10     return artifacts.install_from(source, distdir or command_context.distdir)
 0:09.10            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:09.10   File "/home/gijs/gecko-dev/python/mozbuild/mozbuild/artifacts.py", line 1697, in install_from
 0:09.10     return self.install_from_recent(distdir)
 0:09.10            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:09.10   File "/home/gijs/gecko-dev/python/mozbuild/mozbuild/artifacts.py", line 1593, in install_from_recent
 0:09.10     return self._install_from_hg_pushheads(hg_pushheads, distdir)
 0:09.10            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:09.10   File "/home/gijs/gecko-dev/python/mozbuild/mozbuild/artifacts.py", line 1565, in _install_from_hg_pushheads
 0:09.10     for trees, hg_hash in hg_pushheads:
 0:09.11   File "/home/gijs/gecko-dev/python/mozbuild/mozbuild/artifacts.py", line 1401, in _find_pushheads
 0:09.11     last_revs = self._get_recent_public_revisions()
 0:09.11                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:09.11   File "/home/gijs/gecko-dev/python/mozbuild/mozbuild/artifacts.py", line 1340, in _get_recent_public_revisions
 0:09.11     return self._get_hg_revisions_from_git()
 0:09.11            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:09.11   File "/home/gijs/gecko-dev/python/mozbuild/mozbuild/artifacts.py", line 1288, in _get_hg_revisions_from_git
 0:09.11     hg_hash_list = subprocess.check_output(
 0:09.11                    ^^^^^^^^^^^^^^^^^^^^^^^^
 0:09.11   File "/usr/lib/python3.12/subprocess.py", line 466, in check_output
 0:09.11     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
 0:09.11            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:09.11   File "/usr/lib/python3.12/subprocess.py", line 571, in run
 0:09.11     raise CalledProcessError(retcode, process.args,
 0:09.11 gmake[3]: *** [Makefile:130: recurse_artifact] Error 1
 0:09.11 gmake[2]: *** [/home/gijs/gecko-dev/config/recurse.mk:34: artifact] Error 2
 0:09.11 gmake[1]: *** [/home/gijs/gecko-dev/config/rules.mk:359: default] Error 2
 0:09.11 gmake: *** [client.mk:60: build] Error 2
 0:09.30 W 0 compiler warnings present.

(in fact, reading that log, it complains about several cinnabar commands...)

I expect there would need to be some changes to the various mach bits to not rely on cinnabar, but more abstractly, some way to figure out the "right" artifact from just a git rev or branch/stack (which I expect will need changes elsewhere, like how things get published in taskcluster or whatever).

Stating the obvious, a lot of frontend engineers rely on local artifact builds so it's important that this keeps working in a non-cinnabar git world.

Comment 1

[Mike Hommey [:glandium]]

Since mercurial changesets will have their origin git commit hard-coded as extra data after the switch-over, my suggestion would be to have the decision task add, for each route that currently uses a mercurial sha, another git-sha-based route based on that extra data. Then mach artifact would be able to use that route.