1779939 - Something broke with git-cinnabar due to an indexer re-provisioning that the automation didn't detect

Status: RESOLVED FIXED

(Webtools :: Searchfox, defect)

Description

[Andrew Sutherland [:asuth] (he/him)]

All the indexers failed with a stack like this one:

+ echo Updating git
Updating git
+ pushd /mnt/index-scratch/mozilla-build/git
/mnt/index-scratch/mozilla-build/git ~
+ git pull
Traceback (most recent call last):
  File "/usr/local/bin/git-remote-hg", line 34, in <module>
    from cinnabar.githg import GitHgStore
  File "/home/ubuntu/git-cinnabar/cinnabar/githg.py", line 59, in <module>
    from .hg.objects import (
  File "/home/ubuntu/git-cinnabar/cinnabar/hg/objects.py", line 17, in <module>
    if check_enabled('no-mercurial'):
  File "/home/ubuntu/git-cinnabar/cinnabar/util.py", line 137, in __call__
    config = Git.config(self._key) or self._default
  File "/home/ubuntu/git-cinnabar/cinnabar/git.py", line 153, in config
    value = self._config.get(var.lower())
AttributeError: 'NoneType' object has no attribute 'get'

This is very likely due to the re-provisioning I initiated as part of the (no bug because what could go wrong, right?) https://github.com/mozsearch/mozsearch/pull/540 to switch from virtualenv to venv to make the glean build happy and modernize us.

We provision cinnabar here and I had modernized things to use the "release" branch because our use of older pinned releases download the helpers from taskcluster artifacts that tend to go away and break the provisioning process more actively.

The git-cinnabar "master" branch was merged into "release" yesterday morning well before I re-provisioned, and it wouldn't be surprising for it to want a more modern mercurial? The indexer seems to be on 6.1.1 and I see commits in git-cinnabar that mention 6.2.

I'm going to look at the git-cinnabar repo issues and the code a little but a prime option is to roll back our cinnabar version.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

I filed https://github.com/glandium/git-cinnabar/issues/304 to report the issue.

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Landed PR: pin git-cinnabar at 0.5.8 for now

I re-provisioned with cinnabar pinned to 0.5.8 for now, but this is definitely not a long term solution. Going to leave this bug open for now to track but I have to admit I struggled to find a smoking gun when looking at the cinnabar code unless this iterator that automatically clobbers the "_config" field was coming into play, but I also didn't see how that would happen.

I had looked briefly at trying to hook pdb up, but maybe I should have instead considered from rich.traceback import install and then invoked install(show_locals=True) in one of the scripts. That mechanism has been really good at providing context without being quite as intrusive as dropping into a pdb shell.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Landed PR: Update to 0.6.3

As noted in the PR, I probably could have updated us to just use "release" but there is something to be said for specifying the version for determinism (and so docker can understand when things have changed).