b2g hazard and flame/nexus builds dying after "AttributeError: 'GitConfig' object has no attribute 'json'" repo failures

RESOLVED FIXED

Status

Release Engineering
General
--
major
RESOLVED FIXED
3 years ago
a month ago

People

(Reporter: philor, Unassigned)

Tracking

({intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
e.g.

https://treeherder.mozilla.org/logviewer.html#?job_id=9917582&repo=mozilla-inbound

10:44:13 INFO - Deleting broken projects...
10:44:13 INFO - Running command: ['./repo', 'forall', '-c', 'git show-ref -q --head HEAD || rm -rfv $PWD'] in /builds/slave/b2g_m-in_flm-kk_eng-d_dep-0000/build
10:44:13 INFO - Copy/paste: ./repo forall -c "git show-ref -q --head HEAD || rm -rfv $PWD"
10:44:14 INFO - Traceback (most recent call last):
10:44:14 INFO - File "/builds/slave/b2g_m-in_flm-kk_eng-d_dep-0000/build/.repo/repo/main.py", line 506, in <module>
10:44:14 INFO - _Main(sys.argv[1:])
10:44:14 INFO - File "/builds/slave/b2g_m-in_flm-kk_eng-d_dep-0000/build/.repo/repo/main.py", line 482, in _Main
10:44:14 INFO - result = repo._Run(argv) or 0
10:44:14 INFO - File "/builds/slave/b2g_m-in_flm-kk_eng-d_dep-0000/build/.repo/repo/main.py", line 161, in _Run
10:44:14 INFO - result = cmd.Execute(copts, cargs)
10:44:14 INFO - File "/builds/git-shared/repo/repo/subcmds/forall.py", line 200, in Execute
10:44:14 INFO - mirror = self.manifest.IsMirror
10:44:14 INFO - File "/builds/git-shared/repo/repo/manifest_xml.py", line 358, in IsMirror
10:44:14 INFO - return self.manifestProject.config.GetBoolean('repo.mirror')
10:44:14 INFO - File "/builds/git-shared/repo/repo/git_config.py", line 112, in GetBoolean
10:44:14 INFO - v = self.GetString(name)
10:44:14 INFO - File "/builds/git-shared/repo/repo/git_config.py", line 129, in GetString
10:44:14 INFO - v = self._cache[_key(name)]
10:44:14 INFO - File "/builds/git-shared/repo/repo/git_config.py", line 246, in _cache
10:44:14 INFO - self._cache_dict = self._Read()
10:44:14 INFO - File "/builds/git-shared/repo/repo/git_config.py", line 253, in _Read
10:44:14 INFO - self._SaveJson(d)
10:44:14 INFO - File "/builds/git-shared/repo/repo/git_config.py", line 283, in _SaveJson
10:44:14 INFO - if os.path.exists(self.json):
10:44:14 INFO - AttributeError: 'GitConfig' object has no attribute 'json'
10:44:14 ERROR - Return code: 1
(Reporter)

Comment 1

3 years ago
Hard to beat the fun of a 45 minute cloning failure in a 90 minute job. Since one hazard build has successfully finished (after 9 on various trees failed) and one device build has successfully finished (after 13 failed), it's possible that this was happening for a particular period of time and is now not failing. It's also possible that it happens on some set of slaves, or on slaves in a particular state, or on slaves in a particular State, and not on others. Still closed.
(Reporter)

Comment 2

3 years ago
Comment 1 might make you think it is going away on its own; it is not. Still happening, still tree-closing.

The fact that the pushes where it's happening are all one set of merges of the same code might make you think it's in-tree; it is not. https://treeherder.mozilla.org/logviewer.html#?job_id=1516218&repo=mozilla-central and https://treeherder.mozilla.org/logviewer.html#?job_id=1516217&repo=mozilla-central are the same thing on retriggers of a hazard build and a device build on yesterday's m-c merge which was green before, now failing this same way.
(Reporter)

Comment 3

3 years ago
Well, took me a few hours, but now I'm out of any jobs which haven't managed to build on the second, third, or eighth try, so I'm reopening to see whether I gather up a new set.
Severity: blocker → major
Component: Buildduty → General Automation
QA Contact: bugspam.Callek → catlee
Summary: Trunk trees closed for b2g hazard and flame/nexus builds dying after "AttributeError: 'GitConfig' object has no attribute 'json'" repo failures → b2g hazard and flame/nexus builds dying after "AttributeError: 'GitConfig' object has no attribute 'json'" repo failures
(Reporter)

Comment 4

3 years ago
Still happening, but given our general indifference to failures in b2g builds, I won't close again over this.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Reporter)

Updated

3 years ago
Keywords: intermittent-failure
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
massimo, do you know whats going on here ?
Flags: needinfo?(mgervasini)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
We are using https://gerrit.googlesource.com/git-repo/+/master tag v1.12.20.

This version references 'self.json' (it should be 'self._json') and causes:

10:44:14 INFO - File "/builds/git-shared/repo/repo/git_config.py", line 283, in _SaveJson
10:44:14 INFO - if os.path.exists(self.json):
10:44:14 INFO - AttributeError: 'GitConfig' object has no attribute 'json'

v1.12.16 is the last known version without the typo on line 283. Preparing a patch to use .16
Flags: needinfo?(mgervasini)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Created attachment 8607036 [details] [diff] [review]
[buildbot-configs] Bug 1165727 - using git-repo v1.12.16.patch
Attachment #8607036 - Flags: review?(winter2718)
Attachment #8607036 - Flags: review?(winter2718) → review+
Attachment #8607036 - Flags: checked-in+
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Starting with comment 143, this issue was resolved, but we now need to upgrade git fix the remaining "Return code" errors reported in the subsequent comments. Work being done in bug 1165967.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
See Also: → bug 1166000
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
FYI, https://git.mozilla.org/?p=external/google/gerrit/git-repo.git;a=commit;h=b1d1fd778d5d3d3217023df8b428c5b7fa22a337 should fix this, but hasn't made a stable tag yet. Based on previous form it would be in v1.12.23.
(Assignee)

Updated

a month ago
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.