1263082 - Partner repack failures for Firefox 45.0.2

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

So far on Linux and Linux64, but probably all given they all run on mac:

http://archive.mozilla.org/pub/firefox/candidates/45.0.2-candidates/build1/logs/release-mozilla-release-linux64_partner_repack-bm82-build1-build10.txt.gz
http://archive.mozilla.org/pub/firefox/candidates/45.0.2-candidates/build1/logs/release-mozilla-release-linux_partner_repack-bm84-build1-build11.txt.gz

21:34:08     INFO - retry: Calling _repo_init with args: ('/builds/slave/rel-m-rel-lx_pner_rpk-00000000/build/repo',), kwargs: {}, attempt #1
21:34:08     INFO - Running command: ['/builds/slave/rel-m-rel-lx_pner_rpk-00000000/build/repo', 'init', '--no-repo-verify', '-u', '[e-mail]:mozilla-partners/repack-manifests.git'] in /builds/slave/rel-m-rel-lx_pner_rpk-00000000/build
21:34:08     INFO - Copy/paste: /builds/slave/rel-m-rel-lx_pner_rpk-00000000/build/repo init --no-repo-verify -u [e-mail]:mozilla-partners/repack-manifests.git
21:34:08     INFO -  error: "" unsupported
21:34:08    ERROR - Return code: 1

Then retry until max attempts hit.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Looks like our copy of repo hasn't changed this year, nor the manifest since 45.0.1, nor anything obviously related in mozharness in-tree. And it works on the command line, and after a retry. Hrmph. Any ideas ?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Oops, meant to say it still fails on build machines on a retry.

Comment 3

[Kim Moir [:kmoir] ET]

I don't see the repo here https://github.com/mozilla-partners/ but maybe it is private? Or perhaps the github keys changed?

Comment 4

[Chris Cooper [:coop] (he/him)]

(In reply to Kim Moir [:kmoir] from comment #3)
> I don't see the repo here https://github.com/mozilla-partners/ but maybe it
> is private? Or perhaps the github keys changed?

The repo is indeed private, but accessible from automation using the ffxbld creds. No changes on the github side recently, but perhaps access has changed on our side after the sec incident?

As reported on IRC, I'm able to manually clone the repack-manifests repo from github from one of the affected machines (bld-lion-r5-072) using the ffxbld key.

Comment 5

[Chris Cooper [:coop] (he/him)]

Here's the successful run from 45.0.1 if we want to start digging into differences:

http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/builders/release-mozilla-release-macosx64_partner_repack/builds/13

Comment 6

[Kim Moir [:kmoir] ET]

Perhaps the git upgrade in bug 1257633 caused some issues

<rail> I think this is repo not being able to recognize git version
2:39 PM <rail> https://github.com/mozilla/git-repo/blob/master/repo#L310-L313
2:39 PM <rail> and we upgraded git recently

2:59 PM <rail> something tells me this is a stdout/stderr issue
3:00 PM <rail> _CheckGitVersion expect the version in stdout
3:00 PM <rail> but git prints to stderr

Comment 7

[Chris Cooper [:coop] (he/him)]

Digging in to the error that's being reported, i.e. error: "" unsupported, it looks like it's a git version check that's failing: 

https://github.com/mozilla/git-repo/blob/master/repo#L312

Here's a quick test I ran on bld-lion-r5-002 to see which version of git we're pulling in:

[[e-mail] ~]$ which -a git
/usr/bin/git
/usr/local/bin/git
[[e-mail] ~]$ /usr/bin/git --version
git version 1.7.4.4
[[e-mail] ~]$ /usr/local/bin/git --version
Illegal instruction: 4
[[e-mail] ~]$ ls -la /usr/local/bin/git
lrwxr-xr-x  1 root  wheel  18 30 Mar 17:32 /usr/local/bin/git -> /tools/git/bin/git

PATH for this job is listed as "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11" so we're trying to use a broken version of git in automation and failing.

Did we deploy a new (and broken) version of git on Mac since 45.0.1?

Comment 8

[Jake Watkins [:dividehex]]

(In reply to Chris Cooper [:coop] from comment #7)

> 
> Did we deploy a new (and broken) version of git on Mac since 45.0.1?

Yes, http://hg.mozilla.org/build/puppet/rev/ec6a2cd0cbcf

I'm building and testing new packages now

Comment 9

[Jake Watkins [:dividehex]]

New git (osx specific version) packages have been deployed.  See https://bugzilla.mozilla.org/show_bug.cgi?id=1257633#c5

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

We're hitting a new error now:

15:51:57     INFO - Copy/paste: /builds/slave/rel-m-rel-lx_pner_rpk-00000000/build/repo init --no-repo-verify -u [e-mail]:mozilla-partners/repack-manifests.git
15:51:57     INFO -  dyld: Library not loaded: /opt/local/lib/libiconv.2.dylib
15:51:57     INFO -    Referenced from: /usr/local/bin/git
15:51:57     INFO -    Reason: Incompatible library version: git requires version 8.0.0 or later, but libiconv.2.dylib provides version 7.0.0
15:51:57     INFO -  error: "" unsupported
15:51:57    ERROR - Return code: 1

Comment 11

[Jake Watkins [:dividehex]]

I've traced this to macports installing a new version of libiconv.2 under the vm I use for building OSX 10.7 packages.  After wiping macports and its libraries from the vm, I'm rebuilding git for 10.7.

I've also filed Bug 1263612 so we can prevent this from happening again.

Comment 12

[Jake Watkins [:dividehex]]

The fixed package for 10.7 has been deployed.
See https://bugzilla.mozilla.org/show_bug.cgi?id=1257633#c10

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

46.0 worked fine.