1254016 - mozilla-esr45 switched to doing mozharness builds, which it is incapable of doing

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr45&revision=8d63254ccdfc was the last working push to esr45, which got non-mozharness builds

https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr45&revision=35fddcd22c6a was a DONTBUILD version bump push, but I wanted builds on it so that bug 1253845 could have a nice clean failure, so I triggered them via self-serve. What I got was mozharness builds, which fail because repo_path isn't set so they default to trying to pull from hg.mozilla.org/projects/mozilla-esr45

To eliminate the self-serve issue, https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr45&revision=7b2dfb80cd13 is a clean fresh push, which according to the master I peeked at is getting mozharness builds and failing the same way. Maybe we intended to switch it to mozharness builds, and maybe it would have worked if we had remembered to copy-paste https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/config.py#2665 while creating esr45, dunno.

Tree's closed.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Rather than that version bump, I suspect this is from http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/config.py#l3000 and the current state of gecko_versions.json:
{

    "mozilla-release": ​45,
    "mozilla-beta": ​45,
    "mozilla-aurora": ​46,
    "mozilla-central": ​47,
    "comm-beta": ​45,
    "comm-aurora": ​46

}
and will go away once m-b becomes 46 with attachment 8727200 [details] [diff] [review] (aka when the merge happens).

Comment 2

[Phil Ringnalda (:philor)]

Indeed. Guess I'll let someone else worry about the impact of the rest of that uncopied block, like the way we're building with "--enable-update-channel=".

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

There's still some issue with the mac builds
* they're failing to sign during the packaging step for opt dep & nightly
* debug 'make check' timed out with 'gtest TEST-UNEXPECTED-FAIL | gtest | timed out after 1200 seconds'

Comment 4

[Phil Ringnalda (:philor)]

We are apparently not terribly excited about the gtest timeout: of the 10 runs on release since 45 merged there, 6 have timed out. Of the last 25 runs while it was on beta, 12 timed out.

Failing to sign sounds like we would be very concerned, but I don't see it in either of the green builds, http://archive.mozilla.org/pub/firefox/nightly/2016/03/2016-03-07-00-15-02-mozilla-esr45/mozilla-esr45-macosx64-nightly-bm85-build1-build5.txt.gz and http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-esr45-macosx64/1457329629/mozilla-esr45-macosx64-bm86-build1-build6.txt.gz seem to me to be perfectly happy with their make package buildstep.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

More very slow to sign, the opt dep and nightly build took from 9:29 to 12:53 to do that on mac-v2-signing6.srv.releng.scl3. Ben, could you take a peek at that to see if the machine is OK ? Both jobs were using the dep signer, we don't appear to have set up nightlies properly out side of m-c and some of m-a.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #5)
> More very slow to sign, the opt dep and nightly build took from 9:29 to
> 12:53 to do that on mac-v2-signing6.srv.releng.scl3. Ben, could you take a
> peek at that to see if the machine is OK ? Both jobs were using the dep
> signer, we don't appear to have set up nightlies properly out side of m-c
> and some of m-a.

There's definitely something strange going on. We've got:
libsoftokn3.dylib: signed Mach-O thin (x86_64) [libsoftokn3]
XUL: User interaction is not allowed.
Traceback (most recent call last):
  File "/builds/signing/dep-key-signing-server/tools/lib/python/signing/utils.py", line 346, in dmg_signfile
    check_call(sign_command + [f], cwd=dir_, stdout=stdout, stderr=STDOUT)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['codesign', '-s', 'Mozilla', '-fv', '--keychain', '/builds/signing/dep-key-signing-server/secrets/dmg/signing.keychain', '--requirement', '=designated => ( (anchor apple generic and certificate leaf[field.1.2.840.113635.100.6.1.9] ) or (anchor apple generic and certificate 1[field.1.2.840.113635.100.6.2.6] and certificate leaf[field.1.2.840.113635.100.6.1.13] and certificate leaf[subject.OU] = "Release Engineering"))\n', 'XUL']' returned non-zero exit status 1
2016-03-07 12:30:10,471 - Error acquiring  /builds/signing/dep-key-signing-server/secrets/dmg/.lock for codesign, is something broken?
Traceback (most recent call last):
  File "/builds/signing/dep-key-signing-server/tools/lib/python/signing/utils.py", line 418, in dmg_signpackage
    dmg_signfile(macdir, keychain, mac_id, subject_ou, fake)
  File "/builds/signing/dep-key-signing-server/tools/lib/python/signing/utils.py", line 346, in dmg_signfile
    check_call(sign_command + [f], cwd=dir_, stdout=stdout, stderr=STDOUT)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['codesign', '-s', 'Mozilla', '-fv', '--keychain', '/builds/signing/dep-key-signing-server/secrets/dmg/signing.keychain', '--requirement', '=designated => ( (anchor apple generic and certificate leaf[field.1.2.840.113635.100.6.1.9] ) or (anchor apple generic and certificate 1[field.1.2.840.113635.100.6.2.6] and certificate leaf[field.1.2.840.113635.100.6.1.13] and certificate leaf[subject.OU] = "Release Engineering"))\n', 'XUL']' returned non-zero exit status 1
2016-03-07 12:30:10,721 - Error signing /builds/signing/dep-key-signing-server/unsigned-files/2a3595d67dd8de86a1f1b2c280bb98055844bb9a
Traceback (most recent call last):
  File "/builds/signing/dep-key-signing-server/tools/lib/python/signing/utils.py", line 418, in dmg_signpackage
    dmg_signfile(macdir, keychain, mac_id, subject_ou, fake)
  File "/builds/signing/dep-key-signing-server/tools/lib/python/signing/utils.py", line 346, in dmg_signfile
    check_call(sign_command + [f], cwd=dir_, stdout=stdout, stderr=STDOUT)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['codesign', '-s', 'Mozilla', '-fv', '--keychain', '/builds/signing/dep-key-signing-server/secrets/dmg/signing.keychain', '--requirement', '=designated => ( (anchor apple generic and certificate leaf[field.1.2.840.113635.100.6.1.9] ) or (anchor apple generic and certificate 1[field.1.2.840.113635.100.6.2.6] and certificate leaf[field.1.2.840.113635.100.6.1.13] and certificate leaf[subject.OU] = "Release Engineering"))\n', 'XUL']' returned non-zero exit status 1
2016-03-07 12:30:10,882 - Repacking /builds/signing/dep-key-signing-server/unsigned-files/2a3595d67dd8de86a1f1b2c280bb98055844bb9a to /builds/signing/dep-key-signing-server/signed-files/dmg/2a3595d67dd8de86a1f1b2c280bb98055844bb9a.tmp
  Unpacking /builds/signing/dep-key-signing-server/unsigned-files/2a3595d67dd8de86a1f1b2c280bb98055844bb9a to /tmp/tmphHEnIl
Traceback (most recent call last):
  File "/builds/signing/dep-key-signing-server/tools/release/signing/signscript.py", line 151, in <module>
    dmg_signpackage(inputfile, tmpfile, options.dmg_keychain, options.mac_id, options.mac_cert_subject_ou, options.fake, passphrase)
  File "/builds/signing/dep-key-signing-server/tools/lib/python/signing/utils.py", line 418, in dmg_signpackage
    dmg_signfile(macdir, keychain, mac_id, subject_ou, fake)
  File "/builds/signing/dep-key-signing-server/tools/lib/python/signing/utils.py", line 346, in dmg_signfile
    check_call(sign_command + [f], cwd=dir_, stdout=stdout, stderr=STDOUT)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['codesign', '-s', 'Mozilla', '-fv', '--keychain', '/builds/signing/dep-key-signing-server/secrets/dmg/signing.keychain', '--requirement', '=designated => ( (anchor apple generic and certificate leaf[field.1.2.840.113635.100.6.1.9] ) or (anchor apple generic and certificate 1[field.1.2.840.113635.100.6.2.6] and certificate leaf[field.1.2.840.113635.100.6.1.13] and certificate leaf[subject.OU] = "Release Engineering"))\n', 'XUL']' returned non-zero exit status 1
2016-03-07 12:31:00,400 - lifetime has expired, breaking


And:
XUL: signed Mach-O thin (x86_64) [XUL]
Traceback (most recent call last):
  File "/builds/signing/dep-key-signing-server/tools/lib/python/signing/utils.py", line 355, in dmg_signfile
    check_call(sign_command + [app], cwd=appdir, stdout=stdout, stderr=STDOUT)
  File "/tools/python27/lib/python2.7/subprocess.py", line 506, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 493, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/tools/python27/lib/python2.7/subprocess.py", line 1291, in wait
    pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
  File "/tools/python27/lib/python2.7/subprocess.py", line 478, in _eintr_retry_call
    return func(*args)
KeyboardInterrupt
2016-03-07 12:32:16,515 - Error acquiring  /builds/signing/dep-key-signing-server/secrets/dmg/.lock for codesign, is something broken?
Traceback (most recent call last):
  File "/builds/signing/dep-key-signing-server/tools/lib/python/signing/utils.py", line 418, in dmg_signpackage
    dmg_signfile(macdir, keychain, mac_id, subject_ou, fake)
  File "/builds/signing/dep-key-signing-server/tools/lib/python/signing/utils.py", line 355, in dmg_signfile
    check_call(sign_command + [app], cwd=appdir, stdout=stdout, stderr=STDOUT)
  File "/tools/python27/lib/python2.7/subprocess.py", line 506, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 493, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/tools/python27/lib/python2.7/subprocess.py", line 1291, in wait
    pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
  File "/tools/python27/lib/python2.7/subprocess.py", line 478, in _eintr_retry_call
    return func(*args)
KeyboardInterrupt


The "user interaction error" and KeyboardInterrupts are somewhat alarming, but it could be that these are just the symptoms of a different issue. The machine feels very slow compared to other mac signing machines, so there could be issues with it. I've given it a reboot and we'll see if that helps. The first thing signing operation that ran it went relatively quickly:
2016-03-08 07:13:17,809 - INFO - Signing NightlyDebug.app.tar.gz (dmg - 43d8127b148ea4b3807d799160cfbce7b587e765)
2016-03-08 07:13:17,843 - DEBUG - 458: ['/builds/signing/dep-key-signing-server/bin/python2.7', '/builds/signing/dep-key-signing-server/tools/release/signing/signscript.py', '-c', '/builds/signing/dep-key-signing-server/signscript.ini', 'dmg', u'/builds/signing/dep-key-signing-server/unsigned-files/43d8127b148ea4b3807d799160cfbce7b587e765', u'/builds/signing/dep-key-signing-server/signed-files/dmg/43d8127b148ea4b3807d799160cfbce7b587e765', u'NightlyDebug.app.tar.gz']
2016-03-08 07:13:18,427 - INFO - 10.26.52.73 GET /sign/dmg/43d8127b148ea4b3807d799160cfbce7b587e765
2016-03-08 07:13:18,428 - DEBUG - Waiting for pending job
2016-03-08 07:13:40,012 - DEBUG - 458: Success!

If the problem persists I'm betting there's a hardware issue.

Comment 7

[Phil Ringnalda (:philor)]

Oh, very slow Mac signing? I filed bug 1251116 on that on mac-v2-signing7, then closed it after nobody looked and the slow signing had moved to mac-v2-signing6.