1210686 - mac-v2-signing6 and mac-v2-signing7 failing to sign release jobs

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Lots of this in the log:

2015-10-01 20:50:16,700 - Repacking /builds/signing/rel-key-signing-server/unsigned-files/1687e34c518919c052d2a532ecbcddf7e78cf3a1 to /builds/signing/rel-key-signing-server/s
igned-files/dmg/1687e34c518919c052d2a532ecbcddf7e78cf3a1.tmp
  Unpacking /builds/signing/rel-key-signing-server/unsigned-files/1687e34c518919c052d2a532ecbcddf7e78cf3a1 to /tmp/tmpL0r98S
Traceback (most recent call last):
  File "/builds/signing/rel-key-signing-server/tools/release/signing/signscript.py", line 151, in <module>
    dmg_signpackage(inputfile, tmpfile, options.dmg_keychain, options.mac_id, options.mac_cert_subject_ou, options.fake, passphrase)
  File "/builds/signing/rel-key-signing-server/tools/lib/python/signing/utils.py", line 415, in dmg_signpackage
    raise ValueError("keychain unlock failed")
ValueError: keychain unlock failed

I've stopped the release signing instance on both machines; nagios is not bleating because the process check was already downtimed until Oct 24. Lets undowntime once this is fixed up. Probably should check all nightly/dep for errors too.

Also, we don't have mar and gpg secrets on these boxes ?

Comment 1

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #0)
> Lots of this in the log:
> 
> 2015-10-01 20:50:16,700 - Repacking
> /builds/signing/rel-key-signing-server/unsigned-files/
> 1687e34c518919c052d2a532ecbcddf7e78cf3a1 to
> /builds/signing/rel-key-signing-server/s
> igned-files/dmg/1687e34c518919c052d2a532ecbcddf7e78cf3a1.tmp
>   Unpacking
> /builds/signing/rel-key-signing-server/unsigned-files/
> 1687e34c518919c052d2a532ecbcddf7e78cf3a1 to /tmp/tmpL0r98S
> Traceback (most recent call last):
>   File
> "/builds/signing/rel-key-signing-server/tools/release/signing/signscript.py",
> line 151, in <module>
>     dmg_signpackage(inputfile, tmpfile, options.dmg_keychain,
> options.mac_id, options.mac_cert_subject_ou, options.fake, passphrase)
>   File
> "/builds/signing/rel-key-signing-server/tools/lib/python/signing/utils.py",
> line 415, in dmg_signpackage
>     raise ValueError("keychain unlock failed")
> ValueError: keychain unlock failed
> 
> I've stopped the release signing instance on both machines; nagios is not
> bleating because the process check was already downtimed until Oct 24. Lets
> undowntime once this is fixed up. Probably should check all nightly/dep for
> errors too.

Sounds like they were started with the wrong passphrases...I'm not sure if dividehex has the right ones, come to think of it? Hard to know for sure because we don't check them at start time on mac anymore :(.

> 
> Also, we don't have mar and gpg secrets on these boxes ?

This part is OK, we don't do mar and gpg signing on Mac anymore.

Comment 2

[[e-mail] (:bhearsum)]

https://bugzilla.mozilla.org/show_bug.cgi?id=1210771 was filed this morning about mac l10n nightlies, probably the same issue. I'll try to look at this today...

Comment 3

[[e-mail] (:bhearsum)]

Jake, I think I'll need your help here...I'm not sure what the correct procedure for restarting the signing instance is on these new machines. /usr/libexec/StartupItemContext causes it to start as root...

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum (:bhearsum) (Away until December 7th) from comment #3)
> Jake, I think I'll need your help here...I'm not sure what the correct
> procedure for restarting the signing instance is on these new machines.
> /usr/libexec/StartupItemContext causes it to start as root...

Jake told me on irc that the correct way to start the server is "sudo tools/release/signing/signing_wrapper.sh" as cltsign. I've started up the nightly one on mac-v2-signing6 and kicked one of the failed nightly repack chunks as a test.

I updated https://mana.mozilla.org/wiki/display/RelEng/Signing with these instructions, too.

Comment 6

[[e-mail] (:bhearsum)]

The nightly repacks I kicked worked when signing with the new servers \o/. I restarted the release signing servers w/ the correct passphrases given this.