Closed Bug 983291 Opened 10 years ago Closed 10 years ago

mac l10n nightlies failed on mozilla-central

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 1008052

People

(Reporter: bhearsum, Unassigned)

References

Details

Error is:
cd ../../dist/l10n-stage; \
	  cd ./firefox && python /builds/slave/m-cen-osx64-l10n-ntly-00000000/tools/release/signing/signtool.py --cachedir /builds/slave/m-cen-osx64-l10n-ntly-00000000/signing_cache -t /builds/slave/m-cen-osx64-l10n-ntly-00000000/token -n /builds/slave/m-cen-osx64-l10n-ntly-00000000/nonce -c /builds/slave/m-cen-osx64-l10n-ntly-00000000/tools/release/signing/host.cert -H mac-signing1.srv.releng.scl3.mozilla.com:9100 -H mac-signing2.srv.releng.scl3.mozilla.com:9100 -H mac-signing3.build.scl1.mozilla.com:9100 -H mac-signing4.build.scl1.mozilla.com:9100 -f dmg   FirefoxNightly.app && rm FirefoxNightly.app/Contents/CodeResources && cp FirefoxNightly.app/Contents/_CodeSignature/CodeResources FirefoxNightly.app/Contents && cd /builds/slave/m-cen-osx64-l10n-ntly-00000000/build/mozilla-central/obj-firefox/i386/dist/l10n-stage && /builds/slave/m-cen-osx64-l10n-ntly-00000000/build/mozilla-central/build/package/mac_osx/pkg-dmg --source 'firefox' --target 'firefox-30.0a1.fr.mac.dmg' --volname 'Nightly'  --copy '/builds/slave/m-cen-osx64-l10n-ntly-00000000/build/mozilla-central/obj-firefox/i386/dist/branding/dsstore:/.DS_Store' --mkdir /.background --copy '/builds/slave/m-cen-osx64-l10n-ntly-00000000/build/mozilla-central/obj-firefox/i386/dist/branding/background.png:/.background' --icon '/builds/slave/m-cen-osx64-l10n-ntly-00000000/build/mozilla-central/obj-firefox/i386/dist/branding/disk.icns' --symlink '/Applications:/ '
2014-03-13 04:53:34,814 - 422d5c7f35beef99e33aab39c62ca7fc23322a5d: processing FirefoxNightly.app.tar.gz on https://mac-signing4.build.scl1.mozilla.com:9100
2014-03-13 04:54:12,775 - 422d5c7f35beef99e33aab39c62ca7fc23322a5d: uploading for signing
2014-03-13 04:54:51,666 - 422d5c7f35beef99e33aab39c62ca7fc23322a5d: processing FirefoxNightly.app.tar.gz on https://mac-signing4.build.scl1.mozilla.com:9100
2014-03-13 04:56:06,193 - 422d5c7f35beef99e33aab39c62ca7fc23322a5d: processing FirefoxNightly.app.tar.gz on https://mac-signing4.build.scl1.mozilla.com:9100
2014-03-13 04:58:13,251 - 422d5c7f35beef99e33aab39c62ca7fc23322a5d: processing FirefoxNightly.app.tar.gz on https://mac-signing4.build.scl1.mozilla.com:9100
2014-03-13 04:59:40,886 - 422d5c7f35beef99e33aab39c62ca7fc23322a5d: processing FirefoxNightly.app.tar.gz on https://mac-signing4.build.scl1.mozilla.com:9100
2014-03-13 05:00:49,987 - 422d5c7f35beef99e33aab39c62ca7fc23322a5d: processing FirefoxNightly.app.tar.gz on https://mac-signing4.build.scl1.mozilla.com:9100
2014-03-13 05:02:03,352 - 422d5c7f35beef99e33aab39c62ca7fc23322a5d: processing FirefoxNightly.app.tar.gz on https://mac-signing4.build.scl1.mozilla.com:9100
2014-03-13 05:03:09,993 - 422d5c7f35beef99e33aab39c62ca7fc23322a5d: processing FirefoxNightly.app.tar.gz on https://mac-signing4.build.scl1.mozilla.com:9100
2014-03-13 05:04:56,868 - 422d5c7f35beef99e33aab39c62ca7fc23322a5d: OK
2014-03-13 05:04:56,869 - Copying FirefoxNightly.app.tar.gz to cache /builds/slave/m-cen-osx64-l10n-ntly-00000000/signing_cache/dmg/422d5c7f35beef99e33aab39c62ca7fc23322a5d
FirefoxNightly.app/Contents/MacOS/XUL: (Empty error message)
tar: Error exit delayed from previous errors.


It smells like a problem on the signing servers at first, but it didn't fail for Aurora...so that makes me think it's something in the tree? Maybe somehow related to turning MAR signing?

Regression range is https://hg.mozilla.org/mozilla-central/pushloghtml?tochange=46041cc216fd&fromchange=44ae8462d6ab
It looks like we're getting out of MOZ_SIGN_CMD, and possibly into pkg-dmg (via INNER_MAKE_PACKAGE). But we're clearly not getting as far as the first mktemp call it does: https://mxr.mozilla.org/mozilla-central/source/build/package/mac_osx/pkg-dmg#500
Oh, actually...that error is probably coming from signtool:
https://mxr.mozilla.org/build-central/source/tools/release/signing/signtool.py#188
I grabbed the saved copy of 422d5c7f35beef99e33aab39c62ca7fc23322a5d from the signing server and managed to unpack it fine on the machine that failed to do the repack from comment #0....
Looks like this didn't happen for all locales...just 5 of them. The tar error seems to indicative truncation (according to Google), so perhaps the downloads from the signing got corrupted in some way?

Let's see what happens tomorrow morning.
We saw one of these in 29.0b1. Given that this is the first time we've seen it in a release, I'm guessing that something is tickling an issue starting with Gecko 29.
Haven't seen this again since 29.0b1, maybe it fixed itself?
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → WORKSFORME
I reported a problem with l10n mac builds in bug 989359, not sure if it's the same issue.
Looks like the same symptoms to me.
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
It's not really clear where the issue lays though, it seems to be new in 29.0, which suggests that it may be related to something that landed in that time frame. But then again, the traceback comes from the signing client, which suggests that something is screwed up on either the client or server side of signing.

Build system folks, anything ring any bells to you?
Flags: needinfo?(ted)
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(gps)
Nothing comes to mind, sorry.
Flags: needinfo?(ted)
Nothing off the top of my head. But my memory is bad.
Flags: needinfo?(gps)
I don't think any of us has been much involved with the signing code.
Flags: needinfo?(mh+mozilla)
We hit this again in 30.0b3, and Nick did some better debugging than me in bug 1008052.
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → DUPLICATE
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.