799061 - error uploading file for signing: File too large uploading Thunderbird Daily builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Mark Banner (:standard8)]

Since sometime yesterday, we've been getting this on Mac builds:

2012-10-08 02:43:47,443 - 3395fa97544068444c86a330eb5bf9407569e3e8: processing Daily.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2012-10-08 02:43:47,463 - 3395fa97544068444c86a330eb5bf9407569e3e8: uploading for signing
2012-10-08 02:43:54,232 - 3395fa97544068444c86a330eb5bf9407569e3e8: error uploading file for signing: File too large
2012-10-08 02:43:55,234 - 3395fa97544068444c86a330eb5bf9407569e3e8: giving up after 20 tries

It might be good to also extend the "File too large" message with size & limit.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found in triage.

Comment 2

[Mark Banner (:standard8)]

This is currently stopping us getting Mac Nightlies out on Daily & Earlybird. Sorry I wasn't explicit about that before.

Comment 3

[[e-mail] (:bhearsum)]

Someone needs to verify that this isn't a packaging problem or something else artificially inflating the package. If that's not the case, then this value needs to be bumped up: https://hg.mozilla.org/build/puppet-manifests/file/default/modules/signingserver/templates/signing.ini.erb#l15

master-puppet1 will need to be updated with the change, and the signing servers will automatically reload themselves for it when they pick it up.

Comment 4

[Mark Banner (:standard8)]

(In reply to Ben Hearsum [:bhearsum] from comment #3)
> Someone needs to verify that this isn't a packaging problem or something
> else artificially inflating the package.

According to FTP:

Firefox:     2012-10-07 = 46583640, 2012-10-08 = 48476361.
Thunderbird: 2012-10-07 = 48079959, 2012-10-08 = ?

According to my calculations, the next day should be around 50000000 for Thunderbird, which should still be a little shy of 52428800.

Can someone log onto a builder and confirm the size of the package?

It should be around build/objdir/mozilla/dist/ iirc.

Comment 5

[Rail Aliiev [:rail]]

find . -name Daily.app.tar.gz -ls
237209661   106592 -rw-rw-rw-    1 cltbld           admin            54573314 10 Oct 07:39 ./objdir-tb/i386/mozilla/dist/universal/thunderbird/Daily.app.tar.gz

Comment 6

[Kim Moir [:kmoir] ET]

It looks like the XUL file alone is 144MB unzipped, is that expected?

bld-lion-r5-058:MacOS cltbld$ pwd
/builds/slave/tb-c-cen-osx64/build/objdir-tb/i386/mozilla/dist/universal/thunderbird/Daily.app/Contents/MacOS
bld-lion-r5-058:thunderbird cltbld$ cd Daily.app/Contents/MacOS/
bld-lion-r5-058:MacOS cltbld$ du -sh XUL
144M	XUL

Comment 7

[Mark Banner (:standard8)]

Quite possibly, we've don't strip symbols in nightlies, and the last Daily I have had a XUL of 139804480.

Comment 8

[Mark Banner (:standard8)]

Ok, I think this is down to bug 790517 which added a whole load of webrtc stuff - about 10M of object files on my local 64-bit only directory.

I suspect, multiply that by two and compress into XUL, and you get the increase that pushed us over the limit.

So I think we can safely bump the limit knowing this is an increase in code size.

Comment 9

[Randell Jesup [:jesup] (needinfo me)]

We ran into this briefly on Saturday on one Try push we did of our staging patch-queue for the Webrtc/signaling landing, and we had a bunch of traffic with Callek and others trying to figure out what the limit was and what our options were.  But a later Try (with a few corrections for a bug in a Makefile) went green, as did all the other builds and our landing.

If there isn't a real reason for the limit of ~52MB, I agree with bumping it.

Comment 10

[Kim Moir [:kmoir] ET]

Attachment: patch

Comment 11

[Kim Moir [:kmoir] ET]

deployed to puppet masters

Comment 12

[Mark Banner (:standard8)]

This is now fixed, thanks.