899890 - OS X 10.7 opt UX build: error uploading file for signing: File too large

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Matthew N. [:MattN]]

2013-07-30 21:12:40,075 - 649f012390c5adb478192f234d999c719bf77239: error uploading file for signing: File too large
2013-07-30 21:12:41,077 - 649f012390c5adb478192f234d999c719bf77239: giving up after 20 tries
2013-07-30 21:12:41,078 - Failed to sign FirefoxUX.app.tar.gz with dmg
make[3]: *** [make-package-internal] Error 1
make[2]: *** [make-package] Error 2
make[1]: *** [default] Error 2
make: *** [package] Error 2
program finished with exit code 2
elapsedTime=256.462121

https://tbpl.mozilla.org/php/getParsedLog.php?id=25947370&tree=UX

Raw log: http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/ux-macosx64/1375237252/ux-macosx64-bm57-build1-build35.txt.gz

If I were to guess I'd say that other branches are likely to close to hitting this soon as Australis doesn't add that much new stuff to the UX branch. This started on the following merge: https://hg.mozilla.org/projects/ux/pushloghtml?startID=294&endID=295

I would guess that bug 853301 is the big chunk of code that pushed us over the edge.

I believe this will break our Nightlies until it's fixed.

Comment 1

[Matthew N. [:MattN]]

This seems to break all OS X opt test/talos runs and we're in the middle of perf investigations so it would be great to get this fixed ASAP.

Comment 2

[[e-mail] (:bhearsum)]

Given that UX is currently working on a lot of Australis stuff, I'm guessing that new a bunch of new assets have been added, which is why we're bumping up against the max size again. (I couldn't find the exact size of the package, so it's hard to know how far over the limit we are.)

I'll get the max size bumped shortly, but it would be great if someone could confirm that we have indeed added a bunch of new assets, and also that we've removed any now-unused assets.

Comment 3

[:Gijs (he/him)]

(In reply to Ben Hearsum [:bhearsum] from comment #2)
> Given that UX is currently working on a lot of Australis stuff, I'm guessing
> that new a bunch of new assets have been added, which is why we're bumping
> up against the max size again. (I couldn't find the exact size of the
> package, so it's hard to know how far over the limit we are.)
> 
> I'll get the max size bumped shortly, but it would be great if someone could
> confirm that we have indeed added a bunch of new assets, and also that we've
> removed any now-unused assets.

I think the thing that pushed us over the edge, as noted in comment 0, is bug 853301. A quick check on yesterday's nightlies says m-c is about 0.3MB bigger on OS X than UX is. That's not very significant compared to the increase in size (roughly 6MB) caused by bug 853301.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: bump max size for dmgs

I just noticed that our max size is already set to 70mb. I'm bumping up to 80mb to try to unblock UX, but 70mb is HUGE for an opt build - way too big to actually ship to users. I suspect there's something broken in the build system that's causing them to get this big. Eg, files erroneously being put in the package.

Comment 5

[:Gijs (he/him)]

(In reply to Ben Hearsum [:bhearsum] from comment #4)
> Created attachment 783744 [details] [diff] [review]
> bump max size for dmgs
> 
> I just noticed that our max size is already set to 70mb. I'm bumping up to
> 80mb to try to unblock UX,

Thank you! :-)

> but 70mb is HUGE for an opt build - way too big
> to actually ship to users. I suspect there's something broken in the build
> system that's causing them to get this big. Eg, files erroneously being put
> in the package.

Current m-c opt builds on OS X are already 67MB; this is hardly a UX-specific problem. Current release is apparently 40.1MB, so that is indeed a sizable increase in just a few months, but I don't think this is the right bug in which to worry about this. If you think it's a significant issue, we probably should be filing a bug and taking it up with drivers / owners and figuring out what went wrong here.

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 783744 [details] [diff] [review]
bump max size for dmgs

The signing servers will pick this up in 30min-1h. I've retriggered the OS X build, which is slow enough that it won't get to signing for at least that long.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to :Gijs Kruitbosch from comment #5)
> (In reply to Ben Hearsum [:bhearsum] from comment #4)
> > Created attachment 783744 [details] [diff] [review]
> > bump max size for dmgs
> > 
> > I just noticed that our max size is already set to 70mb. I'm bumping up to
> > 80mb to try to unblock UX,
> 
> Thank you! :-)
> 
> > but 70mb is HUGE for an opt build - way too big
> > to actually ship to users. I suspect there's something broken in the build
> > system that's causing them to get this big. Eg, files erroneously being put
> > in the package.
> 
> Current m-c opt builds on OS X are already 67MB; this is hardly a
> UX-specific problem. Current release is apparently 40.1MB, so that is indeed
> a sizable increase in just a few months, but I don't think this is the right
> bug in which to worry about this. If you think it's a significant issue, we
> probably should be filing a bug and taking it up with drivers / owners and
> figuring out what went wrong here.

Someone just reminded me that we have extra profiling turned on in Nightly and Aurora, which could explain why those are so much bigger...

Comment 8

[(not currently active) Ted Mielczarek]

(In reply to Ben Hearsum [:bhearsum] from comment #7)
> Someone just reminded me that we have extra profiling turned on in Nightly
> and Aurora, which could explain why those are so much bigger...

Specifically, --disable-install-strip:
http://mxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/macosx-universal/nightly#4

OS X probably has the worst of it since it's still a universal binary.

Comment 9

[Jeff Walden [:Waldo]]

Bug 853301 is going to push things up here.  This library/executable size increase was deemed an acceptable tradeoff for adding sensible internationalization APIs to JavaScript, that have been specified and are being implemented by other browsers now as well.

Comment 10

[Matthew N. [:MattN]]

This has started happening again on UX:
https://tbpl.mozilla.org/php/getParsedLog.php?id=26009554&tree=UX

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

2013-07-31 22:41:37,591 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:41:37,617 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:41:47,645 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large

[[e-mail] build]$ find obj-firefox/*/dist/ -name FirefoxUX.app.tar.gz -ls
109448805   143568 -rw-rw-rw-    1 cltbld           staff            73505476 31 Jul 22:40 obj-firefox/i386/dist//universal/firefox/FirefoxUX.app.tar.gz

Looks like mac-signing2 didn't pick up the puppet change. Shouldn't we also try another signing server when retrying a failure ?

Comment 12

[[e-mail] (:bhearsum)]

Ack. I just realized that these aren't hooked up to Puppet anymore. I made this change by hand, and I've triggered a new build that I'll keep an eye on.

Comment 13

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #11)
> [[e-mail] build]$ find
> obj-firefox/*/dist/ -name FirefoxUX.app.tar.gz -ls
> 109448805   143568 -rw-rw-rw-    1 cltbld           staff           
> 73505476 31 Jul 22:40
> obj-firefox/i386/dist//universal/firefox/FirefoxUX.app.tar.gz
> 
> Looks like mac-signing2 didn't pick up the puppet change. Shouldn't we also
> try another signing server when retrying a failure ?

Yeah, we should. I'm surprised we're not, given bug 762922. Maybe it only shuffles on 5xx or connection errors? In any case I filed bug 900546 for this.

The latest build managed to sign OK, so this is fixed, for now.