Last Comment Bug 899890 - OS X 10.7 opt UX build: error uploading file for signing: File too large
: OS X 10.7 opt UX build: error uploading file for signing: File too large
Status: RESOLVED FIXED
[Australis:M8]
:
Product: Release Engineering
Classification: Other
Component: General Automation (show other bugs)
: other
: All Mac OS X
: -- major (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
: Chris AtLee [:catlee]
:
Mentors:
Depends on: 853301
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-30 21:39 PDT by Matthew N. [:MattN] (PM me if requests are blocking you)
Modified: 2013-11-22 06:43 PST (History)
12 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
bump max size for dmgs (587 bytes, patch)
2013-07-31 06:06 PDT, Ben Hearsum (:bhearsum)
rail: review+
bhearsum: checked‑in+
Details | Diff | Splinter Review

Description User image Matthew N. [:MattN] (PM me if requests are blocking you) 2013-07-30 21:39:19 PDT
2013-07-30 21:12:40,075 - 649f012390c5adb478192f234d999c719bf77239: error uploading file for signing: File too large
2013-07-30 21:12:41,077 - 649f012390c5adb478192f234d999c719bf77239: giving up after 20 tries
2013-07-30 21:12:41,078 - Failed to sign FirefoxUX.app.tar.gz with dmg
make[3]: *** [make-package-internal] Error 1
make[2]: *** [make-package] Error 2
make[1]: *** [default] Error 2
make: *** [package] Error 2
program finished with exit code 2
elapsedTime=256.462121

https://tbpl.mozilla.org/php/getParsedLog.php?id=25947370&tree=UX

Raw log: http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/ux-macosx64/1375237252/ux-macosx64-bm57-build1-build35.txt.gz

If I were to guess I'd say that other branches are likely to close to hitting this soon as Australis doesn't add that much new stuff to the UX branch. This started on the following merge: https://hg.mozilla.org/projects/ux/pushloghtml?startID=294&endID=295

I would guess that bug 853301 is the big chunk of code that pushed us over the edge.

I believe this will break our Nightlies until it's fixed.
Comment 1 User image Matthew N. [:MattN] (PM me if requests are blocking you) 2013-07-30 22:24:58 PDT
This seems to break all OS X opt test/talos runs and we're in the middle of perf investigations so it would be great to get this fixed ASAP.
Comment 2 User image Ben Hearsum (:bhearsum) 2013-07-31 04:32:47 PDT
Given that UX is currently working on a lot of Australis stuff, I'm guessing that new a bunch of new assets have been added, which is why we're bumping up against the max size again. (I couldn't find the exact size of the package, so it's hard to know how far over the limit we are.)

I'll get the max size bumped shortly, but it would be great if someone could confirm that we have indeed added a bunch of new assets, and also that we've removed any now-unused assets.
Comment 3 User image :Gijs 2013-07-31 05:15:02 PDT
(In reply to Ben Hearsum [:bhearsum] from comment #2)
> Given that UX is currently working on a lot of Australis stuff, I'm guessing
> that new a bunch of new assets have been added, which is why we're bumping
> up against the max size again. (I couldn't find the exact size of the
> package, so it's hard to know how far over the limit we are.)
> 
> I'll get the max size bumped shortly, but it would be great if someone could
> confirm that we have indeed added a bunch of new assets, and also that we've
> removed any now-unused assets.

I think the thing that pushed us over the edge, as noted in comment 0, is bug 853301. A quick check on yesterday's nightlies says m-c is about 0.3MB bigger on OS X than UX is. That's not very significant compared to the increase in size (roughly 6MB) caused by bug 853301.
Comment 4 User image Ben Hearsum (:bhearsum) 2013-07-31 06:06:39 PDT
Created attachment 783744 [details] [diff] [review]
bump max size for dmgs

I just noticed that our max size is already set to 70mb. I'm bumping up to 80mb to try to unblock UX, but 70mb is HUGE for an opt build - way too big to actually ship to users. I suspect there's something broken in the build system that's causing them to get this big. Eg, files erroneously being put in the package.
Comment 5 User image :Gijs 2013-07-31 06:22:04 PDT
(In reply to Ben Hearsum [:bhearsum] from comment #4)
> Created attachment 783744 [details] [diff] [review]
> bump max size for dmgs
> 
> I just noticed that our max size is already set to 70mb. I'm bumping up to
> 80mb to try to unblock UX,

Thank you! :-)

> but 70mb is HUGE for an opt build - way too big
> to actually ship to users. I suspect there's something broken in the build
> system that's causing them to get this big. Eg, files erroneously being put
> in the package.

Current m-c opt builds on OS X are already 67MB; this is hardly a UX-specific problem. Current release is apparently 40.1MB, so that is indeed a sizable increase in just a few months, but I don't think this is the right bug in which to worry about this. If you think it's a significant issue, we probably should be filing a bug and taking it up with drivers / owners and figuring out what went wrong here.
Comment 6 User image Ben Hearsum (:bhearsum) 2013-07-31 06:50:24 PDT
Comment on attachment 783744 [details] [diff] [review]
bump max size for dmgs

The signing servers will pick this up in 30min-1h. I've retriggered the OS X build, which is slow enough that it won't get to signing for at least that long.
Comment 7 User image Ben Hearsum (:bhearsum) 2013-07-31 06:53:53 PDT
(In reply to :Gijs Kruitbosch from comment #5)
> (In reply to Ben Hearsum [:bhearsum] from comment #4)
> > Created attachment 783744 [details] [diff] [review]
> > bump max size for dmgs
> > 
> > I just noticed that our max size is already set to 70mb. I'm bumping up to
> > 80mb to try to unblock UX,
> 
> Thank you! :-)
> 
> > but 70mb is HUGE for an opt build - way too big
> > to actually ship to users. I suspect there's something broken in the build
> > system that's causing them to get this big. Eg, files erroneously being put
> > in the package.
> 
> Current m-c opt builds on OS X are already 67MB; this is hardly a
> UX-specific problem. Current release is apparently 40.1MB, so that is indeed
> a sizable increase in just a few months, but I don't think this is the right
> bug in which to worry about this. If you think it's a significant issue, we
> probably should be filing a bug and taking it up with drivers / owners and
> figuring out what went wrong here.

Someone just reminded me that we have extra profiling turned on in Nightly and Aurora, which could explain why those are so much bigger...
Comment 8 User image Ted Mielczarek [:ted.mielczarek] 2013-07-31 07:29:26 PDT
(In reply to Ben Hearsum [:bhearsum] from comment #7)
> Someone just reminded me that we have extra profiling turned on in Nightly
> and Aurora, which could explain why those are so much bigger...

Specifically, --disable-install-strip:
http://mxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/macosx-universal/nightly#4

OS X probably has the worst of it since it's still a universal binary.
Comment 9 User image Jeff Walden [:Waldo] (remove +bmo to email) 2013-07-31 13:13:49 PDT
Bug 853301 is going to push things up here.  This library/executable size increase was deemed an acceptable tradeoff for adding sensible internationalization APIs to JavaScript, that have been specified and are being implemented by other browsers now as well.
Comment 10 User image Matthew N. [:MattN] (PM me if requests are blocking you) 2013-07-31 23:22:52 PDT
This has started happening again on UX:
https://tbpl.mozilla.org/php/getParsedLog.php?id=26009554&tree=UX
Comment 11 User image Nick Thomas (UTC+13) [:nthomas] 2013-08-01 02:12:36 PDT
2013-07-31 22:41:37,591 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:41:37,617 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:41:47,645 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large

[cltbld@bld-lion-r5-064.build.releng.scl3.mozilla.com build]$ find obj-firefox/*/dist/ -name FirefoxUX.app.tar.gz -ls
109448805   143568 -rw-rw-rw-    1 cltbld           staff            73505476 31 Jul 22:40 obj-firefox/i386/dist//universal/firefox/FirefoxUX.app.tar.gz

Looks like mac-signing2 didn't pick up the puppet change. Shouldn't we also try another signing server when retrying a failure ?
Comment 12 User image Ben Hearsum (:bhearsum) 2013-08-01 07:02:53 PDT
Ack. I just realized that these aren't hooked up to Puppet anymore. I made this change by hand, and I've triggered a new build that I'll keep an eye on.
Comment 13 User image Ben Hearsum (:bhearsum) 2013-08-01 09:08:21 PDT
(In reply to Nick Thomas [:nthomas] from comment #11)
> [cltbld@bld-lion-r5-064.build.releng.scl3.mozilla.com build]$ find
> obj-firefox/*/dist/ -name FirefoxUX.app.tar.gz -ls
> 109448805   143568 -rw-rw-rw-    1 cltbld           staff           
> 73505476 31 Jul 22:40
> obj-firefox/i386/dist//universal/firefox/FirefoxUX.app.tar.gz
> 
> Looks like mac-signing2 didn't pick up the puppet change. Shouldn't we also
> try another signing server when retrying a failure ?

Yeah, we should. I'm surprised we're not, given bug 762922. Maybe it only shuffles on 5xx or connection errors? In any case I filed bug 900546 for this.

The latest build managed to sign OK, so this is fixed, for now.

Note You need to log in before you can comment on or make changes to this bug.