Open Bug 963104 Opened 6 years ago Updated 2 years ago

Test zips need to go on a diet

Categories

(Firefox Build System :: General, defect)

x86_64
Linux
defect
Not set

Tracking

(Not tracked)

People

(Reporter: catlee, Unassigned)

References

(Depends on 1 open bug)

Details

Attachments

(1 file)

In July last year the test zips for debug linux64 builds were 107MB. Today we're up to 261MB.

e.g. http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1374551088/firefox-25.0a1.en-US.linux-x86_64.tests.zip
vs
http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1390451117/firefox-29.0a1.en-US.linux-x86_64.tests.zip

The larger these are, the more time we spend uploading/downloading/extracting them in our automation.

Can we make an effort to shrink these down?
Depends on: 917999
The increase is probably mostly from bug 811404, see bug 811404 comment 15. However, supposedly we fixed that in bug 889078, bug 889078 comment 7 indicates the zip size was back to normal. Did we regress this at some point?
Bug 920055 made some changes to how the stripping is done. Maybe we're missing some flags?
IIRC the files were as big as 450MB when the striping wasn't working.
So apparently bug 920055 regressed this. Compare:
http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1380330154/ (from 79cba73c77ba, a few changesets before that patch)
[ ]	firefox-27.0a1.en-US.linux-x86_64.tests.zip	28-Sep-2013 02:32 	111M	 
and:
http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1380333677/ (c868eb8d83fd, last changeset in that push)
[ ]	firefox-27.0a1.en-US.linux-x86_64.tests.zip	28-Sep-2013 03:23 	253M
bug 963651 should shrink them back down by about 100MB on debug Linux. I think splitting them into per-testsuite packages is the next logical step but requires a bit more work.
Blocks: 957502
ipsec issues resolved without needing to implement this, no longer blocks
No longer blocks: 957502
While working on some code for bug 1059943, I discovered we have quite a bit of duplicate data in the test zips.

e.g. https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1416229704/firefox-36.0a1.en-US.win32.tests.zip is 120MB. Contained within is 30MB of duplicate data. If we remove these duplicate files and re-compress we get the zip down to 107MB.

There are 4,414 files that are duplicated within this archive. I'll attach a report shortly.
Attached file duplicate files report
The format of the file is <number of duplicate bytes> [list of duplicate files]

The number of duplicate bytes is sizeof(files[0]) * (len(files) - 1)
S3 all of the things and transfer a manifest of the files?
Component: Build Config → General
Product: Firefox → Firefox Build System
See Also: → 1432287
You need to log in before you can comment on or make changes to this bug.