963104 - Test zips need to go on a diet

Status: NEW

(Firefox Build System :: General, defect)

Description

[Chris AtLee [:catlee]]

In July last year the test zips for debug linux64 builds were 107MB. Today we're up to 261MB.

e.g. http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1374551088/firefox-25.0a1.en-US.linux-x86_64.tests.zip
vs
http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1390451117/firefox-29.0a1.en-US.linux-x86_64.tests.zip

The larger these are, the more time we spend uploading/downloading/extracting them in our automation.

Can we make an effort to shrink these down?

Comment 1

[(not currently active) Ted Mielczarek]

The increase is probably mostly from bug 811404, see bug 811404 comment 15. However, supposedly we fixed that in bug 889078, bug 889078 comment 7 indicates the zip size was back to normal. Did we regress this at some point?

Comment 2

[Dan Minor [:dminor]]

Bug 920055 made some changes to how the stripping is done. Maybe we're missing some flags?

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

IIRC the files were as big as 450MB when the striping wasn't working.

Comment 4

[(not currently active) Ted Mielczarek]

So apparently bug 920055 regressed this. Compare:
http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1380330154/ (from 79cba73c77ba, a few changesets before that patch)
[ ]	firefox-27.0a1.en-US.linux-x86_64.tests.zip	28-Sep-2013 02:32 	111M	 
and:
http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1380333677/ (c868eb8d83fd, last changeset in that push)
[ ]	firefox-27.0a1.en-US.linux-x86_64.tests.zip	28-Sep-2013 03:23 	253M

Comment 5

[(not currently active) Ted Mielczarek]

bug 963651 should shrink them back down by about 100MB on debug Linux. I think splitting them into per-testsuite packages is the next logical step but requires a bit more work.

Comment 6

[Hal Wine [:hwine] use NI!]

ipsec issues resolved without needing to implement this, no longer blocks

Comment 7

[Chris AtLee [:catlee]]

While working on some code for bug 1059943, I discovered we have quite a bit of duplicate data in the test zips.

e.g. https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1416229704/firefox-36.0a1.en-US.win32.tests.zip is 120MB. Contained within is 30MB of duplicate data. If we remove these duplicate files and re-compress we get the zip down to 107MB.

There are 4,414 files that are duplicated within this archive. I'll attach a report shortly.

Comment 8

[Chris AtLee [:catlee]]

Attachment: duplicate files report

The format of the file is <number of duplicate bytes> [list of duplicate files]

The number of duplicate bytes is sizeof(files[0]) * (len(files) - 1)

Comment 9

[Gregory Szorc [:gps]]

S3 all of the things and transfer a manifest of the files?

Comment 10

[(not currently active) Ted Mielczarek]

That is exactly bug 1059943 :)

Comment 11

[Marco Castelluccio [:marco]]

:catlee, should we close this or is it still relevant?

Comment 12

[Chris AtLee [:catlee]]

Probably best to DUPE to 1628981. I'm curious about my comment from 6(!!) years ago about duplicate files though!