Open Bug 963104 Opened 11 years ago Updated 2 years ago

Test zips need to go on a diet


(Firefox Build System :: General, defect)



(Not tracked)


(Reporter: catlee, Unassigned)


(Depends on 1 open bug)



(1 file)

In July last year the test zips for debug linux64 builds were 107MB. Today we're up to 261MB.


The larger these are, the more time we spend uploading/downloading/extracting them in our automation.

Can we make an effort to shrink these down?
Depends on: 917999
The increase is probably mostly from bug 811404, see bug 811404 comment 15. However, supposedly we fixed that in bug 889078, bug 889078 comment 7 indicates the zip size was back to normal. Did we regress this at some point?
Bug 920055 made some changes to how the stripping is done. Maybe we're missing some flags?
IIRC the files were as big as 450MB when the striping wasn't working.
So apparently bug 920055 regressed this. Compare: (from 79cba73c77ba, a few changesets before that patch)
[ ]	28-Sep-2013 02:32 	111M	 
and: (c868eb8d83fd, last changeset in that push)
[ ]	28-Sep-2013 03:23 	253M
bug 963651 should shrink them back down by about 100MB on debug Linux. I think splitting them into per-testsuite packages is the next logical step but requires a bit more work.
Blocks: 957502
ipsec issues resolved without needing to implement this, no longer blocks
No longer blocks: 957502
While working on some code for bug 1059943, I discovered we have quite a bit of duplicate data in the test zips.

e.g. is 120MB. Contained within is 30MB of duplicate data. If we remove these duplicate files and re-compress we get the zip down to 107MB.

There are 4,414 files that are duplicated within this archive. I'll attach a report shortly.
Attached file duplicate files report
The format of the file is <number of duplicate bytes> [list of duplicate files]

The number of duplicate bytes is sizeof(files[0]) * (len(files) - 1)
S3 all of the things and transfer a manifest of the files?
Component: Build Config → General
Product: Firefox → Firefox Build System
See Also: → 1432287

:catlee, should we close this or is it still relevant?

Flags: needinfo?(catlee)

Probably best to DUPE to 1628981. I'm curious about my comment from 6(!!) years ago about duplicate files though!

Flags: needinfo?(catlee)
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.