648213 - XPI files should be repacked with zero compression level upon installation

Status: RESOLVED INACTIVE

(Toolkit :: Add-ons Manager, enhancement)

Description

[Wladimir Palant]

My test results (see URL) indicate that having an add-on in a single XPI file isn't always beneficial - at least in case of warm startup on Windows 7 an XPI file packed at maximal compression level degrades startup times. The optimal scenario is an XPI file with zero compression (the long-standing recommendation for JAR files) but add-on authors cannot adopt this approach: the resulting XPI files would be far too large. So ideally the add-on manager would repack XPI files upon installation - even if the source XPI uses maximal compression the installed XPI file could have zero compression and consequently perfect performance.

Comment 1

[(dormant account)]

Typically compression is a tradeoff. It makes cold startup faster(less io) and warm startup slower(more cpu usage).

We should indeed consider some clever way to repack addons. A cool way to go would be to rewrite the jar into an optimized one(bug 559961) and use http://code.google.com/p/snappy/ for compression.

An even better approach would be to combine all of the installed addons into such an optimized jar. Then we'll get awesome IO patterns and low overhead for each installed addon.

Comment 2

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

> We should indeed consider some clever way to repack addons. A cool way to go
> would be to rewrite the jar into an optimized one(bug 559961) and use
> http://code.google.com/p/snappy/ for compression.

Ideally every addon would be signed (by AMO, and optionally by the author) and these signatures should (continue to) be checked at startup because we're loading and executing code from an untrusted location (the user's profile). We install Firefox in "C:\Prograsm Files" (and equiv. on other platforms) specifically to avoid loading code from an untrusted location.

The consequence of this is that any per-XPI repacking should be done by AMO so that AMO can sign the repacked XPI. IMO, a better alternative would be to give addon developers a better packaging tool that correctly optimizes (and signs) their XPI for them, so they don't need to be repacked.

> An even better approach would be to combine all of the installed addons into
> such an optimized jar. Then we'll get awesome IO patterns and low overhead
> for each installed addon.

A good idea. The output of such repacking should be a secured location (non-user-writable) for the above reasons.

Comment 3

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Wladimir Palant from comment #0)
> The optimal scenario is an XPI file with zero compression (the long-standing
> recommendation for JAR files) but add-on authors cannot adopt this approach:
> the resulting XPI files would be far too large.

I am not familiar with addons that much, so please forgive my ignorance here: What do you mean by "too large"? Too large for what?

At least in theory AMO could apply gzip content-encoding to the uncompressed JAR to make uncompressed JARs smaller than compressed JARs would be during transport, so I don't think it is a transport-size issue. Too big on disk?

Comment 4

[Wladimir Palant]

Brian, I doubt that AMO can compress XPIs during transport. It uses releases.mozilla.org for downloads meaning a number of third-party mirrors. Gzip encoding causes enough additional CPU load that these mirrors won't be too willing to change their configuration.

If I were to upload Adblock Plus to AMO with zero compression the download size would increase by factor 3 - many users aren't going to be happy about that. So the obvious solution is to have a compressed download but repack the file on the client side for a better performance.

Comment 5

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Wladimir Palant from comment #4)
> Brian, I doubt that AMO can compress XPIs during transport. It uses
> releases.mozilla.org for downloads meaning a number of third-party mirrors.
> Gzip encoding causes enough additional CPU load that these mirrors won't be
> too willing to change their configuration.

Thanks. Good to know. Most servers have a way to statically pre-Content-Encode a response so there's zero CPU overhead, though you're right that it might be difficult to convince all the mirrors we use to do so.

Comment 6

[(dormant account)]

There is a significant amount of disk overhead from using uncompressed zips on disk. Lets stick to this particular problem in this bug.

Comment 7

[(dormant account)]

Sorry, 'this particular problem' = unexpected disk/network io on main thread during startup

Comment 8

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Taras Glek (:taras) from comment #6)
> There is a significant amount of disk overhead from using uncompressed zips
> on disk. Lets stick to this particular problem in this bug.

(In reply to Taras Glek (:taras) from comment #7)
> Sorry, 'this particular problem' = unexpected disk/network io on main thread
> during startup

This is the bug for uncompressed zips. The bug for unexpected disk/network io is bug 726125.

Comment 9

[BMO Automation]

Per policy at https://wiki.mozilla.org/Bug_Triage/Projects/Bug_Handling/Bug_Husbandry#Inactive_Bugs. If this bug is not an enhancement request or a bug not present in a supported release of Firefox, then it may be reopened.