958312 - Raise maximum packaged app size to 100 Mb

Status: RESOLVED FIXED

(Marketplace Graveyard :: Validation, defect)

Description

[:Harald Kirschner :digitarald]

We have partners upcoming with Emscripten converted games. The packages currently range from 50 to 80 Mb and target the new high-end devices with 512Mb+. One reason for the size are assets for SD and HD graphics, where the latter is required for Helix devices. There are efforts to get the file size down but that will only shave off a few percentiles. Another discussed fix is be downloading assets on launch, but that is a major change in the app architecture and requires to stem up additional CDNs and will impact the user experience considerably.

Bug 794732 limits the package size to 50 Mb and comments already mention to revisit that constrain. Please raise the maximum package size to 100 Mb so the apps can be uploaded.

Comment 1

[Wil Clouser [:clouserw]]

I'd like to see hosted apps, not packaged, but yeah, that's a bigger discussion.

Can you give me one of these packages so I can see how long it takes to process?  I'm worried we're going to send a package off to get validated and it will just timeout (which already happens with some packages today).

Comment 2

[:Harald Kirschner :digitarald]

I just uploaded one of the games with HD assets removed which cut the size down to 40 Mb and I didn't get a timeout: https://marketplace.firefox.com/developers/upload/94821d082bea4505b94e2144464208f7

On a side-note, one warning stuck out: "Warning: A very large JS file was skipped in the validation process. It's over a megabyte.". Emscripten often have the core game code in one 10+ Mb JS file.

To attach the package here I'd need to make the bug confidential or create a bogus package tomorrow.

Comment 3

[Wil Clouser [:clouserw]]

(In reply to Harald Kirschner :digitarald from comment #2)
> On a side-note, one warning stuck out: "Warning: A very large JS file was
> skipped in the validation process. It's over a megabyte.". Emscripten often
> have the core game code in one 10+ Mb JS file.

The validator does static analysis, grepping, and tree walking and doesn't cope well with giant packages (as I recall, it's actually more concerned with # of files than size).  I'm willing to try this on a package and see how it copes, but it may be that we'll need to manually analyze these instead of using the validator in its present form.

Comment 4

[Matt Basta [:basta]]

It might be that the validator just doesn't get enough emotional support.

In all truthfulness, the validator will do fine if there's HD assets in the bundle. Only the first few bytes of most files get read in order to do a magic number test. JS is different, since there's a much more rigorous set of tests that need to be performed. The app-validator's JS engine was rewritten to be less expensive with large numbers of small files, so that shouldn't be an issue either unless there's hundreds of thousands of them.

The real bottleneck is serializing and unserializing the parse tree. Large JS files are *generally* fine, unless they're dense. To get the AST from spidermonkey to Python, we have to serialize it to JSON. Unfortunately, the serialized AST for ~10MB of asm.js can be measured in the gigabytes, which causes the task servers to have out of memory errors. 10MB of JS generally doesn't produce that much AST, but since ASM is mostly very tiny literals and lots of binary operators, it produces an extremely dense tree.

The 1MB cap is a reasonable limit that balances performance with thoroughness. And honestly, if someone sprinkled `window['nav'+'igator']['someNasty'+'API']['doN'+'efarious'+'Things']()` into a 30MB JS file, it's virtually impossible for a human being to notice something like that, much less grep for it.

At the end of the day, if someone's going to hide something in a JS file, there's nothing anyone at Mozilla is going to be able to do to stop them using automated tools. Even if the validator did run, there's plenty of obfuscation patterns that it's incapable of detecting. The real solution is to pay more attention to the permissions that the app requests and eventually do what Google/Apple do: use true dynamic analysis (i.e.: running an app in a VM for a little while) to test that the app doesn't blow up or do anything crazy.

If you experience issues where permissions aren't being detected because they're used in files with large blobs of ASM, simply suggest that devs move the functions that access the permissions to another file (keep ASM and non-ASM separate).

Comment 5

[:Harald Kirschner :digitarald]

Thanks for chiming in basta! No as I understand nothing blocks us from raising the file limit to 100 Mb.

The file limit of large JS files is another issue, apart from being a sane limit.

Comment 6

[Andy McKay]

One other area that needs checking is signing of apps, which copies files around and sends a manifest of it to the signing service.

Comment 7

[Wil Clouser [:clouserw]]

-> Mark to change the cap, but I'd still like to test it with one of your packages.

Comment 8

[:Harald Kirschner :digitarald]

What is the timeline for this? We will have sign off from the partner soon and they'll want to upload the app.

Comment 9

[Mark Striemer [:mstriemer]]

This should make it in our next deploy on Tuesday, January 21st.

Comment 10

[Mark Striemer [:mstriemer]]

Merged to zamboni https://github.com/mozilla/zamboni/commit/3b7ff9dc547366edbe221c1f04795e1397281a5a.

Comment 11

[Iulian Timis]

Please add STR here or mark it with [qa-] if no QA is needed.

Comment 12

[Mark Striemer [:mstriemer]]

I verified this is still working on master locally. It takes far too long to upload these files to staging, no more QA needed.