1068339 - Validation timeout when uploading a large file

Status: RESOLVED WONTFIX

(Marketplace Graveyard :: Validation, defect, P3)

Description

[Lionel Laské]

Attachment: Capture.PNG

User Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0
Build ID: 20140825202822

Steps to reproduce:

Try to upload my packaged app zip (could find it here: http://sugarizer.org/firefoxos/package.zip), size is about 100 Mb


Actual results:

The file seems to be uploaded correctly but there is "Validation timed out" during validation (see screen capture).


Expected results:

I've deployed the same package on a real device, so I expect that it should work.

I've seen a similar bug here https://bugzilla.mozilla.org/show_bug.cgi?id=1019729 but it seems to be fixed.

Comment 1

[Lionel Laské]

Same issue when I'm reduce the package under 100Mb (to 85Mb).

Comment 2

[Andrew Williamson [:eviljeff]]

when I try it doesn't even get to validation timeout - the upload progress bar just stops moving after a while.

Comment 3

[Andy McKay]

I'm surprised it got that far and the max packaged app size doesn't hit us:

https://github.com/mozilla/zamboni/blob/master/mkt/settings.py#L872

Comment 4

[Andrew Williamson [:eviljeff]]

(In reply to Andy McKay [:andym] from comment #3)
> I'm surprised it got that far and the max packaged app size doesn't hit us:
> 
> https://github.com/mozilla/zamboni/blob/master/mkt/settings.py#L872

that limit is only used for hosted app manifests.  I'm not sure we have a defined limit for packaged apps currently - there are a few in the 50mb - 100mb range, though most are less than 5mb I believe.

Comment 5

[Lionel Laské]

No more luck with a 25mb package :-(
So guess that, like the error message suggest, it's more related to a timeout than to a size limitation.
Is there a way to find where the error message come from ?

Comment 6

[Mathieu Pillard [:mat]]

Packaged app upload does have a size limit, see MAX_PACKAGED_APP_SIZE in mkt/constants/submit.py, it is set to 100 * 1024 * 1024  # 100MB.

The timeout probably comes from the app-validator itself chocking on too much code to process or something similar.

Comment 7

[Lionel Laské]

The detailed error message is "Erreur: The validation process took too long to complete. Contact an Firefox Marketplace reviewer for more information. Validation timeout after 110 seconds".
So it probably means that there is really a timeout.

Is there a way to expand the "110" into something bigger.
Plus, I didn't wait 110 seconds before the error message, only few seconds at best.
Does it mean that the starting point of the delay is the beginning of the upload ?

Another thing: searching in the source file mentioned before, I've found https://github.com/mozilla/zamboni/blob/master/mkt/settings.py#L1184

   # Disable timeout code during development because it uses the signal module
   # which can only run in the main thread. Celery uses threads in dev.
   VALIDATOR_TIMEOUT = -1

Does it mean that there is no real timeout and the problem come from elsewhere ?

Any help would be appreciate. I'm stick on the submission of my app due to this issue :-(

Comment 8

[Lionel Laské]

Digging into the Market Place source code, I've found that the validation call come from here https://github.com/mozilla/zamboni/blob/master/mkt/developers/tasks.py#L124 

   with statsd.timer('mkt.developers.validate_packaged_app'):
       return validate_packaged_app(file_path,
           market_urls=settings.VALIDATOR_IAF_URLS,
           timeout=settings.VALIDATOR_TIMEOUT,
           spidermonkey=settings.SPIDERMONKEY)

BTW, not found where is the source code from "validate_package_app" module.

Someone could help here ?

Comment 9

[Mathieu Pillard [:mat]]

In production, VALIDATOR_TIMEOUT is set to 110 in sites/prod/settings_mkt.py so the timeout is working as intended.

validate_packaged_app() comes from https://github.com/mozilla/app-validator/ - you might want to try the standalone validator this repository provides and see if it works on your app on your machine, that might help.

Comment 10

[Lionel Laské]

If 110 is a number of seconds, my feeling is that the validation process stopped before (no more than 2 seconds I think).

Here the output of the validation with app-validator on my machine:

   Summary:
   ------------------------------
   0 Errors, 924 Warnings, 0 Notices
   Test failed! Errors:

Then lot of "Warning: JS: Unknown runtime error".
The detail of the output is here http://sugarizer.org/firefoxos/package.validation.output.txt

Not sure to understand this result. Does it mean my app-validator environment is wrong ?

Comment 11

[Mathieu Pillard [:mat]]

It looks that way, yes. Did you follow the instructions to install spidermonkey ? It doesn't say so on the README (something we should fix) but you can grab the spidermonkey binary from http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/ - also, it should be named 'js' and bin in your PATH, or you need to tweak some settings.

Comment 12

[Lionel Laské]

You're right, spidermonkey was not installed.

Here the result with spidermonkey installed and with a 110 seconds timeout.

   Summary:
   ------------------------------
   0 Errors, 330 Warnings, 0 Notices
   Test failed! Errors:

Strangely, there is 330 warnings (mostly CSP violation) and no error but the test failed.
Detailed report is here: http://sugarizer.org/firefoxos/package.validation.output.txt

Comment 13

[Kumar McMillan [:kumar]]

Since the validation has no errors you should be able to continue submitting your app. It is probably timing out due to a bug. If you're still stuck, a reviewer such as eviljeff or adora (on this thread) can upload the package for you in a way that bypasses the validator.

Comment 14

[Andrew Williamson [:eviljeff]]

(In reply to Kumar McMillan [:kumar] (needinfo for quickness) from comment #13)
> Since the validation has no errors you should be able to continue submitting
> your app. It is probably timing out due to a bug. If you're still stuck, a
> reviewer such as eviljeff or adora (on this thread) can upload the package
> for you in a way that bypasses the validator.

we can?

Comment 15

[Kumar McMillan [:kumar]]

oh, sorry, maybe only admins can do this? https://bugzilla.mozilla.org/show_bug.cgi?id=812538#c5 I don't know if we still have that checkbox

Comment 16

[Mathieu Pillard [:mat]]

IIRC we don't have that functionality in Marketplace, only AMO.

Comment 17

[Lionel Laské]

Yes please, if someone could bypass the validation process and help me to submit it on the marketplace, I will be the happier firefox os developer !

Comment 18

[Andrew Williamson [:eviljeff]]

(In reply to Mathieu Pillard [:mat] from comment #16)
> IIRC we don't have that functionality in Marketplace, only AMO.

yeah, afaik also.  Its been proposed before in response to validator bugs but the preference was always to fix the validator bug instead.  I don't know if this particular case is a validator bug or a genuine timeout due to code complexity though.

Comment 19

[Mathieu Pillard [:mat]]

Looked at it closer today. My guess is, it's a timeout due to code complexity, but maybe there are things we could do better in the validator to avoid it.

On my laptop, with 'js' binary from nightly [0] it takes more than 1 minute at 100% CPU to find that there are no errors, and gives 330 warnings. Somehow it takes longer on prod and dev, don't know if it's because of their js binary or something else. I managed to validate the app on dev in 140 seconds (by manually bypassing the timeout in a shell - tested a few times, I got consistent results).

I still have no idea what we can do to properly fix it, but I don't know the validator very well. A few ideas on what we can do:
- Whitelist some commonly used JS libraries, like enyo and jquery - bug 967724
- Cut down the number of bogus CSP warnings - bug 980342
- Investigate whether a SpiderMonkey update (we can test on dev) would fix the issue
- Consider increasing the timeout to 180 seconds. The current timeout is old, it originated from AMO in bug 776017, where it was bumped from 90 to 110. We have some room since the celery task soft limit is 240, but that's still just a workaround.


[0] http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/jsshell-linux-x86_64.zip

Comment 20

[Mathieu Pillard [:mat]]

https://github.com/mozilla/zamboni/commit/f031179de1ffce9f11d31ac562f366d600bfacec bumps the timeout to 180 seconds.

Comment 21

[Mathieu Pillard [:mat]]

With the new timeout, it now works for me on dev with the package linked in comment 0.

It'll go in production Tuesday around noon PST.

Comment 22

[Lionel Laské]

Just tried, there is now an "Unexpected error during validation" :-(
Both when the package has a 85Mb size and when he has a 25Mb size.

Detail of error message below:

"La validation de votre application a renvoyé 1 erreur.

    Erreur serveur inattendue pendant la validation."

Comment 23

[Mathieu Pillard [:mat]]

Reopening to check comment 22.

Comment 24

[Mathieu Pillard [:mat]]

The actual error is "SoftTimeLimitExceeded: SoftTimeLimitExceeded()". This means we are hitting the validator timeout but the one from our task queue itself.

Comment 25

[Mathieu Pillard [:mat]]

Fixed in https://github.com/mozilla/zamboni/commit/1fb28f5d408bc4350b7962a6fdd1a5f6e7c51d27

STR:
- Try submitting the packaged mentioned in comment 0 (it should take some time once uploaded, but validate anyway)

Comment 26

[Madalin Cotetiu]

Verified in FF36(Win7) in marketplace-dev.allizom.org
I have used the package from comment 0 and the validation was successful but there were lots of warnings (330): http://screencast.com/t/m0TClXJc9m

I have previously verified Bug 967724 where I have used a packaged with jquery file included and no CSP warning was displayed but when uploading the file from comment 0 I see lots of CSP warnings regarding jquery. For example: http://screencast.com/t/7T8eZDRJZ

Mat do you think this is caused by Bug 967724 or is something else?

Comment 27

[Mathieu Pillard [:mat]]

No, that's because the jquery.js file in that package has been re-saved with a windows editor, causing its signature to be slightly different than the original jquery, so that bypasses the whitelist. It's unfortunate, but I'll file another bug for that, thanks.

Comment 28

[Madalin Cotetiu]

Ok Mat thanks for response, I will close this bug then.

Comment 29

[Mathieu Pillard [:mat]]

OP told me it was still happening on prod, reopening :/

Comment 30

[Lionel Laské]

I've retried with the new version of my app with several packaging.
Unfortunately don't work until my package was 12Mb only (89Mb initially).
I've put this same app - Cordova packaged - on Google Play, Amazon Store and Apple Store without any issue. I'm very unhappy with the limited version I must put on the Mozilla Market Place just because it's impossible to upload more than 12Mb without an error.
Read all comments before, you could see that I've done anything possible to help. I'm sorry to say that but Mozilla Market Place should be the #1 Market Place for web app and it's the worst one today for me.