665025 - Very large string literals in XPI package JavaScript files cause validator to fail unexpectedly

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Add-on Validation, defect)

Description

[Zachary Johnson]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.21.1 (KHTML, like Gecko) Version/5.0.5 Safari/533.21.1
Build Identifier: current addons.mozilla.org website

I'm using the Addon SDK (formerly JetPack) to make a JavaScript based plugin for Firefox 4.  The plugin runs perfectly from the command line with 'cfx run' and I can package it with 'cfx xpi'.

When I go to submit my XPI to the website (https://addons.mozilla.org/en-US/developers/addon/submit/2) the Validation steps fails with "Unexpected server error while validating."

I proceeded to delete all my custom JS from my plugin, recreate an XPI, and upload again, and then the validation was successful -- so I knew this was not a temporary server issue.

Through a process of commenting out certain parts of my plugin JS code, recreating the XPI, and reuploading/revalidating I narrowed the problem down to some very large string literals in my plugin JS code.

I have code in the form of:

var x = 'foobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar' +
'foobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar' +
'foobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar' +
...many more lines line that ...;

If the concatenation of the strings being saved into the variable x exceed something greater than 25-35KB then the XPI validator on addons.mozilla.org will fail with "Unexpected server error while validating."

However, if I break the > 30KB string into a few < 20KB strings... there are no problems.  So I can do:

var x = 'foo' + 'foo' + 'foo' + ... 15 more KB;
x += 'foo' + 'foo' + 'foo' + ... 15 more KB;
x += 'foo' + 'foo' + 'foo' + ... 15 more KB;
etc.

That works fine.  Again, however, if I just do:

var x = 'foo' + 'foo' + 'foo' + ... 40 more KB;

Now the validator will fail.



Reproducible: Always

Steps to Reproduce:
1. Create a plugin with the add-on SDK
2. Use the page-mod API to load in a JS file from your plugin's data folder with contentScriptFile
3. Add a JS string variable to the JS code as I described in the Details field of this bug report
4. Package as an XPI and submit to https://addons.mozilla.org/en-US/developers/addon/submit/

Actual Results:  
The XPI uploads and validates, but you get the error:
"Unexpected server error while validating."

Expected Results:  
I should have received a less generic error message about where/how the validator failed.  The validator also should not have failed in the first place.

Comment 1

[krupa raj[:krupa]]

Zachary, do you mind attaching the add-on which generated this error? Thanks!

Comment 2

[Zachary Johnson]

Attachment: Full source of my add-on

In this version of the source I've applied my fix to get past the validation failure.

Comment 3

[Zachary Johnson]

Attachment: XPI with no problems

This XPI already fixed

Comment 4

[Zachary Johnson]

Attachment: XPI WITH Problem

This XPI will create unexpected validation failure.

Comment 5

[Wil Clouser [:clouserw]]

Worked fine for me on the command line:

$ ./addon-validator firebomb-plugin.xpi 

Summary:
------------------------------
Detected type: Extension/Multi-Extension
------------------------------
Test failed! Errors:

Warning: Jetpack module hash unknown
Warning: Jetpack module hash unknown
Warning: Hidden files and folders flagged

Comment 6

[David Chan [:dchan]]

This may be an issue with the prod python setup. The validator JSON output from the bad xpi has the following error

File \"/usr/lib/python2.6/json/scanner.py\", line 55, in iterscan
    rval, next_pos = action(m, context)
File \"/usr/lib/python2.6/json/decoder.py\", line 68, in JSONNumber
    res = fn(integer)
RuntimeError: maximum recursion depth exceeded while calling a Python object

Output cleaned up a little to make readable

Comment 7

[David Chan [:dchan]]

Attachment: output from bad xpi

Comment 8

[Wil Clouser [:clouserw]]

My output was from khan which is supposed to mirror production

Comment 9

[Matt Basta [:basta]]

This problem should have been fixed when we switched from `json` to `simplejson`. If it hasn't been, then this is another reason why we need a streaming JSON parser.

Comment 10

[Kris Maglione [:kmag]]

Don't json and simplejson trivially parse JSON from streams? Or do you mean something SAX-like?

Comment 11

[Matt Basta [:basta]]

SAX-like. We need something that's not based on recursion.