Very large string literals in XPI package JavaScript files cause validator to fail unexpectedly

RESOLVED FIXED

Status

addons.mozilla.org Graveyard
Add-on Validation
RESOLVED FIXED
7 years ago
2 years ago

People

(Reporter: Zachary Johnson, Unassigned)

Tracking

Details

(Whiteboard: [streaming-json-parser], URL)

Attachments

(4 attachments)

(Reporter)

Description

7 years ago
User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.21.1 (KHTML, like Gecko) Version/5.0.5 Safari/533.21.1
Build Identifier: current addons.mozilla.org website

I'm using the Addon SDK (formerly JetPack) to make a JavaScript based plugin for Firefox 4.  The plugin runs perfectly from the command line with 'cfx run' and I can package it with 'cfx xpi'.

When I go to submit my XPI to the website (https://addons.mozilla.org/en-US/developers/addon/submit/2) the Validation steps fails with "Unexpected server error while validating."

I proceeded to delete all my custom JS from my plugin, recreate an XPI, and upload again, and then the validation was successful -- so I knew this was not a temporary server issue.

Through a process of commenting out certain parts of my plugin JS code, recreating the XPI, and reuploading/revalidating I narrowed the problem down to some very large string literals in my plugin JS code.

I have code in the form of:

var x = 'foobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar' +
'foobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar' +
'foobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar' +
...many more lines line that ...;

If the concatenation of the strings being saved into the variable x exceed something greater than 25-35KB then the XPI validator on addons.mozilla.org will fail with "Unexpected server error while validating."

However, if I break the > 30KB string into a few < 20KB strings... there are no problems.  So I can do:

var x = 'foo' + 'foo' + 'foo' + ... 15 more KB;
x += 'foo' + 'foo' + 'foo' + ... 15 more KB;
x += 'foo' + 'foo' + 'foo' + ... 15 more KB;
etc.

That works fine.  Again, however, if I just do:

var x = 'foo' + 'foo' + 'foo' + ... 40 more KB;

Now the validator will fail.



Reproducible: Always

Steps to Reproduce:
1. Create a plugin with the add-on SDK
2. Use the page-mod API to load in a JS file from your plugin's data folder with contentScriptFile
3. Add a JS string variable to the JS code as I described in the Details field of this bug report
4. Package as an XPI and submit to https://addons.mozilla.org/en-US/developers/addon/submit/

Actual Results:  
The XPI uploads and validates, but you get the error:
"Unexpected server error while validating."

Expected Results:  
I should have received a less generic error message about where/how the validator failed.  The validator also should not have failed in the first place.

Comment 1

7 years ago
Zachary, do you mind attaching the add-on which generated this error? Thanks!
(Reporter)

Comment 2

7 years ago
Created attachment 540077 [details]
Full source of my add-on

In this version of the source I've applied my fix to get past the validation failure.
(Reporter)

Comment 3

7 years ago
Created attachment 540078 [details]
XPI with no problems

This XPI already fixed
(Reporter)

Comment 4

7 years ago
Created attachment 540080 [details]
XPI WITH Problem

This XPI will create unexpected validation failure.
Worked fine for me on the command line:

$ ./addon-validator firebomb-plugin.xpi 

Summary:
------------------------------
Detected type: Extension/Multi-Extension
------------------------------
Test failed! Errors:

Warning: Jetpack module hash unknown
Warning: Jetpack module hash unknown
Warning: Hidden files and folders flagged

Comment 6

7 years ago
This may be an issue with the prod python setup. The validator JSON output from the bad xpi has the following error

File \"/usr/lib/python2.6/json/scanner.py\", line 55, in iterscan
    rval, next_pos = action(m, context)
File \"/usr/lib/python2.6/json/decoder.py\", line 68, in JSONNumber
    res = fn(integer)
RuntimeError: maximum recursion depth exceeded while calling a Python object

Output cleaned up a little to make readable
Status: UNCONFIRMED → NEW
Ever confirmed: true

Comment 7

7 years ago
Created attachment 540101 [details]
output from bad xpi
My output was from khan which is supposed to mirror production
Component: Developer Pages → Add-on Validation
QA Contact: developers → add-on-validation

Comment 9

6 years ago
This problem should have been fixed when we switched from `json` to `simplejson`. If it hasn't been, then this is another reason why we need a streaming JSON parser.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Whiteboard: [streaming-json-parser]
Don't json and simplejson trivially parse JSON from streams? Or do you mean something SAX-like?

Comment 11

6 years ago
SAX-like. We need something that's not based on recursion.
(Assignee)

Updated

2 years ago
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.