735473 - RuntimeError: maximum recursion depth exceeded in cmp (validator)

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Developer Pages, defect)

Description

[Kumar McMillan [:kumar]]

STR:
upload this addon: http://people.mozilla.com/~kmcmillan/[e-mail]

Task was called with args: ('cc4214a80142424b8be0e5b737821e70',) kwargs: {}.

The contents of the full traceback was:

Traceback (most recent call last):
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/execute/trace.py", line 34, in trace
   return cls(states.SUCCESS, retval=fun(*args, **kwargs))
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/task/base.py", line 241, in __call__
   return self.run(*args, **kwargs)
 File "/data/www/addons.mozilla.org/zamboni/vendor/lib/python/celery/app/__init__.py", line 141, in run
   return fun(*args, **kwargs)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/nuggets/celeryutils.py", line 35, in wrapped
   return fun(*args, **kw)
 File "/data/www/addons.mozilla.org/zamboni/apps/amo/decorators.py", line 112, in wrapper
   return f(*args, **kw)
 File "/data/www/addons.mozilla.org/zamboni/apps/amo/decorators.py", line 104, in wrapper
   return f(*args, **kw)
 File "/data/www/addons.mozilla.org/zamboni/apps/devhub/tasks.py", line 45, in validator
   result = run_validator(upload.path)
 File "/data/www/addons.mozilla.org/zamboni/apps/devhub/tasks.py", line 154, in run_validator
   timeout=settings.VALIDATOR_TIMEOUT)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/validate.py", line 47, in validate
   timeout=timeout)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/submain.py", line 91, in prepare_package
   for_appversions)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/submain.py", line 171, in test_package
   output = test_inner_package(err, package, for_appversions)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/submain.py", line 346, in test_inner_package
   test_func(err, xpi_package)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/content.py", line 239, in test_packed_scripts
   pollutable=reversed_script in marked_scripts)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/scripting.py", line 58, in test_js_file
   t.run(tree)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 63, in run
   self._traverse_node(data)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 171, in _traverse_node
   self._interpret_block(b)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 197, in _interpret_block
   self._traverse_node(item)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 477, in _expression
   result = traverser._traverse_node(node["expression"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 340, in _call_expression
   member = traverser._traverse_node(node["callee"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 179, in _func_expr
   results = _function(traverser, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 154, in _function
   traverser._traverse_node(node["body"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 171, in _traverse_node
   self._interpret_block(b)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 197, in _interpret_block
   self._traverse_node(item)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 173, in _traverse_node
   self._traverse_node(b)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 171, in _traverse_node
   self._interpret_block(b)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 197, in _interpret_block
   self._traverse_node(item)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 275, in _define_var
   var_value = traverser._traverse_node(declaration["init"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 306, in _define_obj
   var_value = traverser._traverse_node(prop["value"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 179, in _func_expr
   results = _function(traverser, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 154, in _function
   traverser._traverse_node(node["body"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 171, in _traverse_node
   self._interpret_block(b)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 197, in _interpret_block
   self._traverse_node(item)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 166, in _define_function
   me = _function(traverser, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 154, in _function
   traverser._traverse_node(node["body"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 171, in _traverse_node
   self._interpret_block(b)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 197, in _interpret_block
   self._traverse_node(item)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 173, in _traverse_node
   self._traverse_node(b)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 171, in _traverse_node
   self._interpret_block(b)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 197, in _interpret_block
   self._traverse_node(item)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 275, in _define_var
   var_value = traverser._traverse_node(declaration["init"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 149, in _traverse_node
   action_result = action(self, node)
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/actions.py", line 703, in _expr_binary
   left = traverser._traverse_node(node["left"])
...
 File "/data/www/addons.mozilla.org/zamboni/vendor/src/amo-validator/validator/testcases/javascript/traverser.py", line 121, in _traverse_node
   if "type" not in node or not self._can_handle_node(node["type"]):
RuntimeError: maximum recursion depth exceeded in cmp

Comment 1

[Matt Basta [:basta]]

I wonder...is the maximum recursion depth a limit on the number of times any one function appears on the stack or is it a limit on the stack size itself? If it's the latter, I can think of some low-hanging fruit that can buy us some additional stack space.

Comment 2

[Reinhold]

Hello Matt,

did you fixed anything; because now I was successfull in uploading this AddOn Version (unchanged)? Or is the problem related to some ressouces conflicts/problems on the server? But anyway SUCCESS

Comment 3

[Kumar McMillan [:kumar]]

(In reply to Matt Basta from comment #1)
> I wonder...is the maximum recursion depth a limit on the number of times any
> one function appears on the stack or is it a limit on the stack size itself?
> If it's the latter, I can think of some low-hanging fruit that can buy us
> some additional stack space.

It's a hard stack limit of 1000. I can't reproduce it in prod or anywhere so my best guess is that celery introduces some recursion in how it dispatches tasks and we're only hitting the limit after the worker has been around for a while.

Comment 4

[Kumar McMillan [:kumar]]

I think we can mitigate this by limiting the number of tasks the validator worker can execute. I filed bug 735715

Comment 5

[gold]

Attachment: This extension was, which I want to upload

Kumar McMillan, I upload that file.

Comment 6

[Kumar McMillan [:kumar]]

Thanks. 

Matt, unlike the other files this one (attached) reproduces the recursion error when running addon-validator from the command line. Can you take a look?

Comment 7

[Matt Basta [:basta]]

The problem is triggered by the chrome/content/loadedpages.js file. Basically, there's a whole mess load of string concatenations, which all pile up on the left value evaluation. It looks like this:

"foo" + "bar" + "zap" + "baz"

becomes

((("foo" + "bar") + "zap") + "baz")

We need to recurse to perform the evaluation, which means that it gets messy quickly (especially with a few hundred concatenations).

To "fix" the issue, I've tweaked a few things:

* One-off functions in the traverser were inlined into the _traverse_node function (namely _interpret_block)
* Unnecessary JSWrapper re-wrappings were eliminated
* BinaryExpression nodes now evaluate their left branch directly if they detect that it contains another BinaryExpression (since this happens quite often)
* The _debug function is replaced with `lambda x: None` when not in debug mode to prevent any overhead from that.

This prevents the error from occurring from the get-go. I was able to verify for the attached XPI.

https://github.com/mozilla/amo-validator/pull/134

Comment 8

[gold]

(In reply to Matt Basta from comment #7)
> The problem is triggered by the chrome/content/loadedpages.js file.
> Basically, there's a whole mess load of string concatenations, which all
> pile up on the left value evaluation. It looks like this:
> 
> "foo" + "bar" + "zap" + "baz"
> 
> becomes
> 
> ((("foo" + "bar") + "zap") + "baz")
> 
> We need to recurse to perform the evaluation, which means that it gets messy
> quickly (especially with a few hundred concatenations).
> 
> To "fix" the issue, I've tweaked a few things:
> 
> * One-off functions in the traverser were inlined into the _traverse_node
> function (namely _interpret_block)
> * Unnecessary JSWrapper re-wrappings were eliminated
> * BinaryExpression nodes now evaluate their left branch directly if they
> detect that it contains another BinaryExpression (since this happens quite
> often)
> * The _debug function is replaced with `lambda x: None` when not in debug
> mode to prevent any overhead from that.
> 
> This prevents the error from occurring from the get-go. I was able to verify
> for the attached XPI.
> 
> https://github.com/mozilla/amo-validator/pull/134

so, now I can to upload the file? it's everything ok? or I need to change something in the extention?

Comment 9

[Kumar McMillan [:kumar]]

gold, we'll be deploying this fix later today. By tomorrow you should be able to upload your add-on. Thanks for your patience.


Fixed: https://github.com/mozilla/amo-validator/commit/413285101024c9882e069d553d1b472cf4f2206d

Pushed to AMO: https://github.com/mozilla/zamboni/commit/ece4e70