Last Comment Bug 994459 - B2G l10n nightly bustage from "too much recursion" error (make[6]: *** [webapp-optimize] Error 3 and other errors)
: B2G l10n nightly bustage from "too much recursion" error (make[6]: *** [webap...
Status: RESOLVED FIXED
: intermittent-failure, regression
Product: Firefox OS
Classification: Client Software
Component: Gaia::L10n (show other bugs)
: unspecified
: All All
-- blocker (vote)
: ---
Assigned To: Zibi Braniecki [:gandalf][:zibi]
:
:
Mentors:
Depends on:
Blocks: 914414
  Show dependency treegraph
 
Reported: 2014-04-09 19:56 PDT by Ryan VanderMeulen [:RyanVM]
Modified: 2014-04-10 17:42 PDT (History)
7 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
patch (3.44 KB, patch)
2014-04-10 00:10 PDT, Zibi Braniecki [:gandalf][:zibi]
stas: review+
Details | Diff | Splinter Review
Pull request (46 bytes, text/x-github-pull-request)
2014-04-10 04:47 PDT, Staś Małolepszy :stas
bugmail: review+
poirot.alex: review+
Details | Review | Splinter Review

Description User image Ryan VanderMeulen [:RyanVM] 2014-04-09 19:56:03 PDT
This either needs immediate fixing or bug 914414 needs to be backed out.

https://tbpl.mozilla.org/php/getParsedLog.php?id=37533183&tree=Mozilla-Central

b2g_mozilla-central_linux32_gecko_localizer nightly on 2014-04-09 16:09:26 PDT for push 690c810c8e3e
slave: bld-linux64-spot-120

Exception: InternalError: too much recursion

make[6]: *** [webapp-optimize] Error 3
make[5]: *** [libs] Error 2
make[6]: Leaving directory `/builds/slave/m-cen-linux32_g_lz-ntly-000000/build/gaia'
make[5]: Leaving directory `/builds/slave/m-cen-linux32_g_lz-ntly-000000/build/obj-firefox/b2g/gaia'
make[4]: *** [b2g/gaia/libs] Error 2
make[4]: Leaving directory `/builds/slave/m-cen-linux32_g_lz-ntly-000000/build/obj-firefox'
make[3]: *** [libs] Error 2
make[3]: Leaving directory `/builds/slave/m-cen-linux32_g_lz-ntly-000000/build/obj-firefox'
make[2]: *** [default] Error 2
make[2]: Leaving directory `/builds/slave/m-cen-linux32_g_lz-ntly-000000/build/obj-firefox'
make[1]: *** [realbuild] Error 2
make[1]: Leaving directory `/builds/slave/m-cen-linux32_g_lz-ntly-000000/build'
make: *** [build] Error 2
Comment 1 User image Andrew Sutherland [:asuth] 2014-04-09 21:17:08 PDT
It looks like all of the locale-processing code is synchronously generating notifications and the base logic in webapp-optimize.js is just looping through all of the defined locales it knows about:
https://github.com/mozilla-b2g/gaia/blob/master/build/webapp-optimize.js#L573

Per my grep, it has taken care of 41 locales before in 423 frames before it blows the 512k stack limit it has on (non-debug) 32-bit linux.

A setTimeout/do_execute_soon would probably go a long way in that code.  (Specifically, don't touch the setter on https://github.com/mozilla-b2g/gaia/blob/master/build/webapp-optimize.js#L585 in the same loop cycle that dispatchEvent is being called.)
Comment 2 User image Zibi Braniecki [:gandalf][:zibi] 2014-04-09 22:24:11 PDT
looking into it
Comment 3 User image Zibi Braniecki [:gandalf][:zibi] 2014-04-09 22:55:19 PDT
can't reproduce it on my local machine with 62 locales on macos. Is it possible that we have different stack limit per platform?

The logic around locale cycling didn't change, but the stack have got deeper.
Comment 4 User image Andrew Sutherland [:asuth] 2014-04-09 23:05:40 PDT
Yes, you have a 7 megabyte stack on OS X.  You will be unable to reproduce.
http://dxr.mozilla.org/mozilla-central/source/js/xpconnect/src/XPCJSRuntime.cpp#3051
Comment 5 User image Zibi Braniecki [:gandalf][:zibi] 2014-04-09 23:06:08 PDT
One way to get it out of a single loop would be to just launch each app separately in https://github.com/mozilla-b2g/gaia/blob/master/build/webapp-optimize.js#L706

But there's no setTimeout in that code. Any idea what else can I use?
Comment 6 User image Zibi Braniecki [:gandalf][:zibi] 2014-04-10 00:10:35 PDT
Created attachment 8404483 [details] [diff] [review]
patch

This code does exactly the same logic, but skips the pseudo-event dispatching.

It capitalizes on the fact that resource loading in buildtime mode is synchronous.
Comment 7 User image Zibi Braniecki [:gandalf][:zibi] 2014-04-10 00:17:06 PDT
We should overall clean webapp-optimize use of l10n.js. I filed bug 994520.
Comment 8 User image Andrew Sutherland [:asuth] 2014-04-10 00:22:51 PDT
I can confirm based on inspection that this should address the problem as I understood it from the backtraces.

Specifically, the problem used to be caused by a write to:
  mozL10n.language.code = l10nLocales[processedLocales];
which would then do:
  navigator.mozL10n.ctx.requestLocales(lang);

The revised patch directly calls:
  lang = l10nLocales[processedLocales];
  mozL10n.ctx.requestLocales(lang);

And the control flow has been refactored so that the same effective linear code-path is the same, but without the recursion.

So r=asuth for what it's worth assuming travis is green.
Comment 9 User image Staś Małolepszy :stas 2014-04-10 02:28:15 PDT
I'm taking over while Gandalf is asleep on the west coast.  I'll create a pull request soon.  Thanks everyone for investigating this.
Comment 10 User image Staś Małolepszy :stas 2014-04-10 04:29:38 PDT
Comment on attachment 8404483 [details] [diff] [review]
patch

Review of attachment 8404483 [details] [diff] [review]:
-----------------------------------------------------------------

r=me.  I'll create a pull request for you and ask Andrew and Vivien for a review.

::: build/webapp-optimize.js
@@ +570,4 @@
>      };
>    };
>  
> +  win.dispatchEvent = function() {};

I'd move that to the beginning of the file where win is defined.

@@ +596,5 @@
> +          processedLocales + '/' + l10nLocales.length);
> +
> +        lang = l10nLocales[processedLocales];
> +
> +        mozL10n.ctx.requestLocales(lang);

I think this can simply read:

mozL10n.ctx.requestLocales(l10nLocales[processedLocales]);
Comment 11 User image Staś Małolepszy :stas 2014-04-10 04:47:33 PDT
Created attachment 8404638 [details] [review]
Pull request
Comment 12 User image Andrew Sutherland [:asuth] 2014-04-10 05:23:53 PDT
Comment on attachment 8404638 [details] [review]
Pull request

Noting that I don't really have review authority for this code, (although for a nightly bustage fix I think the bar is a lot lower), this looks substantially the same as the other attachment except for the movement of dispatchEvent, so I'll give it my prestigious r=asuth too.
Comment 13 User image Treeherder Robot 2014-04-10 05:39:34 PDT
Tomcat
https://tbpl.mozilla.org/php/getParsedLog.php?id=37566781&tree=Mozilla-Central
b2g_mozilla-central_linux64_gecko_localizer nightly on 2014-04-10 04:03:10
revision: 690c810c8e3e
slave: bld-linux64-spot-404

make[6]: *** [webapp-optimize] Error 3
make[5]: *** [libs] Error 2
make[4]: *** [b2g/gaia/libs] Error 2
make[3]: *** [libs] Error 2
make[2]: *** [default] Error 2
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2
Comment 14 User image Treeherder Robot 2014-04-10 06:43:46 PDT
Tomcat
https://tbpl.mozilla.org/php/getParsedLog.php?id=37568576&tree=Mozilla-Central
b2g_mozilla-central_linux32_gecko_localizer nightly on 2014-04-10 04:04:41
revision: 690c810c8e3e
slave: bld-linux64-spot-192

make[6]: *** [webapp-optimize] Error 3
make[5]: *** [libs] Error 2
make[4]: *** [b2g/gaia/libs] Error 2
make[3]: *** [libs] Error 2
make[2]: *** [default] Error 2
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2
Comment 15 User image Treeherder Robot 2014-04-10 07:05:42 PDT
Tomcat
https://tbpl.mozilla.org/php/getParsedLog.php?id=37569957&tree=Mozilla-Central
b2g_mozilla-central_linux64_gecko_localizer nightly on 2014-04-10 05:39:37
revision: 690c810c8e3e
slave: bld-centos6-hp-015

make[6]: *** [webapp-optimize] Error 3
make[5]: *** [libs] Error 2
make[4]: *** [b2g/gaia/libs] Error 2
make[3]: *** [libs] Error 2
make[2]: *** [default] Error 2
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2
Comment 16 User image Staś Małolepszy :stas 2014-04-10 08:09:21 PDT
Thanks, Andrew :)

The Travis build is green: https://travis-ci.org/mozilla-b2g/gaia/builds/22685615
Comment 17 User image Staś Małolepszy :stas 2014-04-10 08:12:45 PDT
Comment on attachment 8404638 [details] [review]
Pull request

Alexandre, can you take a look?  Thanks!
Comment 18 User image Treeherder Robot 2014-04-10 08:13:55 PDT
edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=37575216&tree=Mozilla-Central
b2g_mozilla-central_linux64_gecko_localizer nightly on 2014-04-10 07:05:55
revision: 690c810c8e3e
slave: bld-linux64-ix-032

make[6]: *** [webapp-optimize] Error 3
make[5]: *** [libs] Error 2
make[4]: *** [b2g/gaia/libs] Error 2
make[3]: *** [libs] Error 2
make[2]: *** [default] Error 2
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2
Comment 19 User image Alexandre Poirot [:ochameau] 2014-04-10 09:21:56 PDT
Comment on attachment 8404638 [details] [review]
Pull request

Looks good, just see my one comment in the pull request.
Comment 21 User image Treeherder Robot 2014-04-10 17:42:38 PDT
RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=37605621&tree=Mozilla-Central
b2g_mozilla-central_linux64_gecko_localizer nightly on 2014-04-10 16:15:48
revision: 83ae54e18689
slave: bld-linux64-ix-037

make[6]: *** [webapp-optimize] Error 3
make[5]: *** [libs] Error 2
make[4]: *** [b2g/gaia/libs] Error 2
make[3]: *** [libs] Error 2
make[2]: *** [default] Error 2
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2

Note You need to log in before you can comment on or make changes to this bug.