1090434 - e10s talos tp5o is failing on holly - appears to be hung on apple.com

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

looking at the latest push:
https://tbpl.mozilla.org/?tree=Holly&jobname=holly%20talos

we have a lot of red tp jobs.  Going into detail of the logs (except for osx), we have a consistent failure pattern:
https://tbpl.mozilla.org/php/getParsedLog.php?id=51155007&tree=Holly&full=1

15:30:50     INFO -  __WARNTimeout (18/20) exceeded on http://localhost/page_load_test/tp5n/store.apple.com/store.apple.com/us@mco=Nzc1MjMwNA.html__WARN
15:30:50     INFO -  RSS: Main: 106823680
15:30:50     INFO -  __WARNTimeout (19/20) exceeded on http://localhost/page_load_test/tp5n/store.apple.com/store.apple.com/us@mco=Nzc1MjMwNA.html__WARN
15:30:50     INFO -  RSS: Main: 105926656

for some reason we hit our internal timeout of 5 seconds with no pageload or mozafterpaint event and terminate the run after 20 attempts.

So why is this happening and it wasn't before?

I think some next steps might be to download a build and run this locally.  Are we executing too many e10s calls?  Is there something special about the store.apple.com webpage that is causing a hang or finding a bug?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

good news! ok, reproced this locally, I get a failure on the store.apple.com page.  I will have to compare the console output with/without e10s as well as try to see if events are making it back.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

and I get this in the console window:
JavaScript error: http://localhost:15707/page_load_test/tp5n/store.apple.com/storeimages.apple.com/1834/store.apple.com/rs1/js/coherent/release/coherent.js, line 18: Error: Permission denied to access property 'apply'


turning e10s off, it works fine.  So somewhere along the lines either this apple website is doing something wrong, or firefox is doing something wrong.

since coherent.js is minified js, it is hard to tell what is really going on, here is the source for it all:
http://people.mozilla.org/~jmaher/taloszips/zips/tp5n.zip

:billm, is there anything you can think of as to why we are getting this error and hanging tp5?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

hacking around in the .js file, I see this as the root cause:
var K=I.prototype.__factory__.apply(I,J);

commenting out that like I am able to continue on and test away.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

some steps to reproduce this:
1) get talos and have it running: https://wiki.mozilla.org/Buildbot/Talos/Running#Running_locally_-_Source_Code
2) get the tp5 pageset (http://people.mozilla.org/~jmaher/taloszips/zips/tp5n.zip), copy it to the talos/page_load_test/ directory, then unzip it there
3) cd talos/
4) run 'python PerfConfigurator.py -v -e /home/jmaher/dump/firefox/firefox --e10s -a tp5o --develop --output tp5.yml --results_url t.out --datazilla-url t.json'  <- NOTE: put a path to your firefox
5) edit page_load_test/tp5n/tp5o.manifest.develop, remove everything but the entry containing store.apple.com
6) run 'python run_tests.py -d -n tp5.yml'

you will see that we don't load and do an internal pageloader timeout.  We actually receive the load event, but we never setup the mozafterpaint event.  If you repeat steps 4-6 but without --e10s, you will see it work just fine.

Comment 5

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Attachment: fix-apple.com

Sorry this took me so long to get to. The problem is that the frame script is using content.wrappedJSObject.setTimeout to do its timing. The coherent.js code is overriding setTimeout with its own version. However, since its version is unprivileged, it's not allowed to call our privileged timeout handler--that's where the failure comes from.

The solution is to use an X-ray wrapper (i.e., don't go through wrappedJSObject) so that we get the default version of setTimeout. With this change, the store.apple.com seems to work.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8517077 [details] [diff] [review]
fix-apple.com

Review of attachment 8517077 [details] [diff] [review]:
-----------------------------------------------------------------

nice, I am glad you could figure this out!

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

please land this on talos and I will work on getting this in production.

Comment 8

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

https://hg.mozilla.org/build/talos/rev/017dca37f35c

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

this is ready for deployment, will be updated this week on inbound.