732206 - [maple] Talos Twinopen ("txul") is broken

Status: RESOLVED WONTFIX

(Firefox for Android Graveyard :: General, defect)

Description

[Matt Brubeck (:mbrubeck)]

Ts and Twinopen are perma-red on Maple since we started running them (bug 731353).

I'm not sure what error if any causes the failure in Ts:
https://tbpl.mozilla.org/php/getParsedLog.php?id=9748345&tree=Maple

Twinopen fails with "timeout exceeded":
https://tbpl.mozilla.org/php/getParsedLog.php?id=9747817&tree=Maple

Comment 1

[Kartikaya Gupta (email:[e-mail])]

Trying a possible fix for this: https://tbpl.mozilla.org/?tree=Try&rev=b0565db2ea63

Comment 2

[Kartikaya Gupta (email:[e-mail])]

Attachment: Guard against nonexistent document element

My fix takes care of Ts. Not sure what's wrong with Twinopen.

Comment 3

[Chris Lord [:cwiiis]]

Comment on attachment 603143 [details] [diff] [review]
Guard against nonexistent document element

Review of attachment 603143 [details] [diff] [review]:
-----------------------------------------------------------------

Looks fine, one small nitpick.

::: mobile/android/chrome/content/browser.js
@@ +1571,5 @@
>  
>    refreshDisplayPort: function(aDisplayPortMargins) {
>      if (this._zoom <= 0)
>        return;
> +    if (! this.browser.contentDocument.documentElement)

Do we usually put spaces after exclamation marks like this?

Comment 4

[Kartikaya Gupta (email:[e-mail])]

(In reply to Chris Lord [:cwiiis] from comment #3)
> Do we usually put spaces after exclamation marks like this?

Nope, just an old habit from the time I wasted half a day because I couldn't see an exclamation mark :p

I'll take it out on landing.

Comment 5

[Kartikaya Gupta (email:[e-mail])]

https://hg.mozilla.org/projects/maple/rev/7f809f941e83

Comment 6

[Joe Drew (not getting mail)]

To land maple, we will hide these tests on mozilla-central.

Comment 7

[Kartikaya Gupta (email:[e-mail])]

It appears as though Ts is pretty broken again. The following output always appears just before it times out:

NOISE: OpenGL LayerManager Initialized Succesfully.
NOISE: Version: OpenGL ES 2.0
NOISE: Vendor: NVIDIA Corporation
NOISE: Renderer: NVIDIA Tegra
NOISE: FBO Texture Target: TEXTURE_2D

I'm not sure where this output is coming from but regardless, the talos suite should be fixed to not die whenever it sees output it doesn't understand.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

I don't believe the talos suite is dying when it sees extra lines.  In this case we are getting a timeout exceeded.  That implies that firefox is running, but not outputting what we expect, therefore not generating the proper log files nor exiting.

What outputs this stuff?  I would think whatever it is is causing fennec to hang.

Comment 9

[Kartikaya Gupta (email:[e-mail])]

I don't see that output anywhere in our source tree. How does talos decide what to output in the "NOISE" section? it's clearly not all of logcat, is there some filter being applied?

Comment 10

[Kartikaya Gupta (email:[e-mail])]

(In reply to Kartikaya Gupta (:kats) from comment #9)
> I don't see that output anywhere in our source tree. How does talos decide
> what to output in the "NOISE" section? it's clearly not all of logcat, is
> there some filter being applied?

Oops, grep fail. I found the output. Still, my question stands.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

talos has a function called noisy which outputs data like the results collected from the pageloader, crashdump information, and the logfile.  In the case of remote talos anything that is written to the browser_output file will show up as noise.

Not sure how that specific data is getting written out to a log file, for the ts tests specifically we read the console messages and output them to the log file, I suspect it is coming from there.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

Ts seems to be working now.  Txul does not work at all and running it locally I am unable to find any log message or other piece of data that leads me towards fixing it.  

What I see is that it launches the proper webpage which is a .xul page and drives opening a .html file in a new tab.  This new file loads and then should terminates giving us a time to open a window.  I don't see it terminating and I don't see console messages in adb logcat.  It just sits there with the throbber spinning nonstop on the main page (urlbar says about:home).  another thing is that I am unable to browse to another page.  

When I do this on a m-c build from yesterday (before the maple merge) I am able to switch the URL to something else, as well as watch the tabs open and close.

Comment 13

[Kartikaya Gupta (email:[e-mail])]

(In reply to Joel Maher (:jmaher) from comment #12)
> Ts seems to be working now.  Txul does not work at all and running it
> locally I am unable to find any log message or other piece of data that
> leads me towards fixing it.  

Do you know the command line that Txul invokes to start fennec? If I can reproduce the behaviour with that command I can add logging or trace with gdb to find out what's happening. Even if I can't repro we should compare it to the ts command line and see if there are differences.

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

I can't seem to find the wiki now, so here are some instructions:
hg clone http://hg.mozilla.org/build/talos
cd talos
python INSTALL.py
. bin/activate
cd talos

#cleanup device
adb shell
$>rm -R /mnt/sdcard/tests
$>exit

#create ts configuration
python remotePerfConfigurator -v -e org.mozilla.fennec --remoteDevice <ip> -a ts --develop --output ts.yml
#create txul configuration
python remotePerfConfigurator -v -e org.mozilla.fennec --remoteDevice <ip> -a txul --develop --output txul.yml

#run tests
python run_tests.py -d -n ts.yml
python run_tests.py -d -n txul.yml

Comment 16

[Armen [:armenzg]]

from bug 736107:
(In reply to Marco Bonardo [:mak] from comment #0)
> https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=Android
> 
> At a first look looks like it started being permanently red from the
> off-main-thread-compositing merge, though this test was already randomly
> timeing out before, that makes the thing a bit harder to say for sure.

Comment 17

[Kartikaya Gupta (email:[e-mail])]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #16)
> from bug 736107:
> (In reply to Marco Bonardo [:mak] from comment #0)
> > https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=Android
> > 
> > At a first look looks like it started being permanently red from the
> > off-main-thread-compositing merge, though this test was already randomly
> > timeing out before, that makes the thing a bit harder to say for sure.

Yeah, I hid that test on central before landing maple because I knew it would go red, but forgot to hide it on inbound. Will do that now.

Comment 18

[Kartikaya Gupta (email:[e-mail])]

When I run this command:

python remotePerfConfigurator.py -v -e org.mozilla.fennec --remoteDevice <ip> -a txul --develop --output txul.yml

I get this:

reconnecting socket
DeviceManager: error pulling file '/data/data/org.mozilla.fennec/application.ini': No such file or directory
remotePerfConfigurator.py: ERROR: Unable to copy master ini file from device - either it doesn't exist yet (have you run fennec at least once?) or you don't have permissions to get it (workaround: extract it from apk locally)

So I pulled the application.ini out of the apk and put at it ./remoteapp.ini and ran it again, now I get a txul.yml file generated, but it has an empty tests section at the end, and running it gives me this error:

(talos)kats@kgupta-pc talos$ python run_tests.py -d -n txul.yml 
setting debug
DEBUG: running test file txul.yml
Traceback (most recent call last):
  File "run_tests.py", line 681, in <module>
    main()
  File "run_tests.py", line 678, in main
    test_file(arg, options, parser.parsed)
  File "run_tests.py", line 440, in test_file
    tests = useBaseTestDefaults(yaml_config.get('basetest', []), tests)
  File "run_tests.py", line 383, in useBaseTestDefaults
    for test in tests:
TypeError: 'NoneType' object is not iterable

I posted the generated txul.yml at https://people.mozilla.com/~kgupta/tmp/txul.yml

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

kats, sorry, the perfconfigurator line should have '-a twinopen' instead of '-a txul'

Comment 20

[Kartikaya Gupta (email:[e-mail])]

Btw, I was finally able to reproduce this. The twinopen test actually seems to be spawning new XUL windows as part of the test, and we don't handle that case very gracefully right now. It should never happen during normal use, though. (Except maybe via addons? Not sure about that). I did test normal window.open calls from a .html page and those work fine, so I'm disinclined to continue debugging this until we've got the beta/blocker list down.

Comment 21

[Matt Brubeck (:mbrubeck)]

Perhaps we should WONTFIX this and disabled the test.  Opening new windows is something we never do in native Fennec, and it's no longer a good proxy for startup time either (as it kind of was in XUL Fennec and desktop Firefox).

Comment 22

[Kartikaya Gupta (email:[e-mail])]

I'm happy with that solution.