773817 - (metro-talos) Get talos tests running in Win8 Metro immersive mode

Status: RESOLVED WONTFIX

(Tracking Graveyard :: Metro Operations, defect)

Description

[Jim Mathies [:jimm]]

+++ This bug was initially created as a clone of Bug #771271 +++

This is for figuring out how to get talos tests running in metro mode. Other bugs will be filed on rel-eng related work.

Comment 1

[Jim Mathies [:jimm]]

For these tests we would like them to be run on touch enabled systems so our accessibility code is in use. The Samsung Series 7 (or 8?) would be a good touch tablet to use.

Comment 2

[Jim Mathies [:jimm]]

Similar to chrome tests, I think the test launch harness can be used here.

Comment 3

[Jim Mathies [:jimm]]

oops, wrong bug.

Comment 4

[Jim Mathies [:jimm]]

From looking over our current set of tests, I think we only need a small subset of these tests:

dromaeojs - good to have
svgr - good to have
tp5o - must have
other - no need, we can't open multiple windows. plus we have startup perf covered by mochiperf. 
dirtypaint - no need, can't open multiple windows.
chromez - no need, tresize not a valid test.

Comment 5

[Jim Mathies [:jimm]]

Looking at the tpo5 logs, it looks like this should be pretty easy to integrate. We just need to hook up the metro test harness to run_tests.py - 

'python' 'run_tests.py' '--noisy' '20130715_0619_config.yml' '--datazilla-url' 'https://datazilla.mozilla.org/talos' '--authfile' 'BuildSlaves.py'

Bhavana, curious if you could help us get this coordinated. We're getting closer to rolling out on the trains and would like to get a subset of talos tests running in the metro browser. I think all the code to get these running is in the tree from our work with browser chrome tests. But there's still some integration/testing work to do in build.

Comment 6

[bhavana bajaj [:bajaj]]

(In reply to Jim Mathies [:jimm] from comment #5)
> Looking at the tpo5 logs, it looks like this should be pretty easy to
> integrate. We just need to hook up the metro test harness to run_tests.py - 
> 
> 'python' 'run_tests.py' '--noisy' '20130715_0619_config.yml'
> '--datazilla-url' 'https://datazilla.mozilla.org/talos' '--authfile'
> 'BuildSlaves.py'
> 
> Bhavana, curious if you could help us get this coordinated.

Sure thing, Jim.
> We're getting
> closer to rolling out on the trains and would like to get a subset of talos
> tests running in the metro browser. I think all the code to get these
> running is in the tree from our work with browser chrome tests. But there's
> still some integration/testing work to do in build.

To get started here I am looping in Armen who I believe has helped a lot in getting the tests running :) Armen given above comment from Jim, can you please outline the next steps here and let us know if releng can help with integration ?

Comment 7

[Armen [:armenzg]]

Hi,
I've looked at this bug and it seems that few things are needed to be clarified.

1) have all the suites needed from comment 4 been run successfully?
2) do we need a-team's support?

Once those are clear, we should file a Releng bug with information on how to run these.

I currently don't have time to help with this. Please follow up with Coop to find someone to help on the releng side.

Comment 8

[Armen [:armenzg]]

On another note, is this supposed to run on the Win8 64-bit machines? like our unit tests?

Comment 9

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) (EDT/UTC-4) from comment #7)
> Hi,
> I've looked at this bug and it seems that few things are needed to be
> clarified.
> 
> 1) have all the suites needed from comment 4 been run successfully?

No, but I can do that. In the past I've always had to run talos tests manually by loading web pages and hacks. You guys have been doing some work to get these in-tree I think, is there an easy way to get these going locally these days?
  
> 2) do we need a-team's support?

Not sure.

Comment 10

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) (EDT/UTC-4) from comment #8)
> On another note, is this supposed to run on the Win8 64-bit machines? like
> our unit tests?

Whatever slave hardware we currently use for Win8 opt / talos, which I think is the same set of slaves we currently use for mochitests.

Comment 11

[Armen [:armenzg]]

If you checkout mozharness you can run this:

C:\slave\test\build\venv\Scripts\talos --noisy --debug -v --executablePath C:\slave\test\build\application\firefox\firefox --title T-W864-IX-038 --symbolsPath http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/cedar-win32/1374518401/firefox-25.0a1.en-US.win32.crashreporter-symbols.zip --activeTests tdhtmlr:tresize --results_url http://graphs.mozilla.org/server/collect.cgi --output talos.yml --branchName Cedar --datazilla-url https://datazilla.mozilla.org/talos --authfile C:\slave\test\oauth.txt --mozAfterPaint --filter ignore_first:5 --filter median --webServer localhost

I think you will have to add the following parameters:
* --no-read-builtbot-configs
* --installer-url <url_to_binary>
* --test-url <url_to_tests.zip>

How are you going to make it trigger the metro version rather than the default?

FYI, mozharness is going to be running talos.
run_tests.py will still get run.

I just read your comment #5 and you will want to throw any talos patches to the a-team. We don't own the talos code. jmaher could direct you to the right person.

Does this help?

Comment 12

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) (EDT/UTC-4) from comment #11)
> If you checkout mozharness you can run this:
> 
> C:\slave\test\build\venv\Scripts\talos --noisy --debug -v --executablePath
> C:\slave\test\build\application\firefox\firefox --title T-W864-IX-038
> --symbolsPath
> http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/cedar-win32/
> 1374518401/firefox-25.0a1.en-US.win32.crashreporter-symbols.zip
> --activeTests tdhtmlr:tresize --results_url
> http://graphs.mozilla.org/server/collect.cgi --output talos.yml --branchName
> Cedar --datazilla-url https://datazilla.mozilla.org/talos --authfile
> C:\slave\test\oauth.txt --mozAfterPaint --filter ignore_first:5 --filter
> median --webServer localhost
> 
> I think you will have to add the following parameters:
> * --no-read-builtbot-configs
> * --installer-url <url_to_binary>
> * --test-url <url_to_tests.zip>
> 
> How are you going to make it trigger the metro version rather than the
> default?

The executablePath param can point to the harness, which will launch the right browser. 

Open issues:

1) make sure the metroharness is packaged in whatever distribution talos uses when it runs on a slave.
2) make sure that we are going to be using win8 slaves that get Q's default browser config setup.
3) make sure output from the browser is getting to the build harness that needs it.

> FYI, mozharness is going to be running talos.

Is it running talos now, or will be at some point (bug #)? I'm not familiar with the build related logic wrapped around talos. (Hence why in testing I've always had to run these tests manually.)

> run_tests.py will still get run.


What is run_tests.py? Up above you used as an example a script called 'talos' - 
located at 'C:\slave\test\build\venv\Scripts\talos'.

> I just read your comment #5 and you will want to throw any talos patches to
> the a-team. We don't own the talos code. jmaher could direct you to the
> right person.

I'm hoping we won't have to change anything the the content for the actual talos test.

Comment 13

[Jim Mathies [:jimm]]

> > FYI, mozharness is going to be running talos.
> 
> Is it running talos now, or will be at some point (bug #)? I'm not familiar
> with the build related logic wrapped around talos. (Hence why in testing
> I've always had to run these tests manually.)
> 
> > run_tests.py will still get run.
> 
> 
> What is run_tests.py? Up above you used as an example a script called
> 'talos' - 
> located at 'C:\slave\test\build\venv\Scripts\talos'.

I think I've found this but it's runtests.py, rather than run_tests.py, located in build under the talos directory -

http://mxr.mozilla.org/build/source/talos/

Comment 14

[Jim Mathies [:jimm]]

(In reply to Jim Mathies [:jimm] from comment #13)
> > > FYI, mozharness is going to be running talos.
> > 
> > Is it running talos now, or will be at some point (bug #)? I'm not familiar
> > with the build related logic wrapped around talos. (Hence why in testing
> > I've always had to run these tests manually.)
> > 
> > > run_tests.py will still get run.
> > 
> > 
> > What is run_tests.py? Up above you used as an example a script called
> > 'talos' - 
> > located at 'C:\slave\test\build\venv\Scripts\talos'.
> 
> I think I've found this but it's runtests.py, rather than run_tests.py,
> located in build under the talos directory -
> 
> http://mxr.mozilla.org/build/source/talos/

Al, might also be - 

http://mxr.mozilla.org/build/source/talos/talos/run_tests.py

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

http://mxr.mozilla.org/build/source/talos/talos/run_tests.py is the main script to run talos, you can pass in all commandline parameters this way.

Comment 16

[Jim Mathies [:jimm]]

> Ah, might also be - 
> 
> http://mxr.mozilla.org/build/source/talos/talos/run_tests.py

I'm not having any luck getting this to run locally. It has various missing dependencies that it doesn't find on its own.

I think I can hack these tests so that I can run them manually, but getting build scripts working will require testing in whatever environment we run this script in normally.

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

talos depends on external dependencies, these are resolved in mozharness via the script and the virtualenv.

prior to mozharness, these are resolved by including all dependencies in talos.zip.

you can create a talos.zip by running create_talos_zip.py from the root directory.

Comment 18

[Jim Mathies [:jimm]]

(In reply to Joel Maher (:jmaher) from comment #17)
> talos depends on external dependencies, these are resolved in mozharness via
> the script and the virtualenv.
> 
> prior to mozharness, these are resolved by including all dependencies in
> talos.zip.
> 
> you can create a talos.zip by running create_talos_zip.py from the root
> directory.

That doesn't run on Windows. :/

I did find some of the missing deps over in the tree in testing/mozbase. Installing those seems to have helped. Although I'm still missing a some things like dzclient. Maybe I can hack around the rest.

Comment 19

[Jim Mathies [:jimm]]

Neat, I managed to get the harness to launch metro and try to run tresize. Which obviously fails but we're making progress.

Anyone know how we serve up page load content? I'm specifying a local server but the harness doesn't appear to set up anything.

Comment 20

[Joel Maher ( :jmaher ) (UTC -8)]

if you run with --develop, it will create a webserver, otherwise it expects a server http://localhost to be running on the system and pointing at the talos/ directory :)

Comment 21

[Armen [:armenzg]]

We have Apache on the test machines.

This is similar (if not exactly the same) as to what we do on the Win8 test machines:
https://github.com/armenzg/bearded-ironman/blob/master/win8_64bit/setup.bat

Comment 22

[Jim Mathies [:jimm]]

Ok, here are the results:

http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.py#166

couldn't test due to missing resources in the repo:

tp5n, tp5o

shouldn't run in metro:

tpaint, tspaint_places_generated_med, tspaint_places_generated_max,
tresize

ran successfully:

a11yr, ts_paint, tsvgr, tsvgr_opacity,
tscrollx, tscrollr, tsvgx, tcanvasmark,
dromaeo_css, dromaeo_dom, kraken, v8_7

The only odd thing I'm not sure of, after the inital pageloader window loads up - 

http://mxr.mozilla.org/build/source/talos/talos/pageloader/chrome/pageloader.xul

I don't see any rendering. All tests run (evident by the test output and drain on the system). Not sure if this is expected or not.

tested using the following command line:

$ python run_tests.py --noisy --debug -v --executablePath /T/Mozilla/MC-REL/dist/bin/metrotestharness.exe --activeTests (testlist) --output jim.yml
--cycles 1 --develop --mozAfterPaint

Comment 23

[Jim Mathies [:jimm]]

(In reply to Jim Mathies [:jimm] from comment #22)
> The only odd thing I'm not sure of, after the inital pageloader window loads
> up - 
> 
> http://mxr.mozilla.org/build/source/talos/talos/pageloader/chrome/pageloader.
> xul
> 
> I don't see any rendering. All tests run (evident by the test output and
> drain on the system). Not sure if this is expected or not.

I guess this isn't quite right. From a run on desktop I see we create two windows, and in metro we load only one. I think we'll need to make a minor modification to touch this up.

Comment 24

[Jim Mathies [:jimm]]

(In reply to Jim Mathies [:jimm] from comment #23)
> (In reply to Jim Mathies [:jimm] from comment #22)
> > The only odd thing I'm not sure of, after the inital pageloader window loads
> > up - 
> > 
> > http://mxr.mozilla.org/build/source/talos/talos/pageloader/chrome/pageloader.
> > xul
> > 
> > I don't see any rendering. All tests run (evident by the test output and
> > drain on the system). Not sure if this is expected or not.
> 
> I guess this isn't quite right. From a run on desktop I see we create two
> windows, and in metro we load only one. I think we'll need to make a minor
> modification to touch this up.

So talos has the option of running tests in a browser with chrome and without. If without, the talos pageloader.xul is loaded in place of chrome://browser/content/browser.xul. With chrome pageloader loads pageloader.xul in the main window and tries to launch a standalone browser window for tests. This succeeds in metro but the window is hidden, since it doesn't have a real window widget.

So to get these running for now use the --noChrome option.

I have a patch that I'm working on that allows us to run with browser chrome by loading pageloader.xul into a browser tab which would be useful in picking up regressions caused by browser chrome landings (like bug 872780).

Comment 25

[Jim Mathies [:jimm]]

couldn't test due to missing resources in the repo:

tp5n, tp5o

shouldn't run in metro:

tpaint, tspaint_places_generated_med, tspaint_places_generated_max,
tresize

broken in metro:

tscanvasmark (bug 897409)

ran successfully:

a11yr, ts_paint, tsvgr, tsvgr_opacity,
tscrollx, tscrollr, tsvgx, dromaeo_css,
dromaeo_dom, kraken, v8_7

Comment 26

[Joel Maher ( :jmaher ) (UTC -8)]

is there a need for tresize or tpaint?  also tsvgr is to be replaced soo with tsvgx and tscrollr is soon to be replaced with tscrollx.  We also have tart coming online soon and canvasmark in staging.

Comment 27

[Jim Mathies [:jimm]]

(In reply to Joel Maher (:jmaher) from comment #26)
> is there a need for tresize or tpaint?  also tsvgr is to be replaced soo
> with tsvgx and tscrollr is soon to be replaced with tscrollx.  We also have
> tart coming online soon and canvasmark in staging.

tresize and tpaint can't run since they create windows. I suppose if tscrollr runs tscrollx will. I don't know about 'tart', it's not in the tree afaict. We can run canvasmark but we shouldn't trust the results until bug 897409 is fixed. (see comment 25)

Comment 28

[Joel Maher ( :jmaher ) (UTC -8)]

great!  Are you using --noChrome on the commandline to avoid pageloader window problems?  This is what we do on Android.

Comment 29

[Jim Mathies [:jimm]]

(In reply to Joel Maher (:jmaher) from comment #28)
> great!  Are you using --noChrome on the commandline to avoid pageloader
> window problems?  This is what we do on Android.

Currently you can only run with --noChrome, bug 897417 has patches to get chrome runs going.

Comment 30

[bhavana bajaj [:bajaj]]

Armen, going by comment #7 , looks like we have test suites run successfully based on Jim's work above. Regarding your 2nd point - to get A team's help, Its unclear what we should be requesting for to them and what is needed, can you help here ?

If we do not need A team's attention what are the next steps here ?

Note we are considering this a high priority given metro will will be riding the trains soon.

Comment 31

[Armen [:armenzg]]

Hi bajaj,
jlund will be working on bug 897420 to get this going.
For now, there's no need to get the A-team involved until we hit issues in our automation.

HTH,
Armen

Comment 32

[Jim Mathies [:jimm]]

Clearly not going to get this done in time.

Comment 33

[Lukas Blakk [:lsblakk] use ?needinfo]

Will track this for FF26 since that is the intended target for Metro release.

Comment 34

[Lukas Blakk [:lsblakk] use ?needinfo]

Jim - FF26 is in beta now, on week 2 - what's the eta for this work?

Comment 35

[Lukas Blakk [:lsblakk] use ?needinfo]

That being said, I realize we are not shipping Metro with FF26 now, it's staying on Aurora (27) so untracking for 26.

Comment 36

[Jim Mathies [:jimm]]

(In reply to [e-mail] [:lsblakk] from comment #34)
> Jim - FF26 is in beta now, on week 2 - what's the eta for this work?

We're rolling out with 28. Looks like releng is really close to finishing this up. They have tests running on cedar currently. Dep bugs track remaining work.

Comment 37

[Jim Mathies [:jimm]]

moving [story] bugs over to tracking product.

Comment 38

[Lukas Blakk [:lsblakk] use ?needinfo]

Jim, there's no one assigned here - is this truly a blocker for release?  Do we have no talos running on Metro at all or is this additional coverage?  What's the risk here if we don't have this on while 28 is in Beta?

Comment 39

[Matt Brubeck (:mbrubeck)]

No, we won't block release for this.  We should do this work on trunk and let it ride the trains.