847442 - Get metro tests integrated with automation

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Description

[Jim Mathies [:jimm]]

Armenz asked me to file this info bug on metro mochitest.

These tests are currently running on mc manually. Basically:

pymake mochitest-metro-chrome 

will invoke them. There will likely be some heavy random orange the first time they run in automation which we can work to patch up pretty quick. So initially they should be hidden.

The way these work:

1) pymake mochitest-metro-chrome invokes a new test harness located here - 

http://mxr.mozilla.org/mozilla-central/source/browser/metro/shell/testing/

2) metrotestharness.cpp writes out test related command line params to an ini

These are written to an ini file (tests.ini) located in the root obj directory the test are to be run in.

http://mxr.mozilla.org/mozilla-central/source/browser/metro/shell/testing/metrotestharness.cpp#176

The reason for this is that the metro browser can't launch from the command line. It must be invoked through a windows api call.

http://mxr.mozilla.org/mozilla-central/source/browser/metro/shell/testing/metrotestharness.cpp#211

3) after Windows launches the browser, nsBrowserApp.cpp reads in the temporary ini file parameters when it launches.

http://mxr.mozilla.org/mozilla-central/source/browser/app/nsBrowserApp.cpp#276

4) the browser starts up and the tests run.

There are a couple setup requirements:

1) The metro browser must be registered as the default system browser before tests can run. You can replace the binaries when the browser isn't running, but the location must be where the default browser was located when it was set. Otherwise step 2 above will fail.

2) Registering the default browser can't be automated. It requires user interaction to click on the appropriate popup and open the default programs control panel to select firefox.

3) the test harness needs write permissions to obj dir.

Comment 1

[Armen [:armenzg]]

Metro specific rather than part of the win8 infra.

Comment 2

[Jim Mathies [:jimm]]

We also have the ability to run mochitest-plain and mochitest-chrome (bug 837772) but the tests have not been triaged. A lot will probably fail. Triaging will happen in bug 843420.

Comment 3

[Armen [:armenzg]]

I'm on buildduty this week and won't be abe to have a look at this before next week.

Comment 4

[Armen [:armenzg]]

Grabbed the builds and tests from here:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1364206260/

jimm, I'm running this but I can't find the metrotestharness.exe:

C:\Users\cltbld.T-W864-IX-042\Desktop>c:\mozilla-build\python27\\python -u c:\Users\cltbld.T-W864-IX-042\Desktop\tests\mochitest\runtests.py --appname=C:\Users\cltbld.T-W864-IX-042\Desktop\firefox-22.0a1.en-US.win32\firefox\firefox.exe --utility-path=tests/bin --extra-profile-file=tests/bin/plugins --symbols-path=http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1364206260/firefox-22.0a1.en-US.win32.crashreporter-symbols.zip --certificate-path=tests/certs --autorun --close-when-done --console-level=INFO --metro-immersive --browser-chrome"
Usage: Usage instructions for runtests.py.
All arguments are optional.
If --chrome is specified, chrome tests will be run instead of web content tests.

If --browser-chrome is specified, browser-chrome tests will be run instead of we
b content tests.
See <http://mochikit.com/doc/html/MochiKit/Logging.html> for details on the logg
ing levels.

runtests.py: error: C:\Users\cltbld.T-W864-IX-042\Desktop\tests\bin\metrotesthar
ness.exe not found, cannot launch immersive tests.

C:\Users\cltbld.T-W864-IX-042\Desktop>

Comment 5

[Jim Mathies [:jimm]]

It's in the dist/bin folder of the zip with the other exes. You can pass this to runtests using the --utility-path command line option. 

http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#57

http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#371

Comment 6

[Jim Mathies [:jimm]]

Looking at a test zip, it looks like we'll have to package this up in the bin dir. Let me see if I can find where we do that.

Comment 7

[Jim Mathies [:jimm]]

(In reply to Jim Mathies [:jimm] from comment #6)
> Looking at a test zip, it looks like we'll have to package this up in the
> bin dir. Let me see if I can find where we do that.

Fixed on mc. Should be in the next zip.

Comment 8

[Armen [:armenzg]]

I've managed to start the tests but it gets stuck in here:
http://cl.ly/NqpY
which seems to wait on:
http://cl.ly/Nqh7

After adding a firewall exception and then trying again I get "Waiting on child process...":
C:\slave\test\build>c:\mozilla-build\python27\\python -u c:\slave\test\build\tests\mochitest/runtests.py --appname=C:\slave\test\build\application\firefox\firefox.exe --utility-path=tests/bin --extra-profile-file=tests/bin/plugins --symbols-path=http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/1364299641/firefox-22.0a1.en-US.win32.crashreporter-symbols.zip --certificate-path=tests/certs --autorun --close-when-done --console-level=INFO --metro-immersive --browser-chrome"                                                 INFO | runtests.py | Installing extension at c:\slave\test\build\tests\mochitest\extensions\specialpowers to c:\users\cltbld~1.t-w\appdata\local\temp\tmpfrqsau.                                                                                INFO | runtests.py | Installing extension at c:\slave\test\build\tests\mochitest\extensions\worker to c:\users\cltbld~1.t-w\appdata\local\temp\tmpfrqsau.       INFO | runtests.py | Installing extension at c:\slave\test\build\tests\mochitest\extensions\workerbootstrap to c:\users\cltbld~1.t-w\appdata\local\temp\tmpfrqsau.                                                                              args: ['C:\\slave\\test\\build\\tests\\bin\\xpcshell.exe', '-g', 'C:\\slave\\test\\build\\application\\firefox', '-v', '170', '-f', './httpd.js', '-e', "const _PROFILE_PATH = 'c:\\\\users\\\\cltbld~1.t-w\\\\appdata\\\\local\\\\temp\\\\tmpfrqsau';const _SERVER_PORT = '8888'; const _SERVER_ADDR = '127.0.0.1';\n                     const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', './server.js']                                                               INFO | runtests.py | Server pid: 2260                                           args: ['c:\\mozilla-build\\python27\\python.exe', 'c:\\slave\\test\\build\\tests\\mochitest\\pywebsocket_wrapper.py', '-p', '9988', '-w', 'c:\\slave\\test\\build\\tests\\mochitest', '-l', 'c:\\slave\\test\\build\\tests\\mochitest\\websock.log', '--log-level=debug', '--allow-handlers-outside-root-dir']                  INFO | runtests.py | Websocket server pid: 944                                  INFO | runtests.py | Running tests: start.                                                                                                                      args: ['C:\\slave\\test\\build\\tests\\bin\\certutil.exe', '-N', '-d', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau', '-f', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau\\.crtdbpw']                                  args: ['C:\\slave\\test\\build\\tests\\bin\\certutil.exe', '-A', '-i', 'C:\\slave\\test\\build\\tests\\certs\\bug483440-attack2b.ca', '-d', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau', '-f', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau\\.crtdbpw', '-n', 'bug483440-attack2b', '-t', 'CT,,']   args: ['C:\\slave\\test\\build\\tests\\bin\\certutil.exe', '-A', '-i', 'C:\\slave\\test\\build\\tests\\certs\\bug483440-attack7.ca', '-d', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau', '-f', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau\\.crtdbpw', '-n', 'bug483440-attack7', '-t', 'CT,,']     args: ['C:\\slave\\test\\build\\tests\\bin\\certutil.exe', '-A', '-i', 'C:\\slave\\test\\build\\tests\\certs\\bug483440-pk10oflo.ca', '-d', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau', '-f', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau\\.crtdbpw', '-n', 'bug483440-pk10oflo', '-t', 'CT,,']   args: ['C:\\slave\\test\\build\\tests\\bin\\certutil.exe', '-A', '-i', 'C:\\slave\\test\\build\\tests\\certs\\jartests-object.ca', '-d', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau', '-f', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau\\.crtdbpw', '-n', 'jartests-object', '-t', 'CT,,CT']       args: ['C:\\slave\\test\\build\\tests\\bin\\pk12util.exe', '-i', 'C:\\slave\\test\\build\\tests\\certs\\mochitest.client', '-w', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau\\.crtdbpw', '-d', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau']                                                        C:\slave\test\build\tests\bin\pk12util.exe: PKCS12 IMPORT SUCCESSFUL            args: ['C:\\slave\\test\\build\\tests\\bin\\certutil.exe', '-A', '-i', 'C:\\slave\\test\\build\\tests\\certs\\pgoca.ca', '-d', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau', '-f', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau\\.crtdbpw', '-n', 'pgoca', '-t', 'CT,,']                             args: ['C:\\slave\\test\\build\\tests\\bin\\ssltunnel.exe', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau\\ssltunnel.cfg']                          INFO | automation.py | SSL tunnel pid: 3608                                     args: ['C:\\slave\\test\\build\\tests\\bin\\metrotestharness.exe', '-no-remote', '-profile', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau/', 'about:blank']                                                                        INFO | automation.py | Application pid: 1928                                    INFO | metrotestharness.exe | args: '-no-remote -profile c:\users\cltbld~1.t-w\appdata\local\temp\tmpfrqsau/ about:blank'                                       INFO | metrotestharness.exe | Launching browser...                              Server listening on port 4443 with cert pgo server certificate                  INFO | metrotestharness.exe | App model id='E4CFE2E6B75AA3A3'                   INFO | metrotestharness.exe | Harness process id: 1928                          INFO | metrotestharness.exe | Activation succeeded. processid=536               INFO | metrotestharness.exe | Waiting on child process...                       

#################################################
For my own future benefit:
unzip -q -o firefox-22.0a1.en-US.win32.tests.zip bin/* certs/* modules/* mozbase/* mochitest/*

Set Firefox as the default browser by opening the app and saying "yes" to become the default browser and click few windows.

Run the tests with this:
c:\mozilla-build\python27\\python -u c:\slave\test\build\tests\mochitest/runtests.py --appname=C:\slave\test\build\application\firefox\firefox.exe --utility-path=tests/bin --extra-profile-file=tests/bin/plugins --symbols-path=http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/1364299641/firefox-22.0a1.en-US.win32.crashreporter-symbols.zip --certificate-path=tests/certs --autorun --close-when-done --console-level=INFO --metro-immersive --browser-chrome"

Comment 9

[Armen [:armenzg]]

Attachment: screenshot - Firefox metro window

Comment 10

[Armen [:armenzg]]

Attachment: screenshot - python firewall prompt

Comment 11

[Armen [:armenzg]]

After the wait for a child process we time out and take a screenshot.

jimm, would you be able to loan the machine and have a look at it?

Comment 12

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #11)
> After the wait for a child process we time out and take a screenshot.
> 
> jimm, would you be able to loan the machine and have a look at it?

Can you post the mochitest-metro-chrome.log here? That should have test run info in it.

Comment 13

[Jim Mathies [:jimm]]

(In reply to Jim Mathies [:jimm] from comment #12)
> (In reply to Armen Zambrano G. [:armenzg] from comment #11)
> > After the wait for a child process we time out and take a screenshot.
> > 
> > jimm, would you be able to loan the machine and have a look at it?
> 
> Can you post the mochitest-metro-chrome.log here? That should have test run
> info in it.

Also is there a tests.ini located in C:\slave\test\build\application or C:\slave\test\build\tests\bin ? We might have to move the tests.ini file over to temp.

Comment 14

[Jim Mathies [:jimm]]

As soon as inbound reopens I'll land a fix for bug 854921. Then we can take this for a spin again in the morning.

Comment 15

[Armen [:armenzg]]

tests.ini is under tests/bin/tests.ini
but I get the same "Waiting on child process..." message.


For my own benefit:
rmdir /s /q application tests
mkdir application
cd application
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1364379546/firefox-22.0a1.en-US.win32.zip
unzip firefox-22.0a1.en-US.win32.zip
cd ..
mkdir tests
cd tests
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1364379546/firefox-22.0a1.en-US.win32.tests.zip
unzip -q -o firefox-22.0a1.en-US.win32.tests.zip bin/* certs/* modules/* mozbase/* mochitest/*
cd ..
c:\mozilla-build\python27\\python -u c:\slave\test\build\tests\mochitest/runtests.py --appname=C:\slave\test\build\application\firefox\firefox.exe --utility-path=tests/bin --extra-profile-file=tests/bin/plugins --symbols-path=http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1364379546/firefox-22.0a1.en-US.win32.crashreporter-symbols.zip --certificate-path=tests/certs --autorun --close-when-done --console-level=INFO --metro-immersive --browser-chrome"

Comment 16

[Jim Mathies [:jimm]]

Can you post the harness output leading up to "Waiting on child process..."

Comment 17

[Jim Mathies [:jimm]]

also, is (workingdir)/application/firefox/firefox.exe set as the default browser?

Comment 18

[Armen [:armenzg]]

After an IRC chat we managed to run these tests.

Our current blocker is making Firefox the default browser (bug 854905).

Comment 19

[Armen [:armenzg]]

Next week I will be getting the current hand modified machine to run this through mozharness.

Comment 20

[Armen [:armenzg]]

Attachment: adding metro immersive config

jimm, would you mind running the following two commands locally and tell me what is waiting on? I can't figure it out.

C:\mozilla-build\hg\hg.exe clone http://hg.mozilla.org/users/armenzg_mozilla.com/mozharness scripts
c:/mozilla-build/python27/python -u scripts/scripts/desktop_unittest.py --cfg  unittests/win_unittest.py --mochitest-suite metro-immersive --download-symbols ondemand --installer-url http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1364379546/firefox-22.0a1.en-US.win32.zip --test-url http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1364379546/firefox-22.0a1.en-US.win32.tests.zip --no-read-buildbot-config

Comment 21

[Jim Mathies [:jimm]]

Hrm, desktop_unittest.py doesn't appear to be very portable. My hg isn't on the c drive, so this bails out before it every gets to trying to run tests. I can try to hack the script. We might want to file a bug on making the harness more developer friendly.

Comment 22

[Jim Mathies [:jimm]]

I copied over my mb dir, but get a failure later on about a missing "buildotve" sub directory. Maybe we can diagnose this from the log, can you post your output?

Comment 23

[Armen [:armenzg]]

Once I get the machine back from Q I will try to run the script and create a developer version of the configs (hopefully) which should not have hardcoded paths.

I will paste the log but it had nothing interesting.

Is there a debug flag that I can pass?

Comment 24

[Jim Mathies [:jimm]]

Maybe --mochitest-suite should be mochitest-metro-chrome? Not sure.

Comment 25

[Armen [:armenzg]]

(In reply to Jim Mathies [:jimm] from comment #24)
> Maybe --mochitest-suite should be mochitest-metro-chrome? Not sure.

It's a general term, all mochitests jobs have the same entry point but we just give it different parameters.

Comment 26

[Jim Mathies [:jimm]]

Are we blocked on something here that I can try to test, or are we waiting on Q for machine config?

Comment 27

[Armen [:armenzg]]

(In reply to Jim Mathies [:jimm] from comment #26)
> Are we blocked on something here that I can try to test, or are we waiting
> on Q for machine config?

We're waiting on the dependent bug.

Comment 28

[Jim Mathies [:jimm]]

The logging patches have a build now over on mc - 

http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1365767540/

We should probably confirm everything is working right with logging output.

Comment 29

[Armen [:armenzg]]

I will be looking into this again tomorrow.
I don't think we will be live this week (as I might need to take Friday off) but I will have all my patches ready.
We will also need to check that deploying the default browser change through PGO does not affect anything else (I will test this on staging).

Comment 30

[Armen [:armenzg]]

Attachment: [buildbot-configs] metro changes

Comment 31

[Armen [:armenzg]]

I'm testing that t-w864-ix-001 with the default browser change done through GPO does not affect other suites. So far I see two oranges which I have re-triggered.

I have also added the metro test to the cedar branch.

I will check on Monday what the results are.

jimm sorry for over-promising that it could be live by the end of the week.
I've completed today all of the machine re-purposing work that I took on past Thursday and all sorts of infra sanity checks (put machines back into the pool).

Sorry again.

Comment 32

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #31)
> I'm testing that t-w864-ix-001 with the default browser change done through
> GPO does not affect other suites. So far I see two oranges which I have
> re-triggered.
> 
> I have also added the metro test to the cedar branch.
> 
> I will check on Monday what the results are.
> 
> jimm sorry for over-promising that it could be live by the end of the week.
> I've completed today all of the machine re-purposing work that I took on
> past Thursday and all sorts of infra sanity checks (put machines back into
> the pool).
> 
> Sorry again.

np! Thanks for the effort.

Comment 33

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #31)
> I'm testing that t-w864-ix-001 with the default browser change done through
> GPO does not affect other suites. So far I see two oranges which I have
> re-triggered.
> 
> I have also added the metro test to the cedar branch.
> 

Would these be visible through cedar's tbpl page?

Comment 34

[Armen [:armenzg]]

(In reply to Jim Mathies [:jimm] from comment #33)
> (In reply to Armen Zambrano G. [:armenzg] from comment #31)
> > I'm testing that t-w864-ix-001 with the default browser change done through
> > GPO does not affect other suites. So far I see two oranges which I have
> > re-triggered.
> > 
> > I have also added the metro test to the cedar branch.
> > 
> 
> Would these be visible through cedar's tbpl page?

Not yet. I'm testing on staging.
Once it works I will ask for reviews.
I will be explicit about them running on Cedar.

I'm also thinking that we should have them running first inside of "mochitest-other" since they take so short to run.

Do you think that would work?

iff --git a/mozilla-tests/config.py b/mozilla-tests/config.py
--- a/mozilla-tests/config.py
+++ b/mozilla-tests/config.py
@@ -309,17 +309,17 @@ MOCHITEST = [
         'use_mozharness': True,
         'script_path': 'scripts/desktop_unittest.py',
         'extra_args': ['--mochitest-suite', 'browser-chrome'],
         'script_maxtime': 7200,
     }),
     ('mochitest-other', {
         'use_mozharness': True,
         'script_path': 'scripts/desktop_unittest.py',
-        'extra_args': ['--mochitest-suite', 'chrome,a11y,plugins'],
+        'extra_args': ['--mochitest-suite', 'metro-immersive,chrome,a11y,plugins'],
         'script_maxtime': 7200,
     }),
 ]
 
 REFTEST_NO_IPC = [
     ('reftest', {
         'use_mozharness': True,
         'script_path': 'scripts/desktop_unittest.py',

Comment 35

[Jim Mathies [:jimm]]

browser chrome might be better since they are a browser chrome test suite. But if there's a good reason why you don't want to run them under browser chrome, oth works too.

Comment 36

[Jim Mathies [:jimm]]

(In reply to Jim Mathies [:jimm] from comment #35)
> browser chrome might be better since they are a browser chrome test suite.
> But if there's a good reason why you don't want to run them under browser
> chrome, oth works too.

If it's not hard to do we might consider adding a new test batch for these ('mc' / 'metro chrome'?). This set currently doesn't run for very long but that will change over time. Also running them under their own batch means we can turn these on for inbound and mc while hidden which would make it easier to work out any random orange issues.

Comment 37

[Armen [:armenzg]]

Attachment: [buildbot-configs] add metro-immersive jobs to Cedar

Comment 38

[Aki Sasaki (not active)]

Comment on attachment 740381 [details] [diff] [review]
[buildbot-configs] add metro-immersive jobs to Cedar

You'll also need to add this suite to the win_unittest.py config file.

Comment 39

[Armen [:armenzg]]

Attachment: [mozharness] add metro-immersive

Comment 40

[Armen [:armenzg]]

The changes will be live in our next reconfiguration.

The jobs will burn on Cedar until the GPO change is deployed to all of our win8 machines (bug 864418).

Comment 41

[Aki Sasaki (not active)]

This is in production.

Comment 42

[Armen [:armenzg]]

Attachment: enable metro accross the board

Comment 43

[Armen [:armenzg]]

Attachment: enable metro on cedar, m-c and m-i

Comment 44

[Armen [:armenzg]]

jimm: I've enabled this on m-i, m-c, cedar and try.
I will have to hide the jobs on tbpl. You can see them with &jobname=opt test metro-immersive&showall=1

Which bug would you like to use to green it out?

Comment 45

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #44)
> jimm: I've enabled this on m-i, m-c, cedar and try.
> I will have to hide the jobs on tbpl. You can see them with &jobname=opt
> test metro-immersive&showall=1
> 
> Which bug would you like to use to green it out?

Bug 880298 is the tracker on current failures.

Thanks for all the effort!

Comment 46

[Armen [:armenzg]]

You're welcome. Please file a bug for when we're ready to enable across the board.

Comment 47

[Jim Mathies [:jimm]]

Did we enable these for debug builds? I'm not seeing those on inbound.

https://tbpl.mozilla.org/?tree=Mozilla-Inbound&showall=1&jobname=winnt%206.2

Comment 48

[Armen [:armenzg]]

No, we have not.

Comment 49

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #48)
> No, we have not.

Thanks. No rush, whenever you have the time. :) Just glad to see them going.

Comment 50

[Milan Sreckovic [:milan] (needinfo for best results)]

Another thing to consider - disable Direct2D (with the preference gfx.direct2d.disabled set to true), as this better simulates some of the tablets that don't have Direct2D support.

Comment 51

[Armen [:armenzg]]

(In reply to Milan Sreckovic [:milan] from comment #50)
> Another thing to consider - disable Direct2D (with the preference
> gfx.direct2d.disabled set to true), as this better simulates some of the
> tablets that don't have Direct2D support.

How do we set this up? On one of the browser settings?
Should this be a different type of job?

What are the remaining set of requirements so I can pass this along properly? and which priority?
A) add debug builds
B) add direct2d disabled pref
C) add RETRY functionality


WRT to C, jimm can you please write a patch that has a return code of 4 if the metrotestharness.exe fails to activate? It will turn the job purple and run it on another win8 machine.

Comment 52

[Jim Mathies [:jimm]]

> WRT to C, jimm can you please write a patch that has a return code of 4 if
> the metrotestharness.exe fails to activate? It will turn the job purple and
> run it on another win8 machine.

Filed bug 881950.

Comment 53

[Jim Mathies [:jimm]]

(In reply to Milan Sreckovic [:milan] from comment #50)
> Another thing to consider - disable Direct2D (with the preference
> gfx.direct2d.disabled set to true), as this better simulates some of the
> tablets that don't have Direct2D support.

I like the idea of massaging this. If the pref takes effect immediately we could write tests that flip it. Otherwise we'd need a new test suite run (similar to our ref tests). I'd suggest filing a separate bug on this.

> A) add debug builds

high priority, and probably pretty easy too right? We just have to touch up that config script you already modified.

Comment 54

[Milan Sreckovic [:milan] (needinfo for best results)]

The pref needs a restart to take effect, so it would need to be a separate run.  Let me know if you want me to file a bug for it.

Comment 55

[Armen [:armenzg]]

(In reply to Milan Sreckovic [:milan] from comment #54)
> The pref needs a restart to take effect, so it would need to be a separate
> run.  Let me know if you want me to file a bug for it.

Yes, please. I won't be able to help as I will be gone until July. You will also need to mention how important it is within company goals and if there are any involved deadlines to meet. The component would be Releng::Automation General.

Comment 56

[Armen [:armenzg]]

What is the purpose of this bug? is it a tracking bug?
Once we take care of points A and C; could we close this bug?
I assume that the test failures on bug 880298 do not necessarily need to keep this bug open as long as we take care of the first 2 points.
I'm heading out on Thursday and I would like to pass any remaining work with a clearly defined scope.

> A) add debug builds
I'm taking care of A on bug 881082.

> B) add direct2d disabled pref
milan will be filing a separate bug.

> C) add RETRY functionality
jimm is taking care of this on bug 881950.

Comment 57

[Jim Mathies [:jimm]]

With those two deps filed, I think we can close this out.

Comment 58

[Jim Mathies [:jimm]]

According to comments from philor in bug 873251, these tests aren't running "on all trees that feed mozilla-central" and therefore not considered top tier. Hence they have been hidden again on mi and bustage will not be dealt with.

Reopening since obviously we want these visible in such a way that checkins are not allowed to break our builds.

Comment 59

[Jim Mathies [:jimm]]

Specifically, I guess we are missing fx team.

Comment 60

[Ed Morley [:emorley]]

(In reply to Jim Mathies [:jimm] from comment #59)
> Specifically, I guess we are missing fx team.

We'll need it switched on for all trunk trees, since people still end up landing generic changes on their team-specific repos.

Comment 61

[Jim Mathies [:jimm]]

Also to prevent confusion, we should disable running these with debug builds. Armen enabled those initially so that we could take a look, but after working on them for a bit it seems pretty hopeless. We can work these on a project branch at some future date.

Comment 62

[Jim Mathies [:jimm]]

aki, curious if you might be able to find someone in releng to help out here since Armen is out till July 7th?

Comment 63

[Aki Sasaki (not active)]

What is needed, precisely?

* enable mochitest-metro-chrome on fx-team
* disable mochitest-metro-chrome on debug builds everywhere

and that's it?

Comment 64

[Jim Mathies [:jimm]]

(In reply to Aki Sasaki [:aki] from comment #63)
> What is needed, precisely?
> 
> * enable mochitest-metro-chrome on fx-team
> * disable mochitest-metro-chrome on debug builds everywhere
> 
> and that's it?

1) enable mochitest-metro-chrome on the repos that currently merge regularly into mc, which included the repos they are currently running on (mc, inbound, try) and the following: 

fx-team
services-central
birch
build-system

Also, through whatever method releng uses to keep track of this type of thing: they should be enabled on new repos created that are based on mc as well.

2) disable mochitest-metro-chrome on debug builds everywhere *except* cedar, so we have a test repo we can use to green those up.

3) philor specifically called out finishing up bug 881950. We can take that discussion there, just want to point it out.

Thanks!

Comment 65

[Jim Mathies [:jimm]]

* try doesn't merge into mc obviously, but we want these test to continue to be available there. :)

Comment 66

[Ed Morley [:emorley]]

(In reply to Aki Sasaki [:aki] from comment #63)
> What is needed, precisely?
> 
> * enable mochitest-metro-chrome on fx-team
> * disable mochitest-metro-chrome on debug builds everywhere
> 
> and that's it?

* enable mochitest-metro-chrome on all trunk trees + try (with the intention of enabling as and when metro is ready to ride the trains)
* disable mochitest-metro-chrome on debug builds everywhere

Thank you :-)

Comment 67

[Ed Morley [:emorley]]

"with the intention of enabling _on release branches_ as and when..."

Comment 68

[Ed Morley [:emorley]]

I think something similar to the jetpack implementation (except with some more release branches added), might be the best way here?
https://hg.mozilla.org/build/buildbot-configs/file/4898edce5204/mozilla-tests/config.py#l1344

Comment 69

[Aki Sasaki (not active)]

Attachment: enable opt metro on everything m-c level, debug on cedar only

Is this going to be following the trains?
If so, we probably want a MERGE DAY comment in here and appropriate merge day bugs.

If it is following the trains, would that be Fx24 ?

Comment 70

[Chris Cooper [:coop] (he/him)]

Comment on attachment 765140 [details] [diff] [review]
enable opt metro on everything m-c level, debug on cedar only

Review of attachment 765140 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +1364,2 @@
>  BRANCHES['cedar']['platforms']['win32']['win8']['debug_unittest_suites'] += METRO[:]
> +for branch in BRANCHES.keys():

Yes, maybe just a placeholder comment for now with the bug # until we decide which train this will be on.

Comment 71

[Aki Sasaki (not active)]

Comment on attachment 765140 [details] [diff] [review]
enable opt metro on everything m-c level, debug on cedar only

With placeholder comment.
https://hg.mozilla.org/build/buildbot-configs/rev/df57d8647cb5

Comment 72

[Jim Mathies [:jimm]]

(In reply to Aki Sasaki [:aki] from comment #69)
> Created attachment 765140 [details] [diff] [review]
> enable opt metro on everything m-c level, debug on cedar only
> 
> Is this going to be following the trains?
> If so, we probably want a MERGE DAY comment in here and appropriate merge
> day bugs.
> 
> If it is following the trains, would that be Fx24 ?

Not yet, we're shooting for around Fx27.

Comment 73

[Aki Sasaki (not active)]

Ok, could you file a bug in advance of the merge day where you want these enabled on Aurora?

This should be fixed now.

Comment 74

[Jim Mathies [:jimm]]

(In reply to Aki Sasaki [:aki] from comment #73)
> Ok, could you file a bug in advance of the merge day where you want these
> enabled on Aurora?
> 
> This should be fixed now.

No problem.