Get metro tests integrated with automation

RESOLVED FIXED

Status

Release Engineering
Platform Support
P2
normal
RESOLVED FIXED
5 years ago
4 years ago

People

(Reporter: jimm, Assigned: armenzg)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: feature=external)

Attachments

(7 attachments, 2 obsolete attachments)

(Reporter)

Description

5 years ago
Armenz asked me to file this info bug on metro mochitest.

These tests are currently running on mc manually. Basically:

pymake mochitest-metro-chrome 

will invoke them. There will likely be some heavy random orange the first time they run in automation which we can work to patch up pretty quick. So initially they should be hidden.

The way these work:

1) pymake mochitest-metro-chrome invokes a new test harness located here - 

http://mxr.mozilla.org/mozilla-central/source/browser/metro/shell/testing/

2) metrotestharness.cpp writes out test related command line params to an ini

These are written to an ini file (tests.ini) located in the root obj directory the test are to be run in.

http://mxr.mozilla.org/mozilla-central/source/browser/metro/shell/testing/metrotestharness.cpp#176

The reason for this is that the metro browser can't launch from the command line. It must be invoked through a windows api call.

http://mxr.mozilla.org/mozilla-central/source/browser/metro/shell/testing/metrotestharness.cpp#211

3) after Windows launches the browser, nsBrowserApp.cpp reads in the temporary ini file parameters when it launches.

http://mxr.mozilla.org/mozilla-central/source/browser/app/nsBrowserApp.cpp#276

4) the browser starts up and the tests run.

There are a couple setup requirements:

1) The metro browser must be registered as the default system browser before tests can run. You can replace the binaries when the browser isn't running, but the location must be where the default browser was located when it was set. Otherwise step 2 above will fail.

2) Registering the default browser can't be automated. It requires user interaction to click on the appropriate popup and open the default programs control panel to select firefox.

3) the test harness needs write permissions to obj dir.
(Assignee)

Comment 1

5 years ago
Metro specific rather than part of the win8 infra.
Blocks: 731280
No longer blocks: 826050
(Reporter)

Comment 2

5 years ago
We also have the ability to run mochitest-plain and mochitest-chrome (bug 837772) but the tests have not been triaged. A lot will probably fail. Triaging will happen in bug 843420.
(Reporter)

Updated

5 years ago
Summary: Info: running mochitest-metro-chrome → Get running metro tests integrated with automation
(Reporter)

Updated

5 years ago
Summary: Get running metro tests integrated with automation → Get metro tests integrated with automation
(Assignee)

Updated

5 years ago
Assignee: nobody → armenzg
(Assignee)

Updated

5 years ago
Priority: -- → P1
(Assignee)

Comment 3

5 years ago
I'm on buildduty this week and won't be abe to have a look at this before next week.
Priority: P1 → P3
(Assignee)

Updated

5 years ago
Priority: P3 → P2
(Assignee)

Comment 4

5 years ago
Grabbed the builds and tests from here:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1364206260/

jimm, I'm running this but I can't find the metrotestharness.exe:

C:\Users\cltbld.T-W864-IX-042\Desktop>c:\mozilla-build\python27\\python -u c:\Users\cltbld.T-W864-IX-042\Desktop\tests\mochitest\runtests.py --appname=C:\Users\cltbld.T-W864-IX-042\Desktop\firefox-22.0a1.en-US.win32\firefox\firefox.exe --utility-path=tests/bin --extra-profile-file=tests/bin/plugins --symbols-path=http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1364206260/firefox-22.0a1.en-US.win32.crashreporter-symbols.zip --certificate-path=tests/certs --autorun --close-when-done --console-level=INFO --metro-immersive --browser-chrome"
Usage: Usage instructions for runtests.py.
All arguments are optional.
If --chrome is specified, chrome tests will be run instead of web content tests.

If --browser-chrome is specified, browser-chrome tests will be run instead of we
b content tests.
See <http://mochikit.com/doc/html/MochiKit/Logging.html> for details on the logg
ing levels.

runtests.py: error: C:\Users\cltbld.T-W864-IX-042\Desktop\tests\bin\metrotesthar
ness.exe not found, cannot launch immersive tests.

C:\Users\cltbld.T-W864-IX-042\Desktop>
(Reporter)

Comment 5

5 years ago
It's in the dist/bin folder of the zip with the other exes. You can pass this to runtests using the --utility-path command line option. 

http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#57

http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#371
(Reporter)

Comment 6

5 years ago
Looking at a test zip, it looks like we'll have to package this up in the bin dir. Let me see if I can find where we do that.
(Reporter)

Updated

5 years ago
Depends on: 854584
(Reporter)

Comment 7

5 years ago
(In reply to Jim Mathies [:jimm] from comment #6)
> Looking at a test zip, it looks like we'll have to package this up in the
> bin dir. Let me see if I can find where we do that.

Fixed on mc. Should be in the next zip.
(Assignee)

Comment 8

5 years ago
I've managed to start the tests but it gets stuck in here:
http://cl.ly/NqpY
which seems to wait on:
http://cl.ly/Nqh7

After adding a firewall exception and then trying again I get "Waiting on child process...":
C:\slave\test\build>c:\mozilla-build\python27\\python -u c:\slave\test\build\tests\mochitest/runtests.py --appname=C:\slave\test\build\application\firefox\firefox.exe --utility-path=tests/bin --extra-profile-file=tests/bin/plugins --symbols-path=http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/1364299641/firefox-22.0a1.en-US.win32.crashreporter-symbols.zip --certificate-path=tests/certs --autorun --close-when-done --console-level=INFO --metro-immersive --browser-chrome"                                                 INFO | runtests.py | Installing extension at c:\slave\test\build\tests\mochitest\extensions\specialpowers to c:\users\cltbld~1.t-w\appdata\local\temp\tmpfrqsau.                                                                                INFO | runtests.py | Installing extension at c:\slave\test\build\tests\mochitest\extensions\worker to c:\users\cltbld~1.t-w\appdata\local\temp\tmpfrqsau.       INFO | runtests.py | Installing extension at c:\slave\test\build\tests\mochitest\extensions\workerbootstrap to c:\users\cltbld~1.t-w\appdata\local\temp\tmpfrqsau.                                                                              args: ['C:\\slave\\test\\build\\tests\\bin\\xpcshell.exe', '-g', 'C:\\slave\\test\\build\\application\\firefox', '-v', '170', '-f', './httpd.js', '-e', "const _PROFILE_PATH = 'c:\\\\users\\\\cltbld~1.t-w\\\\appdata\\\\local\\\\temp\\\\tmpfrqsau';const _SERVER_PORT = '8888'; const _SERVER_ADDR = '127.0.0.1';\n                     const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', './server.js']                                                               INFO | runtests.py | Server pid: 2260                                           args: ['c:\\mozilla-build\\python27\\python.exe', 'c:\\slave\\test\\build\\tests\\mochitest\\pywebsocket_wrapper.py', '-p', '9988', '-w', 'c:\\slave\\test\\build\\tests\\mochitest', '-l', 'c:\\slave\\test\\build\\tests\\mochitest\\websock.log', '--log-level=debug', '--allow-handlers-outside-root-dir']                  INFO | runtests.py | Websocket server pid: 944                                  INFO | runtests.py | Running tests: start.                                                                                                                      args: ['C:\\slave\\test\\build\\tests\\bin\\certutil.exe', '-N', '-d', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau', '-f', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau\\.crtdbpw']                                  args: ['C:\\slave\\test\\build\\tests\\bin\\certutil.exe', '-A', '-i', 'C:\\slave\\test\\build\\tests\\certs\\bug483440-attack2b.ca', '-d', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau', '-f', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau\\.crtdbpw', '-n', 'bug483440-attack2b', '-t', 'CT,,']   args: ['C:\\slave\\test\\build\\tests\\bin\\certutil.exe', '-A', '-i', 'C:\\slave\\test\\build\\tests\\certs\\bug483440-attack7.ca', '-d', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau', '-f', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau\\.crtdbpw', '-n', 'bug483440-attack7', '-t', 'CT,,']     args: ['C:\\slave\\test\\build\\tests\\bin\\certutil.exe', '-A', '-i', 'C:\\slave\\test\\build\\tests\\certs\\bug483440-pk10oflo.ca', '-d', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau', '-f', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau\\.crtdbpw', '-n', 'bug483440-pk10oflo', '-t', 'CT,,']   args: ['C:\\slave\\test\\build\\tests\\bin\\certutil.exe', '-A', '-i', 'C:\\slave\\test\\build\\tests\\certs\\jartests-object.ca', '-d', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau', '-f', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau\\.crtdbpw', '-n', 'jartests-object', '-t', 'CT,,CT']       args: ['C:\\slave\\test\\build\\tests\\bin\\pk12util.exe', '-i', 'C:\\slave\\test\\build\\tests\\certs\\mochitest.client', '-w', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau\\.crtdbpw', '-d', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau']                                                        C:\slave\test\build\tests\bin\pk12util.exe: PKCS12 IMPORT SUCCESSFUL            args: ['C:\\slave\\test\\build\\tests\\bin\\certutil.exe', '-A', '-i', 'C:\\slave\\test\\build\\tests\\certs\\pgoca.ca', '-d', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau', '-f', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau\\.crtdbpw', '-n', 'pgoca', '-t', 'CT,,']                             args: ['C:\\slave\\test\\build\\tests\\bin\\ssltunnel.exe', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau\\ssltunnel.cfg']                          INFO | automation.py | SSL tunnel pid: 3608                                     args: ['C:\\slave\\test\\build\\tests\\bin\\metrotestharness.exe', '-no-remote', '-profile', 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpfrqsau/', 'about:blank']                                                                        INFO | automation.py | Application pid: 1928                                    INFO | metrotestharness.exe | args: '-no-remote -profile c:\users\cltbld~1.t-w\appdata\local\temp\tmpfrqsau/ about:blank'                                       INFO | metrotestharness.exe | Launching browser...                              Server listening on port 4443 with cert pgo server certificate                  INFO | metrotestharness.exe | App model id='E4CFE2E6B75AA3A3'                   INFO | metrotestharness.exe | Harness process id: 1928                          INFO | metrotestharness.exe | Activation succeeded. processid=536               INFO | metrotestharness.exe | Waiting on child process...                       

#################################################
For my own future benefit:
unzip -q -o firefox-22.0a1.en-US.win32.tests.zip bin/* certs/* modules/* mozbase/* mochitest/*

Set Firefox as the default browser by opening the app and saying "yes" to become the default browser and click few windows.

Run the tests with this:
c:\mozilla-build\python27\\python -u c:\slave\test\build\tests\mochitest/runtests.py --appname=C:\slave\test\build\application\firefox\firefox.exe --utility-path=tests/bin --extra-profile-file=tests/bin/plugins --symbols-path=http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/1364299641/firefox-22.0a1.en-US.win32.crashreporter-symbols.zip --certificate-path=tests/certs --autorun --close-when-done --console-level=INFO --metro-immersive --browser-chrome"
(Assignee)

Comment 9

5 years ago
Created attachment 729558 [details]
screenshot - Firefox metro window
(Assignee)

Comment 10

5 years ago
Created attachment 729559 [details]
screenshot - python firewall prompt
(Assignee)

Updated

5 years ago
Depends on: 854905
(Assignee)

Comment 11

5 years ago
After the wait for a child process we time out and take a screenshot.

jimm, would you be able to loan the machine and have a look at it?
(Reporter)

Comment 12

5 years ago
(In reply to Armen Zambrano G. [:armenzg] from comment #11)
> After the wait for a child process we time out and take a screenshot.
> 
> jimm, would you be able to loan the machine and have a look at it?

Can you post the mochitest-metro-chrome.log here? That should have test run info in it.
(Reporter)

Comment 13

5 years ago
(In reply to Jim Mathies [:jimm] from comment #12)
> (In reply to Armen Zambrano G. [:armenzg] from comment #11)
> > After the wait for a child process we time out and take a screenshot.
> > 
> > jimm, would you be able to loan the machine and have a look at it?
> 
> Can you post the mochitest-metro-chrome.log here? That should have test run
> info in it.

Also is there a tests.ini located in C:\slave\test\build\application or C:\slave\test\build\tests\bin ? We might have to move the tests.ini file over to temp.
(Reporter)

Updated

5 years ago
Depends on: 854921

Updated

5 years ago
Whiteboard: feature=story c=testing u=developer p=tbd
(Reporter)

Comment 14

5 years ago
As soon as inbound reopens I'll land a fix for bug 854921. Then we can take this for a spin again in the morning.

Updated

5 years ago
Whiteboard: feature=story c=testing u=developer p=tbd
(Assignee)

Comment 15

5 years ago
tests.ini is under tests/bin/tests.ini
but I get the same "Waiting on child process..." message.


For my own benefit:
rmdir /s /q application tests
mkdir application
cd application
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1364379546/firefox-22.0a1.en-US.win32.zip
unzip firefox-22.0a1.en-US.win32.zip
cd ..
mkdir tests
cd tests
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1364379546/firefox-22.0a1.en-US.win32.tests.zip
unzip -q -o firefox-22.0a1.en-US.win32.tests.zip bin/* certs/* modules/* mozbase/* mochitest/*
cd ..
c:\mozilla-build\python27\\python -u c:\slave\test\build\tests\mochitest/runtests.py --appname=C:\slave\test\build\application\firefox\firefox.exe --utility-path=tests/bin --extra-profile-file=tests/bin/plugins --symbols-path=http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1364379546/firefox-22.0a1.en-US.win32.crashreporter-symbols.zip --certificate-path=tests/certs --autorun --close-when-done --console-level=INFO --metro-immersive --browser-chrome"
(Reporter)

Comment 16

5 years ago
Can you post the harness output leading up to "Waiting on child process..."
(Reporter)

Comment 17

5 years ago
also, is (workingdir)/application/firefox/firefox.exe set as the default browser?
(Assignee)

Comment 18

5 years ago
After an IRC chat we managed to run these tests.

Our current blocker is making Firefox the default browser (bug 854905).
(Reporter)

Updated

5 years ago
Depends on: 855407
(Assignee)

Comment 19

5 years ago
Next week I will be getting the current hand modified machine to run this through mozharness.
(Assignee)

Comment 20

5 years ago
Created attachment 730827 [details] [diff] [review]
adding metro immersive config

jimm, would you mind running the following two commands locally and tell me what is waiting on? I can't figure it out.

C:\mozilla-build\hg\hg.exe clone http://hg.mozilla.org/users/armenzg_mozilla.com/mozharness scripts
c:/mozilla-build/python27/python -u scripts/scripts/desktop_unittest.py --cfg  unittests/win_unittest.py --mochitest-suite metro-immersive --download-symbols ondemand --installer-url http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1364379546/firefox-22.0a1.en-US.win32.zip --test-url http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1364379546/firefox-22.0a1.en-US.win32.tests.zip --no-read-buildbot-config
(Reporter)

Comment 21

5 years ago
Hrm, desktop_unittest.py doesn't appear to be very portable. My hg isn't on the c drive, so this bails out before it every gets to trying to run tests. I can try to hack the script. We might want to file a bug on making the harness more developer friendly.
(Reporter)

Comment 22

5 years ago
I copied over my mb dir, but get a failure later on about a missing "buildotve" sub directory. Maybe we can diagnose this from the log, can you post your output?

Updated

5 years ago
Whiteboard: feature=external
(Assignee)

Comment 23

5 years ago
Once I get the machine back from Q I will try to run the script and create a developer version of the configs (hopefully) which should not have hardcoded paths.

I will paste the log but it had nothing interesting.

Is there a debug flag that I can pass?
(Reporter)

Comment 24

5 years ago
Maybe --mochitest-suite should be mochitest-metro-chrome? Not sure.
(Assignee)

Comment 25

5 years ago
(In reply to Jim Mathies [:jimm] from comment #24)
> Maybe --mochitest-suite should be mochitest-metro-chrome? Not sure.

It's a general term, all mochitests jobs have the same entry point but we just give it different parameters.
(Reporter)

Comment 26

5 years ago
Are we blocked on something here that I can try to test, or are we waiting on Q for machine config?
(Assignee)

Comment 27

5 years ago
(In reply to Jim Mathies [:jimm] from comment #26)
> Are we blocked on something here that I can try to test, or are we waiting
> on Q for machine config?

We're waiting on the dependent bug.
(Reporter)

Updated

5 years ago
Depends on: 860683
(Reporter)

Comment 28

5 years ago
The logging patches have a build now over on mc - 

http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1365767540/

We should probably confirm everything is working right with logging output.
(Reporter)

Updated

5 years ago
Blocks: 852090
(Assignee)

Comment 29

5 years ago
I will be looking into this again tomorrow.
I don't think we will be live this week (as I might need to take Friday off) but I will have all my patches ready.
We will also need to check that deploying the default browser change through PGO does not affect anything else (I will test this on staging).
Priority: P2 → P1
(Assignee)

Comment 30

5 years ago
Created attachment 739222 [details] [diff] [review]
[buildbot-configs] metro changes
(Assignee)

Comment 31

5 years ago
I'm testing that t-w864-ix-001 with the default browser change done through GPO does not affect other suites. So far I see two oranges which I have re-triggered.

I have also added the metro test to the cedar branch.

I will check on Monday what the results are.

jimm sorry for over-promising that it could be live by the end of the week.
I've completed today all of the machine re-purposing work that I took on past Thursday and all sorts of infra sanity checks (put machines back into the pool).

Sorry again.
(Reporter)

Comment 32

5 years ago
(In reply to Armen Zambrano G. [:armenzg] from comment #31)
> I'm testing that t-w864-ix-001 with the default browser change done through
> GPO does not affect other suites. So far I see two oranges which I have
> re-triggered.
> 
> I have also added the metro test to the cedar branch.
> 
> I will check on Monday what the results are.
> 
> jimm sorry for over-promising that it could be live by the end of the week.
> I've completed today all of the machine re-purposing work that I took on
> past Thursday and all sorts of infra sanity checks (put machines back into
> the pool).
> 
> Sorry again.

np! Thanks for the effort.
(Reporter)

Comment 33

5 years ago
(In reply to Armen Zambrano G. [:armenzg] from comment #31)
> I'm testing that t-w864-ix-001 with the default browser change done through
> GPO does not affect other suites. So far I see two oranges which I have
> re-triggered.
> 
> I have also added the metro test to the cedar branch.
> 

Would these be visible through cedar's tbpl page?
(Assignee)

Comment 34

5 years ago
(In reply to Jim Mathies [:jimm] from comment #33)
> (In reply to Armen Zambrano G. [:armenzg] from comment #31)
> > I'm testing that t-w864-ix-001 with the default browser change done through
> > GPO does not affect other suites. So far I see two oranges which I have
> > re-triggered.
> > 
> > I have also added the metro test to the cedar branch.
> > 
> 
> Would these be visible through cedar's tbpl page?

Not yet. I'm testing on staging.
Once it works I will ask for reviews.
I will be explicit about them running on Cedar.

I'm also thinking that we should have them running first inside of "mochitest-other" since they take so short to run.

Do you think that would work?

iff --git a/mozilla-tests/config.py b/mozilla-tests/config.py
--- a/mozilla-tests/config.py
+++ b/mozilla-tests/config.py
@@ -309,17 +309,17 @@ MOCHITEST = [
         'use_mozharness': True,
         'script_path': 'scripts/desktop_unittest.py',
         'extra_args': ['--mochitest-suite', 'browser-chrome'],
         'script_maxtime': 7200,
     }),
     ('mochitest-other', {
         'use_mozharness': True,
         'script_path': 'scripts/desktop_unittest.py',
-        'extra_args': ['--mochitest-suite', 'chrome,a11y,plugins'],
+        'extra_args': ['--mochitest-suite', 'metro-immersive,chrome,a11y,plugins'],
         'script_maxtime': 7200,
     }),
 ]
 
 REFTEST_NO_IPC = [
     ('reftest', {
         'use_mozharness': True,
         'script_path': 'scripts/desktop_unittest.py',
(Reporter)

Comment 35

5 years ago
browser chrome might be better since they are a browser chrome test suite. But if there's a good reason why you don't want to run them under browser chrome, oth works too.
(Reporter)

Comment 36

5 years ago
(In reply to Jim Mathies [:jimm] from comment #35)
> browser chrome might be better since they are a browser chrome test suite.
> But if there's a good reason why you don't want to run them under browser
> chrome, oth works too.

If it's not hard to do we might consider adding a new test batch for these ('mc' / 'metro chrome'?). This set currently doesn't run for very long but that will change over time. Also running them under their own batch means we can turn these on for inbound and mc while hidden which would make it easier to work out any random orange issues.
(Assignee)

Comment 37

5 years ago
Created attachment 740381 [details] [diff] [review]
[buildbot-configs] add metro-immersive jobs to Cedar
Attachment #739222 - Attachment is obsolete: true
Attachment #740381 - Flags: review?(aki)

Comment 38

5 years ago
Comment on attachment 740381 [details] [diff] [review]
[buildbot-configs] add metro-immersive jobs to Cedar

You'll also need to add this suite to the win_unittest.py config file.
Attachment #740381 - Flags: review?(aki) → review+
(Assignee)

Comment 39

5 years ago
Created attachment 740383 [details] [diff] [review]
[mozharness] add metro-immersive
Attachment #730827 - Attachment is obsolete: true
Attachment #740383 - Flags: review?(aki)

Updated

5 years ago
Attachment #740383 - Flags: review?(aki) → review+
(Assignee)

Updated

5 years ago
Attachment #740381 - Flags: checked-in+
(Assignee)

Updated

5 years ago
Attachment #740383 - Flags: checked-in+
(Assignee)

Comment 40

5 years ago
The changes will be live in our next reconfiguration.

The jobs will burn on Cedar until the GPO change is deployed to all of our win8 machines (bug 864418).
Depends on: 864418

Comment 41

5 years ago
This is in production.
(Assignee)

Updated

5 years ago
Priority: P1 → P2
(Assignee)

Comment 42

5 years ago
Created attachment 759245 [details] [diff] [review]
enable metro accross the board
Attachment #759245 - Flags: review?(aki)

Updated

5 years ago
Attachment #759245 - Flags: review?(aki) → review+
(Assignee)

Updated

5 years ago
Attachment #759245 - Flags: checked-in+
(Assignee)

Comment 43

5 years ago
Created attachment 759741 [details] [diff] [review]
enable metro on cedar, m-c and m-i
Attachment #759741 - Flags: review?(bugspam.Callek)
(Assignee)

Updated

5 years ago
Attachment #759741 - Flags: review?(bugspam.Callek) → review+
(Assignee)

Comment 44

5 years ago
jimm: I've enabled this on m-i, m-c, cedar and try.
I will have to hide the jobs on tbpl. You can see them with &jobname=opt test metro-immersive&showall=1

Which bug would you like to use to green it out?
(Reporter)

Comment 45

5 years ago
(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #44)
> jimm: I've enabled this on m-i, m-c, cedar and try.
> I will have to hide the jobs on tbpl. You can see them with &jobname=opt
> test metro-immersive&showall=1
> 
> Which bug would you like to use to green it out?

Bug 880298 is the tracker on current failures.

Thanks for all the effort!
(Assignee)

Comment 46

5 years ago
You're welcome. Please file a bug for when we're ready to enable across the board.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Reporter)

Comment 47

5 years ago
Did we enable these for debug builds? I'm not seeing those on inbound.

https://tbpl.mozilla.org/?tree=Mozilla-Inbound&showall=1&jobname=winnt%206.2

Updated

5 years ago
Depends on: 880298
(Assignee)

Comment 48

5 years ago
No, we have not.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Comment 49

5 years ago
(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #48)
> No, we have not.

Thanks. No rush, whenever you have the time. :) Just glad to see them going.
(Reporter)

Updated

5 years ago
Blocks: 881082
Another thing to consider - disable Direct2D (with the preference gfx.direct2d.disabled set to true), as this better simulates some of the tablets that don't have Direct2D support.
(Assignee)

Comment 51

5 years ago
(In reply to Milan Sreckovic [:milan] from comment #50)
> Another thing to consider - disable Direct2D (with the preference
> gfx.direct2d.disabled set to true), as this better simulates some of the
> tablets that don't have Direct2D support.

How do we set this up? On one of the browser settings?
Should this be a different type of job?

What are the remaining set of requirements so I can pass this along properly? and which priority?
A) add debug builds
B) add direct2d disabled pref
C) add RETRY functionality


WRT to C, jimm can you please write a patch that has a return code of 4 if the metrotestharness.exe fails to activate? It will turn the job purple and run it on another win8 machine.
Flags: needinfo?(milan)
Flags: needinfo?(jmathies)
(Reporter)

Comment 52

5 years ago
> WRT to C, jimm can you please write a patch that has a return code of 4 if
> the metrotestharness.exe fails to activate? It will turn the job purple and
> run it on another win8 machine.

Filed bug 881950.
Flags: needinfo?(jmathies)
(Reporter)

Comment 53

5 years ago
(In reply to Milan Sreckovic [:milan] from comment #50)
> Another thing to consider - disable Direct2D (with the preference
> gfx.direct2d.disabled set to true), as this better simulates some of the
> tablets that don't have Direct2D support.

I like the idea of massaging this. If the pref takes effect immediately we could write tests that flip it. Otherwise we'd need a new test suite run (similar to our ref tests). I'd suggest filing a separate bug on this.

> A) add debug builds

high priority, and probably pretty easy too right? We just have to touch up that config script you already modified.
The pref needs a restart to take effect, so it would need to be a separate run.  Let me know if you want me to file a bug for it.
Flags: needinfo?(milan)
(Assignee)

Comment 55

5 years ago
(In reply to Milan Sreckovic [:milan] from comment #54)
> The pref needs a restart to take effect, so it would need to be a separate
> run.  Let me know if you want me to file a bug for it.

Yes, please. I won't be able to help as I will be gone until July. You will also need to mention how important it is within company goals and if there are any involved deadlines to meet. The component would be Releng::Automation General.
(Assignee)

Updated

5 years ago
No longer blocks: 881082
Depends on: 881082
(Assignee)

Comment 56

5 years ago
What is the purpose of this bug? is it a tracking bug?
Once we take care of points A and C; could we close this bug?
I assume that the test failures on bug 880298 do not necessarily need to keep this bug open as long as we take care of the first 2 points.
I'm heading out on Thursday and I would like to pass any remaining work with a clearly defined scope.

> A) add debug builds
I'm taking care of A on bug 881082.

> B) add direct2d disabled pref
milan will be filing a separate bug.

> C) add RETRY functionality
jimm is taking care of this on bug 881950.
(Reporter)

Comment 57

5 years ago
With those two deps filed, I think we can close this out.
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED
Depends on: 882333
(Reporter)

Updated

5 years ago
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Comment 58

5 years ago
According to comments from philor in bug 873251, these tests aren't running "on all trees that feed mozilla-central" and therefore not considered top tier. Hence they have been hidden again on mi and bustage will not be dealt with.

Reopening since obviously we want these visible in such a way that checkins are not allowed to break our builds.
(Reporter)

Comment 59

5 years ago
Specifically, I guess we are missing fx team.
(In reply to Jim Mathies [:jimm] from comment #59)
> Specifically, I guess we are missing fx team.

We'll need it switched on for all trunk trees, since people still end up landing generic changes on their team-specific repos.
(Reporter)

Comment 61

5 years ago
Also to prevent confusion, we should disable running these with debug builds. Armen enabled those initially so that we could take a look, but after working on them for a bit it seems pretty hopeless. We can work these on a project branch at some future date.
(Reporter)

Comment 62

5 years ago
aki, curious if you might be able to find someone in releng to help out here since Armen is out till July 7th?
Flags: needinfo?(aki)

Comment 63

5 years ago
What is needed, precisely?

* enable mochitest-metro-chrome on fx-team
* disable mochitest-metro-chrome on debug builds everywhere

and that's it?
Flags: needinfo?(aki) → needinfo?(jmathies)
(Reporter)

Comment 64

5 years ago
(In reply to Aki Sasaki [:aki] from comment #63)
> What is needed, precisely?
> 
> * enable mochitest-metro-chrome on fx-team
> * disable mochitest-metro-chrome on debug builds everywhere
> 
> and that's it?

1) enable mochitest-metro-chrome on the repos that currently merge regularly into mc, which included the repos they are currently running on (mc, inbound, try) and the following: 

fx-team
services-central
birch
build-system

Also, through whatever method releng uses to keep track of this type of thing: they should be enabled on new repos created that are based on mc as well.

2) disable mochitest-metro-chrome on debug builds everywhere *except* cedar, so we have a test repo we can use to green those up.

3) philor specifically called out finishing up bug 881950. We can take that discussion there, just want to point it out.

Thanks!
Flags: needinfo?(jmathies)
(Reporter)

Comment 65

5 years ago
* try doesn't merge into mc obviously, but we want these test to continue to be available there. :)
(In reply to Aki Sasaki [:aki] from comment #63)
> What is needed, precisely?
> 
> * enable mochitest-metro-chrome on fx-team
> * disable mochitest-metro-chrome on debug builds everywhere
> 
> and that's it?

* enable mochitest-metro-chrome on all trunk trees + try (with the intention of enabling as and when metro is ready to ride the trains)
* disable mochitest-metro-chrome on debug builds everywhere

Thank you :-)
"with the intention of enabling _on release branches_ as and when..."
I think something similar to the jetpack implementation (except with some more release branches added), might be the best way here?
https://hg.mozilla.org/build/buildbot-configs/file/4898edce5204/mozilla-tests/config.py#l1344

Comment 69

5 years ago
Created attachment 765140 [details] [diff] [review]
enable opt metro on everything m-c level, debug on cedar only

Is this going to be following the trains?
If so, we probably want a MERGE DAY comment in here and appropriate merge day bugs.

If it is following the trains, would that be Fx24 ?
Attachment #765140 - Flags: review?(coop)
Flags: needinfo?(jmathies)
Comment on attachment 765140 [details] [diff] [review]
enable opt metro on everything m-c level, debug on cedar only

Review of attachment 765140 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +1364,2 @@
>  BRANCHES['cedar']['platforms']['win32']['win8']['debug_unittest_suites'] += METRO[:]
> +for branch in BRANCHES.keys():

Yes, maybe just a placeholder comment for now with the bug # until we decide which train this will be on.
Attachment #765140 - Flags: review?(coop) → review+

Comment 71

5 years ago
Comment on attachment 765140 [details] [diff] [review]
enable opt metro on everything m-c level, debug on cedar only

With placeholder comment.
https://hg.mozilla.org/build/buildbot-configs/rev/df57d8647cb5
Attachment #765140 - Flags: checked-in+
(Reporter)

Comment 72

5 years ago
(In reply to Aki Sasaki [:aki] from comment #69)
> Created attachment 765140 [details] [diff] [review]
> enable opt metro on everything m-c level, debug on cedar only
> 
> Is this going to be following the trains?
> If so, we probably want a MERGE DAY comment in here and appropriate merge
> day bugs.
> 
> If it is following the trains, would that be Fx24 ?

Not yet, we're shooting for around Fx27.
Flags: needinfo?(jmathies)

Comment 73

5 years ago
Ok, could you file a bug in advance of the merge day where you want these enabled on Aurora?

This should be fixed now.
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED
(Reporter)

Comment 74

5 years ago
(In reply to Aki Sasaki [:aki] from comment #73)
> Ok, could you file a bug in advance of the merge day where you want these
> enabled on Aurora?
> 
> This should be fixed now.

No problem.
(Reporter)

Updated

5 years ago
No longer depends on: 880298, 882333
Product: mozilla.org → Release Engineering
OS: Windows 8 Metro → Windows 8.1
You need to log in before you can comment on or make changes to this bug.