731280 - (metro-testing) Windows 8 Metro Testing Support

Status: RESOLVED INCOMPLETE

(Firefox for Metro Graveyard :: Tests, defect)

Description

[Chris Cooper [:coop] (he/him)]

We will eventually need a pool of Windows 8 test machines.

Comment 1

[Jim Mathies [:jimm]]

It would be great if we could get some preliminary test runs and talos numbers using the current consumer preview. Any chance we might be able to make that happen?

Comment 3

[Amy Rich [:arr] [:arich]]

From email with Asa:

The current plan is that we would like to support small scale development in Q2 and larger scale alpha and beta level development and testing in Q3 and Q4 with a possible release at the end of the year or early next year.

------

This means that relops will need the windows domain to become a priority project in Q2 since we will be using it to support win8.

Comment 4

[Asa Dotzler [:asa]]

I think my requests weren't as clear or specific as they needed to be.  

We need build and test for the core team in Q2 so that we can make Win 8 a tier one platform at the beginning of Q3. I had hoped we'd be getting nightly builds and test runs and working out the kinks at small scale in Q2 so that we'd be ready for full scale support in Q3.

We'll likely not take Firefox for Metro past Aurora or Beta in Q3 but we need the full setup for build and test in July, ideally up and running by the time we get to Firefox 15 Beta (July 17.)

Windows 8 probably goes to OEMs in early June with consumer PC releases happening in August in time for "back to school". When consumers start buying those machines in August, I want us to have Windows 8 build/test/release humming just like our other primary platforms. 

I'm not sure whether we'll be promoting Nightly, Aurora, or Beta channels to early Windows 8 testers at that point, but we need to be ready, from an infrastructure standpoint, for all of those possibilities.

Comment 5

[Chris Cooper [:coop] (he/him)]

(In reply to Asa Dotzler [:asa] from comment #4) 
> We need build and test for the core team in Q2 so that we can make Win 8 a
> tier one platform at the beginning of Q3. I had hoped we'd be getting
> nightly builds and test runs and working out the kinks at small scale in Q2
> so that we'd be ready for full scale support in Q3.

Given this timing, we need to start looking for hardware ASAP.

Base reqs:

* 100 machines
** based on 80 machines running w7 currently
** assumption: only testing 64-bit version of the OS
*** Asa: do we need to test the 32-bit version of Win8 as well? 
** if IT wants to move away from minis for w7 and XP testing at the same time, we could triple the # of machines ordered, but that would constrain the hardware choice further (assuming it all needs to be the same)

* fastest processors + amount of RAM that make financial sense, i.e. explicitly *not* getting slower machines for testing as we've done in the past

* all the standard IT reqs: rack mountable, remote-imaging/-power capable, space for new machines in scl3, etc.

* graphics: what are we targeting here? Would the same graphics cards (nVidia GT430) we chose in bug 691856 be appropriate? Something more recent? cc-ing Joe and Armen here.

Comment 6

[Joe Drew (not getting mail)]

(In reply to Chris Cooper [:coop] from comment #5)
> * graphics: what are we targeting here? Would the same graphics cards
> (nVidia GT430) we chose in bug 691856 be appropriate? Something more recent?
> cc-ing Joe and Armen here.

That would work fine, as would something more recent. Do what makes most sense for IT/cost/machine requirements.

Comment 7

[Zandr Milewski [:zandr]]

(In reply to Asa Dotzler [:asa] from comment #4)
> We'll likely not take Firefox for Metro past Aurora or Beta in Q3 but we
> need the full setup for build and test in July, ideally up and running by
> the time we get to Firefox 15 Beta (July 17.)

That's roughly 75 days from today. This is not a realistic timeline for a tier one platform.

We'll need to do a real project plan for this deployment, but off the top of my head:

Releng development time-
Coop: How long will it take you guys to go from an initial hardware platform to something ready for production? We'll need to back up the schedule from there.

Hardware lead time-
Assuming we go with something off-the-shelf, (HP DL series) we're probably still looking at a couple of weeks for delivery of machines for development, and a couple more for production quantities.

Datacenter capacity-
That's several racks fully loaded. I think we may have the capacity available, but will need to confirm. 

DC Ops availability for deployment-
RelOps will need to reach out to DCOps and find out what their availability for a large-scale deployment like this will be. I don't anticipate a problem here, but we have no information as yet, and certainly can't commit to a timeline.

This is a major project. We'll work on a real plan, and we're discussing some short-term alternatives, but I don't see how having 100 machines in production as a tier-one platform in 75 days is going to happen.

Comment 8

[Asa Dotzler [:asa]]

(In reply to Zandr Milewski [:zandr] from comment #7)
> (In reply to Asa Dotzler [:asa] from comment #4)
> > We'll likely not take Firefox for Metro past Aurora or Beta in Q3 but we
> > need the full setup for build and test in July, ideally up and running by
> > the time we get to Firefox 15 Beta (July 17.)
> 
> That's roughly 75 days from today. This is not a realistic timeline for a
> tier one platform.

> This is a major project. We'll work on a real plan, and we're discussing
> some short-term alternatives, but I don't see how having 100 machines in
> production as a tier-one platform in 75 days is going to happen.

If we have to prioritize this above other work, I think that's totally reasonable. This is the next version of Windows and we need to do everything in our power to be ready for it.

Comment 9

[Jim Mathies [:jimm]]

Is this bug specific to testing infrastructure aimed at running firefox desktop through various test suites, or is it specific to the experimental win8 metro browser, or both?

FWIW, we should differentiate between desktop testing and metro browser testing. Getting desktop tests running, especially talos, should be the priority.

Comment 10

[Amy Rich [:arr] [:arich]]

This bug is specific to w8 talos machines for firefox desktop.

Comment 11

[Armen [:armenzg]]

I will go through the bug dependency and clear things up.

This is for Windows 8 desktop testing machines.

I don't know what is involved with metro browser testing. We don't produce those, right?

Comment 12

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #11)
> I will go through the bug dependency and clear things up.
> 
> This is for Windows 8 desktop testing machines.
> 
> I don't know what is involved with metro browser testing. We don't produce
> those, right?

We'll want to run metro browser chrome tests in a window on the desktop which can run on existing win7 test slaves.  We'll also want to run the same browser chrome tests in the metro environment. Those will require slaves running win8. Last but not least, we'll want to run talos test in the metro environment as well. I haven't filed a bug on that yet but will.

bug 759905 - get metro browser chrome tests running on the desktop
bug 771271 - get metro browser chrome tests running in metro

Comment 13

[Jim Mathies [:jimm]]

Filed bug 773817 - get talos tests running in metro mode

Comment 14

[bhavana bajaj [:bajaj]]

(In reply to [:armenzg] - gone from Aug. 3rd to Aug. 27th from comment #11)
> I will go through the bug dependency and clear things up.
> 
> This is for Windows 8 desktop testing machines.

Was trying to understand the timeline of win8 testing.Do we have everything needed here in place? We'd(#relman) like to have this ready for FF16, with beta scheduled for Aug 30 . wdyt ?

> 
> I don't know what is involved with metro browser testing. We don't produce
> those, right?

Comment 15

[Jim Mathies [:jimm]]

(In reply to bhavana bajaj [:bajaj] from comment #14)
> (In reply to [:armenzg] - gone from Aug. 3rd to Aug. 27th from comment #11)
> > I will go through the bug dependency and clear things up.
> > 
> > This is for Windows 8 desktop testing machines.
> 
> Was trying to understand the timeline of win8 testing.Do we have everything
> needed here in place? We'd(#relman) like to have this ready for FF16, with
> beta scheduled for Aug 30 . wdyt ?
> 
> > 
> > I don't know what is involved with metro browser testing. We don't produce
> > those, right?

Highest priority is getting tests running / talos numbers for firefox desktop. 

Tests running in the metro browser are secondary although not to be forgotten, we will be release a new browser to the public in the coming months. Doing that without test coverage isn't ideal.

Comment 16

[matthew zeier [:mrz]]

I think this is blocked on bug 780025?

Comment 17

[Armen [:armenzg]]

(In reply to matthew zeier [:mrz] from comment #16)
> I think this is blocked on bug 780025?

Correct. Thanks! arr added it.

Comment 18

[Matt Brubeck (:mbrubeck)]

I imagine we'll eventually want most of the test suites running in Metro mode.  Which are highest priority?  We already have mochitest-browser-chrome and we have a bug filed for Talos.  I expect the graphics team will want reftest support in order to land their OMTC work.  Any others?

Comment 19

[Jim Mathies [:jimm]]

There are a number of bugs related to testing hardware, we should confirm what we are purchasing is compatible with metro win8 and the metrofx backend.

Comment 20

[Armen [:armenzg]]

(In reply to Jim Mathies [:jimm] from comment #19)
> There are a number of bugs related to testing hardware, we should confirm
> what we are purchasing is compatible with metro win8 and the metrofx backend.

How do we do?
We can loan a win8 test machine to be inspected.

Comment 21

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #20)
> (In reply to Jim Mathies [:jimm] from comment #19)
> > There are a number of bugs related to testing hardware, we should confirm
> > what we are purchasing is compatible with metro win8 and the metrofx backend.
> 
> How do we do?
> We can loan a win8 test machine to be inspected.

Can you post the video hardware specs?

Comment 22

[Armen [:armenzg]]

Attachment: dxdiag

We will have a new batch of machines coming in soon which will have a newer graphic card and a newer CPU IIUC.

Would you want us to provide you the output of dxdiag for those machines as well? or is this sufficient?

What do you look for in the file?

Comment 23

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #22)
> Created attachment 710948 [details] [diff] [review]
> dxdiag
> 
> We will have a new batch of machines coming in soon which will have a newer
> graphic card and a newer CPU IIUC.
> 
> Would you want us to provide you the output of dxdiag for those machines as
> well? or is this sufficient?
> 
> What do you look for in the file?

I'm most interested in what version of direct 2d and 3d they support. This looks ok to me, cc'ing bbondy who's been working on our hardware support detection.

Comment 24

[Asa Dotzler [:asa]]

One thing I've discovered is that the Windows 8 Atom-based systems seem different than Intel Core based systems. I'll bet it has to do with the integrated graphics being different and so different D2D and/or D3D versions. We must support both.

The second thing I've discovered is that there are two different DPI modes based on screen pixels and screen size. On most machines, screens are standard 100% DPI. On 1900x1080 resolution screens with a diagonal size of 12.5" or less, Metro switches to a HiDPI mode. For any testing that cares about pixels, this might be relevant.

These are the only two major differences I see in hardware and how it affects Firefox.

Comment 25

[Brian R. Bondy [:bbondy]]

Jim I'll know more once Bug 829127 is done but I think the issue is some machines support D2D10 but not D3D10.  And Firefox works as long as you have D2D10, but our detection detects D3D10.

Comment 26

[Brian R. Bondy [:bbondy]]

I have no idea exactly how the dxdiag output would differ by the way, but perhaps after that bug I'll know.

Comment 27

[Jim Mathies [:jimm]]

We should confirm the hardware is working right with metrofx by taking it for a spin on one of these machines. (metro is now on mc.) I'd be happy to walk someone through the setup process. It should only take a couple minutes.

Independent of this, can we get a status update on win8 testing in general? We would really like to get desktop/metro talos numbers, and the metro browser currently has a set of browser chrome tests that can be run on win8 test hardware which we would love to get hooked up and running in automation so we can catch landings that break the browser.

Comment 28

[Dustin J. Mitchell [:dustin] (he/him)]

Please file a separate releng bug to borrow a machine to test metro on.

Comment 29

[Asa Dotzler [:asa]]

(In reply to Dustin J. Mitchell [:dustin] from comment #28)
> Please file a separate releng bug to borrow a machine to test metro on.

Why are we borrowing machines? We requested Windows 8 hardware for this nearly 6 months ago.

Comment 30

[Dustin J. Mitchell [:dustin] (he/him)]

That's the normal process for developers to get access to test hardware.

Comment 31

[Jim Mathies [:jimm]]

(In reply to Asa Dotzler [:asa] from comment #29)
> (In reply to Dustin J. Mitchell [:dustin] from comment #28)
> > Please file a separate releng bug to borrow a machine to test metro on.
> 
> Why are we borrowing machines? We requested Windows 8 hardware for this
> nearly 6 months ago.

'borrow' as in, get rdp access to it so I can test to make sure the browser launches properly.

Comment 32

[Jim Mathies [:jimm]]

(In reply to Dustin J. Mitchell [:dustin] from comment #28)
> Please file a separate releng bug to borrow a machine to test metro on.

filed bug 841049.

Comment 33

[Asa Dotzler [:asa]]

(In reply to Jim Mathies [:jimm] from comment #31)
> (In reply to Asa Dotzler [:asa] from comment #29)
> > (In reply to Dustin J. Mitchell [:dustin] from comment #28)
> > > Please file a separate releng bug to borrow a machine to test metro on.
> > 
> > Why are we borrowing machines? We requested Windows 8 hardware for this
> > nearly 6 months ago.
> 
> 'borrow' as in, get rdp access to it so I can test to make sure the browser
> launches properly.

Ahh. Sorry for my interjection. I misunderstood.

Comment 34

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Jim Mathies [:jimm] from comment #27)
> We should confirm the hardware is working right with metrofx by taking it
> for a spin on one of these machines. (metro is now on mc.) 
I see bug#841049 filed for access to machine.

> I'd be happy to
> walk someone through the setup process. It should only take a couple minutes.
If you need anything, please let us know.


> Independent of this, can we get a status update on win8 testing in general?
> We would really like to get desktop/metro talos numbers, and the metro
> browser currently has a set of browser chrome tests that can be run on win8
> test hardware which we would love to get hooked up and running in automation
> so we can catch landings that break the browser.

1) win8 ix-4-node-machines have been delivered (on fri 8th), powered, racked.
2) bug#780050 tracks IT work to have installable image to apply on these machines. 
3) bug#836999 tracks RelEng work to very that image does allow tests to pass. 

(2) and (3) are being actively worked by IT(Dustin, Chris, Q) and RelEng (armenzg, tomcat). Most accurate, and timely, status update is in those two bugs.

Comment 35

[Jim Mathies [:jimm]]

Installed and ran fine, here's the about:support. Note the device prompted me with a new nvidia driver. We might want to look at updating it.

  Graphics

        Adapter Description
        NVIDIA GeForce GT 430

        Adapter Drivers
        nvd3dumx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um

        Adapter RAM
        1024

        Device ID
        0x0de1

        Direct2D Enabled
        true

        DirectWrite Enabled
        true (6.2.9200.16384)

        Driver Date
        12-29-2012

        Driver Version
        9.18.13.1090

        GPU #2 Active
        false

        GPU Accelerated Windows
        1/1 Direct3D 10

        Vendor ID
        0x10de

        WebGL Renderer
        Google Inc. -- ANGLE (NVIDIA GeForce GT 430)

        AzureCanvasBackend
        direct2d

        AzureContentBackend
        direct2d

        AzureFallbackCanvasBackend
        cairo

Comment 36

[Armen [:armenzg]]

I would like to move the releng/relops bugs to bug 826050. I would like to keep this one for gathering what is needed for the metro jobs.
This way we know what infra changes are needed for the first part of the project rather than mixing dependencies for both.

Here are the bugs that I believe are only metro related and should stay:
* bug 773817 get talos running in immersive mode
* bug 778493 Add accessibility coverage for Windows tp tests on mc
* bug 837766 get reftests running in metro mode
* bug 838768 metro logs
* bug 843420 annotate failing tests
* bug 843271 CEH

I would like to move the following bugs to bug 826050:
* bug 780050 IT changes for win8 machines
* bug 826050 suggested tracking bug
* bug 844982 mozharness changes (about to be closed)
* bug 846823 tbpl changes (about to be closed)

Comment 37

[Jim Mathies [:jimm]]

Of the resolved bugs, looks like you can also take:

deps:

751224
metro-releng
759905
760440
762526
780024
780025
805950
841049
844130
846550


blocking:

820243

Comment 38

[Jim Mathies [:jimm]]

Please don't disconnect test failure bugs from this tests tracker.

Comment 39

[Firefox Bug Husbandry Bot]

Mass close of bugs in obsolete product https://bugzilla.mozilla.org/show_bug.cgi?id=1350354