If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.
Bug 731280 (metro-testing)

Windows 8 Metro Testing Support

RESOLVED INCOMPLETE

Status

Firefox for Metro
Tests
RESOLVED INCOMPLETE
6 years ago
5 days ago

People

(Reporter: coop, Unassigned)

Tracking

(Depends on: 2 bugs, {meta})

Trunk
x86_64
Windows 8.1
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [win8][testing][platform-support][tracking], URL)

Attachments

(1 attachment)

(Reporter)

Description

6 years ago
We will eventually need a pool of Windows 8 test machines.

Comment 1

6 years ago
It would be great if we could get some preliminary test runs and talos numbers using the current consumer preview. Any chance we might be able to make that happen?

Updated

6 years ago
Duplicate of this bug: 738016
Depends on: 737975
From email with Asa:

The current plan is that we would like to support small scale development in Q2 and larger scale alpha and beta level development and testing in Q3 and Q4 with a possible release at the end of the year or early next year.

------

This means that relops will need the windows domain to become a priority project in Q2 since we will be using it to support win8.

Comment 4

6 years ago
I think my requests weren't as clear or specific as they needed to be.  

We need build and test for the core team in Q2 so that we can make Win 8 a tier one platform at the beginning of Q3. I had hoped we'd be getting nightly builds and test runs and working out the kinks at small scale in Q2 so that we'd be ready for full scale support in Q3.

We'll likely not take Firefox for Metro past Aurora or Beta in Q3 but we need the full setup for build and test in July, ideally up and running by the time we get to Firefox 15 Beta (July 17.)

Windows 8 probably goes to OEMs in early June with consumer PC releases happening in August in time for "back to school". When consumers start buying those machines in August, I want us to have Windows 8 build/test/release humming just like our other primary platforms. 

I'm not sure whether we'll be promoting Nightly, Aurora, or Beta channels to early Windows 8 testers at that point, but we need to be ready, from an infrastructure standpoint, for all of those possibilities.
(Reporter)

Comment 5

6 years ago
(In reply to Asa Dotzler [:asa] from comment #4) 
> We need build and test for the core team in Q2 so that we can make Win 8 a
> tier one platform at the beginning of Q3. I had hoped we'd be getting
> nightly builds and test runs and working out the kinks at small scale in Q2
> so that we'd be ready for full scale support in Q3.

Given this timing, we need to start looking for hardware ASAP.

Base reqs:

* 100 machines
** based on 80 machines running w7 currently
** assumption: only testing 64-bit version of the OS
*** Asa: do we need to test the 32-bit version of Win8 as well? 
** if IT wants to move away from minis for w7 and XP testing at the same time, we could triple the # of machines ordered, but that would constrain the hardware choice further (assuming it all needs to be the same)

* fastest processors + amount of RAM that make financial sense, i.e. explicitly *not* getting slower machines for testing as we've done in the past

* all the standard IT reqs: rack mountable, remote-imaging/-power capable, space for new machines in scl3, etc.

* graphics: what are we targeting here? Would the same graphics cards (nVidia GT430) we chose in bug 691856 be appropriate? Something more recent? cc-ing Joe and Armen here.
(In reply to Chris Cooper [:coop] from comment #5)
> * graphics: what are we targeting here? Would the same graphics cards
> (nVidia GT430) we chose in bug 691856 be appropriate? Something more recent?
> cc-ing Joe and Armen here.

That would work fine, as would something more recent. Do what makes most sense for IT/cost/machine requirements.
(In reply to Asa Dotzler [:asa] from comment #4)
> We'll likely not take Firefox for Metro past Aurora or Beta in Q3 but we
> need the full setup for build and test in July, ideally up and running by
> the time we get to Firefox 15 Beta (July 17.)

That's roughly 75 days from today. This is not a realistic timeline for a tier one platform.

We'll need to do a real project plan for this deployment, but off the top of my head:

Releng development time-
Coop: How long will it take you guys to go from an initial hardware platform to something ready for production? We'll need to back up the schedule from there.

Hardware lead time-
Assuming we go with something off-the-shelf, (HP DL series) we're probably still looking at a couple of weeks for delivery of machines for development, and a couple more for production quantities.

Datacenter capacity-
That's several racks fully loaded. I think we may have the capacity available, but will need to confirm. 

DC Ops availability for deployment-
RelOps will need to reach out to DCOps and find out what their availability for a large-scale deployment like this will be. I don't anticipate a problem here, but we have no information as yet, and certainly can't commit to a timeline.

This is a major project. We'll work on a real plan, and we're discussing some short-term alternatives, but I don't see how having 100 machines in production as a tier-one platform in 75 days is going to happen.
Depends on: 751224

Comment 8

6 years ago
(In reply to Zandr Milewski [:zandr] from comment #7)
> (In reply to Asa Dotzler [:asa] from comment #4)
> > We'll likely not take Firefox for Metro past Aurora or Beta in Q3 but we
> > need the full setup for build and test in July, ideally up and running by
> > the time we get to Firefox 15 Beta (July 17.)
> 
> That's roughly 75 days from today. This is not a realistic timeline for a
> tier one platform.

> This is a major project. We'll work on a real plan, and we're discussing
> some short-term alternatives, but I don't see how having 100 machines in
> production as a tier-one platform in 75 days is going to happen.

If we have to prioritize this above other work, I think that's totally reasonable. This is the next version of Windows and we need to do everything in our power to be ready for it.
(Reporter)

Updated

5 years ago
Depends on: 756465

Updated

5 years ago
Depends on: 755772
No longer depends on: 756465

Updated

5 years ago
No longer depends on: 755772
Depends on: 760440
(Reporter)

Updated

5 years ago
Depends on: 762526

Comment 9

5 years ago
Is this bug specific to testing infrastructure aimed at running firefox desktop through various test suites, or is it specific to the experimental win8 metro browser, or both?

FWIW, we should differentiate between desktop testing and metro browser testing. Getting desktop tests running, especially talos, should be the priority.
This bug is specific to w8 talos machines for firefox desktop.

Comment 11

5 years ago
I will go through the bug dependency and clear things up.

This is for Windows 8 desktop testing machines.

I don't know what is involved with metro browser testing. We don't produce those, right?
Alias: win8-testing
Assignee: nobody → armenzg
Priority: P3 → P1

Comment 12

5 years ago
(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #11)
> I will go through the bug dependency and clear things up.
> 
> This is for Windows 8 desktop testing machines.
> 
> I don't know what is involved with metro browser testing. We don't produce
> those, right?

We'll want to run metro browser chrome tests in a window on the desktop which can run on existing win7 test slaves.  We'll also want to run the same browser chrome tests in the metro environment. Those will require slaves running win8. Last but not least, we'll want to run talos test in the metro environment as well. I haven't filed a bug on that yet but will.

bug 759905 - get metro browser chrome tests running on the desktop
bug 771271 - get metro browser chrome tests running in metro

Comment 13

5 years ago
Filed bug 773817 - get talos tests running in metro mode

Updated

5 years ago
Depends on: 759905, 771271, 773817

Updated

5 years ago
Blocks: 754399

Updated

5 years ago
Depends on: 775378
Depends on: 781277, 780050
(In reply to [:armenzg] - gone from Aug. 3rd to Aug. 27th from comment #11)
> I will go through the bug dependency and clear things up.
> 
> This is for Windows 8 desktop testing machines.

Was trying to understand the timeline of win8 testing.Do we have everything needed here in place? We'd(#relman) like to have this ready for FF16, with beta scheduled for Aug 30 . wdyt ?

> 
> I don't know what is involved with metro browser testing. We don't produce
> those, right?

Comment 15

5 years ago
(In reply to bhavana bajaj [:bajaj] from comment #14)
> (In reply to [:armenzg] - gone from Aug. 3rd to Aug. 27th from comment #11)
> > I will go through the bug dependency and clear things up.
> > 
> > This is for Windows 8 desktop testing machines.
> 
> Was trying to understand the timeline of win8 testing.Do we have everything
> needed here in place? We'd(#relman) like to have this ready for FF16, with
> beta scheduled for Aug 30 . wdyt ?
> 
> > 
> > I don't know what is involved with metro browser testing. We don't produce
> > those, right?

Highest priority is getting tests running / talos numbers for firefox desktop. 

Tests running in the metro browser are secondary although not to be forgotten, we will be release a new browser to the public in the coming months. Doing that without test coverage isn't ideal.
No longer depends on: 781277
I think this is blocked on bug 780025?
Depends on: 781277, 780025

Comment 17

5 years ago
(In reply to matthew zeier [:mrz] from comment #16)
> I think this is blocked on bug 780025?

Correct. Thanks! arr added it.

Updated

5 years ago
Priority: P1 → P4

Updated

5 years ago
Blocks: 800309

Updated

5 years ago
Depends on: 805950

Updated

5 years ago
No longer depends on: 737975

Updated

5 years ago
Depends on: 747347

Updated

5 years ago
No longer blocks: 754399
Depends on: 754399

Updated

5 years ago
Depends on: 780024
(Reporter)

Updated

5 years ago
Blocks: 820243

Updated

5 years ago
Depends on: 778493

Updated

5 years ago
Assignee: armenzg → nobody

Updated

5 years ago
No longer depends on: 781277

Updated

5 years ago
Depends on: 826050
I imagine we'll eventually want most of the test suites running in Metro mode.  Which are highest priority?  We already have mochitest-browser-chrome and we have a bug filed for Talos.  I expect the graphics team will want reftest support in order to land their OMTC work.  Any others?
Blocks: 831495
Depends on: 837766
Depends on: 837772
Depends on: 838768

Updated

5 years ago
Assignee: nobody → mbrubeck

Comment 19

5 years ago
There are a number of bugs related to testing hardware, we should confirm what we are purchasing is compatible with metro win8 and the metrofx backend.

Comment 20

5 years ago
(In reply to Jim Mathies [:jimm] from comment #19)
> There are a number of bugs related to testing hardware, we should confirm
> what we are purchasing is compatible with metro win8 and the metrofx backend.

How do we do?
We can loan a win8 test machine to be inspected.

Comment 21

5 years ago
(In reply to Armen Zambrano G. [:armenzg] from comment #20)
> (In reply to Jim Mathies [:jimm] from comment #19)
> > There are a number of bugs related to testing hardware, we should confirm
> > what we are purchasing is compatible with metro win8 and the metrofx backend.
> 
> How do we do?
> We can loan a win8 test machine to be inspected.

Can you post the video hardware specs?

Comment 22

5 years ago
Created attachment 710948 [details] [diff] [review]
dxdiag

We will have a new batch of machines coming in soon which will have a newer graphic card and a newer CPU IIUC.

Would you want us to provide you the output of dxdiag for those machines as well? or is this sufficient?

What do you look for in the file?
Flags: needinfo?(jmathies)

Comment 23

5 years ago
(In reply to Armen Zambrano G. [:armenzg] from comment #22)
> Created attachment 710948 [details] [diff] [review]
> dxdiag
> 
> We will have a new batch of machines coming in soon which will have a newer
> graphic card and a newer CPU IIUC.
> 
> Would you want us to provide you the output of dxdiag for those machines as
> well? or is this sufficient?
> 
> What do you look for in the file?

I'm most interested in what version of direct 2d and 3d they support. This looks ok to me, cc'ing bbondy who's been working on our hardware support detection.
Flags: needinfo?(jmathies) → needinfo?(netzen)

Comment 24

5 years ago
One thing I've discovered is that the Windows 8 Atom-based systems seem different than Intel Core based systems. I'll bet it has to do with the integrated graphics being different and so different D2D and/or D3D versions. We must support both.

The second thing I've discovered is that there are two different DPI modes based on screen pixels and screen size. On most machines, screens are standard 100% DPI. On 1900x1080 resolution screens with a diagonal size of 12.5" or less, Metro switches to a HiDPI mode. For any testing that cares about pixels, this might be relevant.

These are the only two major differences I see in hardware and how it affects Firefox.
Jim I'll know more once Bug 829127 is done but I think the issue is some machines support D2D10 but not D3D10.  And Firefox works as long as you have D2D10, but our detection detects D3D10.
Flags: needinfo?(netzen)
I have no idea exactly how the dxdiag output would differ by the way, but perhaps after that bug I'll know.
Depends on: 839460

Comment 27

5 years ago
We should confirm the hardware is working right with metrofx by taking it for a spin on one of these machines. (metro is now on mc.) I'd be happy to walk someone through the setup process. It should only take a couple minutes.

Independent of this, can we get a status update on win8 testing in general? We would really like to get desktop/metro talos numbers, and the metro browser currently has a set of browser chrome tests that can be run on win8 test hardware which we would love to get hooked up and running in automation so we can catch landings that break the browser.
Please file a separate releng bug to borrow a machine to test metro on.

Comment 29

5 years ago
(In reply to Dustin J. Mitchell [:dustin] from comment #28)
> Please file a separate releng bug to borrow a machine to test metro on.

Why are we borrowing machines? We requested Windows 8 hardware for this nearly 6 months ago.
That's the normal process for developers to get access to test hardware.

Comment 31

5 years ago
(In reply to Asa Dotzler [:asa] from comment #29)
> (In reply to Dustin J. Mitchell [:dustin] from comment #28)
> > Please file a separate releng bug to borrow a machine to test metro on.
> 
> Why are we borrowing machines? We requested Windows 8 hardware for this
> nearly 6 months ago.

'borrow' as in, get rdp access to it so I can test to make sure the browser launches properly.

Updated

5 years ago
Depends on: 841049

Comment 32

5 years ago
(In reply to Dustin J. Mitchell [:dustin] from comment #28)
> Please file a separate releng bug to borrow a machine to test metro on.

filed bug 841049.

Comment 33

5 years ago
(In reply to Jim Mathies [:jimm] from comment #31)
> (In reply to Asa Dotzler [:asa] from comment #29)
> > (In reply to Dustin J. Mitchell [:dustin] from comment #28)
> > > Please file a separate releng bug to borrow a machine to test metro on.
> > 
> > Why are we borrowing machines? We requested Windows 8 hardware for this
> > nearly 6 months ago.
> 
> 'borrow' as in, get rdp access to it so I can test to make sure the browser
> launches properly.

Ahh. Sorry for my interjection. I misunderstood.
(In reply to Jim Mathies [:jimm] from comment #27)
> We should confirm the hardware is working right with metrofx by taking it
> for a spin on one of these machines. (metro is now on mc.) 
I see bug#841049 filed for access to machine.

> I'd be happy to
> walk someone through the setup process. It should only take a couple minutes.
If you need anything, please let us know.


> Independent of this, can we get a status update on win8 testing in general?
> We would really like to get desktop/metro talos numbers, and the metro
> browser currently has a set of browser chrome tests that can be run on win8
> test hardware which we would love to get hooked up and running in automation
> so we can catch landings that break the browser.

1) win8 ix-4-node-machines have been delivered (on fri 8th), powered, racked.
2) bug#780050 tracks IT work to have installable image to apply on these machines. 
3) bug#836999 tracks RelEng work to very that image does allow tests to pass. 

(2) and (3) are being actively worked by IT(Dustin, Chris, Q) and RelEng (armenzg, tomcat). Most accurate, and timely, status update is in those two bugs.

Comment 35

5 years ago
Installed and ran fine, here's the about:support. Note the device prompted me with a new nvidia driver. We might want to look at updating it.

  Graphics

        Adapter Description
        NVIDIA GeForce GT 430

        Adapter Drivers
        nvd3dumx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um

        Adapter RAM
        1024

        Device ID
        0x0de1

        Direct2D Enabled
        true

        DirectWrite Enabled
        true (6.2.9200.16384)

        Driver Date
        12-29-2012

        Driver Version
        9.18.13.1090

        GPU #2 Active
        false

        GPU Accelerated Windows
        1/1 Direct3D 10

        Vendor ID
        0x10de

        WebGL Renderer
        Google Inc. -- ANGLE (NVIDIA GeForce GT 430)

        AzureCanvasBackend
        direct2d

        AzureContentBackend
        direct2d

        AzureFallbackCanvasBackend
        cairo

Updated

5 years ago
Depends on: 843271
Depends on: 843420
Keywords: meta
Whiteboard: [tracking][win8][testing][platform-support] → [win8][testing][platform-support]

Updated

5 years ago
Depends on: 844130

Updated

5 years ago
No longer blocks: 831495

Updated

5 years ago
Depends on: 846550

Updated

5 years ago
Depends on: 846823

Updated

5 years ago
Depends on: 844982

Comment 36

5 years ago
I would like to move the releng/relops bugs to bug 826050. I would like to keep this one for gathering what is needed for the metro jobs.
This way we know what infra changes are needed for the first part of the project rather than mixing dependencies for both.

Here are the bugs that I believe are only metro related and should stay:
* bug 773817 get talos running in immersive mode
* bug 778493 Add accessibility coverage for Windows tp tests on mc
* bug 837766 get reftests running in metro mode
* bug 838768 metro logs
* bug 843420 annotate failing tests
* bug 843271 CEH

I would like to move the following bugs to bug 826050:
* bug 780050 IT changes for win8 machines
* bug 826050 suggested tracking bug
* bug 844982 mozharness changes (about to be closed)
* bug 846823 tbpl changes (about to be closed)

Comment 37

5 years ago
Of the resolved bugs, looks like you can also take:

deps:

751224
metro-releng
759905
760440
762526
780024
780025
805950
841049
844130
846550


blocking:

820243

Updated

5 years ago
No longer depends on: 780050

Updated

5 years ago
No longer depends on: 844982

Updated

5 years ago
No longer depends on: 846823

Updated

5 years ago
No longer depends on: 751224

Updated

5 years ago
No longer depends on: 760440

Updated

5 years ago
No longer depends on: 762526

Updated

5 years ago
No longer depends on: 780024

Updated

5 years ago
No longer depends on: 780025

Updated

5 years ago
No longer depends on: 805950

Updated

5 years ago
No longer depends on: 844130

Updated

5 years ago
No longer depends on: 846550

Updated

5 years ago
Alias: win8-testing → metro-testing
Summary: [tracking bug] Windows 8 testing support → [tracking bug] Windows 8 Metro testing support

Updated

5 years ago
No longer blocks: 820243

Updated

5 years ago
Component: Release Engineering → General
Product: mozilla.org → Firefox for Metro
Version: other → Trunk

Updated

5 years ago
Component: General → Tests

Updated

5 years ago
Depends on: 847442

Updated

5 years ago
Blocks: 831544

Updated

5 years ago
Blocks: 839336
No longer blocks: 831544
Priority: P4 → P2
QA Contact: jbecerra
Summary: [tracking bug] Windows 8 Metro testing support → Story - Windows 8 Metro Testing Support
Whiteboard: [win8][testing][platform-support] → [win8][testing][platform-support] feature=story c=tbd u=tbd p=0

Updated

5 years ago
No longer depends on: 843271

Updated

5 years ago
Summary: Story - Windows 8 Metro Testing Support → Epic - Windows 8 Metro Testing Support
Whiteboard: [win8][testing][platform-support] feature=story c=tbd u=tbd p=0 → [win8][testing][platform-support] feature=epic

Updated

5 years ago
Blocks: 841999
Assignee: mbrubeck → nobody
Depends on: 850264

Updated

5 years ago
Depends on: 850630

Updated

5 years ago
No longer depends on: 837772

Updated

5 years ago
No longer depends on: 843420

Updated

5 years ago
No longer depends on: 838768

Updated

5 years ago
OS: Windows 8 → Windows 8 Metro

Updated

5 years ago

Updated

5 years ago
Depends on: 855090

Updated

5 years ago
No longer depends on: 850264

Updated

5 years ago
No longer blocks: 839336

Updated

5 years ago
Depends on: 856458

Updated

5 years ago
Depends on: 860899
Depends on: 860994

Updated

5 years ago
Depends on: 845079

Updated

5 years ago
Depends on: 864948

Updated

5 years ago
Depends on: 865451

Updated

4 years ago
Depends on: 850019

Updated

4 years ago
Depends on: 879371
No longer depends on: 879371

Updated

4 years ago
Depends on: 880298

Updated

4 years ago
Depends on: 880426

Updated

4 years ago
No longer depends on: 880298

Updated

4 years ago
Depends on: 880298

Updated

4 years ago
No longer depends on: 880298

Comment 38

4 years ago
Please don't disconnect test failure bugs from this tests tracker.
Depends on: 880298
No longer depends on: 880426

Updated

4 years ago
Depends on: 885503

Updated

4 years ago
No longer depends on: 885503

Updated

4 years ago
Depends on: 882333

Updated

4 years ago
Depends on: 859155

Updated

4 years ago
Depends on: 886280

Updated

4 years ago
Depends on: 881260

Updated

4 years ago
Depends on: 881104

Updated

4 years ago
Depends on: 880704

Updated

4 years ago
Depends on: 880703

Updated

4 years ago
Depends on: 880693

Updated

4 years ago
Depends on: 891892

Updated

4 years ago
Depends on: 884247

Updated

4 years ago
Depends on: 880745

Updated

4 years ago
Depends on: 880739

Updated

4 years ago
Depends on: 881463

Updated

4 years ago
No longer depends on: 881260

Updated

4 years ago
No longer depends on: 881104

Updated

4 years ago
No longer depends on: 880704

Updated

4 years ago
No longer depends on: 880703

Updated

4 years ago
No longer depends on: 880693

Updated

4 years ago
No longer depends on: 881463

Updated

4 years ago
No longer depends on: 884247

Updated

4 years ago
No longer depends on: 891892

Updated

4 years ago
No longer depends on: 880745

Updated

4 years ago
No longer depends on: 880739

Updated

4 years ago
No longer depends on: 886280

Updated

4 years ago
Depends on: 886280

Updated

4 years ago
Depends on: 906726

Updated

4 years ago
No longer depends on: 850019

Updated

4 years ago
Depends on: 927903

Updated

4 years ago
No longer depends on: 886280

Updated

4 years ago
No longer depends on: 850630

Updated

4 years ago
No longer depends on: 837766

Updated

4 years ago
No longer depends on: 773817

Updated

4 years ago
Depends on: 947428

Updated

4 years ago
Blocks: 861680
No longer blocks: 841999
Priority: P2 → --
QA Contact: jbecerra
Summary: Epic - Windows 8 Metro Testing Support → Windows 8 Metro Testing Support
Whiteboard: [win8][testing][platform-support] feature=epic → [win8][testing][platform-support][tracking]

Updated

4 years ago
Depends on: 773817

Updated

4 years ago
Depends on: 850630

Updated

4 years ago
Depends on: 837766

Updated

4 years ago
Depends on: 968802
(Assignee)

Updated

3 years ago
OS: Windows 8 Metro → Windows 8.1
Mass close of bugs in obsolete product https://bugzilla.mozilla.org/show_bug.cgi?id=1350354
Status: NEW → RESOLVED
Last Resolved: 5 days ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.