1191481 - Support windows 10 testing in try

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Justin Wood (:Callek)]

this bug will

* Enable windows 10 in buildbot-configs on try (not by default)
* Setup initial windows 10 machines in slavealloc
* Add initial windows 10 machines to graph server
* Add Slave Health support
* <possibly missed items before this bug is closed>

Explicitly Not: Trychooser Support  [because we're not ready for random devs to use it yet]

NOTE: This doesn't track the related work in getting the systems setup, just getting buildbot automation ready to support them

Comment 1

[Justin Wood (:Callek)]

Attachment: [buildbot-configs] win10

Choosing kim since she had to do new platform work recently, and can also tell me if I missed a step above...

Builder list diff:

+Windows 10 64-bit try debug test cpp_gtest ScriptFactory
+Windows 10 64-bit try debug test cppunit ScriptFactory
+Windows 10 64-bit try debug test crashtest ScriptFactory
+Windows 10 64-bit try debug test jetpack UnittestPackagedBuildFactory
+Windows 10 64-bit try debug test jittest ScriptFactory
+Windows 10 64-bit try debug test jsreftest ScriptFactory
+Windows 10 64-bit try debug test marionette ScriptFactory
+Windows 10 64-bit try debug test mochitest-1 ScriptFactory
+Windows 10 64-bit try debug test mochitest-2 ScriptFactory
+Windows 10 64-bit try debug test mochitest-3 ScriptFactory
+Windows 10 64-bit try debug test mochitest-4 ScriptFactory
+Windows 10 64-bit try debug test mochitest-5 ScriptFactory
+Windows 10 64-bit try debug test mochitest-browser-chrome-1 ScriptFactory
+Windows 10 64-bit try debug test mochitest-browser-chrome-2 ScriptFactory
+Windows 10 64-bit try debug test mochitest-browser-chrome-3 ScriptFactory
+Windows 10 64-bit try debug test mochitest-devtools-chrome-1 ScriptFactory
+Windows 10 64-bit try debug test mochitest-devtools-chrome-2 ScriptFactory
+Windows 10 64-bit try debug test mochitest-devtools-chrome-3 ScriptFactory
+Windows 10 64-bit try debug test mochitest-devtools-chrome-4 ScriptFactory
+Windows 10 64-bit try debug test mochitest-gl ScriptFactory
+Windows 10 64-bit try debug test mochitest-jetpack ScriptFactory
+Windows 10 64-bit try debug test mochitest-other ScriptFactory
+Windows 10 64-bit try debug test mochitest-push ScriptFactory
+Windows 10 64-bit try debug test reftest ScriptFactory
+Windows 10 64-bit try debug test web-platform-tests-1 ScriptFactory
+Windows 10 64-bit try debug test web-platform-tests-2 ScriptFactory
+Windows 10 64-bit try debug test web-platform-tests-3 ScriptFactory
+Windows 10 64-bit try debug test web-platform-tests-4 ScriptFactory
+Windows 10 64-bit try debug test web-platform-tests-5 ScriptFactory
+Windows 10 64-bit try debug test web-platform-tests-6 ScriptFactory
+Windows 10 64-bit try debug test web-platform-tests-7 ScriptFactory
+Windows 10 64-bit try debug test web-platform-tests-8 ScriptFactory
+Windows 10 64-bit try debug test web-platform-tests-reftests ScriptFactory
+Windows 10 64-bit try debug test xpcshell ScriptFactory
+Windows 10 64-bit try opt test cpp_gtest ScriptFactory
+Windows 10 64-bit try opt test cppunit ScriptFactory
+Windows 10 64-bit try opt test crashtest ScriptFactory
+Windows 10 64-bit try opt test jetpack UnittestPackagedBuildFactory
+Windows 10 64-bit try opt test jittest ScriptFactory
+Windows 10 64-bit try opt test jsreftest ScriptFactory
+Windows 10 64-bit try opt test marionette ScriptFactory
+Windows 10 64-bit try opt test mochitest-1 ScriptFactory
+Windows 10 64-bit try opt test mochitest-2 ScriptFactory
+Windows 10 64-bit try opt test mochitest-3 ScriptFactory
+Windows 10 64-bit try opt test mochitest-4 ScriptFactory
+Windows 10 64-bit try opt test mochitest-5 ScriptFactory
+Windows 10 64-bit try opt test mochitest-browser-chrome-1 ScriptFactory
+Windows 10 64-bit try opt test mochitest-browser-chrome-2 ScriptFactory
+Windows 10 64-bit try opt test mochitest-browser-chrome-3 ScriptFactory
+Windows 10 64-bit try opt test mochitest-devtools-chrome ScriptFactory
+Windows 10 64-bit try opt test mochitest-e10s-browser-chrome-1 ScriptFactory
+Windows 10 64-bit try opt test mochitest-e10s-browser-chrome-2 ScriptFactory
+Windows 10 64-bit try opt test mochitest-e10s-browser-chrome-3 ScriptFactory
+Windows 10 64-bit try opt test mochitest-gl ScriptFactory
+Windows 10 64-bit try opt test mochitest-jetpack ScriptFactory
+Windows 10 64-bit try opt test mochitest-other ScriptFactory
+Windows 10 64-bit try opt test mochitest-push ScriptFactory
+Windows 10 64-bit try opt test reftest-no-accel ScriptFactory
+Windows 10 64-bit try opt test reftest ScriptFactory
+Windows 10 64-bit try opt test web-platform-tests-1 ScriptFactory
+Windows 10 64-bit try opt test web-platform-tests-2 ScriptFactory
+Windows 10 64-bit try opt test web-platform-tests-3 ScriptFactory
+Windows 10 64-bit try opt test web-platform-tests-4 ScriptFactory
+Windows 10 64-bit try opt test web-platform-tests-reftests ScriptFactory
+Windows 10 64-bit try opt test xpcshell ScriptFactory
+Windows 10 64-bit try talos chromez ScriptFactory
+Windows 10 64-bit try talos dromaeojs ScriptFactory
+Windows 10 64-bit try talos g1 ScriptFactory
+Windows 10 64-bit try talos g2 ScriptFactory
+Windows 10 64-bit try talos other_nol64 ScriptFactory
+Windows 10 64-bit try talos svgr ScriptFactory
+Windows 10 64-bit try talos tp5o ScriptFactory

Comment 2

[Justin Wood (:Callek)]

Comment on attachment 8643899 [details] [diff] [review]
[buildbot-configs] win10

Note: that list of suites was derived from exactly what we have on win8, presuming that we're swapping all win8 for win10 eventually a 1:1 mapping out-of-the-box felt best.

Comment 3

[Justin Wood (:Callek)]

Attachment: [buildbot-configs] win10_slaves

This adds the first batch of slaves, these numbers are tweakable easily, but I'm assuming <=10 machines in first-pass right now.

Comment 4

[Justin Wood (:Callek)]

Attachment: [buildbot-configs] win10 (v2)

Forgot to refresh the diff :/

Comment 5

[Justin Wood (:Callek)]

Attachment: [buildbot-configs] win10_slaves (v2)

removes a hunk from other patch

Comment 6

[Justin Wood (:Callek)]

Comment on attachment 8643910 [details] [diff] [review]
[buildbot-configs] win10_slaves (v2)

Review of attachment 8643910 [details] [diff] [review]:
-----------------------------------------------------------------

moving review to kim per amy's request.

Comment 7

[Justin Wood (:Callek)]

Attachment: [puppet] add to win10 slave support

We need this for masters to actually be able to reconfig with win10 enabled.

Comment 8

[Justin Wood (:Callek)]

Attachment: raw SQL for slavealloc

This does the slavealloc inserts.

Tested on staging slavealloc, heres the final outcome from that on staging:

mysql> select * from slaves where name like 't-w1064-%';troid=@distro_id, bi
+---------+-----------------+----------+--------+---------+-----------+------+---------+-------+--------+----------+-------------
----+---------+------------------+------------------------+--------------+
| slaveid | name            | distroid | bitsid | speedid | purposeid | dcid | trustid | envid | poolid | basedir  | locked_maste
rid | enabled | current_masterid | notes                  | custom_tplid |
+---------+-----------------+----------+--------+---------+-----------+------+---------+-------+--------+----------+-------------
----+---------+------------------+------------------------+--------------+
|   21522 | t-w1064-ix-0001 |       42 |      2 |       2 |         4 |   10 |       4 |     2 |     63 | C:\slave |            N
ULL |       0 |             NULL | Created in Bug 1191481 |         NULL |
|   21523 | t-w1064-ix-0002 |       42 |      2 |       2 |         4 |   10 |       4 |     2 |     63 | C:\slave |            N
ULL |       0 |             NULL | Created in Bug 1191481 |         NULL |
|   21524 | t-w1064-ix-0003 |       42 |      2 |       2 |         4 |   10 |       4 |     2 |     63 | C:\slave |            N
ULL |       0 |             NULL | Created in Bug 1191481 |         NULL |
|   21525 | t-w1064-ix-0004 |       42 |      2 |       2 |         4 |   10 |       4 |     2 |     63 | C:\slave |            N
ULL |       0 |             NULL | Created in Bug 1191481 |         NULL |
|   21526 | t-w1064-ix-0005 |       42 |      2 |       2 |         4 |   10 |       4 |     2 |     63 | C:\slave |            N
ULL |       0 |             NULL | Created in Bug 1191481 |         NULL |
|   21527 | t-w1064-ix-0006 |       42 |      2 |       2 |         4 |   10 |       4 |     2 |     63 | C:\slave |            N
ULL |       0 |             NULL | Created in Bug 1191481 |         NULL |
|   21528 | t-w1064-ix-0007 |       42 |      2 |       2 |         4 |   10 |       4 |     2 |     63 | C:\slave |            N
ULL |       0 |             NULL | Created in Bug 1191481 |         NULL |
|   21529 | t-w1064-ix-0008 |       42 |      2 |       2 |         4 |   10 |       4 |     2 |     63 | C:\slave |            N
ULL |       0 |             NULL | Created in Bug 1191481 |         NULL |
|   21530 | t-w1064-ix-0009 |       42 |      2 |       2 |         4 |   10 |       4 |     2 |     63 | C:\slave |            N
ULL |       0 |             NULL | Created in Bug 1191481 |         NULL |
|   21531 | t-w1064-ix-0010 |       42 |      2 |       2 |         4 |   10 |       4 |     2 |     63 | C:\slave |            N
ULL |       0 |             NULL | Created in Bug 1191481 |         NULL |
+---------+-----------------+----------+--------+---------+-----------+------+---------+-------+--------+----------+-------------
----+---------+------------------+------------------------+--------------+
10 rows in set (0.01 sec)

mysql>

Comment 9

[Justin Wood (:Callek)]

Attachment: raw SQL for graph server

I chose these values based on the following SQL and these thoughts:

* We should call the OS name for win 10 the full spelled out way, easier to parse
* Speeds chosen based on existing values for the hardware (same hardware as win8)
* We are turning on e10s tests as well, so I presume we need e10s values. but we're not doing metro testing so not adding things for that.

If this SQL passes review+feedback, I'll write an official patch for the graphs repo as well, but this is for now.

mysql> select * from os_list;
+----+-----------------------------------+
| id | name                              |
+----+-----------------------------------+
| 23 | Android 2.2                       |
| 20 | Android 2.2 (Native)              |
| 29 | Android 4.0.4                     |
| 18 | CentOS (x86_64) release 5 (Final) |
|  6 | CentOS release 5 (Final)          |
| 14 | Fedora 12 - Constantine           |
| 15 | Fedora 12 x64 - Constantine       |
| 55 | MacOSX 10.10                      |
| 57 | MacOSX 10.10 (e10s)               |
|  7 | MacOSX 10.5.2                     |
| 13 | MacOSX 10.5.8                     |
| 21 | MacOSX 10.6 (rev4)                |
| 51 | MacOSX 10.6 (rev4) (e10s)         |
| 17 | MacOSX 10.6.2 (rev3)              |
| 22 | MacOSX 10.7                       |
| 24 | MacOSX 10.8                       |
| 53 | MacOSX 10.8 (e10s)                |
| 39 | MacOSX 10.9                       |
|  3 | MacOSX Darwin 8.8.1               |
|  4 | MacOSX Darwin 9.2.2               |
|  9 | Nokia n810                        |
| 16 | Nokia n900                        |
|  5 | Ubuntu 7.10                       |
| 10 | Ubuntu 9.04 (x64)                 |
| 33 | Ubuntu HW 12.04                   |
| 41 | Ubuntu HW 12.04 (e10s)            |
| 35 | Ubuntu HW 12.04 x64               |
| 43 | Ubuntu HW 12.04 x64 (e10s)        |
| 27 | Windows Server 2008 R2 (x64)      |
|  1 | WINNT 5.1                         |
| 45 | WINNT 5.1 (e10s)                  |
| 37 | WINNT 5.1 (ix)                    |
|  8 | WINNT 5.2                         |
|  2 | WINNT 6.0                         |
| 12 | WINNT 6.1                         |
| 25 | WINNT 6.1 (ix)                    |
| 47 | WINNT 6.1 (ix) (e10s)             |
| 19 | WINNT 6.1 x64                     |
| 31 | WINNT 6.2 x64                     |
| 49 | WINNT 6.2 x64 (e10s)              |
| 38 | WINNT 6.2 x64 (metro)             |
+----+-----------------------------------+
41 rows in set (0.00 sec)

mysql> select * from machines where name LIKE "t-w864-ix-138%";
+-------+-------+---------------+-----------+-----------------+-----------+------------+
| id    | os_id | is_throttling | cpu_speed | name            | is_active | date_added |
+-------+-------+---------------+-----------+-----------------+-----------+------------+
|  9769 |    31 |             0 | 2.67      | t-w864-ix-138   |         1 | 1414180388 |
| 11787 |    49 |             0 | 3.14      | t-w864-ix-138.e |         1 | 1415296731 |
|  9849 |    38 |             0 | 2.67      | t-w864-ix-138.m |         1 | 1414180435 |
+-------+-------+---------------+-----------+-----------------+-----------+------------+
3 rows in set (0.00 sec)

Comment 10

[Justin Wood (:Callek)]

Attachment: [slave health] win10

This is *un* tested, however I did confirm that "Windows 10" is an available OS choice in bugzilla.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8644150 [details]
raw SQL for graph server

this looks great.

Comment 12

[Justin Wood (:Callek)]

Comment on attachment 8643922 [details] [diff] [review]
[puppet] add to win10 slave support

Merged to prod:

https://hg.mozilla.org/build/puppet/rev/122d1717e735
https://hg.mozilla.org/build/puppet/rev/4419f51a9cef

Comment 13

[Justin Wood (:Callek)]

Attachment: [buildbot-configs] win8-as-win10

Per Q in an e-mail:

<Callek> Q: so, my buildbot configs are all listing the "proper" hostname for win10 right now, and I know you almost have t-w864-102 ready... are you expecting 1-2 [more] human days from that being ready (or less) before you can get me at least one with the proper hostname, or should I add 102 explicitly to the w10 config for my needs shortterm?

His reply:

>> For now I would add thst host by it's windows 8 name to the win 10 configs. It should be ready to start testing now. There may be a config  change to the graphics but that will come up if gpu accelerated tests fail. <<

Comment 14

[Justin Wood (:Callek)]

Comment on attachment 8644150 [details]
raw SQL for graph server

INSERT INTO os_list VALUES (NULL, "Windows 10 64-bit");
SELECT @win10_id:= LAST_INSERT_ID();

## 58 in both staging and prod

INSERT INTO os_list VALUES (NULL, "Windows 10 64-bit (e10s)");
SELECT @win10_e10s_id:= LAST_INSERT_ID();

## 59 in staging, 60 in prod

Comment 15

[Justin Wood (:Callek)]

Comment on attachment 8643962 [details]
raw SQL for slavealloc

Ran this SQL on prod DB

Comment 16

[Justin Wood (:Callek)]

Pushed the 3 configs patches:

https://hg.mozilla.org/build/buildbot-configs/rev/4978bb48e0fb
https://hg.mozilla.org/build/buildbot-configs/rev/ac988ad46349
https://hg.mozilla.org/build/buildbot-configs/rev/a3c1c41d7a78

Comment 17

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/4978bb48e0fb

Comment 18

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/ac988ad46349

Comment 19

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/a3c1c41d7a78

Comment 20

[Justin Wood (:Callek)]

Attachment: [treeherder] PR for adding win10 support

Comment 21

[Ed Morley [:emorley]]

Comment on attachment 8645078 [details] [review]
[treeherder] PR for adding win10 support

Looks good, ty :-)
Have squashed and tweaked the commit message (it referenced adding support to Try; this PR works for all repos), and will rebase + merge manually.

Comment 22

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/5ae5c4909d183f0f7d6c7c0dc31c2afe6e03b876
Bug 1191481 - Add windows 10 support

Comment 23

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8644152 [details] [diff] [review]
[slave health] win10

Review of attachment 8644152 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/slave_health/rev/920e17d3bd96

Comment 24

[Justin Wood (:Callek)]

Assigning to Q for remaining work of getting windows10 ready for developers to use on Try. if that initial list exceeds this bugs configuration I can assist in extending.

But I am no longer working on or tracking this project.

Comment 25

[Q]

The following tests have passed green on the new mdt based systems:

try_win10_64_test-chromez
try_win10_64_test-chromez-e10s
try_win10_64_test-crashtest
try_win10_64_test-dromaeojs
try_win10_64_test-g1
try_win10_64_test-g1-e10s
try_win10_64_test-g2
try_win10_64_test-jittest
try_win10_64_test-marionette
try_win10_64_test-mochitest-1
try_win10_64_test-mochitest-3
try_win10_64_test-mochitest-browser-chrome-1
try_win10_64_test-mochitest-browser-chrome-2
try_win10_64_test-mochitest-browser-chrome-3
try_win10_64_test-mochitest-browser-chrome-4
try_win10_64_test-mochitest-browser-chrome-5
try_win10_64_test-mochitest-browser-chrome-6
try_win10_64_test-mochitest-devtools-chrome-1
try_win10_64_test-mochitest-devtools-chrome-2
try_win10_64_test-mochitest-devtools-chrome-3
try_win10_64_test-mochitest-devtools-chrome-4
try_win10_64_test-mochitest-devtools-chrome-5
try_win10_64_test-mochitest-devtools-chrome-6
try_win10_64_test-mochitest-devtools-chrome-8
try_win10_64_test-mochitest-e10s-browser-chrome-5
try_win10_64_test-mochitest-e10s-browser-chrome-7
try_win10_64_test-mochitest-jetpack
try_win10_64_test-mochitest-push
try_win10_64_test-svgr
try_win10_64_test-svgr-e10s
try_win10_64_test-tp5o
try_win10_64_test-tp5o-e10s
try_win10_64_test-web-platform-tests-1
try_win10_64_test-web-platform-tests-2
try_win10_64_test-web-platform-tests-3
try_win10_64_test-web-platform-tests-4
try_win10_64_test-web-platform-tests-reftests
try_win10_64-debug_test-cppunit
try_win10_64-debug_test-crashtest
try_win10_64-debug_test-jittest
try_win10_64-debug_test-jsreftest
try_win10_64-debug_test-marionette
try_win10_64-debug_test-mochitest-1
try_win10_64-debug_test-mochitest-3
try_win10_64-debug_test-mochitest-5
try_win10_64-debug_test-mochitest-browser-chrome-1
try_win10_64-debug_test-mochitest-browser-chrome-2
try_win10_64-debug_test-mochitest-browser-chrome-3
try_win10_64-debug_test-mochitest-browser-chrome-4
try_win10_64-debug_test-mochitest-browser-chrome-5
try_win10_64-debug_test-mochitest-browser-chrome-6
try_win10_64-debug_test-mochitest-browser-chrome-7
try_win10_64-debug_test-mochitest-devtools-chrome-1
try_win10_64-debug_test-mochitest-devtools-chrome-2
try_win10_64-debug_test-mochitest-devtools-chrome-3
try_win10_64-debug_test-mochitest-devtools-chrome-4
try_win10_64-debug_test-mochitest-devtools-chrome-5
try_win10_64-debug_test-mochitest-devtools-chrome-6
try_win10_64-debug_test-mochitest-devtools-chrome-7
try_win10_64-debug_test-mochitest-devtools-chrome-8
try_win10_64-debug_test-mochitest-gl
try_win10_64-debug_test-mochitest-jetpack
try_win10_64-debug_test-mochitest-push
try_win10_64-debug_test-web-platform-tests-1
try_win10_64-debug_test-web-platform-tests-2
try_win10_64-debug_test-web-platform-tests-4
try_win10_64-debug_test-web-platform-tests-5
try_win10_64-debug_test-web-platform-tests-6
try_win10_64-debug_test-web-platform-tests-7
try_win10_64-debug_test-web-platform-tests-8
try_win10_64-debug_test-web-platform-tests-reftests

Comment 26

[Q]

The following tests have warnings on the new windows 10 MDT based machines:
try_win10_64-debug_test-reftest
try_win10_64-debug_test-web-platform-tests-3
try_win10_64-debug_test-xpcshell
try_win10_64_test-mochitest-e10s-browser-chrome-4
try_win10_64_test-reftest
try_win10_64_test-reftest-no-accel
try_win10_64_test-xpcshell

Comment 27

[Q]

Of the tests in comment 26  the 2 xpcshell tests are known failures. I am looking into the rest now.

Comment 28

[Nick Thomas [:nthomas] (UTC+12)]

We're ignoring these jobs when Nagios checks what the backlog of pending work is (via bug 1261710). We should remove that again when Win 10 goes live in production.

Comment 29

[Mark Cornmesser [:markco]]

I am not sure why this got moved to me, but we are moving directly to TaskCluster with Win 10 without doing buildbot support. Therefore I am closing the bug won't fix.