770578 - (win7-ix-releng) Run Windows 7 jobs on iX hardware, green the tests out and stop running them on Rev3 minis (releng side)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Description

[Chris Cooper [:coop] (he/him)]

Once the base Win7 OS is installed on the new iX multinode system (bug 765275), releng will need to get the system configured and into the staging testing pool to verify that it works.

Comment 1

[Chris Cooper [:coop] (he/him)]

Sorely needed, but waiting for people to free up to take this on.

Comment 2

[Armen [:armenzg]]

Status:
* waiting on automated steps for Win7 to be in place
* Q says that we're not too far from being able to get a machine that can run buildbot
* I will put such machine into staging once I get my hands on it

Comment 3

[Armen [:armenzg]]

As well as:
* waiting on purchase of machines

Comment 4

[Armen [:armenzg]]

I have connected ix-mn-xp-002 (yes, it is a win7 machine) to my staging master and hope I can make it take jobs by next week. I will work on it and report back later next week.

Comment 5

[Armen [:armenzg]]

I found out from bug 857042 that we will have the machines delivered around 4/22.
DCOps is doing all preparatory work for that.

Once racked, relops will do whatever is left for the machines to be ready for production in bug 857064.

I will have our patches ready on this bug for when the machines are production ready.

Comment 6

[Armen [:armenzg]]

Attachment: [buildbot-configs] add win7-ix

Comment 7

[Armen [:armenzg]]

Attachment: [tools] win7 ix changes

Comment 8

[Armen [:armenzg]]

Attachment: [puppet] win7 ix changes

Comment 9

[Armen [:armenzg]]

Attachment: [buildbotcustom] win7 changes

This makes us run python27 instead of python25.

We don't need the splitting anymore:
http://hg.mozilla.org/build/buildbotcustom/rev/8afa41708c29

Comment 10

[Rail Aliiev [:rail]]

in production

Comment 11

[Armen [:armenzg]]

No need for this dep.

Comment 12

[Armen [:armenzg]]

I'm back to this after b2g prep work and other machine clean up.

I would like to have patches for next week to put ix-mn-xp-002 on Cedar even if talos jobs go red because of a missing slave name (I don't want to put live such hostname on the production graphs DB).

Comment 13

[Armen [:armenzg]]

Attachment: [graphs] add win7 ix machines to graphs

The OS exists in the production DB as this:
| 25 | WINNT 6.1 (ix)                    |

Comment 14

[Armen [:armenzg]]

Attachment: [tools] win7 ix changes

Comment 15

[Armen [:armenzg]]

Attachment: [buildbot-configs] add win7-ix

Comment 16

[Armen [:armenzg]]

Attachment: [slavealloc] add win7 ix machines

Comment 17

[Chris Cooper [:coop] (he/him)]

Comment on attachment 740795 [details] [diff] [review]
[buildbot-configs] add win7-ix

Review of attachment 740795 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with the nit fixed.

::: mozilla-tests/config.py
@@ +86,4 @@
>  PLATFORMS['win32']['env_name'] = 'win32-perf'
>  PLATFORMS['win32']['xp'] = {'name': "Rev3 WINNT 5.1"}
>  PLATFORMS['win32']['win7'] = {'name': "Rev3 WINNT 6.1"}
> +PLATFORMS['win32']['win7-ix'] = {'name': "Windows 7 32-bit"}

I like this change. Is there a bug on file to change the other platform strings similarly?

@@ +1286,5 @@
> +
> +# Let's disable win7-ix while we green things out on Cedar
> +for branch in BRANCHES.keys():
> +    if branch == "cedar":
> +        break

You want to use 'continue' here and not 'break' - 'break' will exit the for loop, probably before you've removed win7-ix from all the branches you need to.

Comment 18

[Armen [:armenzg]]

Attachment: [buildbotcustom] win7 changes

Comment 19

[Armen [:armenzg]]

(In reply to Chris Cooper [:coop] from comment #17)
> Comment on attachment 740795 [details] [diff] [review]
> [buildbot-configs] add win7-ix
> 
> Review of attachment 740795 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> r+ with the nit fixed.
> 
> ::: mozilla-tests/config.py
> @@ +86,4 @@
> >  PLATFORMS['win32']['env_name'] = 'win32-perf'
> >  PLATFORMS['win32']['xp'] = {'name': "Rev3 WINNT 5.1"}
> >  PLATFORMS['win32']['win7'] = {'name': "Rev3 WINNT 6.1"}
> > +PLATFORMS['win32']['win7-ix'] = {'name': "Windows 7 32-bit"}
> 
> I like this change. Is there a bug on file to change the other platform
> strings similarly?
> 
Once I see win7 ix running full in production I will fix win8.
I will not be working on fixing it for rev3 minis unless necessary since we're dropping them.

Comment 20

[Justin Wood (:Callek)]

Comment on attachment 740880 [details] [diff] [review]
[buildbotcustom] win7 changes

in production as of 5p PT yesterday.

Comment 21

[Armen [:armenzg]]

Summary:
########
This bug is waiting on bug 857064 to give us machines that can take production jobs without causing orange tests.
There are 2 known blockers on imaging.
IT has 18 Win7 machines racked waiting to be imaged with correct image.

Comment 22

[Armen [:armenzg]]

Comment on attachment 740793 [details] [diff] [review]
[tools] win7 ix changes

http://hg.mozilla.org/build/tools/rev/7bdc280b0ab0

Comment 23

[Armen [:armenzg]]

Comment on attachment 740880 [details] [diff] [review]
[buildbotcustom] win7 changes

http://hg.mozilla.org/build/buildbotcustom/rev/de40037f81cc

Comment 24

[Armen [:armenzg]]

Comment on attachment 740840 [details] [diff] [review]
[slavealloc] add win7 ix machines

Deployed.

Comment 25

[Armen [:armenzg]]

Next week we will have some machines to add to Cedar.
For now I will wait to land the puppet and buildbot-configs changes until I verify that those machines would be ready in production (I would test them on staging).

Comment 26

[Armen [:armenzg]]

In production.

Comment 27

[Armen [:armenzg]]

These are running on Cedar:
https://tbpl.mozilla.org/?tree=Cedar&jobname=Windows%207%2032-bit%20cedar

Comment 28

[Armen [:armenzg]]

I had to back-out the code as I had made some mistakes that I did not catch on staging. I will do it properly on Monday.

For now, I have discovered 4 test failures on:
* Windows 7 32-bit cedar {opt,debug} test mochitest-1
* Windows 7 32-bit cedar {opt,debug} test reftest

The reftest ones will be fixed by testing a new build of cedar since the one I was testing did not have the fix that I landed on bug 864788.

The mochitest one shows me a different issue.
The problem is that we're using Windows 7 N which means No Media.
This was found in here for win8:
https://bugzilla.mozilla.org/show_bug.cgi?id=847879#c7
We distributed the fix in here:
Bug 849827 - Please distribute to win8 ix machines "Microsoft Windows Media Feature Pack > for N and KN Versions of all Windows 8 Editions"
Error:
09:40:47     INFO -  200413 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_can_play_type_mpeg.html | video/mp4='maybe' - got , expected maybe

Comment 29

[Armen [:armenzg]]

Attachment: [brainddump] buildfaster report

Comment 30

[Armen [:armenzg]]

Attachment: [configs] WIP - add win7 ix

I think I'm getting tunnel vision.
This add the Win7 on ix tests but it messes up the Windows platforms we're supposed to test on some of these release branches.

For instance, win8 gets added to mozilla-b2g18_v1_0_1 even though I have specified only two platforms:
BRANCHES['mozilla-b2g18']['platforms']['win32']['slave_platforms'] = ['xp', 'win7']

rail, can you please help me out?

Comment 31

[Rail Aliiev [:rail]]

Comment on attachment 746633 [details] [diff] [review]
[configs] WIP - add win7 ix

Review of attachment 746633 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +1395,5 @@
> +    pf_win32 = BRANCHES[branch]['platforms'].get('win32', {})
> +    if "win7-ix" in pf_win32:
> +        del BRANCHES[branch]['platforms']['win32']['win7-ix']
> +        if "win7-ix" in pf_win32.get('slave_platforms', []):
> +            BRANCHES[branch]['platforms']['win32']['slave_platforms'].remove("win7-ix")

This won't help you since misc.py uses different logic for mozilla-tests. You have to change BRANCHES[branch]['suites'][s]['options'] (like in http://hg.mozilla.org/build/buildbot-configs/file/59ce2b672588/mozilla-tests/config.py#l1306) and BRANCHES[branch]['%(suite)s_tests'] (like in http://hg.mozilla.org/build/buildbot-configs/file/59ce2b672588/mozilla-tests/config.py#l1311) to remove some platforms.

::: mozilla-tests/config_common.py
@@ +81,1 @@
>  def get_talos_slave_platforms(platforms_dict, platforms):

You don't need this hunk. It breaks PEP8.

Comment 32

[Armen [:armenzg]]

Attachment: [configs] add win7 ix

This only adds Win7 on ix to Cedar and Jetpack tree.

Comment 33

[Rail Aliiev [:rail]]

Comment on attachment 747118 [details] [diff] [review]
[configs] add win7 ix

I know, our configs are great! :) In overall the patch looks ok, except one thing: I'm seeing win7-ix jetpack enabled on m-c, m-a, m-b and m-r. Is that expected? https://gist.github.com/rail/5547097

Comment 34

[Armen [:armenzg]]

(In reply to Rail Aliiev [:rail] from comment #33)
> Comment on attachment 747118 [details] [diff] [review]
> [configs] add win7 ix
> 
> I know, our configs are great! :) In overall the patch looks ok, except one
> thing: I'm seeing win7-ix jetpack enabled on m-c, m-a, m-b and m-r. Is that
> expected? https://gist.github.com/rail/5547097

There are two different jetpacks in this world!
The Jetpack/Addon-SDK project and the jetpack suite that is like any other suite.
I'm enabling the first one!

Comment 35

[Armen [:armenzg]]

Summary:
########
* We have 10 machines running on Cedar
* We're waiting on the delivery of rack mount kits so we can rack remaining win7 machines
** We don't yet have an ETA on when they will be delivered
* Once I have more machines I can enable more branches

Comment 36

[Armen [:armenzg]]

Attachment: [configs] enable win7-ix for all FF23 trees

We're expecting to receive all remaining Windows 7 iX nodes between Tuesday to Wednesday. This patch prepares us to have things ready.

Comment 37

[Armen [:armenzg]]

Summary:
########
* We have 10 machines running on Cedar
* All nodes should be available sometime tomorrow
* I will reconfigure our masters to get them in production

Comment 38

[Armen [:armenzg]]

Comment on attachment 748827 [details] [diff] [review]
[configs] enable win7-ix for all FF23 trees

This is live now.
We have in the production pool the following machines:
* t-w732-ix-0[08-98]

The first 7 will be added to the pool. 99 and 100 will be kept to work on IT's needed graphic cards' solution.

Comment 39

[Armen [:armenzg]]

I can see it running on Jamun:
https://tbpl.mozilla.org/?tree=Jamun&jobname=Windows%207

Comment 40

[Armen [:armenzg]]

Summary
#######
* bug 870453 - fix a bunch of Windows 7 intermittent oranges
* bug 872836 - fix a talos issue
* bug 829126 - 7 more machines need to be added by IT
* bug 873566 - Determine why these 4 machines had graphic issues

Comment 41

[Armen [:armenzg]]

We're looking into disabling Windows 7 builders on Rev3 tonight and evaluate on Monday if Win7 on iX is good enough.

Comment 42

[Armen [:armenzg]]

Since I'm adjusting things for both XP and Win7 I will use the blocking bug (bug 820243) instead.

Comment 43

[Armen [:armenzg]]

I'm adding the following machines that were received later on and that have run a bunch of green tests:
* t-w732-ix-001
* t-w732-ix-002
* t-w732-ix-005
* t-w732-ix-006
* t-w732-ix-007
* t-w732-ix-099

I still have to get this machine into production:
* t-w732-ix-100
* t-w732-ix-003 - loaned in bug 829129
* t-w732-ix-004 - loaned in bug 829129

Comment 44

[Armen [:armenzg]]

We're closing this. All the major work is done. Tomorrow we're stopping rev3 jobs for m-c based project and m-a. We will ride the trains from there on.

We will follow up on individual slaves with our normal problem tracking bugs:
* t-w732-ix-003 - loaned in bug 876013
* t-w732-ix-004 - loaned in bug 876013
* t-w732-ix-100 - iX request for repair in bug 876518

bug 829126 - has been left open to deal to discover the graphic cards solution but nothing left in there for the releng needs.