770579 - (winxp-ix-releng) (winxp-ix-releng) Run Windows XP jobs on iX hardware, green the tests out and stop running them on Rev3 minis (releng side)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Description

[Chris Cooper [:coop] (he/him)]

Once the base WinXP OS is installed on the new iX multinode system (bug 765275), releng will need to get the system configured and into the staging testing pool to verify that it works.

Comment 1

[Chris Cooper [:coop] (he/him)]

Sorely needed, but waiting for people to free up to take this on.

Comment 2

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #1)
> Sorely needed, but waiting for people to free up to take this on.

Apparently we were waiting for me to free up.

Comment 3

[Armen [:armenzg]]

I can probably start this tomorrow/Monday.

Comment 4

[Chris Cooper [:coop] (he/him)]

Details are being recorded here: https://etherpad.mozilla.org/coop-winxp

AFAICT the slave is ready to be connected to my dev master tomorrow.

Comment 5

[Chris Cooper [:coop] (he/him)]

Tests are running now. Those with access can follow along here:

http://dev-master01.build.scl1.mozilla.com:8045/waterfall

Comment 6

[Chris Cooper [:coop] (he/him)]

Attachment: Add xp-ix-mn-2 to graphs db

Comment 7

[Chris Cooper [:coop] (he/him)]

All the tests run against the XP node have returned either green or orange. I'll open a separate bug to collect the test warnings and then close this bug out.

We should move on to the next OS on the iX node here.

Comment 8

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #7)
> All the tests run against the XP node have returned either green or orange.
> I'll open a separate bug to collect the test warnings and then close this
> bug out.

Filed bug 801885.

Comment 9

[Armen [:armenzg]]

Instead of moving to a new bug let's re-use this one and keep on adding bugs as we get machines to put in staging.

Comment 10

[Armen [:armenzg]]

Status:
* waiting on new hire to grab the automation bug to create the imaging steps
* waiting for a machine to be setup with such steps
* waiting on purchase of machines

Comment 11

[Armen [:armenzg]]

Dependency tree:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=770579&hide_resolved=1

Status summary:
###############
* 18 machines have been racked
* We're waiting for more chassis and rack mounts to be delivered
* We need an imaging process (current long pole)


I will prepare the patches to be ready when we have imaged machines.

Comment 12

[Armen [:armenzg]]

Attachment: [slavealloc] add winxp ix machines

Comment 13

[Armen [:armenzg]]

Attachment: [graphs] add winxp ix machines

Comment 14

[Armen [:armenzg]]

Attachment: [configs] add winxp ix jobs

Comment 15

[Armen [:armenzg]]

Attachment: [tbpl] add winxp ix jobs

Comment 16

[Armen [:armenzg]]

Attachment: [puppet-manifests] add winxp-ix as valid platform

Comment 17

[Armen [:armenzg]]

Attachment: [braindump-buildfaster] add winxp-ix to valid buildfaster platforms

Comment 18

[Armen [:armenzg]]

Attachment: [tools] run jetpack on winxp-ix and add to trychooser

Comment 19

[Armen [:armenzg]]

Attachment: [buildapi] add winxp-ix

Comment 20

[Armen [:armenzg]]

Attachment: [puppet] add winxp-ix as a valid platform

Comment 21

[Armen [:armenzg]]

I will be putting the following machines on staging:
* t-xp32-ix-001
* t-xp32-ix-002

Comment 22

[Armen [:armenzg]]

We have failures on:
1) Windows XP 32-bit cedar opt test mochitest-3
    * Due to screen saver bein enabled

2) Windows XP 32-bit cedar debug test mochitest-3
    * It crashes on:
    /tests/dom/media/tests/mochitest/test_peerConnection_basicAudioVideo.html

3) Windows XP 32-bit cedar {opt,debug} test mochitest-other
16:25:56     INFO -  13175 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/layout/inspector/tests/chrome/test_bug467669.xul | number of fonts for mixed Latin & Chinese - got 1, expected 2
16:30:11     INFO -  51002 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/widget/tests/test_keycodes.xul | Thai keyCode=49 (0x31) chars='' [Shift] [Ctrl] activating unshiftedPlus - got 0, expected 1
16:30:12     INFO -  51003 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/widget/tests/test_keycodes.xul | Thai keyCode=187 (0xBB) chars='' [Ctrl] activating unshiftedPlus - got 0, expected 1
16:30:12     INFO -  51004 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/widget/tests/test_keycodes.xul | Thai keyCode=187 (0xBB) chars='' [Shift] [Ctrl] activating unshiftedPlus - got 1, expected 0
1

We might require:
* Install supplemental language support: Regional & Language options -> Languages -> click both RTL and east Asian languages
as indicated in:
https://etherpad.mozilla.org/coop-winxp
https://wiki.mozilla.org/ReferencePlatforms/Test/WinXP#Install_supplemental_language_support

Comment 23

[(not currently active) Ted Mielczarek]

(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #22)
> 2) Windows XP 32-bit cedar debug test mochitest-3
>     * It crashes on:
>     /tests/dom/media/tests/mochitest/test_peerConnection_basicAudioVideo.html

File a separate bug on this and paste in the stack?

Comment 24

[Armen [:armenzg]]

Attachment: crash

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #23)
> (In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment
> #22)
> > 2) Windows XP 32-bit cedar debug test mochitest-3
> >     * It crashes on:
> >     /tests/dom/media/tests/mochitest/test_peerConnection_basicAudioVideo.html
> 
> File a separate bug on this and paste in the stack?

I think it happened because of the screen saver. It is not happening anymore.
"07:50:23     INFO -  18323 INFO Error: Unable to restore focus, expect failures and timeouts."

Comment 25

[Armen [:armenzg]]

I've deployed RTL and east Asian languages and I'm re-running the jobs on those 2 machines.

Comment 26

[Armen [:armenzg]]

All jobs were green. I believe we only have one intermittent job as it was green before.

Comment 28

[Armen [:armenzg]]

Dependency tree:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=770579&hide_resolved=1

Status summary:
###############
* all patches are ready
* 2 machines were given to armenzg to test and the imaging issues have been raised
* releng should have all remaining nodes soon

Comment 29

[Armen [:armenzg]]

Comment on attachment 747445 [details] [diff] [review]
[slavealloc] add winxp ix machines

Deployed.

Comment 30

[Armen [:armenzg]]

Comment on attachment 747446 [details] [diff] [review]
[graphs] add winxp ix machines

Deployed.

Slave 37 was duplicated. start and commit transaction helped prevent any problems.

Comment 31

[Ed Morley [:emorley]]

Comment on attachment 747448 [details] [diff] [review]
[tbpl] add winxp ix jobs

For TBPL patches where the bug isn't in the TBPL component & you are not asking a TBPL maintainer for review, please can you at least make sure I'm CCed? Was a surprise seeing this in the TBPL repo (about to do a production push of what should have just been one changeset - but noticed this which needs testing too pre-push, just in case).

Comment 32

[Ed Morley [:emorley]]

Oh ha, I was CCed, just got lost in all the other reviews/landings. Don't suppose we could have a separate bug in the TBPL component next time? :-)

Comment 33

[Ed Morley [:emorley]]

TBPL part in production :-)

Comment 34

[Armen [:armenzg]]

(In reply to Ed Morley (Away until 17th May) [:edmorley UTC+1] from comment #32)
> Oh ha, I was CCed, just got lost in all the other reviews/landings. Don't
> suppose we could have a separate bug in the TBPL component next time? :-)

Sorry about that. I just assume that it was so similar to the Win7-ix you reviewed that it would be sufficient with just coop. I will make sure I run it through you and file a separate bug next time. I'm very sorry!!

Comment 35

[Armen [:armenzg]]

http://hg.mozilla.org/build/tools/rev/871348f46fad
https://hg.mozilla.org/webtools/tbpl/rev/72ea7abb0bd4
http://hg.mozilla.org/build/puppet-manifests/rev/7ed94c03c549
https://hg.mozilla.org/build/braindump/rev/7eecf176f4d3
http://hg.mozilla.org/build/tools/rev/871348f46fad
http://hg.mozilla.org/build/buildapi/rev/7dbf323b8496

Comment 36

[[e-mail] (:bhearsum)]

Something here is in production

Comment 37

[Armen [:armenzg]]

FYI, I have enabled this on m-i, try and cedar.
I will disable the builders until I have more time to evaluate if there are any bad slaves.

Comment 38

[Armen [:armenzg]]

The jobs are running hidden in here:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=Windows%20XP&show_all=1

Comment 39

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=Windows%20XP&showall=1 - show_all was just trying to show you that you didn't yet hide browser-chrome, the slowest little test.

Comment 40

[Armen [:armenzg]]

(In reply to Phil Ringnalda (:philor) from comment #39)
> https://tbpl.mozilla.org/?tree=Mozilla-
> Inbound&jobname=Windows%20XP&showall=1 - show_all was just trying to show
> you that you didn't yet hide browser-chrome, the slowest little test.

I hid one last builder a little bit ago.

Comment 41

[Armen [:armenzg]]

Attachment: [configs] add xp-ix to every branch

Comment 42

[Armen [:armenzg]]

Live in production.

Comment 43

[Armen [:armenzg]]

We're looking into disabling Windows XP builders on Rev3 tonight and evaluate on Monday if WinXP on iX is good enough.

Comment 44

[Armen [:armenzg]]

Since I'm adjusting things for both XP and Win7 I will use the blocking bug (bug 820243) instead.

Comment 45

[Armen [:armenzg]]

I'm adding the following machines that were received later on and that have run a bunch of green tests:
* t-xp32-ix-096
* t-xp32-ix-097
* t-xp32-ix-098
* t-xp32-ix-100

I still have to get this machine into production:
* t-xp32-ix-099
* t-xp32-ix-001 - loaned in bug 829129
* t-xp32-ix-002 - loaned in bug 829129

Comment 46

[Armen [:armenzg]]

We're closing this. All the major work is done. Tomorrow we're stopping rev3 jobs for m-c based project and m-a. We will ride the trains from there on.

We will follow up on individual slaves with our normal problem tracking bugs:
* t-xp32-ix-001 - loaned in bug 829129
* t-xp32-ix-002 - loaned in bug 829129
* t-xp32-ix-099 - bug 876517