Closed Bug 770579 (winxp-ix-releng) Opened 7 years ago Closed 6 years ago

(winxp-ix-releng) Run Windows XP jobs on iX hardware, green the tests out and stop running them on Rev3 minis (releng side)

Categories

(Infrastructure & Operations :: CIDuty, task, P2)

x86
Windows XP

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: coop, Assigned: armenzg)

References

Details

(Whiteboard: [refplatform] - status on comment 28)

Attachments

(10 files, 2 obsolete files)

7.51 KB, patch
coop
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
8.72 KB, patch
coop
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
18.99 KB, patch
coop
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
1.21 KB, patch
coop
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
1.60 KB, patch
coop
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
1.03 KB, patch
coop
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
4.39 KB, patch
coop
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
2.51 KB, patch
coop
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
167.46 KB, text/plain
Details
2.13 KB, patch
coop
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
Once the base WinXP OS is installed on the new iX multinode system (bug 765275), releng will need to get the system configured and into the staging testing pool to verify that it works.
Sorely needed, but waiting for people to free up to take this on.
Priority: -- → P3
Whiteboard: [refplatform]
(In reply to Chris Cooper [:coop] from comment #1)
> Sorely needed, but waiting for people to free up to take this on.

Apparently we were waiting for me to free up.
Assignee: nobody → coop
Depends on: 786054
I can probably start this tomorrow/Monday.
Assignee: coop → armenzg
Priority: P3 → P2
Priority: P2 → P3
Assignee: armenzg → coop
Status: NEW → ASSIGNED
Priority: P3 → P2
Details are being recorded here: https://etherpad.mozilla.org/coop-winxp

AFAICT the slave is ready to be connected to my dev master tomorrow.
Tests are running now. Those with access can follow along here:

http://dev-master01.build.scl1.mozilla.com:8045/waterfall
Attached patch Add xp-ix-mn-2 to graphs db (obsolete) — Splinter Review
Attachment #670008 - Flags: review?(armenzg)
Attachment #670008 - Flags: review?(armenzg) → review+
Depends on: 801052
All the tests run against the XP node have returned either green or orange. I'll open a separate bug to collect the test warnings and then close this bug out.

We should move on to the next OS on the iX node here.
Blocks: 801885
(In reply to Chris Cooper [:coop] from comment #7)
> All the tests run against the XP node have returned either green or orange.
> I'll open a separate bug to collect the test warnings and then close this
> bug out.

Filed bug 801885.
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Depends on: 829129
Instead of moving to a new bug let's re-use this one and keep on adding bugs as we get machines to put in staging.
Assignee: coop → nobody
No longer blocks: 801885
Status: RESOLVED → REOPENED
Depends on: 801885
Resolution: FIXED → ---
Summary: Configure WinXP on new iX node → [tracker] Configure WinXP on new iX node
Blocks: 820243
Alias: winxp-ix-releng
Assignee: nobody → armenzg
Priority: P2 → P5
Depends on: 857065
Status:
* waiting on new hire to grab the automation bug to create the imaging steps
* waiting for a machine to be setup with such steps
* waiting on purchase of machines
No longer depends on: 829129
Summary: [tracker] Configure WinXP on new iX node → (winxp-ix-releng) Run Windows XP jobs on iX hardware, green the tests out and stop running them on Rev3 minis (releng side)
Dependency tree:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=770579&hide_resolved=1

Status summary:
###############
* 18 machines have been racked
* We're waiting for more chassis and rack mounts to be delivered
* We need an imaging process (current long pole)


I will prepare the patches to be ready when we have imaged machines.
No longer depends on: 857065
Whiteboard: [refplatform] → [refplatform] - status on comment 11
Attachment #670008 - Attachment is obsolete: true
Attachment #670008 - Flags: checked-in+
I will be putting the following machines on staging:
* t-xp32-ix-001
* t-xp32-ix-002
Priority: P5 → P2
We have failures on:
1) Windows XP 32-bit cedar opt test mochitest-3
    * Due to screen saver bein enabled

2) Windows XP 32-bit cedar debug test mochitest-3
    * It crashes on:
    /tests/dom/media/tests/mochitest/test_peerConnection_basicAudioVideo.html

3) Windows XP 32-bit cedar {opt,debug} test mochitest-other
16:25:56     INFO -  13175 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/layout/inspector/tests/chrome/test_bug467669.xul | number of fonts for mixed Latin & Chinese - got 1, expected 2
16:30:11     INFO -  51002 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/widget/tests/test_keycodes.xul | Thai keyCode=49 (0x31) chars='' [Shift] [Ctrl] activating unshiftedPlus - got 0, expected 1
16:30:12     INFO -  51003 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/widget/tests/test_keycodes.xul | Thai keyCode=187 (0xBB) chars='' [Ctrl] activating unshiftedPlus - got 0, expected 1
16:30:12     INFO -  51004 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/widget/tests/test_keycodes.xul | Thai keyCode=187 (0xBB) chars='' [Shift] [Ctrl] activating unshiftedPlus - got 1, expected 0
1

We might require:
* Install supplemental language support: Regional & Language options -> Languages -> click both RTL and east Asian languages
as indicated in:
https://etherpad.mozilla.org/coop-winxp
https://wiki.mozilla.org/ReferencePlatforms/Test/WinXP#Install_supplemental_language_support
(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #22)
> 2) Windows XP 32-bit cedar debug test mochitest-3
>     * It crashes on:
>     /tests/dom/media/tests/mochitest/test_peerConnection_basicAudioVideo.html

File a separate bug on this and paste in the stack?
Attached file crash
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #23)
> (In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment
> #22)
> > 2) Windows XP 32-bit cedar debug test mochitest-3
> >     * It crashes on:
> >     /tests/dom/media/tests/mochitest/test_peerConnection_basicAudioVideo.html
> 
> File a separate bug on this and paste in the stack?

I think it happened because of the screen saver. It is not happening anymore.
"07:50:23     INFO -  18323 INFO Error: Unable to restore focus, expect failures and timeouts."
I've deployed RTL and east Asian languages and I'm re-running the jobs on those 2 machines.
Attachment #747445 - Flags: review?(coop)
Attachment #747446 - Flags: review?(coop)
Attachment #747447 - Flags: review?(coop)
Attachment #747448 - Flags: review?(coop)
Attachment #747449 - Flags: review?(coop)
Attachment #747450 - Flags: review?(coop)
Attachment #747451 - Flags: review?(coop)
Attachment #747452 - Flags: review?(coop)
Attachment #747453 - Flags: review?(coop)
All jobs were green. I believe we only have one intermittent job as it was green before.
Duplicate of this bug: 801885
Depends on: 857065
Dependency tree:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=770579&hide_resolved=1

Status summary:
###############
* all patches are ready
* 2 machines were given to armenzg to test and the imaging issues have been raised
* releng should have all remaining nodes soon
Whiteboard: [refplatform] - status on comment 11 → [refplatform] - status on comment 28
Attachment #747445 - Flags: review?(coop) → review+
Attachment #747446 - Flags: review?(coop) → review+
Attachment #747447 - Flags: review?(coop) → review+
Attachment #747448 - Flags: review?(coop) → review+
Attachment #747449 - Flags: review?(coop) → review+
Attachment #747450 - Flags: review?(coop) → review+
Attachment #747451 - Flags: review?(coop) → review+
Attachment #747452 - Flags: review?(coop) → review+
Attachment #747453 - Flags: review?(coop) → review+
Comment on attachment 747445 [details] [diff] [review]
[slavealloc] add winxp ix machines

Deployed.
Attachment #747445 - Flags: checked-in+
Comment on attachment 747446 [details] [diff] [review]
[graphs] add winxp ix machines

Deployed.

Slave 37 was duplicated. start and commit transaction helped prevent any problems.
Attachment #747446 - Flags: checked-in+
Comment on attachment 747448 [details] [diff] [review]
[tbpl] add winxp ix jobs

For TBPL patches where the bug isn't in the TBPL component & you are not asking a TBPL maintainer for review, please can you at least make sure I'm CCed? Was a surprise seeing this in the TBPL repo (about to do a production push of what should have just been one changeset - but noticed this which needs testing too pre-push, just in case).
Oh ha, I was CCed, just got lost in all the other reviews/landings. Don't suppose we could have a separate bug in the TBPL component next time? :-)
Depends on: 873143
TBPL part in production :-)
(In reply to Ed Morley (Away until 17th May) [:edmorley UTC+1] from comment #32)
> Oh ha, I was CCed, just got lost in all the other reviews/landings. Don't
> suppose we could have a separate bug in the TBPL component next time? :-)

Sorry about that. I just assume that it was so similar to the Win7-ix you reviewed that it would be sufficient with just coop. I will make sure I run it through you and file a separate bug next time. I'm very sorry!!
Attachment #747448 - Flags: checked-in+
Attachment #747449 - Attachment description: [configs] add winxp-ix as valid platform → [puppet-manifests] add winxp-ix as valid platform
Attachment #747449 - Flags: checked-in+
Attachment #747450 - Attachment description: [buildfaster] add winxp-ix to valid buildfaster platforms → [braindump-buildfaster] add winxp-ix to valid buildfaster platforms
Attachment #747451 - Flags: checked-in+
Attachment #747452 - Flags: checked-in+
Attachment #747453 - Attachment is obsolete: true
Attachment #747447 - Flags: checked-in+
Attachment #747450 - Flags: checked-in+
Something here is in production
FYI, I have enabled this on m-i, try and cedar.
I will disable the builders until I have more time to evaluate if there are any bad slaves.
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=Windows%20XP&showall=1 - show_all was just trying to show you that you didn't yet hide browser-chrome, the slowest little test.
(In reply to Phil Ringnalda (:philor) from comment #39)
> https://tbpl.mozilla.org/?tree=Mozilla-
> Inbound&jobname=Windows%20XP&showall=1 - show_all was just trying to show
> you that you didn't yet hide browser-chrome, the slowest little test.

I hid one last builder a little bit ago.
Depends on: 867695
Depends on: 873727
Depends on: 874007
No longer depends on: 874007
Attachment #752248 - Flags: review? → review+
Live in production.
Attachment #752248 - Flags: checked-in+
Depends on: 874619
Depends on: 874922
No longer depends on: 873727
We're looking into disabling Windows XP builders on Rev3 tonight and evaluate on Monday if WinXP on iX is good enough.
Depends on: 875272
Depends on: 874953
Depends on: 875868
Since I'm adjusting things for both XP and Win7 I will use the blocking bug (bug 820243) instead.
I'm adding the following machines that were received later on and that have run a bunch of green tests:
* t-xp32-ix-096
* t-xp32-ix-097
* t-xp32-ix-098
* t-xp32-ix-100

I still have to get this machine into production:
* t-xp32-ix-099
* t-xp32-ix-001 - loaned in bug 829129
* t-xp32-ix-002 - loaned in bug 829129
Depends on: 876514
We're closing this. All the major work is done. Tomorrow we're stopping rev3 jobs for m-c based project and m-a. We will ride the trains from there on.

We will follow up on individual slaves with our normal problem tracking bugs:
* t-xp32-ix-001 - loaned in bug 829129
* t-xp32-ix-002 - loaned in bug 829129
* t-xp32-ix-099 - bug 876517
Blocks: 874922, 876514
Status: REOPENED → RESOLVED
Closed: 7 years ago6 years ago
No longer depends on: 874922, 876514
Resolution: --- → FIXED
Depends on: 877465
Blocks: 900671
Product: mozilla.org → Release Engineering
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.