Closed Bug 827532 Opened 13 years ago Closed 12 years ago

Move Mozmill CI system from MV to SCL3 data center

Categories

(Mozilla QA Graveyard :: Infrastructure, defect)

Product:
Mozilla QA Graveyard
Component:
Infrastructure
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: whimboo, Assigned: whimboo)

References

(
URL
)

Details

The whole Mozmill CI system has to be moved from MV to SCL3. Therefore a lot of work is necessary from IT before we can do something at all. Lets get any work tracked by this bug. Once dependencies have been fixed we can go ahead and make the remaining changes to the configs, and clean-up and release the old system.
I have created a pull on github which will contain the following: * Added config files for all type of builds * Added 3 hosts per platform for Windows, Linux, and Mac - Ubuntu has two versions now, and Fedore will follow later * Let the new CI execute tests for Windows, and Linux - Mac will be added later once we got the machines setup * Stopped Windows and Linux tests in the old CI Here the pull request: https://github.com/mozilla/mozmill-ci/pull/195
Landed pull after a minor update as: https://github.com/mozilla/mozmill-ci/commit/7f004554bad3f816bc955c9920924d51eb546dcc I will go ahead and merge it down to the staging/production branch, and update old production and new production. With that we have Windows and Linux tests live in the new CI.
Component: Mozmill Automation → Infrastructure
QA Contact: hskupin
Windows and Linux slaves have been turned off in the old CI system in favor of our new one in SCL3! Next steps here are: * Waiting for IT to get us setup all the requested Mac minis * Preparing Fedora machines to get them integrated
Remaining steps here are: * Integrate new Mac mini machines into new CI * Get old ESX cluster shutdown * Move l10n testing to new CI * Get rid of old Mac minis * Get rid of old Mac Pro machines * Preparing Fedora machines to get them integrated
Before we can continue here bug 853074 has to be fixed first. What has been changed since the last update: * Integrate new Mac mini machines into new CI Remaining steps here are: * Combine both CI systems into a single one (https://github.com/mozilla/mozmill-ci/pull/234) * Get old ESX cluster shutdown (bug 864732) * Move l10n testing to new CI * Get rid of old Mac minis * Get rid of old Mac Pro machines * Preparing Fedora machines to get them integrated
What has been changed since the last update: * Combine both CI systems into a single one (https://github.com/mozilla/mozmill-ci/pull/234) Remaining steps here are: * Move l10n testing to new CI * Get rid of old Mac minis * Get rid of old Mac Pro machines * Preparing Fedora machines to get them integrated * Get old ESX cluster shutdown (bug 864732)
Moving the l10n tests to the production CI is handled by: https://github.com/mozilla/mozmill-ci/issues/237
I was able to bring the last new mac mini online in the new cluster. So I have disabled the old one. I will wait some days before continuing with the l10n tests.
Given the latest changes as of today I want to give an update about the current state. What has been changed since the last update: * Get old ESX cluster shutdown (bug 864732) * Move l10n testing to new CI Both tasks have been done, whereby we had to put a fair amount of work into the second item. It was not only code in mozmill-ci we had to change but also in pulsebuildmonitor, pulsetranslator, and mozdownload. I'm happy that the l10n tests are now alive in our production mozmill-ci cluster. Remaining steps todo are: * Get rid of old Mac minis * Get rid of old Mac Pro machines * Preparing Fedora machines to get them integrated I will follow-up shortly with newly filed bugs to cover the decommission of the mac minis. Something we might want to find out is if some of those could be used in SCL3, so we can transfer them to test OS X 10.9.
7 of the mac minis have been moved to SCL3 to extend our existing CI machines to 4 nodes per platform and to add support for OS X 10.9. Also 1 more mini has been moved as spare into SCL3. That means the following Mac Minis are remaining in MV: mm-osx-105-1 (release1-osx-105.qa.mtv1.mozilla.com) [unused] mm-osx-105-2 (release2-osx-105.qa.mtv1.mozilla.com) [unused] mm-osx-105-3 (release3-osx-105.qa.mtv1.mozilla.com) [unused] mm-osx-106-1 (release1-osx-106.qa.mtv1.mozilla.com) [unused] mm-osx-106-2 (release2-osx-106.qa.mtv1.mozilla.com) [unused] mm-osx-106-4 (release4-osx-106.qa.mtv1.mozilla.com) [unused] mm-osx-108-3 (release7-osx-107.qa.mtv1.mozilla.com) [unused] I'm not sure about the minis and their hardware specs. Marcia, what would be the best way to get the asset tags assigned to the machine names? I'm not sure if they have their name printed on the body. Would you mind to check that? I think that we do not need all those machines anymore, so if QA has needs you could get some for testing. In terms of Mac Pro machines we have: QA-Horus aka release1.qa.mtv1.mozilla.com QA-Set aka release2.qa.mtv1.mozilla.com (might be used as try server soon) QA-Masterblaster aka release3.qa.mtv1.mozilla.com (staging server for mozmill-ci) Marcia, could QA need a Mac Pro with 32GB of RAM? If not we will find an adequate usage of the box.
Flags: needinfo?(mozillamarcia.knous)
Last time we mentioned the Mac Pro in the QA meeting there was interest - I can email qa-staff about both minis and mac pros and see if there is interest.
Flags: needinfo?(mozillamarcia.knous)
I have need of two of Mac Minis of rev 3 or higher to setup some remote profile testing systems.
Before 7 of the mac minis were moved to SCL3 we had the following set: 4464 - 10.6 2.66ghz core 2 duo 8gb ram 320gb hdd 4588 - 10.6 2.4ghz core 2 duo 4gb ram 320gb hdd 4587 - 10.6 2.4ghz core 2 duo 4gb ram 320gb hdd 4453 - 10.6 2.66ghz core 2 duo 8gb ram 320gb hdd 7638 - 10.7 2.7ghz i7 8gb ram 500gb hdd 7640 - 10.7 2.7ghz i7 8gb ram 500gb hdd 7639 - 10.7 2.7ghz i7 8gb ram 500gb hdd 7644 - 10.7 2.7ghz i7 8gb ram 500gb hdd 9716 - 10.8 2.3ghz i7 16gb ram 250gb ssd 7645 - 10.8 2.7ghz i7 8gb ram 500gb hdd 7642 - 10.8 2.7ghz i7 8gb ram 500gb hdd 7643 - 10.8 2.7ghz i7 8gb ram 500gb hdd 7641 - 10.8 2.7ghz i7 8gb ram 500gb hdd So as mentioned in the meeting we will have to match those asset tags with the DNS names as given above. Marc will try to get this done for all of them which remain in the 2.IDF room. Thanks!
I looked in the IDF on 2 and found the following Mac Mini's still in the racks. 03291 - mm-osx-105-2 (release2-osx-105.qa.mtv1.mozilla.com) [unused] SN H00190NJ9G8 mac mini 2.53|2x 2g /500/sd/ap/bt - Late 2009 03298 - mm-osx-105-3 (release3-osx-105.qa.mtv1.mozilla.com) [unused] SN YM01216P9G5 mac mini 2.26/2x 1G/160/sd/ap/bt - Late 2009 04453 - QA-SEL-STAGE - 10.6 2.66ghz core 2 duo 8gb ram 320gb hdd 04587 - QA-SE3 - 10.6 2.4ghz core 2 duo 4gb ram 320gb hdd 04588 - QE-SE4 - 10.6 2.4ghz core 2 duo 4gb ram 320gb hdd 04464 - QE-SE5 - 10.6 2.66ghz core 2 duo 8gb ram 320gb hdd 09716 - QE-SE6 - 0.8 2.3ghz i7 16gb ram 250gb ssd The following were not in IDF. mm-osx-106-1 (release1-osx-106.qa.mtv1.mozilla.com) [unused] - Not Found mm-osx-106-2 (release2-osx-106.qa.mtv1.mozilla.com) [unused] - Not Found mm-osx-106-4 (release4-osx-106.qa.mtv1.mozilla.com) [unused] - Not Found mm-osx-108-3 (release7-osx-107.qa.mtv1.mozilla.com) [unused] - Not Found QA-SE2 01285 - Not Found QA-SE6 02281 - Not Found
(In reply to Marc Schifer [mschifer] from comment #14) > The following were not in IDF. > mm-osx-106-1 (release1-osx-106.qa.mtv1.mozilla.com) [unused] - Not Found > mm-osx-106-2 (release2-osx-106.qa.mtv1.mozilla.com) [unused] - Not Found > mm-osx-106-4 (release4-osx-106.qa.mtv1.mozilla.com) [unused] - Not Found > mm-osx-108-3 (release7-osx-107.qa.mtv1.mozilla.com) [unused] - Not Found > QA-SE2 01285 - Not Found > QA-SE6 02281 - Not Found I believe (last time we went looking for these minis) that they are in the 3rd floor idf room and are inside the black rack mount chassis for the mac mini (i.e. you won't obviously see a mac mini at first).
(In reply to Marc Schifer [mschifer] from comment #14) > 03291 - mm-osx-105-2 [unused] SN H00190NJ9G8 mac mini 2.53|2x 2g /500/sd/ap/bt - Late 2009 > 03298 - mm-osx-105-3 [unused] SN YM01216P9G5 mac mini 2.26/2x 1G/160/sd/ap/bt - Late 2009 Marc, those Minis you can grab if you want. The hardware doesn't match our minimum requirements. > 04587 - QA-SE3 - 10.6 2.4ghz core 2 duo 4gb ram 320gb hdd > 04588 - QE-SE4 - 10.6 2.4ghz core 2 duo 4gb ram 320gb hdd > 09716 - QE-SE6 - 0.8 2.3ghz i7 16gb ram 250gb ssd > 04453 - QA-SEL-STAGE - 10.6 2.66ghz core 2 duo 8gb ram 320gb hdd > 04464 - QE-SE5 - 10.6 2.66ghz core 2 duo 8gb ram 320gb hdd I don't think that those Minis are ours. `SE` sounds like Selenium. So asking Dave and Stephen. > The following were not in IDF. > mm-osx-106-1 (release1-osx-106.qa.mtv1.mozilla.com) [unused] - Not Found > mm-osx-106-2 (release2-osx-106.qa.mtv1.mozilla.com) [unused] - Not Found > mm-osx-106-4 (release4-osx-106.qa.mtv1.mozilla.com) [unused] - Not Found > mm-osx-108-3 (release7-osx-107.qa.mtv1.mozilla.com) [unused] - Not Found Marc, would you mind to check Room 3.IDF as Clint suggested? It would be really good if we can find those machines. > QA-SE2 01285 - Not Found > QA-SE6 02281 - Not Found Same here. Those should be Selenium nodes.
Flags: needinfo?(stephen.donner)
Flags: needinfo?(dave.hunt)
(In reply to Henrik Skupin (:whimboo) from comment #16) > > 04587 - QA-SE3 - 10.6 2.4ghz core 2 duo 4gb ram 320gb hdd > > 04588 - QE-SE4 - 10.6 2.4ghz core 2 duo 4gb ram 320gb hdd > > 09716 - QE-SE6 - 0.8 2.3ghz i7 16gb ram 250gb ssd > > 04453 - QA-SEL-STAGE - 10.6 2.66ghz core 2 duo 8gb ram 320gb hdd > > 04464 - QE-SE5 - 10.6 2.66ghz core 2 duo 8gb ram 320gb hdd > > I don't think that those Minis are ours. `SE` sounds like Selenium. So > asking Dave and Stephen. These are definitely our Web QA Selenium-running boxes, thanks for checking. > Marc, would you mind to check Room 3.IDF as Clint suggested? It would be > really good if we can find those machines. > > > QA-SE2 01285 - Not Found > > QA-SE6 02281 - Not Found > > Same here. Those should be Selenium nodes. Yes, they are; again, thx for asking :-)
Flags: needinfo?(stephen.donner)
Thanks Stephen. So what we need here is to get the location of the following (or similar) mac minis: mm-osx-106-1 (release1-osx-106.qa.mtv1.mozilla.com) [unused] - Not Found mm-osx-106-2 (release2-osx-106.qa.mtv1.mozilla.com) [unused] - Not Found mm-osx-106-4 (release4-osx-106.qa.mtv1.mozilla.com) [unused] - Not Found mm-osx-108-3 (release7-osx-107.qa.mtv1.mozilla.com) [unused] - Not Found Marc, it would be great if you can have a look again and in room 3.IDF. Thanks.
Flags: needinfo?(dave.hunt) → needinfo?(mschifer)
I think that we can finally close this bug. Any remaining work in figuring out which machines we have might happen when we move to the new office and the room has to be emptied. So I'm closing this bug as fixed now.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Flags: needinfo?(mschifer)
QA Contact: hskupin
Resolution: --- → FIXED
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.