[MDC2] Windows Moonshot nodes fail to PXE boot
Categories
(Infrastructure & Operations :: RelOps: General, task)
Tracking
(Not tracked)
People
(Reporter: markco, Assigned: markco)
References
Details
(Whiteboard: RITM0276335, 5334890094, REQ0259960 )
Attachments
(1 file)
|
572.18 KB,
image/png
|
Details |
| Assignee | ||
Updated•8 years ago
|
| Assignee | ||
Comment 1•8 years ago
|
||
Comment 2•8 years ago
|
||
| Assignee | ||
Comment 3•8 years ago
|
||
| Assignee | ||
Comment 5•8 years ago
|
||
Comment 9•8 years ago
|
||
Comment 10•8 years ago
|
||
Comment 11•8 years ago
|
||
Updated•8 years ago
|
Comment 12•8 years ago
|
||
Comment 13•7 years ago
|
||
Comment 14•7 years ago
|
||
Comment 15•7 years ago
|
||
Comment 16•7 years ago
|
||
Comment 18•7 years ago
|
||
Comment 19•7 years ago
|
||
Comment 20•7 years ago
|
||
| Assignee | ||
Comment 21•7 years ago
|
||
Comment 22•7 years ago
|
||
Comment 23•7 years ago
|
||
Comment 24•7 years ago
|
||
Comment 25•7 years ago
|
||
CIDuty, could you try pxe booting these again? QTS re-seated the cartridges.
Comment 26•7 years ago
|
||
Tried to PXE boot them and without success.
I've also verified the configuration for them, side by side with a known good and healthy machine and everything looks ok at the first look.
Looked after firmware configuration, boot order, system information and and many more.
I'm not sure what's causing the issue..
I'll let the NI in place in case someone has a better idea how to handle them.
| Assignee | ||
Updated•7 years ago
|
Comment 27•7 years ago
|
||
T-w1064-ms-465 and T-W1064-MS-599 do not go past PXE boot(ipv4).
T-W1064-MS-474 reached the point where the \boot\x64\Images\LiteTouchPE_x64.wim file is being loaded. However it failed to complete loading even after 30 minutes (note: this should take no longer than 30-40 seconds).
Looks like these workers can't be brought into a worker state by CiDuty yet.
If cartridge re-seat wasn't helpful, could it be a hard ware problem?
Comment 28•7 years ago
|
||
Let's test swapping these cartridges into other slots that we have cartridges correctly working in. If they still don't work from there, then we can have HPE check or replace them. If they do work from there, then we probably have a problem in the chassis/other.
I'll make a request to QTS today for a swap. I'll note which cartridges I'm having them swap with (and quarantine them in taskcluster).
Comment 29•7 years ago
|
||
Van, could you check the configuration on the moonshot chassis switches? Or is there something you know of that could cause this? When I had QTS swap the cartridges, 2 correctly pxe booted and the third did not. I'm guessing that we have something configured wrong on at least those two (and possibly the one I swapped to that did not work):
moon-chassis-11
cartridge slot 29, 30, 39 (38 is good)
moon-chassis-14
cartridge slot 29 (28 is good)
T-w1064-ms-464 Chassis 11 cartridge 29 fails pxe
T-w1064-ms-465 Chassis 11 cartridge 30 fails pxe
T-W1064-MS-473 Chassis 11 cartridge 38 good (pxe boots)
T-W1064-MS-474 Chassis 11 cartridge 39 fails pxe
T-W1064-MS-598 Chassis 14 cartridge 28 good (pxe boots)
T-W1064-MS-599 Chassis 14 cartridge 29 fails pxe
Comment 30•7 years ago
|
||
for moon11:
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display interface brief
...
XGE1/0/29 UP 10G(a) F(a) A 240
XGE1/0/30 UP 10G(a) F(a) A 240
...
XGE1/0/39 DOWN auto A A 240
cartridge 39 shows as down. the configs look fine. these are the configs on the 3 bad cartridges:
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/29
interface Ten-GigabitEthernet1/0/29
port link-mode bridge
port access vlan 240
stp edged-port
return
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuratio
n interface Ten-GigabitEthernet 1/0/30
interface Ten-GigabitEthernet1/0/30
port link-mode bridge
port access vlan 240
stp edged-port
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/39
interface Ten-GigabitEthernet1/0/39
port link-mode bridge
port access vlan 240
stp edged-port
return
since 38 is good, here is port 38's config:
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/38
interface Ten-GigabitEthernet1/0/38
port link-mode bridge
port access vlan 240
stp edged-port
return
but it doesn't look like any of these MACs are learned again...
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display mac-address interfac
e Ten-GigabitEthernet 1/0/29
MAC Address VLAN ID State Port/NickName Aging
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display mac-address interfac
e Ten-GigabitEthernet 1/0/30
MAC Address VLAN ID State Port/NickName Aging
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display mac-address interfac
e Ten-GigabitEthernet 1/0/39
MAC Address VLAN ID State Port/NickName Aging
for moon14:
[moon14-access.rit46.inband.releng.mdc2.mozilla.net]display interface brief
...
XGE1/0/29 UP 10G(a) F(a) A 240
port 29 shows as up.
[moon14-access.rit46.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/29
interface Ten-GigabitEthernet1/0/29
port link-mode bridge
port access vlan 240
stp edged-port
return
doesn't see a MAC address learned though:
[moon14-access.rit46.inband.releng.mdc2.mozilla.net]display mac-address interface Ten-GigabitEthernet 1/0/29
MAC Address VLAN ID State Port/NickName Aging
and since cartridge 28 was good, here is the config output:
[moon14-access.rit46.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/28
interface Ten-GigabitEthernet1/0/28
port link-mode bridge
port access vlan 240
stp edged-port
return
im guessing we might need to replace the motherboard?
Comment 31•7 years ago
|
||
(In reply to Van Le [:van] from comment #30)
for moon11:
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display interface brief
...
XGE1/0/29 UP 10G(a) F(a) A 240
XGE1/0/30 UP 10G(a) F(a) A 240
...
XGE1/0/39 DOWN auto A A 240cartridge 39 shows as down. the configs look fine. these are the configs on the 3 bad cartridges:
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/29
interface Ten-GigabitEthernet1/0/29
port link-mode bridge
port access vlan 240
stp edged-portreturn
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuratio
n interface Ten-GigabitEthernet 1/0/30interface Ten-GigabitEthernet1/0/30
port link-mode bridge
port access vlan 240
stp edged-port[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/39
interface Ten-GigabitEthernet1/0/39
port link-mode bridge
port access vlan 240
stp edged-portreturn
since 38 is good, here is port 38's config:
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/38
interface Ten-GigabitEthernet1/0/38
port link-mode bridge
port access vlan 240
stp edged-portreturn
but it doesn't look like any of these MACs are learned again...
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display mac-address interfac
e Ten-GigabitEthernet 1/0/29
MAC Address VLAN ID State Port/NickName Aging
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display mac-address interfac
e Ten-GigabitEthernet 1/0/30
MAC Address VLAN ID State Port/NickName Aging
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display mac-address interfac
e Ten-GigabitEthernet 1/0/39
MAC Address VLAN ID State Port/NickName Agingfor moon14:
[moon14-access.rit46.inband.releng.mdc2.mozilla.net]display interface brief
...
XGE1/0/29 UP 10G(a) F(a) A 240port 29 shows as up.
[moon14-access.rit46.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/29
interface Ten-GigabitEthernet1/0/29
port link-mode bridge
port access vlan 240
stp edged-portreturn
doesn't see a MAC address learned though:
[moon14-access.rit46.inband.releng.mdc2.mozilla.net]display mac-address interface Ten-GigabitEthernet 1/0/29
MAC Address VLAN ID State Port/NickName Agingand since cartridge 28 was good, here is the config output:
[moon14-access.rit46.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/28interface Ten-GigabitEthernet1/0/28
port link-mode bridge
port access vlan 240
stp edged-portreturn
im guessing we might need to replace the motherboard?
Thank you Van! I'll open a case with HPE and see if they can send a motherboard or a technician to test&swap it.
Comment 33•7 years ago
|
||
I tried pxe-booting 464 since 465 failed to pxe boot from that location. 464 pxe-boots in that place, so there may be a different problem (at least for 465).
After pxe boot (and image) when swapped, two of 3 cartridges are now active workers:
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc2/T-W1064-MS-474
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc2/T-W1064-MS-599
with 465 remaining stuck at pxe:
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc2/T-W1064-MS-465
I retried pxe booting it, with no success or change.
I left 465 powered off and with the UID on.
Comment 34•7 years ago
|
||
(In reply to Dave House [:dhouse] from comment #33)
I tried pxe-booting 464 since 465 failed to pxe boot from that location. 464 pxe-boots in that place, so there may be a different problem (at least for 465).
So HP replaced the chassis motherboard on moon-chassis-11 once:
11/30/2018 8:20:30 PM
REPLACED BASEBOARD - RESOLVED
I'm opening a case with them referencing the previous case replacing the baseboard and asking for more troubleshooting with the cartridges since the backplane replacement only partially fixed the network issues.
Comment 35•7 years ago
|
||
I created HPE support case #5335544875 to resume investigation of the problem with HPE. Since the backplane was replaced once for moon-chassis-11, I'm not sure what is wrong (maybe needs replaced again along with the failing cartridges).
Comment 36•7 years ago
|
||
(In reply to Dave House [:dhouse] from comment #35)
I created HPE support case #5335544875 to resume investigation of the problem with HPE. Since the backplane was replaced once for moon-chassis-11, I'm not sure what is wrong (maybe needs replaced again along with the failing cartridges).
HPE support asked for details on scheduling an onsite visit to MDC2. I've asked QTS, a general datacenter request, for contact information to provide to HPE for scheduling the visit.
Comment 37•7 years ago
|
||
QTS said:
need to submit a request for temporary physical security access (through the QTS service now portal)
and open a general request with details of date/time and who will be coming
I suggested dates and provided the information to HPE
I'm waiting to hear from HPE for when they can schedule a tech to go to MDC2. When I get that, I'll schedule it with QTS and request the temporary security access.
Comment 38•7 years ago
|
||
Hi Corey,
Could you give my QTS service-now account the permissions to request temp security access? HPE is sending someone on Friday to test and replace a moonshot's backplane, and so I need to request the access for their technician.
Thanks!
Comment 39•7 years ago
|
||
Corey set up my account with access. So I'll make the request for HP. The HPE tech also emailed me about the work that will be done (replied with overview).
Comment 40•7 years ago
|
||
I requested security access for the two technicians (from Unisys) to work on moon-chassis-11 (RITM0284949, RITM0284950).
I also requested that DCO guide the technicians to the chassis (REQ0268566).
HPE specified that they need to work on #11 and not #14. I need to clarify, with HPE support, what is being done for #14.
Comment 41•7 years ago
|
||
The installation of the new backplane was completed for chassis 11. I'm testing pxe boots on 29, 30, 39 to see if it fixed the problem.
Comment 42•7 years ago
|
||
T-w1064-ms-464 Chassis 11 cartridge 29 fails pxe
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc2/T-W1064-MS-464
PXE boots fine now.
T-w1064-ms-465 Chassis 11 cartridge 30 fails pxe
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc2/T-W1064-MS-465
https://bugzilla.mozilla.org/show_bug.cgi?id=t-w1064-ms-465
PXE boot failed the same (no IP from dhcp?)
I performed a bios full reset (settings).
Then PXE boot worked.
I turned SR-IOV back off,
and PXE boot still works.
I kicked off imaging as a w10 worker
T-W1064-MS-474 Chassis 11 cartridge 39 fails pxe
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc2/T-W1064-MS-474
PXE boots fine now
Comment 43•7 years ago
|
||
I removed quarantine on 464 and it is taking tasks. 465 and 474 are not visible (waiting for them to claimWork after imaging completes).
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc2/T-W1064-MS-465
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc2/T-W1064-MS-474
Comment 44•7 years ago
|
||
Both have come online now:
https://papertrailapp.com/groups/1141234?filter=t-w1064-ms-465
https://papertrailapp.com/groups/1141234?filter=t-w1064-ms-474
they are getting set up through OCC
Comment 45•7 years ago
|
||
(In reply to Dave House [:dhouse] from comment #44)
Both have come online now:
https://papertrailapp.com/groups/1141234?filter=t-w1064-ms-465
https://papertrailapp.com/groups/1141234?filter=t-w1064-ms-474they are getting set up through OCC
They've both taken tasks.
So, the problem is fixed on moon-chassis-11 (slots 29,30,39).
I think we still have some issues on moon-chassis-14 and I need to review that and see if HPE support has a follow-up for that chassis.
Comment 46•7 years ago
|
||
I've tried to re-image the machine T-W1064-MS-599, but, the problem seems to persist.
Dave, can you please check ?
Comment 47•7 years ago
|
||
I asked HPE Support about solutions for moon-chassis-14, and if we can try replacing the backplane on it since it fixed the similar problem on moon-chassis-11. I have not heard back yet from them.
Comment 48•7 years ago
|
||
I wrote a follow up email repeating my request for assistance with the other chassis (#14) in the support case.
Comment 49•7 years ago
|
||
I opened a new case with HPE (5337047815) specifically for moon-chassis-14 for cartridge slot 29, T-W1064-MS-599, having no network.
Comment 50•7 years ago
|
||
HPE scheduled for a technician to go onsite to replace the backplane on moon-chassis-14 today
Comment 51•7 years ago
|
||
I powered off all the cartridges and the technician confirmed they're down.
earlier, I down-timed the chassis and children in nagios and alerted CIDuty that it'll go down (and marked ignore in the tracking spreadsheet).
Comment 52•7 years ago
|
||
The unisys technician replaced the backplane. Cartridge 33 would not power on; he reseated it, and it still would not power on. The event log showed the power on action but the cartridge would drop power right away (as he reported when watching it). After more tries, it stayed on but had no network connection (no pxe boot).
We tested swapping cartridges around:
- slot 33 would not power on at first. after multiple attempts (event
log showed power on action each time, but nothing else), it powered on
but had no network- slot 29 had network. verified network on 1 also as reference
- swapping cartridge 29 with 33: slot 33 would not power on. slot 29
had network problems- swapped back (29 and 33)
- swapped cartridge 1 and 33: slots 33 and 1 both powered on, but no
network on eitherSo cartridge 33 (serial: ???CN68440EKY) had no network in any of the 3
slots. It had no problems on the previous backplane.
And slot 29 is now working (except for with cartridge 33).
But slot 33 now has intermittent power issues and no network connectivity.
I collected the show all report for this chassis and reported the above with the report attached to HPE.
Comment 53•7 years ago
|
||
Another replacement of the backplane for moon-chassis-14 is scheduled for tomorrow.
I've scheduled the temp access with QTS, RITM0292439. The same technician will do the swap, and he'll coordinate with me for shutting everything down and testing afterward.
Comment 54•7 years ago
|
||
(In reply to Dave House [:dhouse] from comment #53)
Another replacement of the backplane for moon-chassis-14 is scheduled for tomorrow.
I've scheduled the temp access with QTS, RITM0292439. The same technician will do the swap, and he'll coordinate with me for shutting everything down and testing afterward.
The technician has the replacement backplane and is on his way to the datacenter.
I notified CIDuty in #ci and downtimed moon-chassis-14 (and children) in nagios. https://nagios1.private.releng.mdc2.mozilla.com/releng-mdc2/cgi-bin/extinfo.cgi?type=1&host=moon-chassis-14.inband.releng.mdc2.mozilla.com
Comment 55•7 years ago
|
||
(In reply to Dave House [:dhouse] from comment #54)
The technician has the replacement backplane and is on his way to the datacenter.
QTS reported the technician arrived.
I've requested shutdown on all nodes on chassis14:
$ ssh Administrator@moon-chassis-14.inband.releng.mdc2.mozilla.com
hpiLO->set node power off shutdown all
c1: #Cartridge 1
c1n1: #Node 1 Shutting node down gracefully
[...]
hpiLO->show node power all
c1: #Cartridge 1
c1n1: #Node 1
Power State: Off
c2: #Cartridge 2
c2n1: #Node 1
Power State: Off
c3: #Cartridge 3
c3n1: #Node 1
Power State: On
[...]
hpiLO->set node power off force all
c1: #Cartridge 1
c1n1: #Node 1 Node already off
c2: #Cartridge 2
c2n1: #Node 1 Node already off
c3: #Cartridge 3
c3n1: #Node 1 Forcing power off
[...]
Comment 56•7 years ago
|
||
The technician found again we have problems with power and network on slot 33:
- cartridge in 33 will not power on/off with physical power button, but will power on/off when in another slot.
- Other working cartridge (32) exhibits same power problem when installed in slot 33.
- A cartridge in slot 33 will power on/off from iLO power commands.
- A powered-on cartridge in slot 33, has no network.
So, he is contacting HP and working with them on troubleshooting the problem since multiple replacements of the backplane (and cartridge replacements) has not resolved the problems. (same technician for 4 site visits for these same issues on chassis 11(resolved) and chassis 14(still unresolved))
Comment 57•7 years ago
|
||
HP will send another replacement backplane. When the next site visit is scheduled, I'll schedule access with QTS.
Comment 58•7 years ago
|
||
(In reply to Dave House [:dhouse] from comment #57)
HP will send another replacement backplane. When the next site visit is scheduled, I'll schedule access with QTS.
HP shipped the new backplane and scheduled for the technician to install it tomorrow.
I requested access for the technician with QTS, RITM0292851.
Comment 59•7 years ago
|
||
The technician acquired the part, arrived at the chassis, and started work (I powered off the cartridges and downtimed the chassis+access+children in nagios and notified ciduty in #ci).
He's planned to swap cartridge 33 with another prior to the first power-on with the new backplane. I'll need to update infoblox and inventory if that is permanent.
Comment 60•7 years ago
|
||
We found the same power issues on cartridge slot 33 (and not other slots). For the new backplane, the original cartridge 33 was never installed in this slot and instead was swapped with #35. The technician left a note on the top of the chassis that this cartridge cannot be controlled from the hardware switch (only ilo).
Network does work from cartridge slot 33 on the latest backplane. So I'll ask HPE what they can do to resolve the power switch issue since we have had the backplane replaced 3 times and the issue appeared on the last 3 but not the original (had different problems).
I'll update inventory with our swap of 33 and 35.
Comment 61•7 years ago
|
||
(In reply to Dave House [:dhouse] from comment #60)
I'll update inventory with our swap of 33 and 35.
They're not in inventory. The latest 15 carts added in mdc1 are there, but not these 15 in mdc2. I'll make sure we get them added.
And since swapping 33 and 35 didn't change the power issue. I'll ask QTS to swap these cartridges back to test if network fails again (and to keep our asset tags and tracking data in order).
Comment 62•7 years ago
|
||
T-W1064-MS-598 is running jobs fine Chassis 14 c28.
| Assignee | ||
Updated•7 years ago
|
Description
•