Closed Bug 1496230 Opened 8 years ago Closed 7 years ago

[MDC2] Windows Moonshot nodes fail to PXE boot

Categories

(Infrastructure & Operations :: RelOps: General, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: markco, Assigned: markco)

References

Details

(Whiteboard: RITM0276335, 5334890094, REQ0259960 )

Attachments

(1 file)

T-w1064-ms-465 Chassis 11 cartridge 30 T-W1064-MS-474 Chassis 11 cartridge 39 T-W1064-MS-599 Chassis 14 cartridge 29 T-W1064-MS-600 Chassis 14 cartridge 30 See blocking bugs. These are all failing to pxe boot over IPv4 and then tries over IPv6. Which means it is not getting an answer to the request.
Depends on: 1496234
Depends on: 1496237
The network configuration looks good. I am going to mount an ISO and install an OS to do some additional troubleshooting.
This servers are still unable to boot from pxe. The 600 is booted in windows with the networking details listed in CMD. Van, maybe a reseat should help? Mark, what do you think?
Flags: needinfo?(vle)
Flags: needinfo?(mcornmesser)
Some of the cartridges have been reseaetd. See 1496237 and 1496799. This seems like a possible issue with the hardware. https://bugzilla.mozilla.org/show_bug.cgi?id=1496234#c6: (In reply to Van Le [:van] from comment #6) > it looks like the switch doesn't see the MAC address of the cartridge: > > on the bad examples above: > <moon14-access.rit46.inband.releng.mdc2.mozilla.net>display mac-address > interfac > e Ten-GigabitEthernet 1/0/30 count > 0 mac address(es) found. > > on a working one: > <moon14-access.rit46.inband.releng.mdc2.mozilla.net>display mac-address > interfac > e Ten-GigabitEthernet 1/0/28 count > 1 mac address(es) found. Dhouse: Culd you contact HPE and see if they have any suggestions?
Flags: needinfo?(vle)
Flags: needinfo?(mcornmesser)
Flags: needinfo?(dhouse)
(In reply to Mark Cornmesser [:markco] from comment #0) > T-w1064-ms-465 Chassis 11 cartridge 30 > T-W1064-MS-474 Chassis 11 cartridge 39 > T-W1064-MS-599 Chassis 14 cartridge 29 > T-W1064-MS-600 Chassis 14 cartridge 30 > > > > See blocking bugs. > > These are all failing to pxe boot over IPv4 and then tries over IPv6. Which > means it is not getting an answer to the request. Mark, these machines have never booted (before the firmware update) correctly (had network), right? From the bugs, it looks like the problems with these four were found during or after the initial imaging for MDC2 last month.
Flags: needinfo?(dhouse) → needinfo?(mcornmesser)
They were found during the initial imaging for MDC2.
Flags: needinfo?(mcornmesser)
(In reply to Mark Cornmesser [:markco] from comment #3) > Some of the cartridges have been reseaetd. See 1496237 and 1496799. > > This seems like a possible issue with the hardware. > https://bugzilla.mozilla.org/show_bug.cgi?id=1496234#c6: > (In reply to Van Le [:van] from comment #6) > > it looks like the switch doesn't see the MAC address of the cartridge: > > > > on the bad examples above: > > <moon14-access.rit46.inband.releng.mdc2.mozilla.net>display mac-address > > interfac > > e Ten-GigabitEthernet 1/0/30 count > > 0 mac address(es) found. > > > > on a working one: > > <moon14-access.rit46.inband.releng.mdc2.mozilla.net>display mac-address > > interfac > > e Ten-GigabitEthernet 1/0/28 count > > 1 mac address(es) found. > > > Dhouse: Culd you contact HPE and see if they have any suggestions? I opened a case with HPE Support: Case ID: 5333366943 Case title: no network, no mac on NIC1
(In reply to Dave House [:dhouse] from comment #6) > I opened a case with HPE Support: > Case ID: 5333366943 > Case title: no network, no mac on NIC1 Van, is this something that QTS could do, or can we schedule it for the next on-site visit? HPE Support suggested: "[...] Please confirm if it is possible for you to move the impacted cartridges onto different slots and check if the issue sticks to the slot or follows the cartridge? Also, Would it be possible for you to swap the switches and check if the issue sticks to a particular switch slot or follows the impacted switch? This will help us in isolating the issue on the chassis among the components."
Flags: needinfo?(vle)
HPE Support gave a follow-up suggestion that they could send a technician to perform the testing: "From the entries for the cartridges listed by you on the previous email, we see the MAC Addresses of both the NICs for the cartridges. If its a particular switch that is unable to see the MAC Addresses, we will set up a technician visit with one switch module and one cartridge to isolate the issue. Please confirm if the server location listed below is accurate and provide the requested onsite contact details." Van, if you would like for HPE to do the testing could you coordinate with them? I think I can add you onto the support case.
this will require someone with switch access too so i think it'll be best if i test it on my next mdc2 trip in about 3 weeks or so.
(In reply to Van Le [:van] from comment #9) > this will require someone with switch access too so i think it'll be best if > i test it on my next mdc2 trip in about 3 weeks or so. That sounds great. Can you see the support case on the HPE site? I can add your information or CC you on their emails.
no, it doesn't link or show me the cases. ill provide info my next trip and you can add it to the case log.
Flags: needinfo?(vle)
Flags: needinfo?(vle)
Van, is this test plan thorough enough or do we need more information from HPE? > Thank you for your email. The test that needs to be performed is as follows. > > Swap the impacted switch onto a different slot and confirm if the issue follows the switch or sticks to the slot. > Swap one of the good known cartridges onto the impacted cartridge slot and check if the MAC of the new cartridge is visible on the switch or if the issue still persists on that particular switch. > > Let us know how these tests go. When you go to do the test, I'll plan to be available if you can use any remote help checking or monitoring the machines when testing.
I have tested the cartridges by moving them to a known good slot. It looks like the motherboard might be bad. I have replied to HPE with the following information: Here is my test procedure: 1) cartridge/slot #30 and #39 are bad, cartridge/slot #33 is good. 2) moved c#30 to slot 33 and c#33 to slot 30. c#30 now gets a MAC and c#33 doesn't. 3) restore cartridges in their original slots. 4) moved c#39 to slot 33 and c#33 to slot 39. c#39 now gets a MAC and c#33 doesn't. 5) restore cartridges in their original slots CLI output: [moon11-access.rit47.inband.releng.mdc2.mozilla.net]display mac-address interface Ten-GigabitEthernet 1/0/30 MAC Address VLAN ID State Port/NickName Aging [moon11-access.rit47.inband.releng.mdc2.mozilla.net]display mac-address interface Ten-GigabitEthernet 1/0/33 MAC Address VLAN ID State Port/NickName Aging f403-43df-3925 240 Learned XGE1/0/33 Y [moon11-access.rit47.inband.releng.mdc2.mozilla.net]display mac-address interface Ten-GigabitEthernet 1/0/39 MAC Address VLAN ID State Port/NickName Aging These are the current configs for each cartridge: [moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/30 # interface Ten-GigabitEthernet1/0/30 port link-mode bridge port access vlan 240 stp edged-port # return [moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/33 # interface Ten-GigabitEthernet1/0/33 port link-mode bridge port access vlan 240 stp edged-port # return [moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/39 # interface Ten-GigabitEthernet1/0/39 port link-mode bridge port access vlan 240 stp edged-port # return So this looks like a bad motherboard unless you guys have other ideas. Also, it looks like these slots can not pick up the cartridge information. This is what I see in iLO: System Firmware System ROM firmware Unavailable <<----- different from working cartridges CPLD 14 System ME ROM Firmware Unavailable <<----- different from working cartridges Cartridge data 11/13/2017 iLO v 2.55.1 31-May-2018 Mellanox ConnectX3Pro
we are good now. the 2 bad cartridges in moon-chassis-14 was due to rmfd's config. the 2 cartridges show a mac address learned from the switch after i removed LACP. the 2 bad cartridges in moon-chassis-11 was due to a bad motherboard. after replacing the motherboard, the issue is resolved. MAC Address VLAN ID State Port/NickName Aging f403-43df-429d 240 Learned XGE1/0/39 Y MAC Address VLAN ID State Port/NickName Aging f403-43df-3abd 240 Learned XGE1/0/30 Y
Flags: needinfo?(vle)
This two are still stuck at the PXE boot. The other two are reimaging now. T-w1064-ms-465 Chassis 11 cartridge 30 T-W1064-MS-599 Chassis 14 cartridge 29
Flags: needinfo?(vle)
this machine : T-W1064-MS-474 Chassis 11 cartridge 39 , is still stuck at the PXE boot. Can you please check it ?
opened REQ0259960 with QTS to reseat blade.
Whiteboard: REQ0259960
i think cartridge 39 needs to be replaced. -%Dec 14 22:47:14:692 2018 moon11-access.rit47.inband.releng.mdc2.mozilla.net IFNET/3/PHY_UPDOWN: Ten-GigabitEthernet1/0/39 link status is up. %Dec 14 22:47:14:692 2018 moon11-access.rit47.inband.releng.mdc2.mozilla.net IFNET/5/LINK_UPDOWN: Line protocol on the interface Ten-GigabitEthernet1/0/39 is up. %Dec 14 22:47:15:006 2018 moon11-access.rit47.inband.releng.mdc2.mozilla.net IFNET/3/PHY_UPDOWN: Ten-GigabitEthernet1/0/39 link status is down. %Dec 14 22:47:15:006 2018 moon11-access.rit47.inband.releng.mdc2.mozilla.net IFNET/5/LINK_UPDOWN: Line protocol on the interface Ten-GigabitEthernet1/0/39 is down. it won't stay up. hopefully it's not going to be an issue where we need to replace the whole motherboard. ill open a case with HPE and QTS for an RMA.
5334890094 opened with HPE for an RMA. i will need to coordinate with the field tech for access to our space in QTS.
Whiteboard: REQ0259960 → 5334890094, REQ0259960
RITM0276335 opened with QTS to replace cartridge.
Whiteboard: 5334890094, REQ0259960 → RITM0276335, 5334890094, REQ0259960
See Also: → 1495972
Van: is the rma for one cartridge or the 3 that are still down?
>Van:is the rma for one cartridge or the 3 that are still down? which 3 are still down? we replaced cartridge 39 on moon 11. i see 1/0/39 as up now with a MAC. the others started working after we reseated them or at least the MACs were seen by the switch. <moon11-access.rit47.inband.releng.mdc2.mozilla.net>display mac-address interfac e Ten-GigabitEthernet 1/0/39 MAC Address VLAN ID State Port/NickName Aging 1402-ec00-280b 240 Learned XGE1/0/39 Y XGE1/0/39 UP 10G(a) F(a) A 240 i've updated the master moonshoot spreadsheet with the new MACs.
Flags: needinfo?(vle) → needinfo?(mcornmesser)
Attached image pxe.PNG —
I have checked the bellow 3 servers, they are still unable to PXE boot. Please see the capture. T-w1064-ms-465 Chassis 11 cartridge 30 T-W1064-MS-474 Chassis 11 cartridge 39 T-W1064-MS-599 Chassis 14 cartridge 29
the MACs are learned by the switch so it's not the usual issue we see and doesn't appear we need to replace the cartridges. Can you try opening a ticket with QTS for a reseat? i've confirmed they match up with the spreadsheet and i've updated the MAC for ms-474 in the master spreadsheet and infoblox. display mac-address interface Ten-GigabitEthernet 1/0/39 MAC Address VLAN ID State Port/NickName Aging 1402-ec00-280b 240 Learned XGE1/0/39 Y Y display mac-address interface Ten-GigabitEthernet 1/0/30 MAC Address VLAN ID State Port/NickName Aging f403-43df-3abd 240 Learned XGE1/0/30 Y display mac-address interface Ten-GigabitEthernet 1/0/29 MAC Address VLAN ID State Port/NickName Aging f403-43df-4929 240 Learned XGE1/0/29
Blocks: 1518729

CIDuty, could you try pxe booting these again? QTS re-seated the cartridges.

Flags: needinfo?(ciduty)

Tried to PXE boot them and without success.
I've also verified the configuration for them, side by side with a known good and healthy machine and everything looks ok at the first look.

Looked after firmware configuration, boot order, system information and and many more.

I'm not sure what's causing the issue..

I'll let the NI in place in case someone has a better idea how to handle them.

Flags: needinfo?(mcornmesser)

T-w1064-ms-465 and T-W1064-MS-599 do not go past PXE boot(ipv4).
T-W1064-MS-474 reached the point where the \boot\x64\Images\LiteTouchPE_x64.wim file is being loaded. However it failed to complete loading even after 30 minutes (note: this should take no longer than 30-40 seconds).

Looks like these workers can't be brought into a worker state by CiDuty yet.

If cartridge re-seat wasn't helpful, could it be a hard ware problem?

Flags: needinfo?(ciduty) → needinfo?(dhouse)

Let's test swapping these cartridges into other slots that we have cartridges correctly working in. If they still don't work from there, then we can have HPE check or replace them. If they do work from there, then we probably have a problem in the chassis/other.

I'll make a request to QTS today for a swap. I'll note which cartridges I'm having them swap with (and quarantine them in taskcluster).

Van, could you check the configuration on the moonshot chassis switches? Or is there something you know of that could cause this? When I had QTS swap the cartridges, 2 correctly pxe booted and the third did not. I'm guessing that we have something configured wrong on at least those two (and possibly the one I swapped to that did not work):

moon-chassis-11
cartridge slot 29, 30, 39 (38 is good)

moon-chassis-14
cartridge slot 29 (28 is good)

T-w1064-ms-464 Chassis 11 cartridge 29 fails pxe
T-w1064-ms-465 Chassis 11 cartridge 30 fails pxe
T-W1064-MS-473 Chassis 11 cartridge 38 good (pxe boots)
T-W1064-MS-474 Chassis 11 cartridge 39 fails pxe

T-W1064-MS-598 Chassis 14 cartridge 28 good (pxe boots)
T-W1064-MS-599 Chassis 14 cartridge 29 fails pxe

Flags: needinfo?(dhouse) → needinfo?(vle)

for moon11:

[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display interface brief
...
XGE1/0/29 UP 10G(a) F(a) A 240
XGE1/0/30 UP 10G(a) F(a) A 240
...
XGE1/0/39 DOWN auto A A 240

cartridge 39 shows as down. the configs look fine. these are the configs on the 3 bad cartridges:

[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/29

interface Ten-GigabitEthernet1/0/29
port link-mode bridge
port access vlan 240
stp edged-port

return

[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuratio
n interface Ten-GigabitEthernet 1/0/30

interface Ten-GigabitEthernet1/0/30
port link-mode bridge
port access vlan 240
stp edged-port

[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/39

interface Ten-GigabitEthernet1/0/39
port link-mode bridge
port access vlan 240
stp edged-port

return

since 38 is good, here is port 38's config:

[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/38

interface Ten-GigabitEthernet1/0/38
port link-mode bridge
port access vlan 240
stp edged-port

return

but it doesn't look like any of these MACs are learned again...

[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display mac-address interfac
e Ten-GigabitEthernet 1/0/29
MAC Address VLAN ID State Port/NickName Aging
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display mac-address interfac
e Ten-GigabitEthernet 1/0/30
MAC Address VLAN ID State Port/NickName Aging
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display mac-address interfac
e Ten-GigabitEthernet 1/0/39
MAC Address VLAN ID State Port/NickName Aging

for moon14:
[moon14-access.rit46.inband.releng.mdc2.mozilla.net]display interface brief
...
XGE1/0/29 UP 10G(a) F(a) A 240

port 29 shows as up.

[moon14-access.rit46.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/29

interface Ten-GigabitEthernet1/0/29
port link-mode bridge
port access vlan 240
stp edged-port

return

doesn't see a MAC address learned though:
[moon14-access.rit46.inband.releng.mdc2.mozilla.net]display mac-address interface Ten-GigabitEthernet 1/0/29
MAC Address VLAN ID State Port/NickName Aging

and since cartridge 28 was good, here is the config output:
[moon14-access.rit46.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/28

interface Ten-GigabitEthernet1/0/28
port link-mode bridge
port access vlan 240
stp edged-port

return

im guessing we might need to replace the motherboard?

Flags: needinfo?(vle)

(In reply to Van Le [:van] from comment #30)

for moon11:

[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display interface brief
...
XGE1/0/29 UP 10G(a) F(a) A 240
XGE1/0/30 UP 10G(a) F(a) A 240
...
XGE1/0/39 DOWN auto A A 240

cartridge 39 shows as down. the configs look fine. these are the configs on the 3 bad cartridges:

[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/29

interface Ten-GigabitEthernet1/0/29
port link-mode bridge
port access vlan 240
stp edged-port

return

[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuratio
n interface Ten-GigabitEthernet 1/0/30

interface Ten-GigabitEthernet1/0/30
port link-mode bridge
port access vlan 240
stp edged-port

[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/39

interface Ten-GigabitEthernet1/0/39
port link-mode bridge
port access vlan 240
stp edged-port

return

since 38 is good, here is port 38's config:

[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/38

interface Ten-GigabitEthernet1/0/38
port link-mode bridge
port access vlan 240
stp edged-port

return

but it doesn't look like any of these MACs are learned again...

[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display mac-address interfac
e Ten-GigabitEthernet 1/0/29
MAC Address VLAN ID State Port/NickName Aging
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display mac-address interfac
e Ten-GigabitEthernet 1/0/30
MAC Address VLAN ID State Port/NickName Aging
[moon11-access.rit47.inband.releng.mdc2.mozilla.net]display mac-address interfac
e Ten-GigabitEthernet 1/0/39
MAC Address VLAN ID State Port/NickName Aging

for moon14:
[moon14-access.rit46.inband.releng.mdc2.mozilla.net]display interface brief
...
XGE1/0/29 UP 10G(a) F(a) A 240

port 29 shows as up.

[moon14-access.rit46.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/29

interface Ten-GigabitEthernet1/0/29
port link-mode bridge
port access vlan 240
stp edged-port

return

doesn't see a MAC address learned though:
[moon14-access.rit46.inband.releng.mdc2.mozilla.net]display mac-address interface Ten-GigabitEthernet 1/0/29
MAC Address VLAN ID State Port/NickName Aging

and since cartridge 28 was good, here is the config output:
[moon14-access.rit46.inband.releng.mdc2.mozilla.net]display current-configuration interface Ten-GigabitEthernet 1/0/28

interface Ten-GigabitEthernet1/0/28
port link-mode bridge
port access vlan 240
stp edged-port

return

im guessing we might need to replace the motherboard?

Thank you Van! I'll open a case with HPE and see if they can send a motherboard or a technician to test&swap it.

I tried pxe-booting 464 since 465 failed to pxe boot from that location. 464 pxe-boots in that place, so there may be a different problem (at least for 465).

After pxe boot (and image) when swapped, two of 3 cartridges are now active workers:
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc2/T-W1064-MS-474
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc2/T-W1064-MS-599

with 465 remaining stuck at pxe:
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc2/T-W1064-MS-465
I retried pxe booting it, with no success or change.
I left 465 powered off and with the UID on.

(In reply to Dave House [:dhouse] from comment #33)

I tried pxe-booting 464 since 465 failed to pxe boot from that location. 464 pxe-boots in that place, so there may be a different problem (at least for 465).

So HP replaced the chassis motherboard on moon-chassis-11 once:
11/30/2018 8:20:30 PM
REPLACED BASEBOARD - RESOLVED

I'm opening a case with them referencing the previous case replacing the baseboard and asking for more troubleshooting with the cartridges since the backplane replacement only partially fixed the network issues.

I created HPE support case #5335544875 to resume investigation of the problem with HPE. Since the backplane was replaced once for moon-chassis-11, I'm not sure what is wrong (maybe needs replaced again along with the failing cartridges).

Summary: MDC 2 Windows Moonshot nodes fail to PXE boot → [MDC2] Windows Moonshot nodes fail to PXE boot

(In reply to Dave House [:dhouse] from comment #35)

I created HPE support case #5335544875 to resume investigation of the problem with HPE. Since the backplane was replaced once for moon-chassis-11, I'm not sure what is wrong (maybe needs replaced again along with the failing cartridges).

HPE support asked for details on scheduling an onsite visit to MDC2. I've asked QTS, a general datacenter request, for contact information to provide to HPE for scheduling the visit.

QTS said:
need to submit a request for temporary physical security access (through the QTS service now portal)
and open a general request with details of date/time and who will be coming

I suggested dates and provided the information to HPE

I'm waiting to hear from HPE for when they can schedule a tech to go to MDC2. When I get that, I'll schedule it with QTS and request the temporary security access.

Hi Corey,

Could you give my QTS service-now account the permissions to request temp security access? HPE is sending someone on Friday to test and replace a moonshot's backplane, and so I need to request the access for their technician.

Thanks!

Flags: needinfo?(cshields)

Corey set up my account with access. So I'll make the request for HP. The HPE tech also emailed me about the work that will be done (replied with overview).

Flags: needinfo?(cshields)

I requested security access for the two technicians (from Unisys) to work on moon-chassis-11 (RITM0284949, RITM0284950).
I also requested that DCO guide the technicians to the chassis (REQ0268566).

HPE specified that they need to work on #11 and not #14. I need to clarify, with HPE support, what is being done for #14.

Depends on: 1526323

The installation of the new backplane was completed for chassis 11. I'm testing pxe boots on 29, 30, 39 to see if it fixed the problem.

T-w1064-ms-464 Chassis 11 cartridge 29 fails pxe
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc2/T-W1064-MS-464
PXE boots fine now.

T-w1064-ms-465 Chassis 11 cartridge 30 fails pxe
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc2/T-W1064-MS-465
https://bugzilla.mozilla.org/show_bug.cgi?id=t-w1064-ms-465
PXE boot failed the same (no IP from dhcp?)
I performed a bios full reset (settings).
Then PXE boot worked.
I turned SR-IOV back off,
and PXE boot still works.
I kicked off imaging as a w10 worker

T-W1064-MS-474 Chassis 11 cartridge 39 fails pxe
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc2/T-W1064-MS-474
PXE boots fine now

(In reply to Dave House [:dhouse] from comment #44)

Both have come online now:
https://papertrailapp.com/groups/1141234?filter=t-w1064-ms-465
https://papertrailapp.com/groups/1141234?filter=t-w1064-ms-474

they are getting set up through OCC

They've both taken tasks.

So, the problem is fixed on moon-chassis-11 (slots 29,30,39).
I think we still have some issues on moon-chassis-14 and I need to review that and see if HPE support has a follow-up for that chassis.

I've tried to re-image the machine T-W1064-MS-599, but, the problem seems to persist.

Dave, can you please check ?

Flags: needinfo?(dhouse)

I asked HPE Support about solutions for moon-chassis-14, and if we can try replacing the backplane on it since it fixed the similar problem on moon-chassis-11. I have not heard back yet from them.

Flags: needinfo?(dhouse)

I wrote a follow up email repeating my request for assistance with the other chassis (#14) in the support case.

I opened a new case with HPE (5337047815) specifically for moon-chassis-14 for cartridge slot 29, T-W1064-MS-599, having no network.

HPE scheduled for a technician to go onsite to replace the backplane on moon-chassis-14 today

I powered off all the cartridges and the technician confirmed they're down.

earlier, I down-timed the chassis and children in nagios and alerted CIDuty that it'll go down (and marked ignore in the tracking spreadsheet).

The unisys technician replaced the backplane. Cartridge 33 would not power on; he reseated it, and it still would not power on. The event log showed the power on action but the cartridge would drop power right away (as he reported when watching it). After more tries, it stayed on but had no network connection (no pxe boot).
We tested swapping cartridges around:

  1. slot 33 would not power on at first. after multiple attempts (event
    log showed power on action each time, but nothing else), it powered on
    but had no network
  2. slot 29 had network. verified network on 1 also as reference
  3. swapping cartridge 29 with 33: slot 33 would not power on. slot 29
    had network problems
  4. swapped back (29 and 33)
  5. swapped cartridge 1 and 33: slots 33 and 1 both powered on, but no
    network on either

So cartridge 33 (serial: ???CN68440EKY) had no network in any of the 3
slots. It had no problems on the previous backplane.
And slot 29 is now working (except for with cartridge 33).
But slot 33 now has intermittent power issues and no network connectivity.

I collected the show all report for this chassis and reported the above with the report attached to HPE.

Another replacement of the backplane for moon-chassis-14 is scheduled for tomorrow.

I've scheduled the temp access with QTS, RITM0292439. The same technician will do the swap, and he'll coordinate with me for shutting everything down and testing afterward.

(In reply to Dave House [:dhouse] from comment #53)

Another replacement of the backplane for moon-chassis-14 is scheduled for tomorrow.

I've scheduled the temp access with QTS, RITM0292439. The same technician will do the swap, and he'll coordinate with me for shutting everything down and testing afterward.

The technician has the replacement backplane and is on his way to the datacenter.

I notified CIDuty in #ci and downtimed moon-chassis-14 (and children) in nagios. https://nagios1.private.releng.mdc2.mozilla.com/releng-mdc2/cgi-bin/extinfo.cgi?type=1&host=moon-chassis-14.inband.releng.mdc2.mozilla.com

(In reply to Dave House [:dhouse] from comment #54)

The technician has the replacement backplane and is on his way to the datacenter.

QTS reported the technician arrived.

I've requested shutdown on all nodes on chassis14:

$ ssh Administrator@moon-chassis-14.inband.releng.mdc2.mozilla.com

hpiLO->set node power off shutdown all
  c1: #Cartridge 1
    c1n1: #Node 1 Shutting node down gracefully

[...]
hpiLO->show node power all


  c1: #Cartridge 1
    c1n1: #Node 1
      Power State: Off

  c2: #Cartridge 2
    c2n1: #Node 1
      Power State: Off

  c3: #Cartridge 3
    c3n1: #Node 1
      Power State: On

[...]
hpiLO->set node power off force all


  c1: #Cartridge 1
    c1n1: #Node 1 Node already off

  c2: #Cartridge 2
    c2n1: #Node 1 Node already off

  c3: #Cartridge 3
    c3n1: #Node 1 Forcing power off
[...]

The technician found again we have problems with power and network on slot 33:

  1. cartridge in 33 will not power on/off with physical power button, but will power on/off when in another slot.
  2. Other working cartridge (32) exhibits same power problem when installed in slot 33.
  3. A cartridge in slot 33 will power on/off from iLO power commands.
  4. A powered-on cartridge in slot 33, has no network.

So, he is contacting HP and working with them on troubleshooting the problem since multiple replacements of the backplane (and cartridge replacements) has not resolved the problems. (same technician for 4 site visits for these same issues on chassis 11(resolved) and chassis 14(still unresolved))

HP will send another replacement backplane. When the next site visit is scheduled, I'll schedule access with QTS.

(In reply to Dave House [:dhouse] from comment #57)

HP will send another replacement backplane. When the next site visit is scheduled, I'll schedule access with QTS.

HP shipped the new backplane and scheduled for the technician to install it tomorrow.
I requested access for the technician with QTS, RITM0292851.

The technician acquired the part, arrived at the chassis, and started work (I powered off the cartridges and downtimed the chassis+access+children in nagios and notified ciduty in #ci).

He's planned to swap cartridge 33 with another prior to the first power-on with the new backplane. I'll need to update infoblox and inventory if that is permanent.

We found the same power issues on cartridge slot 33 (and not other slots). For the new backplane, the original cartridge 33 was never installed in this slot and instead was swapped with #35. The technician left a note on the top of the chassis that this cartridge cannot be controlled from the hardware switch (only ilo).

Network does work from cartridge slot 33 on the latest backplane. So I'll ask HPE what they can do to resolve the power switch issue since we have had the backplane replaced 3 times and the issue appeared on the last 3 but not the original (had different problems).

I'll update inventory with our swap of 33 and 35.

(In reply to Dave House [:dhouse] from comment #60)

I'll update inventory with our swap of 33 and 35.

They're not in inventory. The latest 15 carts added in mdc1 are there, but not these 15 in mdc2. I'll make sure we get them added.

And since swapping 33 and 35 didn't change the power issue. I'll ask QTS to swap these cartridges back to test if network fails again (and to keep our asset tags and tracking data in order).

T-W1064-MS-598 is running jobs fine Chassis 14 c28.

Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: