switch2.r102-1.ops.scl1 rebooting

RESOLVED FIXED

Status

RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: adam, Assigned: adam)

Tracking

Details

(Assignee)

Description

5 years ago
We have observed this switch rebooting itself twice so far today. The first time was at approximately 12:35 PDT, the second at approximately 18:40 PDT.

The logs indicate that interfaces were flapping constantly before the switch finally fell over:


[...]
Jul  2 01:34:08  switch2.r102-1.ops.scl1 chassism[795]: ifd_process_flaps IFD: ge-0/0/16, sent flap msg to RE, Downstate
Jul  2 01:34:08  switch2.r102-1.ops.scl1 rpd[924]: EVENT <UpDown> index 146 <Broadcast Multicast> address #0 78.19.f7.9a.b1.90
Jul  2 01:34:11  switch2.r102-1.ops.scl1 rpd[924]: EVENT <UpDown> ge-0/0/16.0 index 84 <Up Broadcast Multicast> address #0 78.19.f7.9a.b1.90
Jul  2 01:34:11  switch2.r102-1.ops.scl1 rpd[924]: EVENT <UpDown> index 146 <Up Broadcast Multicast> address #0 78.19.f7.9a.b1.90
Jul  2 01:34:11  switch2.r102-1.ops.scl1 mib2d[936]: SNMP_TRAP_LINK_UP: ifIndex 515, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/0/16
Jul  2 01:34:11  switch2.r102-1.ops.scl1 mib2d[936]: SNMP_TRAP_LINK_UP: ifIndex 516, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/0/16.0
Jul  2 01:34:19  switch2.r102-1.ops.scl1 rpd[924]: EVENT <UpDown> ge-0/0/16.0 index 84 <Broadcast Multicast> address #0 78.19.f7.9a.b1.90
Jul  2 01:34:19  switch2.r102-1.ops.scl1 chassism[795]: ifd_process_flaps IFD: ge-0/0/16, sent flap msg to RE, Downstate
Jul  2 01:34:19  switch2.r102-1.ops.scl1 mib2d[936]: SNMP_TRAP_LINK_DOWN: ifIndex 515, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/16
Jul  2 01:34:19  switch2.r102-1.ops.scl1 rpd[924]: EVENT <UpDown> index 146 <Broadcast Multicast> address #0 78.19.f7.9a.b1.90
Jul  2 01:34:21  switch2.r102-1.ops.scl1 rpd[924]: EVENT <UpDown> ge-0/0/16.0 index 84 <Up Broadcast Multicast> address #0 78.19.f7.9a.b1.90
Jul  2 01:34:21  switch2.r102-1.ops.scl1 rpd[924]: EVENT <UpDown> index 146 <Up Broadcast Multicast> address #0 78.19.f7.9a.b1.90
Jul  2 01:34:21  switch2.r102-1.ops.scl1 mib2d[936]: SNMP_TRAP_LINK_UP: ifIndex 515, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/0/16
Jul  2 01:34:21  switch2.r102-1.ops.scl1 mib2d[936]: SNMP_TRAP_LINK_UP: ifIndex 516, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/0/16.0
Jul  2 01:34:31  switch2.r102-1.ops.scl1 rpd[924]: EVENT <UpDown> ge-0/0/16.0 index 84 <Broadcast Multicast> address #0 78.19.f7.9a.b1.90
Jul  2 01:34:31  switch2.r102-1.ops.scl1 mib2d[936]: SNMP_TRAP_LINK_DOWN: ifIndex 515, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/16
Jul  2 01:34:31  switch2.r102-1.ops.scl1 rpd[924]: EVENT <UpDown> index 146 <Broadcast Multicast> address #0 78.19.f7.9a.b1.90
Jul  2 01:34:31  switch2.r102-1.ops.scl1 chassism[795]: ifd_process_flaps IFD: ge-0/0/16, sent flap msg to RE, Downstate
Jul  2 01:34:33  switch2.r102-1.ops.scl1 rpd[924]: EVENT <UpDown> ge-0/0/16.0 index 84 <Up Broadcast Multicast> address #0 78.19.f7.9a.b1.90
Jul  2 01:34:33  switch2.r102-1.ops.scl1 mib2d[936]: SNMP_TRAP_LINK_UP: ifIndex 515, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/0/16
Jul  2 01:34:33  switch2.r102-1.ops.scl1 rpd[924]: EVENT <UpDown> index 146 <Up Broadcast Multicast> address #0 78.19.f7.9a.b1.90
Jul  2 01:34:33  switch2.r102-1.ops.scl1 mib2d[936]: SNMP_TRAP_LINK_UP: ifIndex 516, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/0/16.0
Jul  2 01:35:00  switch2.r102-1.ops.scl1 /usr/sbin/cron[2034]: (root) CMD (newsyslog)
Jul  2 01:38:54  switch2.r102-1.ops.scl1 /kernel: GDB: debug ports: uart
Jul  2 01:38:54  switch2.r102-1.ops.scl1 eventd: sendto: Can't assign requested address
Jul  2 01:38:54  switch2.r102-1.ops.scl1 /kernel: GDB: current port: uart
Jul  2 01:38:54  switch2.r102-1.ops.scl1 /kernel: KDB: debugger backends: ddb gdb
Jul  2 01:38:54  switch2.r102-1.ops.scl1 /kernel: KDB: current backend: ddb
Jul  2 01:38:54  switch2.r102-1.ops.scl1 /kernel: Copyright (c) 1996-2011, Juniper Networks, Inc.
Jul  2 01:38:54  switch2.r102-1.ops.scl1 /kernel: All rights reserved.
Jul  2 01:38:54  switch2.r102-1.ops.scl1 /kernel: Copyright (c) 1992-2006 The FreeBSD Project.
[...]

I am uploading our current code to the switch now to perform an upgrade.
(Assignee)

Comment 1

5 years ago
The switch is currently running 11.1R3.5, but has been upgraded to 11.4R5.5. Upon reboot, intentional or otherwise, it should load the new image. This will cause a delay in the switch returning to service, but will hopefully remedy the issue.
Switch rebooted a few minutes ago (down 07:39, up 07:48) and is now on JUNOS 11.4R5.5
Upgrade didn't help,
13:39 < nagios-releng> Tue 06:39:29 PDT [425] switch2.r102-1.ops.scl1.mozilla.net is DOWN :PING CRITICAL - Packet loss = 100%
13:42 < nagios-releng> Tue 06:42:59 PDT [427] switch2.r102-1.ops.scl1.mozilla.net is UP :PING OK - Packet loss = 0%, RTA = 1139.33 ms
Depends on: 889407
(Assignee)

Comment 4

5 years ago
The switch has been replaced. Resolving unless other issues with the new switch pop up.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.