Closed Bug 473084 Opened 11 years ago Closed 11 years ago

core1/core2 IOS upgrade

Categories

(mozilla.org Graveyard :: Server Operations, task, minor)

All
Other
task
Not set
minor

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mrz, Assigned: dmoore)

References

Details

DURATION:

Sunday, January 11th from 8:00 PM to 11:00 PM.


PURPOSE:

Scheduled sequential downtime of the MPT core1 and core2 switches for IOS software upgrades. This IOS upgrade is required to support our next generation of integrated load balancing hardware modules.

SCHEDULE:

Prior to upgrade: Load 12.2(18)SXF11 firmware to local flash storage on each switch. Create running-config backups on local flash storage on each switch. Archive output of the "show module", "show test all", "show port status", "show channel", and "show trunk" commands for later review.

8:00 PM 01/11/09: Perform IOS upgrade of core1 by copying firmware to the bootflash and updating necessary boot variables. Reload core1 and monitor boot progress. Validate output of show module, test, port, channel, and trunk. Confirm ACE module is properly detected and initialized. Soak and monitor network for 1 hour.

9:00 PM 01/11/09: Perform IOS upgrade of core2, as above.

10:00PM 01/11/09: Upgrade complete.
Assignee: server-ops → dmoore
Flags: needs-downtime+
Upgrade process has been postponed for one hour (to 9:00 PM), pending resolution of unrelated networking issues.
Depends on: 473113
Depends on: 473116
Upgrade for core1 was completed within window. Core2 upgrade is postponed until dependencies have been resolved.
More detail on core1 upgrade failure:

The IOS upgrade image was stored on core1's 1GB Compact Flash. Compact Flash was used because the internal supervisor bootflash was limited to 64MB, making it unable to hold two images simultaneously. The old IOS image was left untouched on the internal bootflash.

Core1 was reloaded with the intent of booting off of Compact Flash. Rommon was unable to read the Compact Flash card, halting the boot process. A manual attempt to revert back to the internal bootflash revealed the old image to be corrupt or inaccessible. At that time, no images were available to boot core1.

By formatting core2's Compact Flash card in IOS (as opposed to using the default format from the factory), it became readable by core1's rommon. The Compact Flash card was formatted in core2, images were loaded via TFTP, and then the card was manually transferred to core1. Core1 was able to boot from the newly-formatted card and returned to service.
Yep.  Log caught this when you moved cards around:

Jan 11 22:44:23 core2 3927937: Jan 11 22:44:21.978 PDT: %PCMCIAFS-SP-5-DIBERR: PCMCIA disk 0 is formatted from a different router or PC. A format in this router is required before  an image can be booted from this device
Depends on: 473122
Blocks: 475371
core2 upgrade proceeding as planned, beginning at 17:00
core2 upgrade completed.

Notes:

* A configuration mismatch prevented certain portchannels (7,8,10) from rebuilding properly. This was corrected manually.

* ns-15k is single-homed to core2, so it (and all VIPs it services) were temporarily unavailable during the reload.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.