473084 - core1/core2 IOS upgrade

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[matthew zeier [:mrz]]

DURATION:

Sunday, January 11th from 8:00 PM to 11:00 PM.


PURPOSE:

Scheduled sequential downtime of the MPT core1 and core2 switches for IOS software upgrades. This IOS upgrade is required to support our next generation of integrated load balancing hardware modules.

SCHEDULE:

Prior to upgrade: Load 12.2(18)SXF11 firmware to local flash storage on each switch. Create running-config backups on local flash storage on each switch. Archive output of the "show module", "show test all", "show port status", "show channel", and "show trunk" commands for later review.

8:00 PM 01/11/09: Perform IOS upgrade of core1 by copying firmware to the bootflash and updating necessary boot variables. Reload core1 and monitor boot progress. Validate output of show module, test, port, channel, and trunk. Confirm ACE module is properly detected and initialized. Soak and monitor network for 1 hour.

9:00 PM 01/11/09: Perform IOS upgrade of core2, as above.

10:00PM 01/11/09: Upgrade complete.

Comment 1

[Derek Moore [:dmoore]]

Upgrade process has been postponed for one hour (to 9:00 PM), pending resolution of unrelated networking issues.

Comment 2

[Derek Moore [:dmoore]]

Upgrade for core1 was completed within window. Core2 upgrade is postponed until dependencies have been resolved.

Comment 3

[Derek Moore [:dmoore]]

More detail on core1 upgrade failure:

The IOS upgrade image was stored on core1's 1GB Compact Flash. Compact Flash was used because the internal supervisor bootflash was limited to 64MB, making it unable to hold two images simultaneously. The old IOS image was left untouched on the internal bootflash.

Core1 was reloaded with the intent of booting off of Compact Flash. Rommon was unable to read the Compact Flash card, halting the boot process. A manual attempt to revert back to the internal bootflash revealed the old image to be corrupt or inaccessible. At that time, no images were available to boot core1.

By formatting core2's Compact Flash card in IOS (as opposed to using the default format from the factory), it became readable by core1's rommon. The Compact Flash card was formatted in core2, images were loaded via TFTP, and then the card was manually transferred to core1. Core1 was able to boot from the newly-formatted card and returned to service.

Comment 4

[matthew zeier [:mrz]]

Yep.  Log caught this when you moved cards around:

Jan 11 22:44:23 core2 3927937: Jan 11 22:44:21.978 PDT: %PCMCIAFS-SP-5-DIBERR: PCMCIA disk 0 is formatted from a different router or PC. A format in this router is required before  an image can be booted from this device

Comment 5

[Derek Moore [:dmoore]]

core2 upgrade proceeding as planned, beginning at 17:00

Comment 6

[Derek Moore [:dmoore]]

core2 upgrade completed.

Notes:

* A configuration mismatch prevented certain portchannels (7,8,10) from rebuilding properly. This was corrected manually.

* ns-15k is single-homed to core2, so it (and all VIPs it services) were temporarily unavailable during the reload.