971445 - reboot github-sync3 during next TCW

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Hal Wine [:hwine] use NI!]

github-sync3 is our primary production vcs-sync machine at this time. The MOTD currently states:

Please run /usr/local/sbin/install_hp_firmware.sh before rebooting this 
machine to install:
   bios_update.sh storage_p410_update.sh

Rebooting will take coordinated shutdown of the services running on the machine, and manual startup after. TCW is a good time.

This primarily will require releng folks.

Comment 1

[Shyam Mani [:fox2mike]]

Approved by the CAB on Feb 12th to be done during the next TCW on Feb 22nd. This needs a MOC person on hand, as the box has pending hardware updates.

Comment 2

[Hal Wine [:hwine] use NI!]

$ ssh gd3
Last login: Fri Feb 21 22:01:29 2014 from 10.22.248.70
Please run /usr/local/sbin/install_hp_firmware.sh before rebooting this
machine to install:
   bios_update.sh storage_p410_update.sh
[[e-mail]3 ~]$ sudo su -
Please run /usr/local/sbin/install_hp_firmware.sh before rebooting this
machine to install:
   bios_update.sh storage_p410_update.sh
[[e-mail]3 ~]# set -o vi
[[e-mail]3 ~]# logout
[[e-mail]3 ~]$ vcs2vcs_screen -x
[screen is terminating]
[[e-mail]3 ~]$ sudo su -
Please run /usr/local/sbin/install_hp_firmware.sh before rebooting this
machine to install:
   bios_update.sh storage_p410_update.sh
[[e-mail]3 ~]# /usr/local/sbin/install_hp_firmware.sh
Running "/usr/local/hp//manual_updates/storage_p410_update.sh"

        This program consists of two phases: device discovery and device update.
        No device will be updated until you answer.

Do you want to run device discovery?
(yes/no) yes
Finding hardware. This may take a few minutes.
Found 1 devices.

Do you want to upgrade the device that has older ROM?
(yes/no) yes
1 devices will be updated.
Updating: P410i Slot: 0 from [5.70] to [6.40]
Updating: P410i Slot: 0 from [5.70] to [6.40]

As part of the reboot process, you must power cycle the server and any external array storage devices.

Running "/usr/local/hp//manual_updates/bios_update.sh"
Flash Engine Version: Linux-4.4.1-1

Name: Online ROM Flash Component for Linux - HP ProLiant BL460c G7 (I27) Servers
New Version: 07/02/2013

Current Version: 05/05/2011

The software is installed but is not up to date.

Do you want to upgrade the software to a newer version (y/n) ?^C
[[e-mail]3 ~]# /usr/local/sbin/install_hp_firmware.sh
Running "/usr/local/hp//manual_updates/storage_p410_update.sh"

        This program consists of two phases: device discovery and device update.
        No device will be updated until you answer.

Do you want to run device discovery?
(yes/no) yes
Finding hardware. This may take a few minutes.
Found 1 devices.

Do you want to upgrade the device that has older ROM?
(yes/no) yes
1 devices will be updated.
Updating: P410i Slot: 0 from [5.70] to [6.40]
Updating: P410i Slot: 0 from [5.70] to [6.40]

As part of the reboot process, you must power cycle the server and any external array storage devices.

Running "/usr/local/hp//manual_updates/bios_update.sh"
Flash Engine Version: Linux-4.4.1-1

Name: Online ROM Flash Component for Linux - HP ProLiant BL460c G7 (I27) Servers
New Version: 07/02/2013

Current Version: 05/05/2011

The software is installed but is not up to date.

Do you want to upgrade the software to a newer version (y/n) ?y

Flash in progress do not interrupt or your system may become unusable.
Working.........................................................................
The installation procedure completed successfully.

A reboot is required to finish the installation completely.
Do you want to reboot your system now? yes

Broadcast message from [e-mail]
        (/dev/pts/1) at 20:35 ...

The system is going down for reboot NOW!

REMINDER. The login warning will remain until puppet has run after a reboot.

[[e-mail]3 ~]# Connection to github-sync3.dmz.scl3.mozilla.com closed by remote host.
Connection to github-sync3.dmz.scl3.mozilla.com closed.
255 [Hal@Hals-MacBook-Air ~]
$ ping -c 5 github-sync3.dmz.scl3.mozilla.com
5 packets transmitted, 0 packets received, 100.0% packet loss
PING github-sync3.dmz.scl3.mozilla.com (10.22.74.134): 56 data bytes
64 bytes from 10.22.74.134: icmp_seq=0 ttl=62 time=23.374 ms
64 bytes from 10.22.74.134: icmp_seq=1 ttl=62 time=24.189 ms
64 bytes from 10.22.74.134: icmp_seq=2 ttl=62 time=22.597 ms
64 bytes from 10.22.74.134: icmp_seq=3 ttl=62 time=21.786 ms
64 bytes from 10.22.74.134: icmp_seq=4 ttl=62 time=34.256 ms

--- github-sync3.dmz.scl3.mozilla.com ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 21.786/25.240/34.256/4.578 ms
^C
1 [Hal@Hals-MacBook-Air ~]
$ while true; do nc -vz github-sync3.dmz.scl3.mozilla.com 22 ; sleep 30; done
Connection to github-sync3.dmz.scl3.mozilla.com 22 port [tcp/ssh] succeeded!
^C
1 [Hal@Hals-MacBook-Air ~]
$ ssh gd3
Last login: Sat Feb 22 20:08:29 2014 from 10.22.248.70
Please run /usr/local/sbin/install_hp_firmware.sh before rebooting this
machine to install:
   bios_update.sh storage_p410_update.sh
[[e-mail]3 ~]$ sudo su -
Please run /usr/local/sbin/install_hp_firmware.sh before rebooting this
machine to install:
   bios_update.sh storage_p410_update.sh
[[e-mail]3 ~]# /usr/local/sbin/install_hp_firmware.sh
Running "/usr/local/hp//manual_updates/storage_p410_update.sh"

        This program consists of two phases: device discovery and device update.
        No device will be updated until you answer.

Do you want to run device discovery?
(yes/no) yes
Finding hardware. This may take a few minutes.
Found 1 devices.

No devices updated.
Running "/usr/local/hp//manual_updates/bios_update.sh"
Flash Engine Version: Linux-4.4.1-1

Name: Online ROM Flash Component for Linux - HP ProLiant BL460c G7 (I27) Servers
New Version: 07/02/2013

Current Version: 07/02/2013

The software is installed and up to date.
The installation does not need to continue,
however, you may still install the software.

Do you want to install this version (y/n) ?n

REMINDER. The login warning will remain until puppet has run after a reboot.

[[e-mail]3 ~]# logout