708013 - turn off updates while phx datacentre is having issues

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

We turned these off around 7am this morning.

Comment 1

[[e-mail] (:bhearsum)]

14:10 < bhearsum> cshields: is it safe to turn updates on again?
14:10 < cshields> no
14:10 < bhearsum> k
14:10 < cshields> indefintely
14:10 < cshields> until we can fix vamo
14:10 < bhearsum> because phx may go down again?
14:10 < bhearsum> ah
14:10 < cshields> yeah
14:10 < cshields> demand exceeds supply  :(
14:10 < cshields> working on it though


(times are in ET)

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

from irc: these updates were disabled (again) this morning because:

1) users were getting (incorrect) warnings about incompatible addons.
2) users were having long delays checking for addon compatibility immediately after updating.

Updates were disabled for all users, not just the 3.6.24->8.0.1 MUs.

Comment 3

[[e-mail] (:bhearsum)]

Also, for the record, I turned these off through
chmod 700 /opt/aus2/incoming/3

Run as ffxbld@dm-ausstage01, and also ffxbld@dp-ausstage01.

To re-enable, chmod back to 775.

Comment 4

[Alex Keybl [:akeybl]]

Let's turn this on once somebody from desktop let's us know that the VAMO servers are back up and stable.

Comment 5

[Alex Keybl [:akeybl]]

(In reply to Alex Keybl [:akeybl] from comment #4)
> Let's turn this on once somebody from desktop let's us know that the VAMO
> servers are back up and stable.

s/desktop/IT/

Comment 6

[matthew zeier [:mrz]]

From #vamo, we're holding until Wed morning.  Netops is up at 4a to monitor load without MU and will advise.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to matthew zeier [:mrz] from comment #6)
> From #vamo, we're holding until Wed morning.  Netops is up at 4a to monitor
> load without MU and will advise.

Just to be clear, I disabled all release updates, not just the 3.6.24 -> 8.0.1 MU.

Comment 8

[Corey Shields [:cshields]]

Go ahead and advertise the updates again.  We feel we are in the clear, netops is happy taking the extra load, and wil watch closely as the updates start to go out.

Comment 9

[[e-mail] (:bhearsum)]

09:09 < cshields> bhearsum: please turn updates back on now
09:09 < bhearsum> will do
09:10 < bhearsum> okay, pending nfs and other caching clearing, they're back on