1237135 - Rectify time zone discrepancies in peach cluster's hosts

Status: RESOLVED FIXED

(Infrastructure & Operations :: Change Requests, task)

Description

[Matt Pressman [:mpressman]]

nodes on peach are set with different TZ's. This will require setting and may require rebooting the nodes with disparate TZ's. Pythian will be handling the administration of this.

This can be done at any time
The system affected is the peach cluster
No end-user impact as the system will still remain up and available
PLAN?
ROLLBACK PLAN?
NOTIFICATION MECHANISMS?
Pythian will be point and handle the upgrade

Comment 1

[Peter Radcliffe [:pir]]

Will this be done through puppet to make sure that even if hosts are replaced/reinstalled they remain consistent?

Comment 2

[Matt Pressman [:mpressman]]

I don't see why not. I'll pass this along to pythian to make sure it is handled through puppet

Comment 3

[Linda Ypulong [:unixfairy]]

Reviewed and deferred 1/6/2016 CAB - for plan details - please needinfo me when those are updated

Comment 5

[Matt Pressman [:mpressman]]

Information Needed for CAB review:

* date, time, duration of maintenance:
  TBD, let us know what date suits you, we would need 15 minutes, no downtime required.

* system(s) affected:
  cloudera-cm4.metrics.scl3.mozilla.com (Cloudera Manager Console)
  
* end-user impact
  No impact to end users, Cloudera Manager Console will be unavailable during the maitenance window

* maintenance plan and timeline (link to a wiki or etherpad is fine)
 1 - Copy correct TZ file to system configuration
 # cp /usr/share/zoneinfo/UTC /etc/localtime
 2 - Set ZONE variable to correct TZ
 # echo 'ZONE="UTC"' > /etc/sysconfig/clock
 3 - Verify TZ has changed
 # date
 4 - Verify CM is functional and has no errors

* rollback plan / rollback point (at which point will you determine to roll back)
 
 1 - Set TZ back to before changes
 # cp /usr/share/zoneinfo/America/Los_Angeles /etc/localtime
 # echo 'ZONE="America/Los_Angeles"' > /etc/sysconfig/clock
 
* notification mechanisms:
  Is email at the beginning and end of the window ok?
  
* who will be point, who else will be involved
  Pythian SRE only, the assignee will depend on the timeframe when the change is executed

Comment 6

[Peter Radcliffe [:pir]]

(In reply to Matt Pressman [:mpressman] from comment #5)
> * maintenance plan and timeline (link to a wiki or etherpad is fine)
>  1 - Copy correct TZ file to system configuration
>  # cp /usr/share/zoneinfo/UTC /etc/localtime

You may want to read digi's writeup on how not to do this, here:
  https://bugzilla.mozilla.org/show_bug.cgi?id=1234907#c1

I also don't see reboots in the plan, when changing a timezone it is strongly recommended.

Comment 7

[u550631]

There is only monitoring services running, so I don't think it is needed. But if you prefer server can be rebooted just in case quickly during the window with no impact to the users (this server just handles the CM console)

Comment 8

[Peter Radcliffe [:pir]]

There's still the kernel, syslogd, anything else running... never just one thing one a unix box. It can get very confusing when some processes are set to one timezone and others another, a reboot is by far safest.

Comment 9

[u550631]

Ok, let me know if/when you want this to be performed.

Comment 10

[Matt Pressman [:mpressman]]

* date, time, duration of maintenance:
  TBD, let us know what date suits you, we would need 15 minutes, no downtime required.

* system(s) affected:
  cloudera-cm4.metrics.scl3.mozilla.com (Cloudera Manager Console)
  
* end-user impact
  No impact to end users, Cloudera Manager Console will be unavailable during the maitenance window

* maintenance plan and timeline (link to a wiki or etherpad is fine)
 1 - Copy correct TZ file to system configuration
 # cp /usr/share/zoneinfo/UTC /etc/localtime
 2 - Set ZONE variable to correct TZ
 # echo 'ZONE="UTC"' > /etc/sysconfig/clock
 3 - Verify TZ has changed
 # date
 4 - Verify CM is functional and has no errors

* rollback plan / rollback point (at which point will you determine to roll back)
 
 1 - Set TZ back to before changes
 # cp /usr/share/zoneinfo/America/Los_Angeles /etc/localtime
 # echo 'ZONE="America/Los_Angeles"' > /etc/sysconfig/clock
 
* notification mechanisms:
  Is email at the beginning and end of the window ok?
  
* who will be point, who else will be involved
  Pythian SRE only, the assignee will depend on the timeframe when the change is executed

Comment 11

[Peter Radcliffe [:pir]]

So this plan hasn't been updated to make the changes through puppet, which means if any of the hosts get replaced/re-imaged/etc they will not stay consistent.

Comment 12

[Ludovic Hirlimann [:Usul]]

Also the new cab process says this should go thru service now , I don't see this one at https://mozilla.service-now.com/change_request_list.do?sysparm_userpref_module=cd579a82c0a8016400aa77d97a4d70a8&sysparm_query=active=true^EQ&active=true see Jason Crain's email and https://mana.mozilla.org/wiki/display/MOC/Quick+Start+-+Requesting+a+Change on how to submit this in Service now.