1016369 - Change zlb[3-5].ops.phx1.mozilla.com timezone to UTC

Status: RESOLVED FIXED

(Cloud Services :: Operations: Marketplace, task, P2)

Description

[Jason Thomas [:jason]]

Currently configured to:
[[e-mail]1 zeus]# date +%Z
PDT

Comment 1

[Jason Thomas [:jason]]

This will affect the following zeus logs:

versiocheck.addons.mozilla.org
pfs.mozilla.org
blocklist.addons.mozilla.org

Comment 2

[Anurag Phadke[:[e-mail]]]

Jason - does this affect all vamo, bamo and pfs logs?

Comment 3

[Anurag Phadke[:[e-mail]]]

Jason - afaik we have 12 Zeus clusters. Is it possible to have all 12 in same time zone?

Comment 4

[Jason Thomas [:jason]]

(In reply to Anurag Phadke[:[e-mail]] from comment #2)
> Jason - does this affect all vamo, bamo and pfs logs?

If all logs mean zeus load balancer logs, then yes, see comment 1.

Comment 5

[Jason Thomas [:jason]]

(In reply to Anurag Phadke[:[e-mail]] from comment #3)
> Jason - afaik we have 12 Zeus clusters. Is it possible to have all 12 in
> same time zone?

I believe IT has bug 899852 for a general UTC change on all hosts but you will need to talk to IT Webops for more information.

Comment 6

[Anurag Phadke[:[e-mail]]]

jason - can we run this through CAB? I would prefer to have all zeus LBs on same timezone. Doing few LBs on different TZs have hurt us badly in the past.

Comment 7

[Jason Thomas [:jason]]

(In reply to Anurag Phadke[:[e-mail]] from comment #6)
> jason - can we run this through CAB? 

Yes. This will definitely require a change window since I am not seeing a application level configuration and a timezone change would require a restart of Zeus service. We will most likely bundle this change with Bug 1013437.

(In reply to Anurag Phadke[:[e-mail]] from comment #6)
> I would prefer to have all zeus LBs on same timezone. Doing few LBs on different TZs have hurt us badly in the past.

We already know that this change will only affect 3 properties (see comment 1) and I believe you are able to adjust processing based on timezone because you are doing that already for marketplace.firefox.com and addons.mozilla.org. Why do the other zlb's need to be in the same timezone for this request?

Comment 8

[Anurag Phadke[:[e-mail]]]

jason - currently zlb[3-5] serve AMO, what happens when zlb[4-12] shift the TZ, start serving AMO and BAMO and misses notifying via email/bug until much later wherein its really difficult to backfill. 

Will be lot lot easier if all ZLBs were changed at same time.

Comment 9

[Jason Thomas [:jason]]

As per our conversation we will proceed with only changing zlb[3-5].ops.phx1.mozilla.com to UTC. If for whatever reason in the future we decide to move hosting away for the sites listed in comment 1 we will communicate it to the metrics team before hand so there aren't any log processing surprises.

Comment 10

[Anurag Phadke[:[e-mail]]]

verified that BAMO pings are in UTC and are working as expected. Thx Jason!

-anurag

Comment 11

[Jason Thomas [:jason]]

This change has not been implemented yet. I will let you know once completed.

Comment 12

[Jason Thomas [:jason]]

Done.

63.245.216.100 blocklist.addons.mozilla.org - [17/Jun/2014:21:14:37 +0000] "GET /blocklist/3/x/x/ HTTP/1.1" 200 86411 "-" "check_http/v1.4.16 (nagios-plugins 1.4.16)" "-" "DNT:-"