975001 - Log large-difference HAWK clock mismatches

Status: VERIFIED FIXED

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[Toby Elliott [:telliott]]

We should start logging large (>2 min?) time differentials between clients and servers to get an idea of the scope of the mismatch problem. 

A large change in the frequency of these may indicate a problem on the server side.

Comment 1

[Ryan Kelly [:rfkelly]]

Moving to Server:Sync since tokenserver is the one thing in this mix that doesn't speak hawk.  Is this covered by the changes in Bug 975214?

Comment 2

[Toby Elliott [:telliott]]

I think it's supplemental. We should still be tracking the big mismatches, both so we would have an idea of scope if we shortened the hawk window and also because a rise in them might indicate a clock problem on the server.

Comment 3

[Toby Elliott [:telliott]]

ah, but you do have the logging in there, so yes, we're covered.

Comment 4

[Ryan Kelly [:rfkelly]]

Marking resolved/fixed since this is covered by Bug 975214.  This code is now live on stage, we should verify that large timestamp differences do in fact show up in the logs before marking it as verified.

Comment 5

[James Bonacci [:jbonacci]]

We can't really do this accurately since the host times and log times are different.
What is the status of that issue?

Comment 6

[James Bonacci [:jbonacci]]

Here it is (now): https://github.com/mozilla-services/puppet-config/issues/172

Comment 7

[Ryan Kelly [:rfkelly]]

ISTM we don't have to have the times lining up, we just need to confirm whether the particular log lines produced by Bug 975214 are actually appearing in the log.  (Correlated times would be *better* of course, but not necessary)

Comment 8

[James Bonacci [:jbonacci]]

I think we are good here.