1339235 - Convert Sync to a Logging 2.0 infrastructure

Status: RESOLVED WONTFIX

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Bob Micheletto [:bobm]]

Convert Sync to a Logging 2.0 infrastructure.  This is slightly more complicated for Sync because both local and upstream heka filters must remain running.  The local heka daemons are required for http.request.count metrics.  The upstream heka filters are required reporting and must be redirected to consume events from Logging 2.0.

Comment 1

[Ryan Kelly [:rfkelly]]

Will this require any work on the dev side, or is it purely operational?

Comment 2

[Bob Micheletto [:bobm]]

(In reply to Ryan Kelly [:rfkelly] from comment #1)
> Will this require any work on the dev side, or is it purely operational?

If the sync application logs are mozlog compliant, this should be purely operational.  Can you verify that is true?

Comment 3

[Ryan Kelly [:rfkelly]]

I vaguely recall them *not* being so, I'll dig in and see if I can confirm...

Comment 4

[Bob Micheletto [:bobm]]

(In reply to Ryan Kelly [:rfkelly] from comment #3)
> I vaguely recall them *not* being so, I'll dig in and see if I can confirm...

:whd has informed me that since the logstamp format is not nanoseconds it is not mozlog format compliant.  :relud can Logging 2.0 handle arbitrary json log data?

Comment 5

[Daniel Thorn [:relud]]

Logging 2.0 is currently being designed with an assumption that all logs are mozlog compliant. Changing that assumption would be very difficult.

Comment 6

[Bob Micheletto [:bobm]]

(In reply to Bob Micheletto [:bobm] from comment #4)

> :whd has informed me that since the logstamp format is not nanoseconds it is
> not mozlog format compliant.  :relud can Logging 2.0 handle arbitrary json
> log data?

It turns out that we need to follow the mozlog standard to maintain forward compatibility with Logging 2.0, and for message routing to work.  We would probably go about converting doing something like this:

1. Update Sync to newest mozlog format and simultaneously switch current downstream log processing to handle it.  It might be possible to allow for handling two separate formats, in which case they don't need to happen at the same time.

2. Run gollem and Heka at the same time on Sync servers.  This will double the output, but allows for switching over without losing data.

3. Switch the current downstream message processing over to Kafka.

4. Turn off the Heka log aggregation output, but keep the Nginx output because gollem won't give us the statsd that we need for real time graphs and alerting.

Comment 7

[Bob Micheletto [:bobm]]

Will be moving to Stackdriver.