1494336 - [Tracking] Unify the papertrail logs for Moonshots and OSX-10.10

Status: RESOLVED INVALID

(Infrastructure & Operations :: RelOps: General, task)

Description

[Zsolt Fay [:zfay]]

At the moment each worker has two places where they send logs to in papertrail: a short and a long fqdn. E.g:
 T-W1064-MS-165.mdc1.mozilla.com 
 t-w1064-ms-165.wintest.releng.mdc1.mozilla.com

The goal is to have only one entry per worker. Filing this bug to track this progress.

Comment 1

[Rob Thijssen [:grenade (EET/UTC+0300)]]

pmoore: i'm planning on adding a hosts file to hardware workers in the same way we did for ec2 workers in bug 1497308. before i do, do you know where hardware workers get their generic-worker.config file from? if it is generated by generic-worker.exe during the --install phase, is there somewhere i can check if it includes a base64 encoded hosts file? i'd need to remove that before proceeding with this patch.

Comment 2

[Pete Moore [:pmoore][:pete]]

Hi Rob,

I'm not sure where they get their generic-worker.config file from, Mark might know.

Comment 3

[Mark Cornmesser [:markco]]

During the initial deployment a master config file, including secrets, is copied to the node. Then the generic-worker wrapper script edits the file to add in machine specific info, https://github.com/mozilla-releng/OpenCloudConfig/blob/e80ed61d2e8eba2761392c2f01624723b81587cd/userdata/Configuration/GenericWorker/run-hw-generic-worker-10-and-reboot.bat#L33-L36 .

Comment 4

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Mark: could you check if the master config file contains a base64 encoded hosts file? It would have a path of: "C:\Windows\system32\drivers\etc\hosts". Alternatively, let me know where the file comes from and I'll check it.

If the config file does contain a base64 encoded hosts file, we'll need to remove it.

Comment 5

[Mark Cornmesser [:markco]]

All the content in that file is commented out. It pretty much looks like the default/example file.

Comment 6

[Jake Watkins [:dividehex]]

Status? Is this still an issue?

Comment 7

[Rob Thijssen [:grenade (EET/UTC+0300)]]

looks like yes.
https://papertrailapp.com/groups/1141234/events?q=system%3Awintest

the problem manifests when hardware logs are routed through the log aggregators.
when logs are routed directly to papertrail we don't see the issue.

Comment 8

[Jake Watkins [:dividehex]]

(In reply to Rob Thijssen [:grenade (EET/UTC+0300)] from comment #7)

looks like yes.
https://papertrailapp.com/groups/1141234/events?q=system%3Awintest

the problem manifests when hardware logs are routed through the log aggregators.
when logs are routed directly to papertrail we don't see the issue.

I believe, in the long run, the goal is to get rid of the log aggregators and route logs directly to papertrail or stackdriver. From the windows perspective, can that be done now?

Comment 9

[Mark Cornmesser [:markco]]

I believe, in the long run, the goal is to get rid of the log aggregators and route logs directly to papertrail or stackdriver. From the windows perspective, can that be done now?

As far as I know there is no real blocker on it except for available time to work on it.

Comment 10

[Rob Thijssen [:grenade (EET/UTC+0300)]]

dhouse: fubar mentioned you might have figured out the settings to get nxlog to forward to stackdriver. is my understanding correct? can you point me at settings that will work?

Comment 11

[:dhouse]

(In reply to Rob Thijssen [:grenade (EET/UTC+0300)] from comment #10)

dhouse: fubar mentioned you might have figured out the settings to get nxlog to forward to stackdriver. is my understanding correct? can you point me at settings that will work?

I'm sorry I found some documents on configuring nxlog to send metrics to influxdb, but I didn't look at logs. I think Andrew set up straight python to send logs into stackdriver (and I'm tempted to change the linux/posix to do that also because of dependency issues I've had on centos with the google fluentd plugin).

Comment 12

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Andrew: did you find a way to send logs to stackdriver without fluentd?
If there's any code or config you can point me at, I'd be grateful. I'm just looking for a way to forward nxlog output to stackdriver. So even a port number and url would be helpful. Any pointers on the log format that stackdriver will accept is a bonus too.

Comment 13

[Jake Watkins [:dividehex]]

See next comment

Comment 14

[Jake Watkins [:dividehex]]

It is too bad stackdriver doesn't provide syslog endpoints like papartrail does. I think running fluentd or some other middleware (python?) might be the only choice here. This also starts to make the argument of keeping and fixing log aggregators to handle the syslog<->stackdriver flow. Good grief.
https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#om_exec

Comment 15

[Andrew Erickson [:aerickson]]

Rob: I send logs to Stackdriver via python. It's pretty simple. I guess the client has a hostname and port hard-coded in it. It will take any text format, there's no structure to it (I know papertrail you can set tags and such).

https://github.com/bclary/mozilla-bitbar-docker/blob/master/scripts/run_gw.py

Comment 16

[ablew]

backlog cleanup