1205640 - investigate low frequency of event log messages to papertrail from ec2 2008 instances

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Rob Thijssen [:grenade (EET/UTC+0300)]]

there seems to be more than one issue:
- nxlog internal log is recording multiple connection attempts to log-aggregator.srv.releng.use1.mozilla.com which resolves to internal-LogAggregatorLB-1909308518.us-east-1.elb.amazonaws.com (must be a load balancer). when changing nxlog config to point to log-aggregator1 (skipping the lb) this problem seems to go away.
- the filter in use in ec2 was generated in ix and is missing lots of useful log categories. we should generate an ec2 specific filter.
- there is a lot of noise in the event log from the parent (golden/base) amis. we should add a purge step to the ami generator to discard this noise before generating clean amis.
- we should identify event log messages other than those in the admin filter which are useful to debugging build processes and add these to the filter.

Comment 1

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: filter generated for ec2 2008 instances

this patch differentiates win 2008 event log filters in ec2 from ix instances since the events of interest are different on the slightly different windows version configurations. it should result in better (more) papertrail data for ec2 instances (and no change to papertrail data for ix instances)

Comment 2

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: with client config update

Comment 3

[Rob Thijssen [:grenade (EET/UTC+0300)]]

pushed https://hg.mozilla.org/build/puppet/rev/9fe4b8a20ffa

Comment 4

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: https://github.com/mozilla/build-cloud-tools/pull/118

there is a lot of noise in the event log from the parent (golden/base) amis. this pr adds a purge step to the ami capture to discard this noise before generating clean amis.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

oh sorry didn't see who the r? was directed to :)

Comment 6

[Rob Thijssen [:grenade (EET/UTC+0300)]]

all patches pushed