investigate low frequency of event log messages to papertrail from ec2 2008 instances

RESOLVED FIXED

Status

Infrastructure & Operations
RelOps: Puppet
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: grenade, Assigned: grenade)

Tracking

Details

(Whiteboard: [windows][aws])

Attachments

(2 attachments, 1 obsolete attachment)

(Assignee)

Description

2 years ago
there seems to be more than one issue:
- nxlog internal log is recording multiple connection attempts to log-aggregator.srv.releng.use1.mozilla.com which resolves to internal-LogAggregatorLB-1909308518.us-east-1.elb.amazonaws.com (must be a load balancer). when changing nxlog config to point to log-aggregator1 (skipping the lb) this problem seems to go away.
- the filter in use in ec2 was generated in ix and is missing lots of useful log categories. we should generate an ec2 specific filter.
- there is a lot of noise in the event log from the parent (golden/base) amis. we should add a purge step to the ami generator to discard this noise before generating clean amis.
- we should identify event log messages other than those in the admin filter which are useful to debugging build processes and add these to the filter.
(Assignee)

Comment 1

2 years ago
Created attachment 8662367 [details] [diff] [review]
filter generated for ec2 2008 instances

this patch differentiates win 2008 event log filters in ec2 from ix instances since the events of interest are different on the slightly different windows version configurations. it should result in better (more) papertrail data for ec2 instances (and no change to papertrail data for ix instances)
Attachment #8662367 - Flags: review?(arich)
(Assignee)

Comment 2

2 years ago
Created attachment 8662369 [details] [diff] [review]
with client config update
Attachment #8662367 - Attachment is obsolete: true
Attachment #8662367 - Flags: review?(arich)
Attachment #8662369 - Flags: review?(arich)
Attachment #8662369 - Flags: review?(arich) → review+
(Assignee)

Comment 4

2 years ago
Created attachment 8662372 [details] [diff] [review]
https://github.com/mozilla/build-cloud-tools/pull/118

there is a lot of noise in the event log from the parent (golden/base) amis. this pr adds a purge step to the ami capture to discard this noise before generating clean amis.
Attachment #8662372 - Flags: review?(mcornmesser)
oh sorry didn't see who the r? was directed to :)
Attachment #8662372 - Flags: review?(mcornmesser) → review+
(Assignee)

Comment 6

2 years ago
all patches pushed
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.