1073907 - High sys% cpu usage on node[1-30].peach.metrics.scl3.mozilla.com with auditd enabled

Status: RESOLVED FIXED

(Data & BI Services Team :: Data Warehouse, task)

Description

[T [:tmary] Meyarivan]

Please review attached graph which shows the effect of disabling auditing (specifically: 'auditctl -D') on a IO intensive job.

'auditctl -D' was issued at ~01:54 UTC and 'service auditd restart' was issued at ~02:05 UTC.

-

Running 'perf top' on one of these hosts seemed to indicate that 'get_task_cred' and 'audit_filter_rules' were the culprits.

--

Comment 1

[T [:tmary] Meyarivan]

Attachment: Aggregate sys% cpu graph

Comment 2

[T [:tmary] Meyarivan]

Attachment: Aggregate sys% cpu graph (line graph) - 2014-09-29 (UTC)

sys% increases as soon as the IO intensive job starts running at ~00:00 (UTC). The job finished at ~01:40 (UTC)

Comment 3

[T [:tmary] Meyarivan]

Re the outliers in Comment#1:

node26 -> configured to not run jobs, auditing is enabled
node25 -> configured to run jobs, auditing is disabled (auditctl -D)

--

Comment 4

[T [:tmary] Meyarivan]

Attachment: Job details

Jobs being referred to in Comment#0 and Comment#2 are same - it is a job which reads FHR data from HBase and writes to HDFS. Re #s, it reads (and decompresses) approximately 7TB of compressed data, writes approximately 21TB of compressed data

--

Comment 5

[T [:tmary] Meyarivan]

sys% down by half after pruning auditing rules via BUG 1074478 - TY kang@, digi@

--