Create a shell wrapper script to output puppet changes on jumphost

RESOLVED FIXED

Status

Infrastructure & Operations
RelOps: Puppet
RESOLVED FIXED
10 months ago
9 months ago

People

(Reporter: dragrom, Assigned: dragrom)

Tracking

(Depends on: 1 bug, Blocks: 1 bug)

Details

Attachments

(1 attachment, 3 obsolete attachments)

(Assignee)

Description

10 months ago
Create a shell wrapper script to output puppet changes on jumphosts and send email notification
(Assignee)

Updated

10 months ago
Blocks: 1387390
(Assignee)

Updated

10 months ago
No longer blocks: 1387390

Updated

10 months ago
Blocks: 1379671
(Assignee)

Comment 1

9 months ago
Fail to use sudo on rejh1.srv.releng.mdc1.mozilla.com for user dcrisan
(Assignee)

Updated

9 months ago
Flags: needinfo?(jwatkins)
(Assignee)

Updated

9 months ago
Flags: needinfo?(jwatkins)
(Assignee)

Comment 2

9 months ago
Created attachment 8899867 [details] [diff] [review]
Bug_1388282_Shell_script_to_output_puppet_changes.patch

Added a shell script that check puppet changes on jumphosts and sent email with these changes
The script will be runn by a cron job, every day at 7 am.
Attachment #8899867 - Flags: review?(jwatkins)
(Assignee)

Comment 3

9 months ago
Created attachment 8900286 [details] [diff] [review]
Bug_1388282_Shell_script_to_output_puppet_changes.patch

Added a shell script that check puppet changes on jumphosts and sent email with these changes
Added a filter to remove Yumrepo information
The script will be run by a cron job, every day at 7 am.
Attachment #8899867 - Attachment is obsolete: true
Attachment #8899867 - Flags: review?(jwatkins)
Attachment #8900286 - Flags: review?(jwatkins)
(Assignee)

Comment 4

9 months ago
Created attachment 8900295 [details] [diff] [review]
Bug_1388282_Shell_script_to_output_puppet_changes.patch

Added a shell script that check puppet changes on jumphosts and sent email with these changes
Added a filter to remove Yumrepo information
The script will be run by a cron job, every day at 7 am.
Attachment #8900286 - Attachment is obsolete: true
Attachment #8900286 - Flags: review?(jwatkins)
Attachment #8900295 - Flags: review?(jwatkins)
Comment on attachment 8900295 [details] [diff] [review]
Bug_1388282_Shell_script_to_output_puppet_changes.patch

Review of attachment 8900295 [details] [diff] [review]:
-----------------------------------------------------------------

I've very wary of filter the log output here.  There is a big risk to accidentally filtering out stuff you didn't want to filter.  I think a better approach would be to solve the non-deterministic and failure to ever reach state errors.  The two errors I see should be fixable.  I also think we need to come to the conclusion that we are not going to get away with only sending email when things change.  It is better to be expecting an email everyday so you are assured things are working instead of assuming.

So pull the filters out, fix the issues mentioned inline and let's open other bugs to fix the issues that are making the logs unnecessarily verbose.

::: modules/puppet/manifests/check_changes.pp
@@ +11,5 @@
> +    # The file where we will store the output for puppet agent -t --noop command. This output will be sent to the email body
> +    $logfile = '/tmp/puppet.txt'
> +
> +    case $::operatingsystem {
> +        # On junphosts we have only CentOS

Jumphosts is mispelled

@@ +16,5 @@
> +        CentOS: {
> +            $hour = 7
> +            file {
> +                # This is done via crontab due to a memory leak in puppet identified by
> +                # Mozilla IT.  There is enough splay here to avoid killing the master

The first sentence of the comments here does not apply in this case.  The issue with the memory leak had to do with running puppet as a daemon.

::: modules/puppet/templates/puppetcheck_changes.sh.erb
@@ +12,5 @@
> +FILTERS="/bin/grep -v 'Yumrepo' | /bin/grep -v 'yum' | /bin/grep -v '/tmp/puppet-' | /bin/grep -v '@@' | /bin/grep -v 'Info:' | /bin/grep -v 'Notice: Finished catalog run' | /bin/sed '/^\s*$/d'"
> +RUN_COMMAND="$PUPPET_COMMAND | $FILTERS > $LOGFILE"
> +
> +# eval - construct command by concatenating arguments
> +eval $RUN_COMMAND

This command should be executed in a loop in case it fails which will happen if puppet is already running and holding a lock open.  The loop should backoff and retry, and if it fails x number of times, skip it and email it failed to run.

@@ +14,5 @@
> +
> +# eval - construct command by concatenating arguments
> +eval $RUN_COMMAND
> +
> +# If there are changes, sent the email 

trailing whitespace

@@ +19,5 @@
> +LINES=$( /bin/cat $LOGFILE|/usr/bin/wc -l )
> +
> +if [ $LINES -gt 0 ]; then
> +    # sent the email
> +    mail -s "Puppet changes on `facter fqdn`" -r "root@`facter fqdn`<root@`facter fqdn`>"  $EMAIL_ADDRESS< $LOGFILE

Use @::fqdn to embed the string instead of calling facter every time
Attachment #8900295 - Flags: review?(jwatkins) → review-
Depends on: 1393138
Depends on: 1393139
(Assignee)

Comment 6

9 months ago
Created attachment 8900728 [details] [diff] [review]
Bug_1388282_Shell_script_to_output_puppet_changes.patch

With this patch I want to implement a wrapper script to check for puppet changes and sent the email
From the previous patch, I changed the following:
* removed the filter for puppet result
* added the check into the loop
* sent email if the check command failed
Attachment #8900295 - Attachment is obsolete: true
Attachment #8900728 - Flags: review?(jwatkins)
Attachment #8900728 - Flags: review?(jwatkins) → review+
(Assignee)

Comment 7

9 months ago
Comment on attachment 8900728 [details] [diff] [review]
Bug_1388282_Shell_script_to_output_puppet_changes.patch

https://hg.mozilla.org/build/puppet/rev/a1c21297321a2b694754ea446c0cf38dbbf8bb57
Attachment #8900728 - Flags: checked-in+
(Assignee)

Updated

9 months ago
Status: ASSIGNED → RESOLVED
Last Resolved: 9 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.