1388282 - Create a shell wrapper script to output puppet changes on jumphost

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Dragos Crisan [:dragrom]]

Create a shell wrapper script to output puppet changes on jumphosts and send email notification

Comment 1

[Dragos Crisan [:dragrom]]

Fail to use sudo on rejh1.srv.releng.mdc1.mozilla.com for user dcrisan

Comment 2

[Dragos Crisan [:dragrom]]

Attachment: Bug_1388282_Shell_script_to_output_puppet_changes.patch

Added a shell script that check puppet changes on jumphosts and sent email with these changes
The script will be runn by a cron job, every day at 7 am.

Comment 3

[Dragos Crisan [:dragrom]]

Attachment: Bug_1388282_Shell_script_to_output_puppet_changes.patch

Added a shell script that check puppet changes on jumphosts and sent email with these changes
Added a filter to remove Yumrepo information
The script will be run by a cron job, every day at 7 am.

Comment 4

[Dragos Crisan [:dragrom]]

Attachment: Bug_1388282_Shell_script_to_output_puppet_changes.patch

Added a shell script that check puppet changes on jumphosts and sent email with these changes
Added a filter to remove Yumrepo information
The script will be run by a cron job, every day at 7 am.

Comment 5

[Jake Watkins [:dividehex]]

Comment on attachment 8900295 [details] [diff] [review]
Bug_1388282_Shell_script_to_output_puppet_changes.patch

Review of attachment 8900295 [details] [diff] [review]:
-----------------------------------------------------------------

I've very wary of filter the log output here.  There is a big risk to accidentally filtering out stuff you didn't want to filter.  I think a better approach would be to solve the non-deterministic and failure to ever reach state errors.  The two errors I see should be fixable.  I also think we need to come to the conclusion that we are not going to get away with only sending email when things change.  It is better to be expecting an email everyday so you are assured things are working instead of assuming.

So pull the filters out, fix the issues mentioned inline and let's open other bugs to fix the issues that are making the logs unnecessarily verbose.

::: modules/puppet/manifests/check_changes.pp
@@ +11,5 @@
> +    # The file where we will store the output for puppet agent -t --noop command. This output will be sent to the email body
> +    $logfile = '/tmp/puppet.txt'
> +
> +    case $::operatingsystem {
> +        # On junphosts we have only CentOS

Jumphosts is mispelled

@@ +16,5 @@
> +        CentOS: {
> +            $hour = 7
> +            file {
> +                # This is done via crontab due to a memory leak in puppet identified by
> +                # Mozilla IT.  There is enough splay here to avoid killing the master

The first sentence of the comments here does not apply in this case.  The issue with the memory leak had to do with running puppet as a daemon.

::: modules/puppet/templates/puppetcheck_changes.sh.erb
@@ +12,5 @@
> +FILTERS="/bin/grep -v 'Yumrepo' | /bin/grep -v 'yum' | /bin/grep -v '/tmp/puppet-' | /bin/grep -v '@@' | /bin/grep -v 'Info:' | /bin/grep -v 'Notice: Finished catalog run' | /bin/sed '/^\s*$/d'"
> +RUN_COMMAND="$PUPPET_COMMAND | $FILTERS > $LOGFILE"
> +
> +# eval - construct command by concatenating arguments
> +eval $RUN_COMMAND

This command should be executed in a loop in case it fails which will happen if puppet is already running and holding a lock open.  The loop should backoff and retry, and if it fails x number of times, skip it and email it failed to run.

@@ +14,5 @@
> +
> +# eval - construct command by concatenating arguments
> +eval $RUN_COMMAND
> +
> +# If there are changes, sent the email 

trailing whitespace

@@ +19,5 @@
> +LINES=$( /bin/cat $LOGFILE|/usr/bin/wc -l )
> +
> +if [ $LINES -gt 0 ]; then
> +    # sent the email
> +    mail -s "Puppet changes on `facter fqdn`" -r "root@`facter fqdn`<root@`facter fqdn`>"  $EMAIL_ADDRESS< $LOGFILE

Use @::fqdn to embed the string instead of calling facter every time

Comment 6

[Dragos Crisan [:dragrom]]

Attachment: Bug_1388282_Shell_script_to_output_puppet_changes.patch

With this patch I want to implement a wrapper script to check for puppet changes and sent the email
From the previous patch, I changed the following:
* removed the filter for puppet result
* added the check into the loop
* sent email if the check command failed

Comment 7

[Dragos Crisan [:dragrom]]

Comment on attachment 8900728 [details] [diff] [review]
Bug_1388282_Shell_script_to_output_puppet_changes.patch

https://hg.mozilla.org/build/puppet/rev/a1c21297321a2b694754ea446c0cf38dbbf8bb57