1270315 - puppet freshness check sometimes parses configuration version wrong from state file

Status: RESOLVED WONTFIX

(Infrastructure & Operations :: RelOps: General, task)

Description

[Amy Rich [:arr] [:arich]]

I've noticed that sometimes the freshness check alerts with: "Catalog version is unknown" on all the hosts at once for some period of time. And then it magically corrects itself.

Looking at the code, it's failing because the line that parses the version is using " as a separator:
 
configuration_version=$(awk -F '"' '/\s*configuration_version:/ {print $2}' $reportfile) 

The line in the report file (at least when this fails) doesn't have any quotes:

# grep configuration_version /var/lib/puppet/state/last_run_report.yaml 
  configuration_version: f62811ebdea4

The puppet_version (which is also checked for by the script) does in the same file:

  puppet_version: "3.7.0"

Not sure if the yaml file is being written out with quotes and sometimes without.

Comment 1

[Amy Rich [:arr] [:arich]]

Attachment: puppet_freshness_awk.patch

This removes the field separator of " from the script.

Comment 2

[Jake Watkins [:dividehex]]

It's quote possible that values are being written with quotes and sometimes without.  I just ran a diff between a working report and a non working report and noticed some fields changed to being quoted.

Although the configuration_version we are checking for didn't

Comment 3

[Amy Rich [:arr] [:arich]]

Another issue I noticed... the awk line also matches message: when the message includes the string configuration_version. This script could probably do with a rewrite in something that parses json better.

In the meantime, I've also upped the nagios check to only alert after 8 failures, and I've created a servicegroup for puppet-freshness so we can downtime them all at once more easily.

Comment 4

[Amy Rich [:arr] [:arich]]

Not going to have cycles to fix this better, and we haven't gotten alerted since I bumped up the nagios limit.