842704 - Increase the puppet::atboot retry interval

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

In the vast majority of cases where a puppet run fails, the next one's going to fail too.  And the same on a few hundred machines.  Which makes for hundreds of emails per minute in the current incarnation.

The retry logic is currently to retry every 60s for 10m, then reboot.  The reboot is in case the host doesn't have an IP.  We haven't seen hosts without IPs in a long time.

Let's bump the delay back a bit: perhaps wait 2 minutes after the first failure and increase exponentially from there, and only reboot after an hour?

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug842704.patch

Comment 2

[Justin Wood (:Callek)]

Comment on attachment 721308 [details] [diff] [review]
bug842704.patch

Review of attachment 721308 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/puppet/templates/puppet-atboot-common.erb
@@ +2,5 @@
>  # License, v. 2.0. If a copy of the MPL was not distributed with this
>  # file, You can obtain one at http://mozilla.org/MPL/2.0/.
>  
> +# number of tries to make between attempts to run puppet.  7 waits about 2h
> +REBOOT_AFTER=7

count is 0-order not 1-order, so we actually try 8 times with this change, adding up to 4.25 hours! drop this to 6 and you have my r+.