848488 - release runner shouldn't send e-mail about slow reconfig until it's more sure there's a problem

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[[e-mail] (:bhearsum)]

Currently, release runner sends warnings about reconfigs not completing every 5min. However, the AWS masters can take upwards of 20min to reconfig (as I understand it). This is problematic because it makes it appear like someone needs to take action when they often don't. We should either:
1) Make release runner wait longer before sending mail
2) Make the AWS masters reconfig faster

Or maybe do both.

Comment 1

[Chris AtLee [:catlee]]

I don't think 2) is going to be easy

5 minutes is a bit aggressive for 1). let's change it to 10 or 15, and adjust the message to say that you shouldn't be alarmed until 30 minutes have gone by?

Comment 2

[[e-mail] (:bhearsum)]

Thanks! Should be easy to do #1.

Comment 3

[[e-mail] (:bhearsum)]

Attachment: adjust message; times

Implements #1. I got rid of the timedelta because I think "600 seconds" is more readable than "0:05:00.117830"

Comment 4

[[e-mail] (:bhearsum)]

Comment on attachment 722801 [details] [diff] [review]
adjust message; times

Landed, updated the checkout on bm36, and restarted release runner through supervisorctl.