1463508 - refactor instance age check logic to determine precise instance age

Status: RESOLVED WONTFIX

(Infrastructure & Operations :: RelOps: OpenCloudConfig, task)

Description

[Rob Thijssen [:grenade (EET/UTC+0300)]]

windows instances are terminating due to being 24 hours old at peak times of activity. when they mass terminate at busy times of day it causes issues with our ability to spawn sufficient new instances and results in high pending counts at the same busy time of each day.

see: https://papertrailapp.com/login?return_to=%2Fgroups%2F3538343%2Fevents%3Fq%3Dprogram%253Auser32%2520%2522initiated%2520the%2520shutdown%2522

Comment 1

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: https://github.com/mozilla-releng/OpenCloudConfig/pull/140

Comment 2

[Rob Thijssen [:grenade (EET/UTC+0300)]]

allowable age altered from 24 hours to 29 hours in order to stagger terminations to different times of day
https://github.com/mozilla-releng/OpenCloudConfig/commit/81db13c6780ce5d2dadf89a48697647807dcd112

Comment 3

[Rob Thijssen [:grenade (EET/UTC+0300)]]

there seems to be an issue with whether or not the instances have actually been around for the period of time allowed. they may be getting terminated early due to the pre-existence of renaming events (from the base ami) in the event log. we need to find a better way of checking and validating instance age.

Comment 4

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: https://github.com/mozilla-releng/OpenCloudConfig/pull/142

temporarily disable age based deletions while we find a better way to check instance age.

Comment 5

[Rob Thijssen [:grenade (EET/UTC+0300)]]

please see bug 1480398, comment 2