1574164 - high retry counts for Windows x64 reftests including webgl

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Windows OS, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Sheriffs are noticing much higher retry rate rate for the last 2-3 hours
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&resultStatus=retry%2Cusercancel%2Crunning%2Cpending%2Crunnable&searchStr=windows%2Cx64&tochange=0db07ff50ab5c9b655ba7e23788f68a2b9d66e71&selectedJob=261771581

From Rob: "last change to that [Windows x64 gpu workers] was 2 weeks ago": https://github.com/mozilla-releng/OpenCloudConfig/commits/master/userdata/Manifest/gecko-t-win10-64-gpu.json

Comment 1

[Rob Thijssen [:grenade (EET/UTC+0300)]]

i have found the culprit. it's down to a high number of spot terminations (when amazon terminates instances, not us).
https://papertrailapp.com/groups/13938351/events?q=spot%20termination%20notice%20received

this could be mitigated by having multiple instance types defined. something that we've been talking about doing in bug 1572089

Comment 2

[Intermittent Failures Robot]

4 failures in 4839 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4

Platform breakdown:

• windows10-64-qr: 1
• windows10-64: 1
• windows10-64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1574164&startday=2019-08-12&endday=2019-08-18&tree=all