1671033 - OCC terminates instances for missing task drives before ec2config has mapped task drives

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: OpenCloudConfig, task)

Description

[Rob Thijssen [:grenade (EET/UTC+0300)]]

for some reason on newly deployed win2012 amis, the occ task drive check is happening before ec2config has completed drive mapping, causing a race condition that commonly resolves badly when occ terminates the instance right before ec2config would have completed the mappings.

Comment 1

[Rob Thijssen [:grenade (EET/UTC+0300)]]

patch: https://github.com/mozilla-releng/OpenCloudConfig/commit/745e0f349126ffe37a838c9e6b1f5bcfae266044

on win2012 worker types:

• wait 5 minutes for ec2config mappings
• if mappings do not occur but mapping config exists, format D drive to change it's label from '' (blank) to 'task' and reboot to force ec2config to try again

Comment 2

[Rob Thijssen [:grenade (EET/UTC+0300)]]

new patch: https://github.com/mozilla-releng/OpenCloudConfig/commit/83ededbae1e83f1b03c47537c084750c6828c05a
image build: https://firefox-ci-tc.services.mozilla.com/tasks/ZIJ0hSRLT8ah6_zQc2pCbg (unfortunately no livelogs, since the image build worker was made l3)

my earlier patch neglected to check for a cache drive (y:) and only fixed the missing task drive (z:). i still don't understand why ec2-config drive mappings are occurring later than we are used to or why the 2012 configs only contain two drives instead of 3 but i'm probably forgetting something that changed in the 6 months or so since we last updated these amis.

we'll also need to figure out how to update ci-config to include the 3rd drive (c:, y:, z:). win 7 &/or 10 should have a config we can copy.

Comment 3

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: 2012 images with drive map patch

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/894ee90b38fc
2012 images with drive map patch r=aki

Comment 5

[Aki Sasaki (not active)]

Aiui, status: gecko-3/b-win2012 should be all good. t-win10-64-source may still need some attention, but aiui those are all tier 2 tasks; relops will look at that tomorrow.

Comment 6

[Rob Lemley [:rjl]]

Attachment: Bug 1671033 - Follow-up: add drive to comm3/b-win2012 workers. r=aki

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/75b8482842f1
Follow-up: add drive to comm3/b-win2012 workers. r=aki

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This failure still hits gecko-1/b-win2012, e.g. https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=B4pTNkiQR3m3xbasQ-ej4w.0&revision=a61c4514299fced454b3bb7e7b86de2fd988c194

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Mark, can you assist with the gecko-1/b-win2012 workers? It blocks testing current mozilla-central code as mozilla-beta (last successful build is from Tuesday) and that merge is on Monday.

Comment 11

[Rob Thijssen [:grenade (EET/UTC+0300)]]

gecko-1/b-win2012 image rebuilds in progress:

• https://github.com/mozilla-releng/OpenCloudConfig/commit/af1b426
• https://firefox-ci-tc.services.mozilla.com/tasks/aqcii1bfRG-CAwzZSkYTRg

i'll follow up with a ci-config patch when we've seen working examples using the updated images.

Comment 12

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: rebuilt 2012 amiswith:- updated hg cert pin- patched task/cache drive mapping

Comment 13

[Intermittent Failures Robot]

4 failures in 3694 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 4

Platform breakdown:

• windows2012-64-shippable: 1
• windows2012-32-shippable: 1
• windows2012-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1671033&startday=2020-10-12&endday=2020-10-18&tree=all