898227 - Mozharness should release mozpool requests when it gives up waiting for the device to be ready

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Justin Wood (:Callek)]

Attachment: [mozharness] cancel the requests when not ready!

So, currently we issue mozpool requests, valid for 4 hours.

When mozpool takes a long time to get us a device, mozharness gives up. However mozpool still thinks the request-for-device is valid and will happily hold the device aside for the full 4 hours, even if it became ready after the fact.

This prevents future jobs from getting the device. And afaict the requests then stack up... We need to close the request when waiting-for-ready fails.

This won't fix any of the issues that cause the device to not be ready, but hopefully this will allow us to not have 100s of RETRIES when it happens.

Comment 1

[Aki Sasaki (not active)]

Comment on attachment 781385 [details] [diff] [review]
[mozharness] cancel the requests when not ready!

>+        if fail_cb:
>+            fail_cb()

Maybe |if fail_cb and callable(fail_cb):|, but this is fine too.

Comment 2

[Justin Wood (:Callek)]

Landed with |assert callable(fail_cb)| instead after IRC chat.

https://hg.mozilla.org/build/mozharness/rev/c1a932eb4239

Awaits mozharness merge

Comment 3

[John Hopkins (:jhopkins)]

in production

Comment 4

[Kim Moir [:kmoir] ET]

Can this be closed? Just looking at this since it's a dependency for one of my bugs.

Comment 5

[Justin Wood (:Callek)]

indeed I believe it is fixed. (sorry)

Comment 6

[Kim Moir [:kmoir] ET]

yay:-) thanks Callek