893998 - move all 'locked_out' panda to a ready state in mozpool

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Jake Watkins [:dividehex]]

In order to get better visibility on bad pandas, we should move all devices in mozpool currently 'locked_out' to the 'Ready' state.  This will allow self tests to run and find pandas that are misbehaving.  Doing so should have no impact on production android pandas since they are already set in a 'Ready' state (or other non locked_out state).

Comment 1

[Armen [:armenzg]]

Yes, please! Thanks Jake.

Comment 2

[Justin Wood (:Callek)]

To be explicit here, I'm working on a patch that will require us to lock out a swatch of pandas again, in order to support talos (not yet using mozpool) and unittests (using mozpool) at the same time.

Comment 3

[Jake Watkins [:dividehex]]

(In reply to Justin Wood (:Callek) from comment #2)
> To be explicit here, I'm working on a patch that will require us to lock out
> a swatch of pandas again, in order to support talos (not yet using mozpool)
> and unittests (using mozpool) at the same time.

We can place the talos pandas into the 'locked_out' state when that range is defined and allocated but until then, we should move them to 'Ready' so mozpool can help us identify trouble before they end up in production.

Comment 4

[Jake Watkins [:dividehex]]

I've moved all the pandas that were 'locked_out' into the 'ready' state (except for panda-0307).  It is very likely that some pandas in this set will fail the self test and end up in a 'failed_*' state.  Expect nagios alerts.

Comment 5

[Jake Watkins [:dividehex]]

There is a comment on panda-0307 saying it is being used but there is no bug # referenced.  It remains 'locked_out'  Anyone still using it?

/cc jmaher kmoir

Comment 6

[Jake Watkins [:dividehex]]

:jmaher

Were you using panda-0307 for anything?

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

I am not using panda-0307 for anything, take it away!

Comment 8

[Jake Watkins [:dividehex]]

Thanks Joel.  All pandas have been moved out of the locked_out state EXCEPT for the ones we just moved back into the locked_out state in Bug894512.  (but they have a good reason) ;-)