In order to get better visibility on bad pandas, we should move all devices in mozpool currently 'locked_out' to the 'Ready' state. This will allow self tests to run and find pandas that are misbehaving. Doing so should have no impact on production android pandas since they are already set in a 'Ready' state (or other non locked_out state).
Yes, please! Thanks Jake.
To be explicit here, I'm working on a patch that will require us to lock out a swatch of pandas again, in order to support talos (not yet using mozpool) and unittests (using mozpool) at the same time.
(In reply to Justin Wood (:Callek) from comment #2) > To be explicit here, I'm working on a patch that will require us to lock out > a swatch of pandas again, in order to support talos (not yet using mozpool) > and unittests (using mozpool) at the same time. We can place the talos pandas into the 'locked_out' state when that range is defined and allocated but until then, we should move them to 'Ready' so mozpool can help us identify trouble before they end up in production.
I've moved all the pandas that were 'locked_out' into the 'ready' state (except for panda-0307). It is very likely that some pandas in this set will fail the self test and end up in a 'failed_*' state. Expect nagios alerts.
There is a comment on panda-0307 saying it is being used but there is no bug # referenced. It remains 'locked_out' Anyone still using it? /cc jmaher kmoir
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
:jmaher Were you using panda-0307 for anything?
I am not using panda-0307 for anything, take it away!
Thanks Joel. All pandas have been moved out of the locked_out state EXCEPT for the ones we just moved back into the locked_out state in Bug894512. (but they have a good reason) ;-)
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.