If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

move all 'locked_out' panda to a ready state in mozpool

RESOLVED FIXED

Status

Infrastructure & Operations
RelOps
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: dividehex, Assigned: dividehex)

Tracking

Details

(Assignee)

Description

4 years ago
In order to get better visibility on bad pandas, we should move all devices in mozpool currently 'locked_out' to the 'Ready' state.  This will allow self tests to run and find pandas that are misbehaving.  Doing so should have no impact on production android pandas since they are already set in a 'Ready' state (or other non locked_out state).

Comment 1

4 years ago
Yes, please! Thanks Jake.
To be explicit here, I'm working on a patch that will require us to lock out a swatch of pandas again, in order to support talos (not yet using mozpool) and unittests (using mozpool) at the same time.
(Assignee)

Comment 3

4 years ago
(In reply to Justin Wood (:Callek) from comment #2)
> To be explicit here, I'm working on a patch that will require us to lock out
> a swatch of pandas again, in order to support talos (not yet using mozpool)
> and unittests (using mozpool) at the same time.

We can place the talos pandas into the 'locked_out' state when that range is defined and allocated but until then, we should move them to 'Ready' so mozpool can help us identify trouble before they end up in production.
(Assignee)

Comment 4

4 years ago
I've moved all the pandas that were 'locked_out' into the 'ready' state (except for panda-0307).  It is very likely that some pandas in this set will fail the self test and end up in a 'failed_*' state.  Expect nagios alerts.
(Assignee)

Comment 5

4 years ago
There is a comment on panda-0307 saying it is being used but there is no bug # referenced.  It remains 'locked_out'  Anyone still using it?

/cc jmaher kmoir
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
(Assignee)

Comment 6

4 years ago
:jmaher

Were you using panda-0307 for anything?
Flags: needinfo?(jmaher)
I am not using panda-0307 for anything, take it away!
Flags: needinfo?(jmaher)
(Assignee)

Comment 8

4 years ago
Thanks Joel.  All pandas have been moved out of the locked_out state EXCEPT for the ones we just moved back into the locked_out state in Bug894512.  (but they have a good reason) ;-)
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.