818906 - We cannot image anymore more boards since there are lots of pending requests

Status: RESOLVED DUPLICATE of bug 817762

(Infrastructure & Operations :: RelOps: General, task)

Description

[Armen [:armenzg]]

There are few dozens jobs that are "pending" and it is not clearing up:
http://mobile-imaging-001.p1.releng.scl1.mozilla.com/ui/mozpool.html
I can't re-image more boards in this state.

I could close all requests manually and I'm sure it would work but I would like to determine what did I do wrong and why the system get into this state.

I tried deleting the request for panda-0082 and start again
http://mobile-imaging-001.p1.releng.scl1.mozilla.com/ui/log.html?device=panda-0082
which allowed me to request the board again but it never changed the state from "pending".

In other words, the system has come to a halt IIUC.

I looked at the SOP document and this seems an unusual behavior; hence filing it as a separate bug:
https://wiki.mozilla.org/ReleaseEngineering/Mozpool/Handling_Panda_Failures