Closed Bug 1022438 Opened 10 years ago Closed 10 years ago

https://tbpl.mozilla.org/?tree=Try&rev=8fa5dc94a2d2 has more than 1500 pending jobs

Categories

(Release Engineering :: General, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: rail, Unassigned)

Details

Not sure what's going on there but this push generated more than 1500 pending b2g_emulator_vm try opt test marionette-webapi jobs
Funny story, I tried to trigger 999 jobs via the box on self-serve (taking advantage of a slow weekend and needing to collect some data), but somehow that got doubled to 1998. Not sure how it happened, FWIW. I definitely didn't double-submit.
So, it was expected, phew. I still wonder whether we can do something more effective to figure out the root clause of the failures...
Severity: blocker → normal
Priority: P1 → --
It's the first time I've ever seen requested number of jobs doubled like that. Suppose it couldn't have been on a worse one either :P This also exposes a need for a better way to selectively mass-cancel jobs in self-serve :)
FTR, we still have 916 jobs pending...
BTW, this also affects our usage of on-demand instances. We start all of them since we hit spot instance capacity (our own limit set in http://hg.mozilla.org/build/cloud-tools/file/acac9151bbb8/configs/watch_pending.cfg#l70).
I canceled all running jobs in self-serve since the push already completed way more than 999.
As I mentioned on IRC, it would have been nice if the running jobs had been left to complete instead of showing up as canceled and leaving me to have to sort through a mess of canceled jobs skewing the numbers I was trying to get.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → WORKSFORME
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.