854130 - big backlog of linux64/ubuntu64 jobs

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[Joe Drew (not getting mail)]

It's not clear what's going on with the linux64 jobs, but:

[3:14 PM] <joe> are we running low on linux64 test slaves, or something?
[3:15 PM] <joe> linux64 jobs are stacking up, with only 2 running
[3:16 PM] <philor> hmm
[3:17 PM] <joe> 28 running on non-try it seems
[3:20 PM] <philor> 10, actually
[3:20 PM] <philor> 9 non-try
[3:21 PM] <philor> "Ubuntu 12.04 x64" is a zombie from Thursday; "Ubuntu VM 12.04 x64" is a live job
[3:29 PM] <philor> and given how we coalesce away jobs no matter what, I can't really tell whether it was awful yesterday like it would have been if it was the name change, or not

And from #releng:
[3:18 PM] <philor> 113 pending ubuntu vm jobs; 24 running ubuntu vm jobs
[3:18 PM] <philor> does whatever decides about starting up slaves not realize that all the ubuntu ?!vm jobs that are "running" are actually zombies?

Comment 1

[Justin Wood (:Callek)]

So right now I see all jobs running.

I checked the http://mxr.mozilla.org/build/source/cloud-tools/aws/watch_pending.cfg.template (non template ver) and they are indeed listed as VMs right now.

I also just fixed a bug that faulted reporting for a bit with buildAPI (specifically I noticed that with the wait times report) but it probably affects a few other things.

I'm going to mark this WFM, lets reopen if we see a surprising story again.


I'll note (times PT):
2013-03-23 13:25:19,121 - tst-linux32 - started 23 instances; need 0
2013-03-23 13:26:01,636 - tst-linux64 - started 88 instances; need 0
2013-03-23 13:30:16,876 - tst-linux32 - started 12 instances; need 0
2013-03-23 13:30:37,097 - tst-linux64 - started 28 instances; need 33
2013-03-23 13:35:10,080 - tst-linux64 - started 0 instances; need 11
2013-03-23 13:40:10,099 - tst-linux64 - started 0 instances; need 6
2013-03-23 14:15:23,259 - tst-linux32 - started 28 instances; need 0
...

I'm not really certain what the 'need' column is, perhaps 'additional needed at script run end' And it is possible :rail or someone dived in and fixed this before I got a chance to look.