seems somehow b2g-ics opt builds fail to start and are still pending also after a long time example url https://tbpl.mozilla.org/?tree=B2g-Inbound&rev=3314ca6fc2a5 Seems b2g-i specific since other trees build ics opt builds
(In reply to Carsten Book [:Tomcat] from comment #0) > Seems b2g-i specific since other trees build ics opt builds Don't forget release trees etc have a higher priority.
I checked aws pending logs - all looked fine there. Currently trying to establish if: a) No masters are picking up these jobs or b) Masters can pick them up, but no slaves are available e.g. this has been waiting over 6 hours: https://secure.pub.build.mozilla.org/buildapi/self-serve/b2g-inbound/request/41985526
Simone is going to run dump master script to see which masters should be able to take this job: "b2g_b2g-inbound_emulator_dep"
I'm also looking in the production scheduler database to see if I can see whether any masters have been able to claim any jobs, or if they haven't been claiming jobs for a while etc. I haven't managed to find these particular jobs in the database so far, due to my limited knowledge of the schema.
I'll take a look at reconfig history on the Maintenance wiki page to see if I can see any recent changes that might have caused impact.
We've established there are 12 buildbot masters that can take this job: http://buildbot-master70.srv.releng.use1.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep http://buildbot-master71.srv.releng.use1.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep http://buildbot-master72.srv.releng.usw2.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep http://buildbot-master73.srv.releng.usw2.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep http://buildbot-master74.srv.releng.usw2.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep http://buildbot-master77.srv.releng.use1.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep http://buildbot-master82.srv.releng.scl3.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep http://buildbot-master84.srv.releng.scl3.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep http://buildbot-master85.srv.releng.scl3.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep http://buildbot-master86.srv.releng.scl3.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep http://buildbot-master91.srv.releng.usw2.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep http://buildbot-master94.srv.releng.use1.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep
It is currently not clear why none of the buildbot masters have taken these jobs, when they have connected slaves, and the jobs have been sat there a very long time in pending state (8h54m at the moment).
So at the off chance my theory was right (based on all the above) I looked at: http://jacuzzi-allocator.pub.build.mozilla.org/v1/builders/b2g_b2g-inbound_emulator_dep And its assigned ONLY one spot instance.... can't find docs on how to adjust, SMS'd catlee to come by and help depsite canada holiday
n-i on this bug too, since unsure how long the SMS will take to deliver
[11:47:52] Callek_Buildduty |2014-05-19 08:35:33,040 - bld-linux64 - started 1 spot instances for slaveset frozenset([u'bld-linux64-spot-107']); need 3| [11:48:19] Callek_Buildduty says its not even attached, this oughta be interesting [11:48:20] Callek_Buildduty https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=bld-linux64-spot-107 AWS console shows "terminated" for this spot instance when I search for it in use1...
...catlee came around and updated the config.json in https://github.com/bhearsum/static-jacuzzis/blob/master/config.json to have us allocate it against more spot instances. Still unsure whats up with -107, but thats less urgent. And came to light that I don't have access to that repo. Proper deployment was ongoing last week however the master issues took precedent and the deployment slipped down priority tree a bit. Things look good enough that the immediately blocking issue for tree-closure should be clear. I'll hold resolution of this bug on verifying that everything clears up.
Reopened b2g-inbound at 2014-05-19T11:15:32 since ICS builds seem to be building again.