The default bug view has changed. See this FAQ.

B2g-inbound tree closed due to pending B2G ICS Emulator Opt builds

RESOLVED FIXED

Status

Release Engineering
Buildduty
--
blocker
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: Tomcat, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
seems somehow b2g-ics opt builds fail to start and are still pending also after a long time example url https://tbpl.mozilla.org/?tree=B2g-Inbound&rev=3314ca6fc2a5

Seems b2g-i specific since other trees build ics opt builds
http://builddata.pub.build.mozilla.org/reports/pending/pending.html
http://builddata.pub.build.mozilla.org/reports/pending/running.html
(In reply to Carsten Book [:Tomcat] from comment #0)
> Seems b2g-i specific since other trees build ics opt builds

Don't forget release trees etc have a higher priority.
I checked aws pending logs - all looked fine there.

Currently trying to establish if:

a) No masters are picking up these jobs or
b) Masters can pick them up, but no slaves are available

e.g. this has been waiting over 6 hours:
https://secure.pub.build.mozilla.org/buildapi/self-serve/b2g-inbound/request/41985526
Simone is going to run dump master script to see which masters should be able to take this job:
"b2g_b2g-inbound_emulator_dep"
I'm also looking in the production scheduler database to see if I can see whether any masters have been able to claim any jobs, or if they haven't been claiming jobs for a while etc.

I haven't managed to find these particular jobs in the database so far, due to my limited knowledge of the schema.
I'll take a look at reconfig history on the Maintenance wiki page to see if I can see any recent changes that might have caused impact.
(Reporter)

Updated

3 years ago
Summary: B2g-inbound tree closed tue to pending B2G ICS Emulator Opt builds → B2g-inbound tree closed due to pending B2G ICS Emulator Opt builds
We've established there are 12 buildbot masters that can take this job:

http://buildbot-master70.srv.releng.use1.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep
http://buildbot-master71.srv.releng.use1.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep
http://buildbot-master72.srv.releng.usw2.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep
http://buildbot-master73.srv.releng.usw2.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep
http://buildbot-master74.srv.releng.usw2.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep
http://buildbot-master77.srv.releng.use1.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep
http://buildbot-master82.srv.releng.scl3.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep
http://buildbot-master84.srv.releng.scl3.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep
http://buildbot-master85.srv.releng.scl3.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep
http://buildbot-master86.srv.releng.scl3.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep
http://buildbot-master91.srv.releng.usw2.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep
http://buildbot-master94.srv.releng.use1.mozilla.com:8001/builders/b2g_b2g-inbound_emulator_dep
It is currently not clear why none of the buildbot masters have taken these jobs, when they have connected slaves, and the jobs have been sat there a very long time in pending state (8h54m at the moment).
So at the off chance my theory was right (based on all the above) I looked at:

	http://jacuzzi-allocator.pub.build.mozilla.org/v1/builders/b2g_b2g-inbound_emulator_dep

And its assigned ONLY one spot instance.... can't find docs on how to adjust, SMS'd catlee to come by and help depsite canada holiday
n-i on this bug too, since unsure how long the SMS will take to deliver
Flags: needinfo?(catlee)
[11:47:52]	Callek_Buildduty	|2014-05-19 08:35:33,040 - bld-linux64 - started 1 spot instances for slaveset frozenset([u'bld-linux64-spot-107']); need 3|
[11:48:19]	Callek_Buildduty	says its not even attached, this oughta be interesting
[11:48:20]	Callek_Buildduty	https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=bld-linux64-spot-107

AWS console shows "terminated" for this spot instance when I search for it in use1...
...catlee came around and updated the config.json in https://github.com/bhearsum/static-jacuzzis/blob/master/config.json to have us allocate it against more spot instances.

Still unsure whats up with -107, but thats less urgent. And came to light that I don't have access to that repo. Proper deployment was ongoing last week however the master issues took precedent and the deployment slipped down priority tree a bit.

Things look good enough that the immediately blocking issue for tree-closure should be clear. I'll hold resolution of this bug on verifying that everything clears up.
Flags: needinfo?(catlee)
Reopened b2g-inbound at  2014-05-19T11:15:32 since ICS builds seem to be building again.

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.