Closed Bug 1012633 Opened 10 years ago Closed 10 years ago

B2g-inbound tree closed due to pending B2G ICS Emulator Opt builds

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Platform:
x86
Gonk (Firefox OS)
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cbook, Unassigned)

Details

seems somehow b2g-ics opt builds fail to start and are still pending also after a long time example url https://tbpl.mozilla.org/?tree=B2g-Inbound&rev=3314ca6fc2a5

Seems b2g-i specific since other trees build ics opt builds
(In reply to Carsten Book [:Tomcat] from comment #0)
> Seems b2g-i specific since other trees build ics opt builds

Don't forget release trees etc have a higher priority.
I checked aws pending logs - all looked fine there.

Currently trying to establish if:

a) No masters are picking up these jobs or
b) Masters can pick them up, but no slaves are available

e.g. this has been waiting over 6 hours:
https://secure.pub.build.mozilla.org/buildapi/self-serve/b2g-inbound/request/41985526
Simone is going to run dump master script to see which masters should be able to take this job:
"b2g_b2g-inbound_emulator_dep"
I'm also looking in the production scheduler database to see if I can see whether any masters have been able to claim any jobs, or if they haven't been claiming jobs for a while etc.

I haven't managed to find these particular jobs in the database so far, due to my limited knowledge of the schema.
I'll take a look at reconfig history on the Maintenance wiki page to see if I can see any recent changes that might have caused impact.
Summary: B2g-inbound tree closed tue to pending B2G ICS Emulator Opt builds → B2g-inbound tree closed due to pending B2G ICS Emulator Opt builds
It is currently not clear why none of the buildbot masters have taken these jobs, when they have connected slaves, and the jobs have been sat there a very long time in pending state (8h54m at the moment).
So at the off chance my theory was right (based on all the above) I looked at:

	http://jacuzzi-allocator.pub.build.mozilla.org/v1/builders/b2g_b2g-inbound_emulator_dep

And its assigned ONLY one spot instance.... can't find docs on how to adjust, SMS'd catlee to come by and help depsite canada holiday
n-i on this bug too, since unsure how long the SMS will take to deliver
Flags: needinfo?(catlee)
[11:47:52]	Callek_Buildduty	|2014-05-19 08:35:33,040 - bld-linux64 - started 1 spot instances for slaveset frozenset([u'bld-linux64-spot-107']); need 3|
[11:48:19]	Callek_Buildduty	says its not even attached, this oughta be interesting
[11:48:20]	Callek_Buildduty	https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=bld-linux64-spot-107

AWS console shows "terminated" for this spot instance when I search for it in use1...
...catlee came around and updated the config.json in https://github.com/bhearsum/static-jacuzzis/blob/master/config.json to have us allocate it against more spot instances.

Still unsure whats up with -107, but thats less urgent. And came to light that I don't have access to that repo. Proper deployment was ongoing last week however the master issues took precedent and the deployment slipped down priority tree a bit.

Things look good enough that the immediately blocking issue for tree-closure should be clear. I'll hold resolution of this bug on verifying that everything clears up.
Flags: needinfo?(catlee)
Reopened b2g-inbound at  2014-05-19T11:15:32 since ICS builds seem to be building again.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.