1022267 - B2G non-unified builds are falling way behind

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Throughout the day, I've seen multiple instances where pending jobs from the *prior* periodic run haven't started yet before the next periodic run kicks off. Right now, I'm looking at m-c tip where the Windows PGO build has completed and finished running tests, but 5 out of 6 of the periodic B2G non-unified builds requested at the same time are still pending.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

All 5 of the jobs in comment 0 are still pending.

Comment 2

[Rail Aliiev [:rail]]

Looks like we don't start new instances for nonunified builders. I pushed https://hg.mozilla.org/build/cloud-tools/rev/5f550c1b1762.
Keeping an eye on it now.

Comment 3

[Rail Aliiev [:rail]]

Looks better now:

014-06-08 07:22:44,697 - b2g_mozilla-central_emulator-debug_nonunified instance type bld-linux64 slaveset frozenset([u'bld-linux64-spot-131', u'bld-linux64-spot-138', u'bld-linux64-spot-139'])
(what aligns with to http://jacuzzi-allocator.pub.build.mozilla.org/v1/builders/b2g_mozilla-central_emulator-debug_nonunified)

2014-06-08 07:22:51,851 - 0 running for spot bld-linux64 frozenset([u'bld-linux64-spot-131', u'bld-linux64-spot-138', u'bld-linux64-spot-139']) (0 fresh)
2014-06-08 07:23:48,966 - need 1 spot bld-linux64 for slaveset frozenset([u'bld-linux64-spot-131', u'bld-linux64-spot-138', u'bld-linux64-spot-139'])
2014-06-08 07:23:52,824 - Spot request for bld-linux64-spot-131.build.releng.use1.mozilla.com (0.18)
2014-06-08 07:23:54,719 - bld-linux64 - started 1 spot instances for slaveset frozenset([u'bld-linux64-spot-131', u'bld-linux64-spot-138', u'bld-linux64-spot-139']); need 0

Comment 4

[Rail Aliiev [:rail]]

https://tbpl.mozilla.org/?jobname=.*_nonunified looks much better now, at least no pending.

Comment 5

[Rail Aliiev [:rail]]

I see that we started some jacuzzied instances for m-i non-unified builds as well

Comment 6

[Rail Aliiev [:rail]]

I just checked the only pending non-unified build on m-i. All of the slaves in http://jacuzzi-allocator.pub.build.mozilla.org/v1/builders/b2g_mozilla-inbound_emulator-jb-debug_nonunified are up and doing something. Sounds expected.

Comment 7

[Rail Aliiev [:rail]]

Same here: http://jacuzzi-allocator.pub.build.mozilla.org/v1/builders/b2g_b2g-inbound_emulator-kk-debug_nonunified

Comment 8

[Rail Aliiev [:rail]]

Also added flame builds in https://hg.mozilla.org/build/cloud-tools/rev/6e474160aa7b

Comment 9

[Rail Aliiev [:rail]]

B2g-Inbound has no pending builds now...

Comment 10

[Rail Aliiev [:rail]]

Same for m-i. The issue should be fixed now. I'll keep the bug open till tomorrow to make sure everything is OK.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Thanks for the quick work, Rail. Much appreciated :)

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

How can we track this kind of problem more generally ? In theory it should be possible to take the builder list from allthethings.json, determine which jobs run on AWS slaves (at least in part), and check the buildermap handles them. Possibly running on jenkins.

Comment 13

[Rail Aliiev [:rail]]

(In reply to Nick Thomas [:nthomas] from comment #12)
> How can we track this kind of problem more generally ? In theory it should
> be possible to take the builder list from allthethings.json, determine which
> jobs run on AWS slaves (at least in part), and check the buildermap handles
> them. Possibly running on jenkins.

I think we can use allthethings.json to build the map dynamically. I'll file a follow up bug.