641260 - No Mobile on-change build machines are on Tinderbox

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Mark Finkle (:mfinkle) (use needinfo?)]

Isee nightly build machines, but no on-change build machines are listed. We have checked in code, but we are not getting any build - and more importantly, no Talos runs.

Comment 1

[Mark Finkle (:mfinkle) (use needinfo?)]

Making a blocker cause we plan to spin RC builds on Monday and need the Talos numbers.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Could you give the names of the builders that are missing, and the tree you normally see them on ? 

Looking at http://tbpl.mozilla.org/?tree=Mobile, I see on-change builds for 
* Linux Mobile Desktop mozilla-central build
* Android R7 mozilla-central build
* Maemo 5 GTK mozilla-central build
* Maemo 5 QT mozilla-central build
on the current tip of m-c (d8fe8514d7e6). That looks to be the set going back several days. There are also nightly builds for the same + Mac/win32 destkop + thumbless Android.

It's possible bug 639980 regressed something, even if the patch looks mostly additive.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Are you referring to the lack of builds from these changes ?
 http://hg.mozilla.org/mobile-browser/pushloghtml?fromchange=907bc2950d31&tochange=9b20fa85a5cb

Comment 4

[Matt Brubeck (:mbrubeck)]

(In reply to comment #3)
> Are you referring to the lack of builds from these changes ?
> 
> http://hg.mozilla.org/mobile-browser/pushloghtml?fromchange=907bc2950d31&tochange=9b20fa85a5cb

Yes.  ""Maemo 5 GTK mozilla-central build" is the one we care about most for Talos numbers.  It looks like it hasn't built anything for mobile-browser changes since last night.  Is that because there are no mozilla-central changes since then?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

IIRC we are polling both mozilla-central and mozilla-browser and triggering jobs from changes in both.

From pm02:8010's log:
2011-03-12 07:27:57-0800 [-] Starting factory <HTTPClientFactory: http://hg.mozilla.org/mobile-browser/pushlog?fromchange=907bc2950d318772ee7321b8989a26db23216ee0>
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] adding change, who [e-mail], 1 files, rev=3869704a33c75ee4664a0c0dbda7bf09a8d2fa4c, branch=mobile-browser, comments http://hg.mozilla.org/mobile-browser/rev/3869704a33c75ee4664a0c0dbda7bf09a8d2fa4c, category None
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] <Scheduler 'linux_l10n_verification' at 171737804> ignoring off-branch <buildbot.changes.changes.Change instance at 0x127983cc>
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] <Scheduler 'win32_l10n_verification' at 171738348> ignoring off-branch <buildbot.changes.changes.Change instance at 0x127983cc>
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] <Scheduler 'macosx_l10n_verification' at 171738060> ignoring off-branch <buildbot.changes.changes.Change instance at 0x127983cc>
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] <Scheduler 'updates' at 171683596> ignoring off-branch <buildbot.changes.changes.Change instance at 0x127983cc>
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] Nightly Scheduler <Linux Fennec Desktop mozilla-1.9.2 nightly>: no add change
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] Nightly Scheduler <Win32 Fennec Desktop mozilla-1.9.2 nightly>: no add change
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] Nightly Scheduler <OS X Fennec Desktop mozilla-1.9.2 nightly>: no add change
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] <Scheduler 'tag' at 348494636> ignoring off-branch <buildbot.changes.changes.Change instance at 0x127983cc>
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] <Scheduler 'mobile_tag' at 218236620>: change is not important, adding <buildbot.changes.changes.Change instance at 0x127983cc>
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] Nightly Scheduler <Maemo 5 GTK mozilla-1.9.2 nightly>: no add change
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] Nightly Scheduler <Maemo mozilla-1.9.2 nightly>: no add change
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] <Scheduler 'mobile mobile-1.9.2' at 227556780> ignoring off-branch <buildbot.changes.changes.Change instance at 0x127983cc>
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] <Scheduler 'mobile mobile-trunk' at 217395948> ignoring off-branch <buildbot.changes.changes.Change instance at 0x127983cc>
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] <Scheduler 'mobile-browser' at 321678284>: change is important, adding <buildbot.changes.changes.Change instance at 0x127983cc>
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] <Scheduler 'mobile-browser' at 321678284>: setting timer to 07:30:50
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] pruning 1 changes
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] last changeset 3869704a33c75ee4664a0c0dbda7bf09a8d2fa4c on http://hg.mozilla.org/mobile-browser
2011-03-12 07:27:58-0800 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: http://hg.mozilla.org/mobile-browser/pushlog?fromchange=907bc2950d318772ee7321b8989a26db23216ee0>
2011-03-12 07:28:57-0800 [-] Polling Hg server at http://hg.mozilla.org/mobile-browser/pushlog?fromchange=3869704a33c75ee4664a0c0dbda7bf09a8d2fa4c

ie it notices 3869704a33c7, adds a change, which lots of unrelated schedulers ignore (OK), the mobile-browser scheduler thinks it is important, then we poll for changes since 3869704a33c7. So, something not starting builds based on the change ?

Aki, you know how this stuff is put together much better than I do, and what should appear on tbpl?Tree=Mobile. Could you take a look ?

Fired off a build on Maemo 5 GTK mozilla-central build, using tips of m-c + m-b, as an interim measure. Dunno if that will appear on tbpl or not.

Comment 6

[Mark Finkle (:mfinkle) (use needinfo?)]

I see the Maemo 5 GTK build starting on tinderbox (http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mobile)

Comment 7

[Matt Brubeck (:mbrubeck)]

(In reply to comment #5)
> Fired off a build on Maemo 5 GTK mozilla-central build, using tips of m-c +
> m-b, as an interim measure. Dunno if that will appear on tbpl or not.

Thanks. Can we get builds for 3869704a33c7 too?  (I'd like to verify that that changeset fixed a performance issue.)

Comment 8

[Aki Sasaki (not active)]

All nightly/depend builds are on buildbot 0.8 now, so comment 5 is looking in the wrong place.  I'll look to see if the scheduler master is picking up changes from mobile-browser.

Comment 9

[Aki Sasaki (not active)]

(In reply to comment #7)
> (In reply to comment #5)
> > Fired off a build on Maemo 5 GTK mozilla-central build, using tips of m-c +
> > m-b, as an interim measure. Dunno if that will appear on tbpl or not.
> 
> Thanks. Can we get builds for 3869704a33c7 too?  (I'd like to verify that that
> changeset fixed a performance issue.)

This is going to be difficult, as a) depend builds don't have a way for me to force a revision, afaik, and b) nightly builds most likely allow me to force a mozilla-central revision but not a mobile-browser revision. (Maybe I should add this to https://wiki.mozilla.org/User:Asasaki:MBMerge ; this bug would not have surfaced at all had mobile-browser been a part of mozilla-central.)

Comment 10

[Aki Sasaki (not active)]

We think we've never had mobile-browser changes trigger 0.8 nightlies/depend builds.  It's only noticeable now because a) all nightlies/depend builds are now on 0.8, and b) there are no pushes to mozilla-central.

The quick fix is to back out http://hg.mozilla.org/build/buildbot-configs/rev/cfbe699f4ff6, which means I'm turning 0.7 maemo m-c builds back on.

The longer term fix will be to revisit our mobile-browser polling to make sure that works properly.

The best long term fix is to merge mobile-browser in.

Comment 11

[Aki Sasaki (not active)]

(In reply to comment #7)
> Can we get builds for 3869704a33c7 too?  (I'd like to verify that that
> changeset fixed a performance issue.)

To do this,

1) I changed the mobileRevision at
http://hg.mozilla.org/build/buildbotcustom/file/c8d37e44217e/process/factory.py#l5397

to be 3869704a33c7 instead of default.

2) reconfig
3) kick off maemo5 gtk build
4) revert
5) reconfig

Comment 12

[Aki Sasaki (not active)]

Filed bug 641276 for 0.8 scheduler fixes.

Comment 13

[Matt Brubeck (:mbrubeck)]

Fixed by backout in http://hg.mozilla.org/build/buildbot-configs/rev/eb610dd9d191 - bug 641276 is tracking the medium-term fix, and I agree that moving to a single repo is a good long-term fix.