834222 - Nightly jobs on mozilla-central and mozilla-beta didn't start on Jan 24

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Comment 1

[[e-mail] (:bhearsum)]

The ASan builds have been orange for awhile, but I don't think they would prevent a nightly from being triggered...

Looking through the scheduler master's logs I don't see anything at 3:02am, when the nightly should be started. However, I see a 4 minute gap between 3am and 3:04:
2013-01-24 03:00:10-0800 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: http://hg.mozilla.org/projects/profiling/json-pushes?full=1&fromchange=56dd7a73a6ffa39fdcda62380a3e5703e53517fd> 
2013-01-24 03:04:20-0800 [-] lastBuiltRevision was 6b5016ab9ebbb2ec884c4d865266446adebbc162


Which makes me wonder if we had some time adjustment that happened in that period that screwed things up. Looking into that theory now.

Comment 2

[[e-mail] (:bhearsum)]

Doesn't look like a strong theory, /var/log/messages has stuff around that time:
Jan 24 03:00:35 buildbot-master36 last message repeated 4 times
Jan 24 03:01:42 buildbot-master36 last message repeated 5 times
Jan 24 03:02:48 buildbot-master36 last message repeated 5 times
Jan 24 03:03:55 buildbot-master36 last message repeated 5 times

I don't see anything in any logs about ntp either.

Comment 3

[[e-mail] (:bhearsum)]

Just found this:
2013-01-24 03:04:51-0800 [-] lastGoodRev: took 0.34 seconds to run; returned None
2013-01-24 03:04:51-0800 [-] mozilla-central nightly: No sourcestamp returned from ssfunc; not scheduling a build

So, something in our should-we-make-a-nightly-or-not logic decided that we shouldn't.

Comment 4

[[e-mail] (:bhearsum)]

Based on my reading of the code, it appears that this query returned no revs at all:
https://github.com/mozilla/build-buildbotcustom/blob/master/misc_scheduler.py#L159
https://github.com/mozilla/build-buildbotcustom/blob/master/misc_scheduler.py#L283

It wants revs with green or orange results from the builders in the previous 24h. I'm trying to find out which builders this query runs with.

Comment 5

[[e-mail] (:bhearsum)]

Some printf debugging locally is showing the following builders being passed to lastGoodFunc:
['Android mozilla-central build', 'Android Armv6 mozilla-central build', 'Android Debug mozilla-central build', 'Android no-ionmonkey mozilla-central build', 'Android X86 mozilla-central build', 'Linux mozilla-central build', 'Linux mozilla-central leak test build', 'Linux x86-64 mozilla-central build', 'Linux x86-64 mozilla-central asan build', 'Linux x86-64 mozilla-central debug asan build', 'Linux x86-64 mozilla-central leak test build', 'OS X 10.7 mozilla-central build', 'OS X 10.7 64-bit mozilla-central leak test build', 'WINNT 5.2 mozilla-central build', 'WINNT 5.2 mozilla-central leak test build', 'WINNT 6.1 x86-64 mozilla-central build']

The fact that win64 is in there makes me suspicious of my own debugging though, because it's been burning for quite a few days.

Comment 6

[[e-mail] (:bhearsum)]

I'm not sure what to do from here. I tried poking around the manhole and ended up hanging the scheduler master (though in retrospect, it may have just been running a synchronous query). I wanted to figure out what the lastGoodRev query returned, but I'm not sure how now.

Catlee, any ideas about this?

Comment 7

[Jim Jeffery not reading bug-mail 1/2/11]

Nightly failed again this morning is appears 
Jan 27, 2013

Comment 8

[Jim Jeffery not reading bug-mail 1/2/11]

(In reply to Jim Jeffery not reading bug-mail 1/2/11 from comment #7)
> Nightly failed again this morning is appears 
> Jan 27, 2013

Well, guess it did build, cset: https://hg.mozilla.org/mozilla-central/rev/f18b12139151 
just not where I thought it would, as there were two green PGO builds after that cset

Comment 9

[Bill Gianopoulos [:WG9s]]

(In reply to Ben Hearsum [:bhearsum] from comment #5)
> Some printf debugging locally is showing the following builders being passed
> to lastGoodFunc:
> ['Android mozilla-central build', 'Android Armv6 mozilla-central build',
> 'Android Debug mozilla-central build', 'Android no-ionmonkey mozilla-central
> build', 'Android X86 mozilla-central build', 'Linux mozilla-central build',
> 'Linux mozilla-central leak test build', 'Linux x86-64 mozilla-central
> build', 'Linux x86-64 mozilla-central asan build', 'Linux x86-64
> mozilla-central debug asan build', 'Linux x86-64 mozilla-central leak test
> build', 'OS X 10.7 mozilla-central build', 'OS X 10.7 64-bit mozilla-central
> leak test build', 'WINNT 5.2 mozilla-central build', 'WINNT 5.2
> mozilla-central leak test build', 'WINNT 6.1 x86-64 mozilla-central build']
> 
> The fact that win64 is in there makes me suspicious of my own debugging
> though, because it's been burning for quite a few days.

win64 builds are currently marked as hidden here in tbpl, because we have decided we don't currently care about redness on those.

I would think the correct thing to do is to not be paying attention to any builds in your list above that are currently marked as hidden.  If we don't care to display the build status, then the build status should not be considered in picking a nightly.

Of course, it might already be doing that.

Comment 10

[Bill Gianopoulos [:WG9s]]

The reason I think the lack of builds may have been caused by errors on a hidden build (like win-64) is that looking at the visible builds, I can not come up with any explanation for why the January 27th nightly picked changset f18b12139151 rather than either of the 2 later ones, d802d6faa080 or 47684913d63d.  If it is not based on anything visible, it must be based on something hidden.

Comment 11

[Chris AtLee [:catlee]]

Pretty sure this was fixed by the dependent bug.