482123 - No 20090308 nightlies for all mercurial-based builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Peter van der Woude [:Peter6]]

nighly builds just haven't started (yet ?), right now 3 hours late.

does this have to do with the change from PST to PDT ?

Comment 1

[Bill Gianopoulos [:WG9s]]

It would seem this is most likely the cause as the nightly builds that did not run are the ones that normally start between 2AM and 3AM.

Comment 2

[WildcatRay]

Not sure if this applicable or appropriate, but the same seems to apply to the Shiretoko nightly, too.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #0)
> does this have to do with the change from PST to PDT ?

Seems likely. Nightlies are triggered at 2am California local time, just when the daylight savings jump occurs.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Buildbot log goes directly from "2009-03-08 01:59:50-0800" to "2009-03-08 03:00:05-0700" (had to go all the way to twisted.log.77!! l10n sure is verbose)

I've manually triggered builds for all platforms on mozilla-central / mozilla-1.9.1 / tracemonkey / mobile to fill in the gap. 

Leaving this open to adjust the build hour
 http://mxr.mozilla.org/build/source/buildbot-configs/mozilla2/master.cfg#164

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

We should handle the clock going both forward and back, which means these two transitions:
 Sunday, March 8, 2009 at 1:59:59 AM  (PST)
 Sunday, March 8, 2009 at 3:00:00 AM  (PDT)
 ------------------------------------------------
 Sunday, November 1, 2009 at 1:59:59 AM	 (PDT)
 Sunday, November 1, 2009 at 1:00:00 AM	 (PST)

So we can trigger builds 
* a little earlier, say 1:55am, and decide not to mind bogus elapsed times twice a year. I wonder if buildbot would handle negative elapsed times properly when a step starts before 2am on November 1 and finishes after 1am
* an hour later, say 3:05am, and avoid the jumps completely. Ben, do you recall why you set 2am in the scheduler ? I think we used 3 or 4 am in tinderbox land but had fewer people in Canada then.

Comment 6

[Bill Gianopoulos [:WG9s]]

(In reply to comment #5)
> We should handle the clock going both forward and back, which means these two
> transitions:
>  Sunday, March 8, 2009 at 1:59:59 AM  (PST)
>  Sunday, March 8, 2009 at 3:00:00 AM  (PDT)
>  ------------------------------------------------
>  Sunday, November 1, 2009 at 1:59:59 AM     (PDT)
>  Sunday, November 1, 2009 at 1:00:00 AM     (PST)
> 
> So we can trigger builds 
> * a little earlier, say 1:55am, and decide not to mind bogus elapsed times
> twice a year. I wonder if buildbot would handle negative elapsed times properly

I think the bigger issue would be to avoid having the build triggered twice in the fall since 1:55 comes 2 times on the same day in the fall.

> when a step starts before 2am on November 1 and finishes after 1am
> * an hour later, say 3:05am, and avoid the jumps completely. Ben, do you recall
> why you set 2am in the scheduler ? I think we used 3 or 4 am in tinderbox land
> but had fewer people in Canada then.


I suspect this might be because the Windows builds taking longer due to PGO.

Comment 7

[Bill Gianopoulos [:WG9s]]

I think the real answer here is to change to using UTC for builds and just not do Daylight Savings at all.

The current situation really does not work for developers not in the US trying to avoid doing check-ins near when the nightlies start.  It is just way too confusing as to what times nightly builds will start in your timezone because of different countries doing Daylight savings on different days.  The worst case is for Australia where the change goes by 2 hours becuase they are on Daylight savings when the US is not and vice-versa.

If the builds are just done on UTC then you only need to figure out your own offset instead of worrying about the Mozilla time offset and your local offset.

Just my $0.02.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #6)
> I think the bigger issue would be to avoid having the build triggered twice in
> the fall since 1:55 comes 2 times on the same day in the fall.

Ah yes, that's a good point. We'd have to trigger before 1am or after 3am.

Comment 9

[Bill Gianopoulos [:WG9s]]

Wow about adding a second scheduled build for 3AM on the 2nd Sunday in March in addition to the 2AM every day build?  Hat would seem to fix the missing build issue while avoiding the duplicate build possibility.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Move nightlies to 3:02am

Start hourlies an hour later to dodge the time-jumps, and allow a couple of minutes for the system to update the time.

Comment 11

[Brian Carpenter [:geeknik]]

Branch build is out.

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1b4pre) Gecko/20090308 Shiretoko/3.1b4pre

Comment 12

[-fullmetaljacket-]

trunk build is out [win]

Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2a1pre) Gecko/20090308 Minefield/3.2a1pre ID:20090308162952

Comment 13

[Chris AtLee [:catlee]]

Sounds like this could be a buildbot bug as well?

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #13)
Assuming the scheduler is just polling the current time it might be quite hard to deal with that changing instantaneously. Did you have thoughts about how buildbot could behave better ?

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Master reconfig'd.