998035 - emergency change to b2g midnight PT start times

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[hwine]

To avoid the large number of b2g jobs starting at midnight PT, which is strongly correlated to the memory issues, stagger the start times.

Approved by QA:
On 2014-04-17 14:32 , Jason Smith wrote:
> Looks good to me. This is good to put into production.
>
> ----- Original Message -----
>> From: "Hal Wine" 
>> To: "Tony Chung", "Jason Smith"
>> Cc: "Aki Sasaki", "Laura Thomson"
>> Sent: Thursday, April 17, 2014 2:22:49 PM
>> Subject: URGENT FYI: Emergency nightly b2g run time changes being made TODAY!
>>
>> tl;dr: we are experiencing nightly tree closures due to load on git.m.o
>> While we find a long term solution, we'd like to spread out the start
>> time of nightly b2g builds to minimize the chance of a failed build when
>> sheriffs aren't around. (Especially with Easter weekend - there is no
>> sheriff coverage tonight, afaik.)
>>
>> This only occurs with the jobs starting at midnight PT, those are:
>> 1051:BRANCHES['mozilla-aurora']['start_hour'] = [0, 16]
>> 1079:BRANCHES['mozilla-b2g28_v1_3t']['start_hour'] = [0]
>> 1094:BRANCHES['mozilla-b2g28_v1_3']['start_hour'] = [0, 16]
>> 1142:BRANCHES['mozilla-b2g26_v1_2']['start_hour'] = [0, 16]
>>
>> When the system overloads, it takes about 90 minutes of manual work and
>> waiting to clear to the point where trees can be reopened. I'm hoping
>> that spreading start times by one hour will be sufficient to never cause
>> the overload.
>>
>> Based on the priorities we've received from relman, we'd like to shift
>> the midnight runs as follows:
>> 1051:BRANCHES['mozilla-aurora']['start_hour'] = [0, 16]
>> 1079:BRANCHES['mozilla-b2g28_v1_3t']['start_hour'] = [1]
>> 1094:BRANCHES['mozilla-b2g28_v1_3']['start_hour'] = [2, 16]
>> 1142:BRANCHES['mozilla-b2g26_v1_2']['start_hour'] = [3, 16]
>>
>> If another order would be better for when QA expects to see the
>> binaries, please let me know ASAP.
>>
>> Thanks for your help through this challenge,
>> --Hal
>>

Comment 1

[hwine]

Attachment: b2g_changes.diff

As approved in concept.

Comment 2

[Jordan Lund (:jlund)]

in production :)

Comment 3

[hwine]

Comment on attachment 8408575 [details] [diff] [review]
b2g_changes.diff

http://hg.mozilla.org/build/buildbot-configs/rev/c8620b84d544

Comment 4

[hwine]

Checking on QA whether to revert these changes or make permanent. If permanent, the 0200 & 0300 ones need to be checked and/or moved to ensure they behave properly at change of daylight savings time.

Comment 5

[hwine]

This is too old to be relevant