899407 - change timing of some b2g nightlies

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Spinning out from bug#899170.

...
A request was made by tchung:

can they change the build kickoff times of 1.2 unagi master/M-C builds to closer to 7am PST?   The current priority for testing would be 1.1 leos and 1.2 unagis.
...

As priorities have changed for QA recently, they'd now prefer to get the 1.2 unagi nightly builds first , then 1.1 leo nightlies, then 1.1hd helix nightlies... in that order... and then any/all other b2g device builds.

If its easier for automation to schedule these builds by branch, instead of by device-on-branch, then please change scheduling to do 1.2, 1.1, 1.1hd nightlies before nightly for any other b2g branches.

Comment 1

[John Hopkins (:jhopkins)]

Attachment: [buildbot-configs] change nightly start times

The updated nightly start times amount to:

ionmonkey: 7:42
mozilla-b2g18: 7:12 23:12
mozilla-b2g18_v1_0_1: 7:32 23:32
mozilla-b2g18_v1_1_0_hd: 7:22 23:22
mozilla-central: 7:2
oak: 7:42

Comment 2

[Chris AtLee [:catlee]]

Comment on attachment 783262 [details] [diff] [review]
[buildbot-configs] change nightly start times

Review of attachment 783262 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 3

[John Hopkins (:jhopkins)]

Comment on attachment 783262 [details] [diff] [review]
[buildbot-configs] change nightly start times

https://hg.mozilla.org/build/buildbot-configs/rev/2a4b34286e9c

Comment 4

[hwine]

in production, closing

Comment 5

[Zac C (:zac)]

I wish I had known about this sooner. This is inconvenient for us running gaia-ui-tests automation. It means we can't get a (2 hour) test run in before 9am PST.

I will talk to tchung about this directly.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Zac C (:zac) from comment #0)
> Bug #899407 changed the m-c nightly builds to release around 9am PST.
> 
> This reduces the ability for EU staff and particularly the UI smoke test
> automation to be run before the PST/manual test team start work. (the test
> run takes 2 hrs, +result analysis and filing bugs).
Urgh. Sorry Zac, didnt realize that time-change caused headaches for you and other EU-based folks. Reopening, after reading bug#901020, as this has context for why change of when-to-start-nightly-build was made, so is best place to figure out what time does work for everyone.

tchung, zac: what time of day would be ideal for each of you?



> Twice-daily builds, perhaps staggered at 12 hour intervals, would allow the
> automation to have an impact on a fresh build without doubling up work with
> the manual test team.

We can generate "nightly" builds twice a day if that helps. However, I note that we *used* to generate nightly builds twice a day in the rush-to-v1.0-shipping, and were asked to stop doing that because they caused a bunch of confusion when regression hunting problems with nightlies. 

Is there enough volume of checkins that you believe its worthwhile generating two nightlies-each-night? I suspect that the root of the problem could be solved by finding a time-to-generate-nightly that gives EU people, and also PDT people, enough time to test the same nightly.

Comment 8

[Zac C (:zac)]

Np, not your fault John!

I did speak to Tony via email before filing bug'901020 but as he's out of the office he may not have given it clear thought. 

It's not urgent so we can defer this to next week when we can settle on a time or solution.

Comment 9

[Tony Chung [:tchung]]

(In reply to John O'Duinn [:joduinn] from comment #6)
> (In reply to Zac C (:zac) from comment #0)
> > Bug #899407 changed the m-c nightly builds to release around 9am PST.
> > 
> > This reduces the ability for EU staff and particularly the UI smoke test
> > automation to be run before the PST/manual test team start work. (the test
> > run takes 2 hrs, +result analysis and filing bugs).
> Urgh. Sorry Zac, didnt realize that time-change caused headaches for you and
> other EU-based folks. Reopening, after reading bug#901020, as this has
> context for why change of when-to-start-nightly-build was made, so is best
> place to figure out what time does work for everyone.
> 
> tchung, zac: what time of day would be ideal for each of you?
> 
> 
> 
> > Twice-daily builds, perhaps staggered at 12 hour intervals, would allow the
> > automation to have an impact on a fresh build without doubling up work with
> > the manual test team.
> 
> We can generate "nightly" builds twice a day if that helps. However, I note
> that we *used* to generate nightly builds twice a day in the
> rush-to-v1.0-shipping, and were asked to stop doing that because they caused
> a bunch of confusion when regression hunting problems with nightlies. 
> 
> Is there enough volume of checkins that you believe its worthwhile
> generating two nightlies-each-night? I suspect that the root of the problem
> could be solved by finding a time-to-generate-nightly that gives EU people,
> and also PDT people, enough time to test the same nightly.

Hey guys, i assume i'm being summon'd to comment here.  I only asked Zac to request twice-a-days because we could use the automation testrun in zacs timezone.  we want his test results to execute around the time we wake up in PST.   however, we can't afford to have him execute tests on an outdated build in his timezone. 

If the ask from releng here is to stick with one build, how about we compromise and generate one build around 4am PST?   This would close to zac's end of day, and still late enough to pick up any checkins that Europe devs make before we wake up on the west coast.   Would that be a fair compromise to everyone?

Comment 10

[Zac C (:zac)]

4am PST is fine by me, but it is setting it back to just what it was before!

It is still later than the v1-train build and thus picking up more commits.

Comment 11

[John Hopkins (:jhopkins)]

Can you please let me know what the final decision is?

Comment 12

[Zac C (:zac)]

Sorry John this got buried under 100+ bugmail last night!

I think the previous two comments are agreeance upon 4am PST.

Comment 13

[John Hopkins (:jhopkins)]

Attachment: [buildbot-configs] start b2g nightlies after 4am PT

zac - can you confirm whether this captures what was discussed?  This patch switches all b2g nightlies to start after 4am PT and switches from 2/day to 1/day nightlies (where applicable).

Comment 14

[Zac C (:zac)]

Comment on attachment 786951 [details] [diff] [review]
[buildbot-configs] start b2g nightlies after 4am PT

Thanks, the start hour is good.

Comment 15

[John Hopkins (:jhopkins)]

Comment on attachment 786951 [details] [diff] [review]
[buildbot-configs] start b2g nightlies after 4am PT

https://hg.mozilla.org/build/buildbot-configs/rev/4e6661b1e631

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

In production

Comment 17

[Ryan VanderMeulen [:RyanVM]]

Whatever's being done here was clearly insufficient if B2G QA feels the need to trigger nightlies through self-serve every day, wasting a huge amount of build and test resources along the way.

Comment 18

[Zac C (:zac)]

new builds are triggered when bad regressions slip in :(