527447 - some unittests burn every Monday morning because of the NightlyRebuild scheduler

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

They kicked off a couple of runs early this morning, but because there were no builds over the weekend they attempted to test builds that had already been cleaned up.

With the death of the Idle builds we're going to have to be a bit smarter here.

Comment 1

[Chris Cooper [:coop] (he/him)]

(In reply to comment #0)
> With the death of the Idle builds we're going to have to be a bit smarter here.

Ben: what version of 'smarter' did you have in mind? 

Are you talking about bringing back idle builds? Running tests only when builds are found? Trying to run tests regardless but not failing outright when builds aren't found?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Force a unit test build as the first step ?

Comment 3

[[e-mail] (:bhearsum)]

(In reply to comment #1)
> (In reply to comment #0)
> > With the death of the Idle builds we're going to have to be a bit smarter here.
> 
> Ben: what version of 'smarter' did you have in mind? 
> 
> Are you talking about bringing back idle builds? Running tests only when builds
> are found? Trying to run tests regardless but not failing outright when builds
> aren't found?

I mean that we're going to have to be smarter when trying to run these tests. If a branch hasn't done a dep build in a few days we have to either a) bail, or b) pull a nightly instead.

We could make this less painful in the meantime by moving the NightlyRebuild Scheduler to a day not preceded by a weekend - that'd lower our chances of hitting this state.

Comment 4

[[e-mail] (:bhearsum)]

I'm going to switch these to happen on a different day to reduce the frequency of this.

Comment 5

[[e-mail] (:bhearsum)]

Attachment: do nightly builds on friday rather than monday

This doesn't fix the root problem but there's a higher chance of us having
recent builds later in the week than there is on a Monday morning.

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 412686 [details] [diff] [review]
do nightly builds on friday rather than monday

changeset:   500:e686b3cdeaa0

Comment 7

[[e-mail] (:bhearsum)]

The workaround is in, back to the future for this.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

catlee is in the process of disabling these tests again (in a bug which is currently hiding from me).

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Fixed by bug 488732.

Comment 10

[Chris Cooper [:coop] (he/him)]

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.