904063 - Jetpack test intermittent failure rate too high in automation

Status: RESOLVED FIXED

(Add-on SDK Graveyard :: General, defect, P1)

Description

[Ed Morley [:emorley]]

The intermittent failure rate for jetpack is extremely high in automation, and as such is risking the removal of jetpack from the default TBPL view ().

Please can someone drive the dependant bugs forwards? Also, each time we've had a jetpack uplift of late, it's seemed that it's brought with it as many new intermittent failures as it fixes of the old ones.. please can we try to minimise the new regressions?

For the frequency counts of the intermittent failures for the last week, see:
http://brasstacks.mozilla.com/orangefactor/?display=OrangeFactor&includefiltertype=quicksearch&includefilterdetailsexcludeResolved=false&includefilterdetailsexcludeDisabled=false&includefilterdetailsquicksearch=&includefilterdetailsnumbugs=0&includefilterdetailsresolvedIds=&excludefiltertype=quicksearch&excludefilterdetailsquicksearch=&excludefilterdetailsnumbugs=0&excludefilterdetailsresolvedIds=&plat=All&test=jetpack&type=All&startday=2013-08-06&endday=2013-08-12&tree=trunk

Many thanks :-)

Comment 1

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley UTC+1] from comment #0)
> The intermittent failure rate for jetpack is extremely high in automation,
> and as such is risking the removal of jetpack from the default TBPL view ().

Missing link:
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#7.29_Low_intermittent_failure_rate

Comment 2

[Dave Townsend [:mossop]]

(In reply to Ed Morley [:edmorley UTC+1] from comment #0)
> Please can someone drive the dependant bugs forwards? Also, each time we've
> had a jetpack uplift of late, it's seemed that it's brought with it as many
> new intermittent failures as it fixes of the old ones.. please can we try to
> minimise the new regressions?

The biggest problem that we have is that there are very few test runs on our own tree so intermittent tests don't tend to show up there ahead of time. Is there anything else we can be doing in this regard?

Comment 3

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

(In reply to Dave Townsend (:Mossop) from comment #2)
> (In reply to Ed Morley [:edmorley UTC+1] from comment #0)
> > Please can someone drive the dependant bugs forwards? Also, each time we've
> > had a jetpack uplift of late, it's seemed that it's brought with it as many
> > new intermittent failures as it fixes of the old ones.. please can we try to
> > minimise the new regressions?
> 
> The biggest problem that we have is that there are very few test runs on our
> own tree so intermittent tests don't tend to show up there ahead of time. Is
> there anything else we can be doing in this regard?

I'm wondering if it'd be possible for the build system to run each platform's tests five times for each push on our tree...

Comment 4

[Dave Townsend [:mossop]]

(In reply to Wes Kocher (:KWierso) from comment #3)
> (In reply to Dave Townsend (:Mossop) from comment #2)
> > (In reply to Ed Morley [:edmorley UTC+1] from comment #0)
> > > Please can someone drive the dependant bugs forwards? Also, each time we've
> > > had a jetpack uplift of late, it's seemed that it's brought with it as many
> > > new intermittent failures as it fixes of the old ones.. please can we try to
> > > minimise the new regressions?
> > 
> > The biggest problem that we have is that there are very few test runs on our
> > own tree so intermittent tests don't tend to show up there ahead of time. Is
> > there anything else we can be doing in this regard?
> 
> I'm wondering if it'd be possible for the build system to run each
> platform's tests five times for each push on our tree...

I'm sure it's possible, it's up to you guys and releng I guess to figure out if you think the additional load is worth it here

Comment 5

[Ed Morley [:emorley]]

One option would be to perform a bunch of retriggers on tip of the jetpack repo prior to performing an uplift (or alternatively pushing the uplift to try first, and requesting retriggers there). :-)

Comment 6

[Dave Townsend [:mossop]]

Aside from bug 905338, all the high frequency failures should be gone now. We're going to continue to work on these and try to be better at jumping on them in the future. Do you all think we're at an acceptable level and can close this bug?

One of the main tests that we had to disable is blocked on getting a loaner slave for the engineer, bug 904219. That it takes over a week to get loaners is a real roadblock to tracking these things down sometimes.

Comment 7

[Ed Morley [:emorley]]

The overall jetpack failure rate looking much better now, thank you :-)

-> Resolving for now