1186078 - [Meta] Tracking bug to bring 24 hours backouts a reality

Status: RESOLVED WONTFIX

(Testing :: Talos, defect)

Description

[Vaibhav (:vaibhav1994)]

We currently have some things in place to make 24-hour backouts a reality in perf regressions, but a lot of work is still left. Lets use this bug as a tracker.

:jmaher points out the stages in the life of perf sheriff:

>  -1: needs attention
>   0: new
>   - possibly do something different if this is a merge (look on other >branches, etc. - for automation we don't need to)
>   1: backfilling: needs backfilling (could be the same as #1)
>   - mozci to verify rev +- 2 (rev-2, rev-1, rev, rev+1, rev+2) has data
>   - mozci to schedule 6 data points builds/jobs for the rev a+- 2 (might need a repeat if there are no builds)
>   - need to do this in 2 parts, 1 ensure we have builds, 2, ensure we have tests
>   - move to stage -1 if we cannot fill in the holes 100% (i.e. build bustage, dontbuild, trees closed, etc.)
>  2: has more data for specific test
>   - somehow verify we have a non merge revision and that revision 'a' is where we shift (we could script this in perfherder/alertmanager)
>  3: needs all-talos run
>   - mozci: given revision A showing a regression, schedule all-talos (6 runs) for all tests/platforms for Rev A and A-1.
>   - mozci: might have to wait for builds
>  4: has all-talos data for revision a and a-1
>   - sanity check we have the full set of data
>  5: bug filed
>  6: closed (wontfix, backout, fixed)

Comment 1

[Vaibhav (:vaibhav1994)]

A rough state machine suggested by :jmaher


for alert in alerts:
    startRev = getPushLog(alert.rev) - 2
    endRev = getPushLog(alert.rev) + 2
    dataPoints = perfherder.query(alert.branch, alert.platform, alert.test, startRev, endRev)
    switch alert.stage:
        case 0: #new
            if getRevision(alert.rev).merge:
                case = -1
                break
            if alert.branch.endswith('pgo'):
                case = -1
                break
            alert.stage = 1
       case 1: #backfilling
            if len(dataPoints) < 5:
                status = mozci.trigger(alert.buildername, startRev, endRev, times=6)
                if len(status.builds) > 0:
                    alert.stage = 1 # we are waiting on builds, need to run this again
                else:
                    alert.stage = 2
                break
            alert.stage = 2
        case 2: # enough data after initial backfilling, verify
            status = mozci.trigger(alert.buildername, startRev, endRev, times=6)
            if status.builds > 0 or status.pending > 0 or status.running > 0:
                alert.stage = 1 # waiting on builds/tests
                break
               
            if len(dataPoints) < 5:
                alert.stage = -1  # all builds are done, missing jobs for revisions
                break
           
            for data in dataPoints:
                if len(data) < 6:
                    alert.stage = -1 # all builds are done, missing data for jobs
                    break

            # analyze the data, find specific revision:
            pl = getPushLog()
            badRevisions = []
            for rev in pl[startRev:endRev]:
                results = perfherder.compare(pl[rev], pl[rev-1], alert.branch, alert.platform, alert.test)
                if results.change < -2.0:
                    badRevisions.append(rev)

            if len(badRevisions) != 1:
                alert.stage = -1 # too noisy, other issues
                break

            if getRevision(badRevisions[0]).merge:
                case = -1
                break

            if alert.rev != badRevisions[0]:
                alert.rev = badRevisions[0] # we misreported initially, possibly update other tools/status
            alert.stage = 3
        case 3:
            mozci.trigger_all_talos(alert.rev, alert.branch, times=6)
            previous_rev = getPushLog(alert.rev) - 2
            mozci.trigger_all_talos(previous_rev, alert.branch, times=6)
            alert.stage = 4
            break
        case 4:
             # verify all data exists, i.e. jobs are completed

Comment 2

[Vaibhav (:vaibhav1994)]

We had a meeting, and these are some things to take action on: https://etherpad.mozilla.org/perf-backouts

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

closing out old bugs that haven't been a priority