1171167 - [raptor] simulate app launch regression and verify raptor catches it

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[Robert Wood [:rwood]]

Make a local change to the test-startup-limit app that introduces a significant startup delay. Make a PR and verify on the gaia run that raptor catches the regression (don't merge the PR!).

Comment 1

[:Eli Perelman]

I guess you could just:

setTimeout(function() {
  performance.mark('visuallyLoaded');
}, 500);

Comment 2

[Robert Wood [:rwood]]

This PR introduces a startup performance regression of approx. 25% in the test-startup-limit-app (when ran locally):

https://github.com/mozilla-b2g/gaia/pull/30428

Comment 3

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #2)
> This PR introduces a startup performance regression of approx. 25% in the
> test-startup-limit-app (when ran locally):
> 
> https://github.com/mozilla-b2g/gaia/pull/30428

On TC the results task says there is a 9% regression:
https://tools.taskcluster.net/task-inspector/#F3seZkF-TaSXsdtEbJNjLA/0

Which must be relative to the speed of the machine (?) because... on my local machine the regular app (no delay) visuallyLoaded p95 was 9985; with the delay was 15029.  VS on TC, median with no delay 4101, median with the 700ms delay 4472.

Comment 4

[Robert Wood [:rwood]]

Updated PR to add a 1500ms startup delay in test-startup-limit:

https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=91e48c1d6ea823a28d3698a2d74ae5b52ec77c28

Comment 5

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #4)
> Updated PR to add a 1500ms startup delay in test-startup-limit:
> 
> https://treeherder.mozilla.org/#/
> jobs?repo=gaia&revision=91e48c1d6ea823a28d3698a2d74ae5b52ec77c28

Yep regression detected:

https://tools.taskcluster.net/task-inspector/#9BE0qPx5ReWtalJGQeCwoA/2

Comment 6

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #5)
> (In reply to Robert Wood [:rwood] from comment #4)
> > Updated PR to add a 1500ms startup delay in test-startup-limit:
> > 
> > https://treeherder.mozilla.org/#/
> > jobs?repo=gaia&revision=91e48c1d6ea823a28d3698a2d74ae5b52ec77c28
> 
> Yep regression detected:
> 
> https://tools.taskcluster.net/task-inspector/#9BE0qPx5ReWtalJGQeCwoA/2

Re-triggered 5 more times and the regression was detected each time (25%, 26%, 29% 26%, 33%) a bit of a range though.

https://tools.taskcluster.net/task-inspector/#iUbn8G19TI6VLrFrfLwNbA/2
https://tools.taskcluster.net/task-inspector/#WAi-ruZ_QB69EnzR6-TMMw/2
https://tools.taskcluster.net/task-inspector/#MubSFO5vTRKEaH7lTqXYJA/2
https://tools.taskcluster.net/task-inspector/#KPH1wwmNSTa5omcCD8iCCw/2
https://tools.taskcluster.net/task-inspector/#6LyYVIZ_T3ykw_USxin8Cw/2

Comment 7

[Robert Wood [:rwood]]

Tried the same test again (1500ms startup delay in test-startup-limit app) after switching to the c4.large instance type for raptor tasks.

https://github.com/mozilla-b2g/gaia/pull/30428

Regression detected as expected (29%):

https://tools.taskcluster.net/task-inspector/#dkX_39jFTE2EolUgtzqMuQ/2

Comment 8

[Robert Wood [:rwood]]

Repeat this test on the newly refactored setup on gaia-ci. Now there the Raptor launch test runs on both base and patch revs on the same AWS/TC instance, and there is one task per app.

Comment 9

[GitHub Autolander Bot]

Attachment: [gaia] rwood-moz:bug1171167-two > mozilla-b2g:master

Comment 10

[GitHub Autolander Bot]

Attachment: [gaia] rwood-moz:fakeregression > mozilla-b2g:master

Comment 11

[Robert Wood [:rwood]]

(In reply to Autolander from comment #10)
> Created attachment 8638710 [details] [review]
> [gaia] rwood-moz:fakeregression > mozilla-b2g:master

Raptor did flag a performance regression on app startup for test-startup-limit with the test patch, on the first run and on each of the 3 retries. There is still variance in that the reported regression values were an increase from 26% to 39%, but the regression was caught.

Comment 12

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #11)
> (In reply to Autolander from comment #10)
> > Created attachment 8638710 [details] [review]
> > [gaia] rwood-moz:fakeregression > mozilla-b2g:master
> 
> Raptor did flag a performance regression on app startup for
> test-startup-limit with the test patch, on the first run and on each of the
> 3 retries. There is still variance in that the reported regression values
> were an increase from 26% to 39%, but the regression was caught.

The actual intentional-delay introduced represents approximately a 30% increase in launch time