1210704 - Add service worker benchmark for Flame and Aries

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Performance, defect, P2)

Description

[Bobby Chien]

As gaia development, I would like to know what is performance of service worker. Also, I want to track performance regression.

Goal:
Integrate service worker benchmark with raptor, and perform benchmark per daily based.

Devices:
- Flame
- Aries

Comment 1

[Oleg Zasypkin [:azasypkin]]

ni? myself to do the following:

* Remove manual "Install Service Worker" step from benchmarking app [1];

* Find out where in Gaia tree we can place this app. Looks like "dev_apps" folder is going to die (see bug 1133711), so we need long term plan.

[1] https://github.com/nga-crash-tests/service-worker-benchmarks

Comment 2

[Bobby Chien]

[Tracking Requested - why for this release]:

Comment 3

[Oleg Zasypkin [:azasypkin]]

(In reply to Oleg Zasypkin [:azasypkin][⏰UTC+1] from comment #1)
> * Find out where in Gaia tree we can place this app. Looks like "dev_apps"
> folder is going to die (see bug 1133711), so we need long term plan.

Discussing in bug 1133711.

Comment 4

[Oleg Zasypkin [:azasypkin]]

Trying to make dynamically preloaded apps work with Raptor in bug 1219176.

Comment 5

[Oleg Zasypkin [:azasypkin]]

bug 1219176 is in review now, so I can start preparing the test app itself here.

Comment 6

[GitHub Autolander Bot]

Attachment: [gaia] azasypkin:bug-1210704-sw-benchmarks > mozilla-b2g:master

Comment 7

[Oleg Zasypkin [:azasypkin]]

Waiting for bug 1225121 to be resolved.

Comment 8

[Oleg Zasypkin [:azasypkin]]

Comment on attachment 8686576 [details] [review]
[gaia] azasypkin:bug-1210704-sw-benchmarks > mozilla-b2g:master

Okay, all dependencies are resolved and it seems working fine for me locally.

Hey Eli, how does it look to you? Could you please check if it works fine for the Raptor Dashboard as well?

I use the following commands to run 3 benchmarking apps that are included into the package:

* raptor test coldlaunch --app fxos-performance.github.io --entry-point service-worker-baseline --runs 30

* raptor test coldlaunch --app fxos-performance.github.io --entry-point service-worker-empty --runs 30

* raptor test coldlaunch --app fxos-performance.github.io --entry-point service-worker-simple-fetch --runs 30

Thanks!

Comment 9

[:Eli Perelman]

Comment on attachment 8686576 [details] [review]
[gaia] azasypkin:bug-1210704-sw-benchmarks > mozilla-b2g:master

Looks good to me. I'll try and get this up in automation next week some time.

Comment 10

[:Eli Perelman]

ni to add these apps to coldlaunch automation.

Comment 11

[Oleg Zasypkin [:azasypkin]]

(In reply to :Eli Perelman from comment #9)
> Comment on attachment 8686576 [details] [review]
> [gaia] azasypkin:bug-1210704-sw-benchmarks > mozilla-b2g:master
> 
> Looks good to me. I'll try and get this up in automation next week some time.

Thanks! So let's see if it sticks.

Master: https://github.com/mozilla-b2g/gaia/commit/d2ff5923a622b398d4f55b4f3d93b7728028cc51

Not marking this bug as FIXED until we have automation support for this.

Comment 12

[:Eli Perelman]

Benchmark apps have been added to Jenkins automation configuration for:

flame-kk-1024.b2g-inbound.tinderbox.perf.coldlaunch.v2
flame-kk-512.b2g-inbound.tinderbox.perf.coldlaunch.v2

Apps have also been added to the dashboard configuration for raptor.mozilla.org. Awaiting builds to ensure tests and running and reporting successfully.

Comment 13

[Oleg Zasypkin [:azasypkin]]

(In reply to :Eli Perelman from comment #12)
> Benchmark apps have been added to Jenkins automation configuration for:
> 
> flame-kk-1024.b2g-inbound.tinderbox.perf.coldlaunch.v2
> flame-kk-512.b2g-inbound.tinderbox.perf.coldlaunch.v2
> 
> Apps have also been added to the dashboard configuration for
> raptor.mozilla.org. Awaiting builds to ensure tests and running and
> reporting successfully.

Sweet, thanks a lot for your help, Eli!

Comment 14

[Oleg Zasypkin [:azasypkin]]

I see at least two datapoints at Dashboard so marking bug as resolved :)

Comment 15

[Oleg Zasypkin [:azasypkin]]

Hey Mike,

Could you please include apps from comment 8 to the daily Raptor runs on Aries as well?

We have had them for Flame for some time already and didn't have any noticeable issues so far.

Thanks!

Comment 16

[Mike Lien[:mlien]]

just add three tests from comment 8, leave ni to see if results are posted to dashboard correctly

Comment 17

[Mike Lien[:mlien]]

Now Aries has three service worker cases and records

Comment 18

[Oleg Zasypkin [:azasypkin]]

(In reply to Mike Lien[:mlien] from comment #17)
> Now Aries has three service worker cases and records

Nice, thanks a lot!