1642329 - [meta][android] Migrate benchmarks from raptor to browsertime on android

Status: RESOLVED FIXED

(Testing :: Raptor, task, P2)

Description

[Greg Mierzwinski [:sparky]]

This bug is for migrating all mobile benchmarks from raptor to browsertime.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Similar to bug 1585015, which is tracking desktop benchmarks. Perhaps there's some overlap here? If not let's clarify the other bug as being desktop only.

Comment 2

[Greg Mierzwinski [:sparky]]

It's possible that most of the desktop tasks would be addressed here but I'll leave both of these bugs for now.

Comment 3

[Alexandru Ionescu (needinfo me) [:alexandrui]]

As far as I can see, all the mobile benchmarks were added to browsertime. I don't see any remaining raptor benchmark not added to browsertime. Document here. The bold ones are the raptor benchmarks that are on both mobile and desktop and the green already on btime. We have some problems with unity-webgl, that's why I don't think it is approachable. Dave, sparky, thoughts?

Comment 4

[Greg Mierzwinski [:sparky]]

(In reply to Alexandru Ionescu (needinfo me) [:alexandrui] from comment #3)

As far as I can see, all the mobile benchmarks were added to browsertime. I don't see any remaining raptor benchmark not added to browsertime. Document here. The bold ones are the raptor benchmarks that are on both mobile and desktop and the green already on btime. We have some problems with unity-webgl, that's why I don't think it is approachable. Dave, sparky, thoughts?

Nice that there's only one left for mobile. Unity-webgl will need to be fixed and migrated.

Comment 5

[Alexandru Ionescu (needinfo me) [:alexandrui]]

According to this comment we abandoned for the moment the idea of fixing unity-webgl. The proposed variant by :luke was to transform this to a pageload type but this wasn't bringing value to us, so we set it as P3 until we decide its worth the effort.
IMO this is still not bringing us that much value, and the effort for fixing this is getting higher as the time passes by.
Sparky, still believe we have to fix this?

Comment 6

[Greg Mierzwinski [:sparky]]

Yes, we have it running in mozilla-central so we need to get it running in browsertime at least or completely disable the test (which is not an ideal solution): https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=unity-web&selectedTaskRun=M7wm0bViQ6GCg9KVQqArLA.0

Maybe we can convert it to a pageload test at the same time and solve the issue in that bug.

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This bug should cover not only support for running the benchmarks using raptor-browsertime, but also replacing scheduled raptor-webext jobs with raptor-browsertime jobs. It appears that at least speedometer is still running on autoland/m-c/m-b using raptor-webext, and only running on m-c using raptor-browsertime.

Alex: could you take a look through the jobs and provide a summary of where we are in this migration?

Comment 8

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Working on it.

Comment 9

[Florin Strugariu [:Bebe]]

Attachment: Bug 1642329 - Migrate speedometer benchmarks from raptor to browsertime on android

Comment 10

[Florin Strugariu [:Bebe]]

Attachment: Bug 1642329 - Migrate unity-webgl benchmarks from raptor to browsertime on android

Depends on D94292

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/815817bdeb1a
Migrate speedometer benchmarks from raptor to browsertime on android r=perftest-reviewers,sparky
https://hg.mozilla.org/integration/autoland/rev/7d0449457119
Migrate unity-webgl benchmarks from raptor to browsertime on android r=perftest-reviewers,sparky

Comment 12

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚BST from comment #7)

This bug should cover not only support for running the benchmarks using raptor-browsertime, but also replacing scheduled raptor-webext jobs with raptor-browsertime jobs. It appears that at least speedometer is still running on autoland/m-c/m-b using raptor-webext, and only running on m-c using raptor-browsertime.

Alex: could you take a look through the jobs and provide a summary of where we are in this migration?

Before bebe's patch landed, the following speedometer tests were still running:

• raptor-speedometer-fenix-e10s
• raptor-speedometer-fennec68-1proc
• raptor-speedometer-geckoview-cpu-memory-e10s
• raptor-speedometer-geckoview-cpu-memory-power-e10s
• raptor-speedometer-geckoview-e10s
• raptor-speedometer-refbrow-e10s

on the following devices:

• test-android-hw-g5-7-0-arm7-api-16
• test-android-hw-g5-7-0-arm7-api-16-shippable
• test-android-hw-p2-8-0-android-aarch64
• test-android-hw-p2-8-0-android-aarch64-shippable
• test-android-hw-p2-8-0-arm7-api-16
• test-android-hw-p2-8-0-arm7-api-16-shippable

Comment 13

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/815817bdeb1a
https://hg.mozilla.org/mozilla-central/rev/7d0449457119