Closed Bug 1552484 Opened 5 years ago Closed 5 years ago

Run YouTube playback performance tests muted on Android devices

Categories

(Testing :: Raptor, task, P1)

Product:
Testing
Component:
Raptor
Version:
Version 3
Type:
task

Tracking

(firefox69 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla69
Tracking Flags:
Tracking Status
firefox69 --- fixed

People

(Reporter: whimboo, Assigned: whimboo)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

Attachments

(2 files)

Bug 1552484 - [raptor] Add geckoview support to raptor-youtube-playback test suite. r=#perftest
47 bytes, text/x-phabricator-request
Details | Review
Bug 1552484 - [raptor] Enable raptor-youtube-playback test suite for geckoview in CI. r=#perftest
47 bytes, text/x-phabricator-request
Details | Review

We want to run the upcoming YouTube playback performance tests (bug 1548845) also on Android devices for apps which are based on GeckoView. Sadly those tests won't work because autoplay is blocked. And as long as bug 1547717 hasn't been fixed, we need a workaround.

I would propose that for those apps we simply run the suite with audio muted, which results in no blocking of autoplay.

Note, the performance numbers might be slightly different, which I will check next on desktop at least.

Blocks: 1552738

As discussed last Friday in the streaming meeting using that page with muted audio shouldn't have an impact on the performance results. As such I will go ahead soon and also enable jobs on Android.

Chris, on which Android devices it would be good to get those tests running in CI? Do we have some which have HW decoding for H265 and VP9?

Flags: needinfo?(cpearce)

I assume running these tests on emulated devices doesn't make that much sense. So I wonder which hardware mobile devices we have in our CI which could be used to run the playback performance tests on. As best with hardware video decoding for the above codecs. It will also affect the question I raised for Chris.

Florin and/or Dave, could you give some feedback?

Flags: needinfo?(fstrugariu)
Flags: needinfo?(dave.hunt)

At least when checking Treeherder I can see Moto G5, and Pixel 2:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=raptor%2Candroid

As confirmed with Florin on IRC those are the two supported real devices. There are no FireTV stick 4k yet in CI.

Flags: needinfo?(fstrugariu)
Flags: needinfo?(dave.hunt)

The Pixel2 is probably a good candidate. The Moto G5 doesn't have hardware VP9, so won't do well in the tests. It would be good to have a reasonable lower spec device as well as the Pixel to run the tests over.

Flags: needinfo?(cpearce)

Perfect. So I will enable CI builds for both of them, but only on mozilla-central for now. Once we know more about the noise level we can then decide to also let them run on integration branches with alerting turned on.

Here you can find the perfherder data for the G5 and Pixel 2:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=try,2049061,1,10&series=try,2049350,1,10

As you can see there is quite a difference between both devices. G5 shows a lot of noise as already expected.

Adding those tests with the premise to run all the videos as muted,
at least as long as bug 1547717 hasn't been fixed.

Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c04159a7f012
[raptor] Add geckoview support to raptor-youtube-playback test suite. r=perftest-reviewers,rwood
https://hg.mozilla.org/integration/autoland/rev/53ead6f38bd7
[raptor] Enable raptor-youtube-playback test suite for geckoview in CI. r=perftest-reviewers,stephendonner

https://hg.mozilla.org/mozilla-central/rev/c04159a7f012
https://hg.mozilla.org/mozilla-central/rev/53ead6f38bd7

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
status-firefox69: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla69

Chris, do you have an idea why the tests on Moto G5 are faster than the ones on the Google Pixel 2? Given the missing hardware decoders I would expect a longer duration. Also the G5 jobs are even faster than desktop!

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=ytp%2Candroid

Flags: needinfo?(cpearce)

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #14)

Chris, do you have an idea why the tests on Moto G5 are faster than the ones on the Google Pixel 2? Given the missing hardware decoders I would expect a longer duration. Also the G5 jobs are even faster than desktop!

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=ytp%2Candroid

I'm seeing the Pixel2 scoring about 20 minutes, and the G5 scoring about 55 minutes, so that's the opposite of what you reported here? Am I misreading the results? I assume taking more minutes to run means it's slower?

Flags: needinfo?(cpearce) → needinfo?(hskupin)

I wonder where you see the 55 minutes for G5? When I check on Treeherder all the jobs are still below of 25 minutes and as such faster than the Pixel 2 which is between 25 and 30 minutes.

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=ytp%2Candroid

Flags: needinfo?(hskupin) → needinfo?(cpearce)

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #16)

I wonder where you see the 55 minutes for G5? When I check on Treeherder all the jobs are still below of 25 minutes and as such faster than the Pixel 2 which is between 25 and 30 minutes.

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=ytp%2Candroid

Pixel 2:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=ytp%2Candroid&selectedJob=250252517
Says: raptor-youtube-playback-geckoview-live pgo: 21.54

Moto G5:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=ytp%2Candroid&selectedJob=250252438
raptor-youtube-playback-geckoview-live pgo: 59.15

Click on the performance tab? I could be looking in the wrong spot?

Flags: needinfo?(cpearce) → needinfo?(hskupin)

Oh, now I see the confusion! My question was not about the mean value of dropped frames for that test suite, but the run time. This value can be seen at the bottom left panel.

It's basically not important but I would expect that a phone with a better hardware would also finish the job faster.

Flags: needinfo?(hskupin) → needinfo?(cpearce)

Thanks for clearing that up.

If you open a log for a MotoG5 run, and search for "u'decodedFrames': 0", you'll see a lot of hits, all on the 1440p and 2160p. You don't get hits on the Pixel2 log. So what I think's happening is that on the MotoG5 the in-built decoder is either refusing to decode, or we're expressing that we can't decode, resolutions greater than 1080p and the test is detecting that and not running these tests.

So I think the Moto G5 runs are faster because they're not running the 1440p and 2160p cases.

Flags: needinfo?(cpearce)

Interesting. Is that something which should be highlighted in the perf results? Right now this is a case where we don't actually see valid results. Imagine if we don't play any video at all, it would have a mean of 0 dropped frames, which gives totally a wrong assumption of the healthiness of the test.

At least for the health dashboard we report 100% failure rate if 0 decoded frames are detected. But not sure what to do with the plain decoded frames value.

Flags: needinfo?(cpearce)

The tests where we can't decode on that device, we'll likely never be able to decode on that device, as we're limited by the hardware. So there's not much point in tracking those tests on that device. I would suggest having some kind of "expected fail" for the > 1080p cases on the MotoG5.

Flags: needinfo?(cpearce)

(In reply to Chris Pearce [:cpearce (GMT+13)] from comment #21)

The tests where we can't decode on that device, we'll likely never be able to decode on that device, as we're limited by the hardware. So there's not much point in tracking those tests on that device. I would suggest having some kind of "expected fail" for the > 1080p cases on the MotoG5.

Thanks! Alexandru, could you or Arnold please have a look at this? A new bug blocking bug 1552738 would be good.

Flags: needinfo?(airimovici)

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #22)

(In reply to Chris Pearce [:cpearce (GMT+13)] from comment #21)

The tests where we can't decode on that device, we'll likely never be able to decode on that device, as we're limited by the hardware. So there's not much point in tracking those tests on that device. I would suggest having some kind of "expected fail" for the > 1080p cases on the MotoG5.

Thanks! Alexandru, could you or Arnold please have a look at this? A new bug blocking bug 1552738 would be good.

I'm on it. I created the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1552484

Flags: needinfo?(airimovici)
See Also: → 1558456
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: