consider turning off talos media_tests as they are noisy and have found no regressions or improvments

RESOLVED FIXED in Firefox 46

Status

Testing
Talos
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: jmaher, Unassigned)

Tracking

unspecified
mozilla46
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox46 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
A little over a year ago we turned on the webrtc media_tests in bug 909524.  Looking at the graph over a year:
http://graphs.mozilla.org/graph.html#tests=[[317,63,35],[317,131,35]]&sel=none&displayrange=365&datatype=geo

you can see a lot of noise, this is hard to track and we will only catch large regressions.  In fact if you zoom in, you will see the values fall into one of 5 nice lines:
http://graphs.mozilla.org/graph.html#tests=[[317,63,35],[317,131,35]]&sel=1430229989574.8628,1438085527481,4.455704759431365,5.482550397015258&displayrange=365&datatype=geo

We need to get this test working better or turn it off.  We have a lot of hacks in place to have this running and I don't see any value from it.  It does show that any and all changes made in the last year are not causing regressions or improvements, but is this test really necessary?

If there is a good reason to keep this on, lets do it, otherwise- we can leave the code in place and just stop running it for 6 months, then properly remove it.
(Reporter)

Comment 1

3 years ago
:jesup, I would really like your input here!
Flags: needinfo?(rjesup)
There was a follow-on we ran out of resources to do (and still actually need), which is to use a good metric for this value - SNR is an awful metric, though it would catch total failure.  PESQ had been planned, but is encumbered.  We need to use the Opus metrics, but we need them to pull them out for us.

Also, another followon that's needed is to track video.

Given the above, I'm ok with turning it off for now until we can get these (which also makes it much easier to try improvements or additions).  I'd like to be pinged before removing this entirely.

We're also doing cross-machine tests as part of steeplechase which may want to run these tests, or variants of them.  (such that real networking is in the loop).
Flags: needinfo?(rjesup)
(Reporter)

Updated

3 years ago
Blocks: 1189375
(Reporter)

Comment 3

2 years ago
I am circling back here to consider removing the code from the talos tree.  We are just about 6 months out from actually turning off the test, do we have anyone who is working on getting these tests running again this quarter?
Flags: needinfo?(rjesup)
(Reporter)

Comment 4

2 years ago
Created attachment 8709547 [details]
MozReview Request: Bug 1188321 - remove media_tests code from talos. r?parkouss

Review commit: https://reviewboard.mozilla.org/r/31471/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/31471/
Attachment #8709547 - Flags: review?(j.parkouss)
Comment on attachment 8709547 [details]
MozReview Request: Bug 1188321 - remove media_tests code from talos. r?parkouss

https://reviewboard.mozilla.org/r/31471/#review28173

This looks great!

A small question, there is some preferences in the config.py file that are not removed. Do we still need them ?

talos/config.py:        'media.gmp-manager.url':
talos/config.py:        'media.navigator.enabled': True,
talos/config.py:        'media.peerconnection.enabled': True,
talos/config.py:        'media.navigator.permission.disabled': True,
talos/config.py:        'media.capturestream_hints.enabled': True,

Also I did not tested, try will say if everything is good - but to me all looks fine.
Attachment #8709547 - Flags: review?(j.parkouss) → review+
(Reporter)

Comment 6

2 years ago
try looks great:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f8c3322bd8f7

looking back in talos history, we have 4 of those preferences related to media:
http://hg.mozilla.org/build/talos/diff/27eb2c6ea066/talos/PerfConfigurator.py#l279

I have pushed to try again with those 4 preferences removed- will land when all is looking good.
Flags: needinfo?(rjesup) → needinfo?(jmaher)
(Reporter)

Comment 7

2 years ago
ok, the new try push looks good :)
Flags: needinfo?(jmaher)

Comment 9

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/2dc0cb4fdad6
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox46: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla46
You need to log in before you can comment on or make changes to this bug.