Closed Bug 1188321 Opened 5 years ago Closed 5 years ago

consider turning off talos media_tests as they are noisy and have found no regressions or improvments


(Testing :: Talos, defect)

Not set


(firefox46 fixed)

Tracking Status
firefox46 --- fixed


(Reporter: jmaher, Unassigned)




(1 file)

A little over a year ago we turned on the webrtc media_tests in bug 909524.  Looking at the graph over a year:[[317,63,35],[317,131,35]]&sel=none&displayrange=365&datatype=geo

you can see a lot of noise, this is hard to track and we will only catch large regressions.  In fact if you zoom in, you will see the values fall into one of 5 nice lines:[[317,63,35],[317,131,35]]&sel=1430229989574.8628,1438085527481,4.455704759431365,5.482550397015258&displayrange=365&datatype=geo

We need to get this test working better or turn it off.  We have a lot of hacks in place to have this running and I don't see any value from it.  It does show that any and all changes made in the last year are not causing regressions or improvements, but is this test really necessary?

If there is a good reason to keep this on, lets do it, otherwise- we can leave the code in place and just stop running it for 6 months, then properly remove it.
:jesup, I would really like your input here!
Flags: needinfo?(rjesup)
There was a follow-on we ran out of resources to do (and still actually need), which is to use a good metric for this value - SNR is an awful metric, though it would catch total failure.  PESQ had been planned, but is encumbered.  We need to use the Opus metrics, but we need them to pull them out for us.

Also, another followon that's needed is to track video.

Given the above, I'm ok with turning it off for now until we can get these (which also makes it much easier to try improvements or additions).  I'd like to be pinged before removing this entirely.

We're also doing cross-machine tests as part of steeplechase which may want to run these tests, or variants of them.  (such that real networking is in the loop).
Flags: needinfo?(rjesup)
Blocks: 1189375
I am circling back here to consider removing the code from the talos tree.  We are just about 6 months out from actually turning off the test, do we have anyone who is working on getting these tests running again this quarter?
Flags: needinfo?(rjesup)
Comment on attachment 8709547 [details]
MozReview Request: Bug 1188321 - remove media_tests code from talos. r?parkouss

This looks great!

A small question, there is some preferences in the file that are not removed. Do we still need them ?

talos/        'media.gmp-manager.url':
talos/        'media.navigator.enabled': True,
talos/        'media.peerconnection.enabled': True,
talos/        'media.navigator.permission.disabled': True,
talos/        'media.capturestream_hints.enabled': True,

Also I did not tested, try will say if everything is good - but to me all looks fine.
Attachment #8709547 - Flags: review?(j.parkouss) → review+
try looks great:

looking back in talos history, we have 4 of those preferences related to media:

I have pushed to try again with those 4 preferences removed- will land when all is looking good.
Flags: needinfo?(rjesup) → needinfo?(jmaher)
ok, the new try push looks good :)
Flags: needinfo?(jmaher)
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla46
You need to log in before you can comment on or make changes to this bug.