Closed Bug 1307616 Opened 6 years ago Closed 6 years ago
_PLAY _TIME _MS regressed on Beta50
I used the evo telemetry dashboard to compare VIDEO_PLAY_TIME_MS probe across Beta releases and it seems to have regressed badly in Beta50 cycle. Beta 49 (Build 2016-09-16) Media 21.6k Mean 517.9k 25th percentile 0 95th percentile 3.15M Beta50 (Build 2015-09-26) Media 1.86k Mean 165.92k 25th percentile 0 95th percentile 679.82l
6 years ago
I think the main reason for this big change is that we used to report the play time for traditional video sources only, but since 50 we report this telemetry for MSE videos as well, see bug 1287684. It landed on 2016-07-20, and if you look at the evolution graph around that time (looking at nightly 50 and 51), it went from a steady-ish ~15k submissions per day to ~40k! I'm guessing MSE videos like those on YouTube are probably shorter on average, which would explain the shift. I suppose we could have created a separate probe for MSE videos, so the old one would not have been disturbed. But on the other hand, the old graph was giving us a short-sighted view of the whole video-playback usage. In the end, we are mostly interested in the actual numbers and their distribution, not the average.
Assignee: nobody → gsquelart
Status: NEW → RESOLVED
Closed: 6 years ago
OS: Unspecified → All
Hardware: Unspecified → All
Resolution: --- → WONTFIX
Hi Gerald, I noticed a steady downward trend for this probe on Firefox Mobile. Mean, Median, 75th percentile and 95th percentile are all trending down. Is that a cause for concern? Just wanted to make sure you review it as well.
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
Ritu, I am not too sure how to really be able to answer that satisfactorily. First, would you have a link to the numbers/graphs you are looking at? (I have limited experience using telemetry.m.o, I don't see how to get anything else than the median.) From looking at the evolution of nightlies 50-52 for mobile ( http://tinyurl.com/j3u9xxu ), there is the obvious change that happened on 2016-07-20 (when we started collecting numbers for MSE playback), where there was a spike of submissions (3k-4k) until the end of 50, 51 goes down to a more steady ~1.5k but higher than at the start of 50, and then 52 a bit higher again. But the main point I would consider is that the number of submissions seems a bit low to give much trust about the numbers, especially when looking at the wild jigsawing of the median, with a wide range of numbers between 3 and 40 seconds. (But I know enough about stats to know not to trust myself, so we should find an actual statistician to analyze these numbers!) Visually it does look like 52 is slowly trending down. If that's true, then how could we possibly find a regression point in this sea of spikes? Some theories: - Because we are getting better at showing MSE videos, they may get played more often; but these videos are supposedly shorter on average than non-MSE videos, so they skew the average-play-time down. - We are getting better at not even starting playing videos when not visible, so there are more zeroes, which naturally pushes the trend down. - People are getting more and more impatient, and move on more quickly. - Of course it could be a real issue, maybe we are getting worse at showing videos, so we are not playing them (zeroes) or people abandon sooner (shorter play duration). I could probably come up with others that could explain these numbers, but I think it would be difficult to prove or discard such theories. Also, as Anthony pointed out in an email discussion, "The issue with this probe is that we're interested in the total not the average. The average just tells us that people are watching shorter videos. We're actually interested in total viewing time." So this probe is probably not an appropriate choice to monitor the general health of media playback, and to potentially block releases because of how it trends -- barring obvious catastrophic changes. We like to have these numbers internally to guide us towards better supporting what our users are actually doing. And part of our efforts will probably continue to bring these numbers down (e.g., suspending video playback when video is in the background), so a downward trend is not necessarily a bad thing. Ritu (and others reading this), what do you think? Should we invest more time into trying to find out if there are real issues behind this probe's trend?
Video play time is a very tricky thing to look at without a well-defined sample set of users, and you really can't look at it across nightly/aurora/beta at all. And this will change according to all sorts of outside uses (how many ads are used in hte middle of pages with video (tend to get closed quickly, if the option is available), etc. That said this may have some useful monitoring validity; I'm talking in the abstract.
Component: Audio/Video → Audio/Video: Playback
Hi Gerald, thank you for such a detailed response. I am fine with your decision to wontfix this if you believe that this probe isn't the right one to monitor. The real question (at least for me) is: which probes should we use instead to catch unusual trends around video playback? I see several *video* and *webrtc* related probes but unable to figure out which ones to use.
IIUC the total is what we should care about, not the average.
Status: REOPENED → RESOLVED
Closed: 6 years ago → 6 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.