Closed Bug 1450808 Opened 2 years ago Closed 2 years ago

Hours and page loads between Flash CTA interaction spiked unexpectedly in Beta 60

Categories

(Core :: Plug-ins, defect, P1)

defect

Tracking

()

RESOLVED FIXED
mozilla61
Tracking Status
firefox-esr52 --- unaffected
firefox59 --- unaffected
firefox60 + fixed
firefox61 --- fixed

People

(Reporter: cpeterson, Assigned: Felipe)

References

Details

Attachments

(1 file)

[Tracking Requested - why for this release]:

Romain noticed the following surprising telemetry spike when 60 rode from Nightly to Beta on March 15.

Both hours and page loads between CTA in the Beta channel spiked unexpectedly when Firefox 60 entered Beta on March 15:

https://sql.telemetry.mozilla.org/queries/4813/source#9786
https://sql.telemetry.mozilla.org/queries/4813/source#9783

But hours and page loads between CTA in the Release channel have been pretty stable since February:

https://sql.telemetry.mozilla.org/queries/39051#104930
https://sql.telemetry.mozilla.org/queries/39051#104929

This all suggests a code change in Beta 60 is responsible for the CTA spike and not an external factor like a website removing Flash. Something might be causing the CTA UI to not be shown. Felipe suspects this might be a regression from bug 1192927, which added Flash to the Site Permissions door hanger.
Romain said he would review the telemetry data to answer these questions:

* Was there a similar spike during Nightly 60? Bug 1192927 landed on March 3, so we might see a CTA spike in Nightly then.

* Did the spike in Beta 60 affect Windows, Mac, and Linux equally?
Flags: needinfo?(rtestard)
Priority: -- → P1
Emil tested the Flash CTA permissions when updating from Beta 59 (b10) to Beta 60 (b8) on Windows, macOS, and Linux. There were no obvious CTA bugs. So this telemetry spike is probably a corner case that only affects some users (but a lot of them) or might even be a bug in our Flash telemetry reporting?

Emil's test results:

https://docs.google.com/document/d/1iwBQbkZtg-8sFx2DseWtcAnqoc_3p3mSBG95WYyeU7g/
The issue is apparently coming from the probes, Frank pointed me to https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Histograms.json#8131
Are we now too late to cancel the probe retirement?
Flags: needinfo?(rtestard)
(In reply to Romain Testard [:RT] from comment #3)
> Are we now too late to cancel the probe retirement?

I don't think it's too late. The data for beta 60 won't be useful, but for release, and newer betas, it should be fine again. We'll want to keep observing these numbers as we take more steps in the Flash deprecation timeline.
OK great let's make sure it's still there for release. Also for info bug 1451071 is about changing the owner of all of :bsmedberg's probes. Felipe would you be the new owner for these?
(In reply to Romain Testard [:RT] from comment #5)
> Felipe would you be the new owner for these?

Chris and I talked about it and decided to use the flash mailing list for those notifications. I'll do that change in bug 1451071 since it doesn't need to be uplifted, and will use this bug just to do the version bump on the probes
Assignee: nobody → felipc
Status: NEW → ASSIGNED
So we looked at the planned Flash deprecation timeline and calculated the version to expire these to be around ~75.
This is far too distant in the future to be subject to changes, and there are a number of other plugin-related probes that are already set to never expire, that I think that bumping these to never expire too is the best approach.

We'll want to watch these numbers for as long as we support plugins, and once we remove that support, these probes will naturally be removed too.

(Also, a number of these probes can be removed already when we work on bug 1438857, but that should be left for that bug)
Comment on attachment 8965858 [details]
Bug 1450808 - Unexpire plugin-related telemetry probes.

https://reviewboard.mozilla.org/r/234682/#review240664

This r+ is for the code changes.

Extending the expiry of probes (especially to "never") requires Data Collection Review: https://wiki.mozilla.org/Firefox/Data_Collection

Once you fill out the form (just dump it into a comment or something here on the bug and ni?me), I can conduct the review.
Attachment #8965858 - Flags: review?(chutten) → review+
---- 1 - What questions will you answer with this data?

These probes are primarily used to help us understand how often are users activating Flash ("Hours between Flash Click-to-Activate interactions)

---- 2 - Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

There are continuous changes applied to the Flash Click-to-Activate UI and UX, based on the long-term plan to deprecate Flash, and we want to continue to monitor and validate these changes as we make them.

---- 3 - What alternative methods did you consider to answer these questions? Why were they not sufficient?

We could fly data-blind and rely on bug reports, but we are already collecting this data and this is just a request to continue collecting it

---- 4 - Can current instrumentation answer these questions?

Yes

---- 5 - List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories on the Mozilla wiki.

From the categories on the wiki, I'd classify all of them as "Interaction Data":

PLUGINS_NOTIFICATION_SHOWN
PLUGINS_NOTIFICATION_PLUGIN_COUNT [1]
PLUGINS_NOTIFICATION_USER_ACTION_2
PLUGINS_INFOBAR_SHOWN [2]
PLUGINS_INFOBAR_BLOCK [2]
PLUGINS_INFOBAR_ALLOW [2]
PLUGINS_INFOBAR_DISMISSED [2]

[1] even though this sounds like the count of plugins on the page, this is actually the count of plugin _types_. Since we only support Flash now, this is always 1. (actually 0, because it's count - 1)
I'll file a separate follow-up bug to remove this if none of the queries that people are watching is making use of this.

[2] The infobar is currently unused (pref'ed off), and these probes should go away soon in bug 1438857

---- 6 - How long will this data be collected? Choose one of the following:

I want to permanently monitor this data. (the Flash plugin mailing list will watch it)

According to the deprecation plan, Chris Peterson calculate that this should be measured until ~2020, or version ~75. Since this is far ahead in the future to make any concrete plans, I think it'll be more churn to say "75" and have to bump it again (or freak out about this again), so that's why I'm setting it to "never" expire.


---- 7 - What populations will you measure?

All users and channels (these probes were made opt-out in bug 1345894)

---- 8 - If this data collection is default on, what is the opt-out mechanism for users?

Opting out of telemetry, or not using or installing Flash

---- 9 - Please provide a general description of how you will analyze this data.

This data is analyzed mostly as "Hours between Flash activations", through the links provided in comment 0.

---- 10 - Where do you intend to share the results of your analysis?

The data is discussed on a bi-weekly meeting
Flags: needinfo?(chutten)
(In reply to :Felipe Gomes (needinfo me!) from comment #10)
> PLUGINS_NOTIFICATION_PLUGIN_COUNT [1]

> I'll file a separate follow-up bug to remove this if none of the queries
> that people are watching is making use of this.

(Filed bug 1452739 for that)
Data Collection Review Response:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate? 

Yes, Telemetry, the Probe Dictionary, and the rest of the usual suspected for Telemetry Histograms.

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, Telemetry's usual about:preferences checkbox.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

There is, :felipe. He will need to add himself to the alert_emails fields (and remove BDS, as that email address is defunct)

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2

    Is the data collection request for default-on or default-off?

Default-on, though approval for that was covered under an early version of this process.

    Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No.

    Is the data collection covered by the existing Firefox privacy notice? If unsure: escalate to legal if:

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No.

---
Result: data-review+ if you replace BDS' email with your own address in alert_emails.
Flags: needinfo?(chutten)
(In reply to Chris H-C :chutten from comment #12)
> ---
> Result: data-review+ if you replace BDS' email with your own address in
> alert_emails.

Thank you! I forgot to mention, I'm already doing that in a separate bug, bug 1451071.

I'll land both together
Pushed by felipc@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/2b82c51a5926
Unexpire plugin-related telemetry probes. r,data-review=chutten
Comment on attachment 8965858 [details]
Bug 1450808 - Unexpire plugin-related telemetry probes.

Approval Request Comment
[Feature/Bug causing the regression]: Flash Click-to-Activate UI telemetry
[User impact if declined]: These probes will expire and not send data
[Is this code covered by automated tests?]: no
[Has the fix been verified in Nightly?]: not yet
[Needs manual test from QE? If yes, steps to reproduce]: no
[List of other uplifts needed for the feature/fix]: bug 1451071 (so that we get alerts on it)
[Is the change risky?]: no
[Why is the change risky/not risky?]: no code changes, just declarative changes
[String changes made/needed]: none
Attachment #8965858 - Flags: approval-mozilla-beta?
https://hg.mozilla.org/mozilla-central/rev/2b82c51a5926
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
Comment on attachment 8965858 [details]
Bug 1450808 - Unexpire plugin-related telemetry probes.

Thanks for fixing this! Approved for 60.0b12.
Attachment #8965858 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
You need to log in before you can comment on or make changes to this bug.