840828 - Add metrics to FHR for SocialAPI

Status: RESOLVED FIXED

(Privacy Graveyard :: Product Review, task, P2)

Description

[John Jensen]

The SocialAPI is a significant, recent new feature in Firefox. We have large partners (eg Facebook, others) making use of it, but it is difficult for us to know whether the API is working well or if it is helping/hurting our users' browser experience.

Since it, by design, runs non-Mozilla Javascript code as a background worker services, we need diagnostic tools to show whether these services are working well or causing problems.

Suggested metrics:

1) Whether the SocialAPI has been toggled on 
2) For which whitelisted partners it has been enabled
3) Number of worker service starts
4) Number of socialAPI widget interactions

Using these, in combination with the other performance metrics in FHR, we will be able to see whether or not the use of SocialAPI is correlated with browser stability, performance, increased/decreased use, etc. This will help us better ensure that the functionality created by our partners works well for our users.

Comment 1

[Richard Newman [:rnewman]]

Starting the process to work with Privacy on the details.

(If you've already started working with them, then great!)

Comment 2

[Gregory Szorc [:gps]]

Tom instructed me to assign new probes requests to Privacy first.

Comment 3

[Johnathan Nightingale [:johnath]]

Pinging this since it's been in privacy without response for most of a month.

Comment 4

[Gregory Szorc [:gps]]

Presumably these two bugs are coupled.

Comment 5

[Asa Dotzler [:asa]]

This is critical to our ability to move this feature forward and I'd love to see it make first FHR roll-out. Shane has offered some of his time to help implement.

(Social API should be treated much like add-ons in terms of privacy so I would expect this to be a quick and easy privacy review.)

Alex, Tom, Alina, we're under very tight time constraints here. Will you all be able to complete this review today or tomorrow?

Comment 6

[David Slater]

+1 of the comments from the fine Mr. Dotzler.

Comment 7

[Shane Caraveo (:mixedpuppy)]

(In reply to Gregory Szorc [:gps] from comment #4)
> Presumably these two bugs are coupled.

yes, they are.

Comment 8

[Tom Lowenthal]

I'm not sure I understand why it's critical to get data from all Firefox users (IE: FHR) rather than just those who want to share info with us (IE: telemetry). Is SocialAPI usage data critical to calibrating telemetry across our entire user population?

Comment 9

[Asa Dotzler [:asa]]

(In reply to Tom Lowenthal [:StrangeCharm] from comment #8)
> I'm not sure I understand why it's critical to get data from all Firefox
> users (IE: FHR) rather than just those who want to share info with us (IE:
> telemetry). Is SocialAPI usage data critical to calibrating telemetry across
> our entire user population?

For the same reasons we want FHR data on add-ons. This is just like add-ons. It's as simple as that.

Comment 10

[Richard Newman [:rnewman]]

(In reply to Tom Lowenthal [:StrangeCharm] from comment #8)
> I'm not sure I understand why it's critical to get data from all Firefox
> users (IE: FHR) rather than just those who want to share info with us (IE:
> telemetry). Is SocialAPI usage data critical to calibrating telemetry across
> our entire user population?

I think the answer is in Comment 0: the justification isn't about calibrating telemetry (a benefit only for Mozilla), it's that we can offer health-related advice to users (a user benefit as well as information we can use).

For example, if we see a strong correlation between crashes or slowness and a particular whitelisted social partner, we can share that information with the user in about:healthreport -- "your browser is running slowly; maybe you want to turn off Joe's Chat". We can also use the information on the server side or in client development to fix the root cause, but there's a direct user benefit.

As Asa says: it's just like add-ons.

Comment 11

[Alex Fowler]

Sorry for the delayed response. Here's my current position on this:

I'd like to see us complete v1 development, testing and user engagement with the current spec for FHR *before* adding new data elements to the system.

Here's why:

FHR data set is intended to provide a baseline set of metrics for Firefox with clear user facing benefits. We went through much discussion and vetting to come up with a minimal set of data elements to: (1) measure use of Firefox with the least possible risk of fingerprintability; and (2) power a health check feature for our general users that provides tangible benefits to their experience with Firefox.

FHR is not fully baked today. We're still refining data quality and developing the user facing feature. I'd like to see us complete this phase before we start contemplating the addition of new data elements.

There are probably lots of other features and services within Firefox that we'd love to include as part of FHR. Each team here probably has their own wishlist. We need to have a process by which we evaluate additions and subtractions to FHR that consider priority, value, risks, tradeoffs, privacy and security.

I'd like to see our metrics and telemetry groups able to justify why some data elements require reporting from 100% of our user base while others are fine as probes in telemetry. I know from my own review of the current set of data elements that we have duplication across FHR, pings and telemetry. I understand that it's likely there are duplicates within telemetry itself that haven't been removed. This all needs to be rationalized and cleaned up from an ongoing data governance perspective, again, before we start changing our plans at this point.

For the above reasons and without seeing more maturity in our data management and decision making process, I am not comfortable with us adding SocialAPI to FHR at this point in time. Happy to discuss more and am open to other input and suggestions.

Comment 12

[Johnathan Nightingale [:johnath]]

Alex, what are your thoughts on comment 5 and comment 9? I find I agree with Asa that we'd want this information for precisely the same (user-facing!) reasons as Add-ons - namely that participation in it could harm your user experience in ways we'd want to be able to diagnose and remedy.

I appreciate your concern that we get this right for the long run, and we should, but I'm pretty reluctant to have data about whether, e.g. a particular SocialAPI provider is causing performance problems, block on rationalizing and cleaning up our ongoing data governance globally.

(I also think we should separate out talk of "v1" from this discussion. The patch doesn't exist right now, and until it does, the question of whether it's in v1 or a follow-on is moot. My real question is: is it a waste of Shane's time to write the patch now, while we sort things through? I don't think it is, because I agree with Asa that this is basically like measuring add-on impact. I can't tell if you're suggesting otherwise above.)

Comment 13

[Anurag Phadke[:[e-mail]]]

The BizDev team is lining up tons of providers for SocialAPI and internally, we want to understand, at the very basic, the metrics described in this bug. This measurement helps us understand how users perceive our product and provides us more data points when we talk to partners.

Based on comment #11 and #12, its understandable that adding new probes to FHR will involve some cycles. 

At same time, based on comment #12, we need to understand how it affects performance and given it should be treated like an "addon", can we make it part of blocklist ping?

-anurag

Comment 14

[Richard Newman [:rnewman]]

(In reply to aphadke from comment #13)

> At same time, based on comment #12, we need to understand how it affects
> performance and given it should be treated like an "addon", can we make it
> part of blocklist ping?

From a privacy perspective: I'm not sure how we could justify 'leaking' information through the (non-opt-out) blocklist ping that we're unwilling to submit through (opt-out) FHR.

In essence, if it's 'weak' enough to send as part of blocklist, it's weak enough to include in FHR, particularly if there are health-related aspects (which I think we've established that there are).

Part of the original motivation for FHR was to eliminate the overloading of other services like blocklist ping into ad hoc user tracking.

Comment 15

[Alex Fowler]

(In reply to Johnathan Nightingale [:johnath] from comment #12)
> Alex, what are your thoughts on comment 5 and comment 9? I find I agree with
> Asa that we'd want this information for precisely the same (user-facing!)
> reasons as Add-ons - namely that participation in it could harm your user
> experience in ways we'd want to be able to diagnose and remedy.

Personally, I'm having a hard time understanding which features are critically important to measure via FHR from those that are not or are already being measured through another ping. To me, one way to create a threshold is to ask, "what are the elements that will provide our general user the most benefit when s/he runs a health check of her/his instance of Firefox?" Our opt-out position is predicated on the direct user benefit, and the data it utilizes, this service provides.

Again, do we need data from 100% of our user base to evaluate the use of SocialAPI? Also, how will the addition of the SocialAPI element/s to FHR be presented back to the user? For those who don't use it, will the report be blank for this element and not show it at all?

Comment 16

[Asa Dotzler [:asa]]

(In reply to Alex Fowler from comment #15)
> (In reply to Johnathan Nightingale [:johnath] from comment #12)
> > Alex, what are your thoughts on comment 5 and comment 9? I find I agree with
> > Asa that we'd want this information for precisely the same (user-facing!)
> > reasons as Add-ons - namely that participation in it could harm your user
> > experience in ways we'd want to be able to diagnose and remedy.
> 
> Personally, I'm having a hard time understanding which features are
> critically important to measure via FHR from those that are not or are
> already being measured through another ping.

We already made this decision for Add-ons. Social API is just like Add-ons. It's actually a type of Add-on. Users would be confused if we didn't have Social API in with their other Add-ons.

I know we want to be thorough, but more than a month has passed since this bug was filed and there is no good reason for a question as simple as this one to take that long.

Comment 17

[Asa Dotzler [:asa]]

(In reply to Alex Fowler from comment #15)
> Again, do we need data from 100% of our user base to evaluate the use of
> SocialAPI? Also, how will the addition of the SocialAPI element/s to FHR be
> presented back to the user? For those who don't use it, will the report be
> blank for this element and not show it at all?

Sorry, missed this part.  

We'll handle it the way we handle Add-ons. Not all users have installed XUL Add-ons, or themes, or plug-ins.

Comment 18

[Richard Newman [:rnewman]]

(In reply to Alex Fowler from comment #15)

> Personally, I'm having a hard time understanding which features are
> critically important to measure via FHR from those that are not or are
> already being measured through another ping.

I don't see how this is an open question next to your point about "most benefit" -- social has a direct impact on browser health, beyond merely being something from which Mozilla derives a benefit. Yes, Mozilla would derive a benefit from FHR reporting social usage, but the recorded data is of direct value to the user even if submission is disabled.

We should absolutely record, correlate, and present how your use of social is impacting the health of your browser.

Additionally, that a feature is being measured through another ping is essentially irrelevant for two reasons:

* Other pings are not able to offer a health benefit to users, and we wish to provide users with that benefit.
* We should be actively centralizing other kinds of reports, eliminating them (and their ad hoc, low-benefit, non-opt-out attributes) in favor of (user-benefiting, well-documented, etc. etc.) FHR.


> Again, do we need data from 100% of our user base to evaluate the use of
> SocialAPI?

100% of users who have enabled a social provider will benefit from understanding its impact on their browser health.

Comment 19

[Alex Fowler]

(In reply to aphadke from comment #13)

> At same time, based on comment #12, we need to understand how it affects
> performance and given it should be treated like an "addon", can we make it
> part of blocklist ping?

I don't think this is the right answer, either. All our pings and the data metrics they provide need to be rationalized, de-duped and optimized, in my opinion.

Comment 20

[Alex Fowler]

(In reply to Richard Newman [:rnewman] from comment #18)
> (In reply to Alex Fowler from comment #15)
> 
> > Personally, I'm having a hard time understanding which features are
> > critically important to measure via FHR from those that are not or are
> > already being measured through another ping.
> 
> I don't see how this is an open question next to your point about "most
> benefit" -- social has a direct impact on browser health, beyond merely
> being something from which Mozilla derives a benefit. Yes, Mozilla would
> derive a benefit from FHR reporting social usage, but the recorded data is
> of direct value to the user even if submission is disabled.
> 
> We should absolutely record, correlate, and present how your use of social
> is impacting the health of your browser.

I don't mean to suggest that the SocialAPI isn't important to both Mozilla, our partners, and our users. What I'm trying to get at, though, is what do we consider in/out of FHR? How do we make decisions, justify our rationale, consider potential risks and communicate changes to the data service to our stakeholders?

We set out with a plan for FHR to consist of a baseline set of elements core to Firefox, necessary for the Project and valuable for our general users. Not all possible data elements, not most, just the right, minimal set. The Metrics team came up with that list and vetted it with a number of people inside/outside Mozilla. We had an understanding that once the system was fully functional we could revisit the data elements and tweak the set based on what we were seeing, false positives, etc.

Jonathan's point is the baseline is arbitrary in that it doesn't exist yet, so why not just add an element for SocialAPI and say that it's part of our baseline. I want the rigor to be there in documenting and evaluating an additional data element, especially one that would include tracking engagement with partner services.

> Additionally, that a feature is being measured through another ping is
> essentially irrelevant for two reasons:
> 
> * Other pings are not able to offer a health benefit to users, and we wish
> to provide users with that benefit.
> * We should be actively centralizing other kinds of reports, eliminating
> them (and their ad hoc, low-benefit, non-opt-out attributes) in favor of
> (user-benefiting, well-documented, etc. etc.) FHR.

I completely agree with your points! We need to restore the pings and associated metrics to their primary purposes, remove duplications and clean up our back end data practices associated to how pings have evolved over the past many years. We can't do that, however, until FHR is up and running, stable and providing the value we want for our users.

Comment 21

[Asa Dotzler [:asa]]

(In reply to Alex Fowler from comment #20)
> Jonathan's point is the baseline is arbitrary in that it doesn't exist yet,
> so why not just add an element for SocialAPI and say that it's part of our
> baseline. I want the rigor to be there in documenting and evaluating an
> additional data element, especially one that would include tracking
> engagement with partner services.

If you are suggesting that there's a higher bar for Social API than for other types of add-ons because Social API includes 3rd party services, then you're making a distinction which doesn't hold up because there are plenty of XUL extensions and even plug-ins that include third party services. 

Again, social API is an add-on API for Firefox. It's a slightly different API than the XUL extensions API, but XUL extensions is also slightly different API from the Jetpack API, or NPAPI plug-ins, or the Themes API. These are all add-ons and all of them can include third party services.

If that's all off the mark and what you're actually saying is "Other add-on types made it in before we had a process but nothing else will make it in until we have a process" then my response is similar but even simpler: Social API should be grandfathered in with all the other add-on APIs because it's not materially different.

If I can help by sitting down with you and showing you what Social API is, I'm happy to do so. I get the feeling that you think it's something that it isn't and maybe a product demonstration would help.

Comment 22

[John Jensen]

Hi all, 

We had the opportunity to talk about this in person today -- Alex, Chad, Taras, and Laura Forrest. The short summary is that we will proceed in parallel with two initiatives:

1) SocialAPI metrics in FHR
2) A larger discussion, with all the relevant stakeholders, to finalize the process and owner(s) for making changes to FHR metrics collection and reporting.

Ideally we can combine the two together but it is not necessary.

Comment 23

[Shane Caraveo (:mixedpuppy)]

implementation will proceed on bug 851653

Comment 24

[Alina Hua]

Closing this bug as implementation is in process. A policy will be developed about collecting metrics and will be tracked in a new ticket.