Closed Bug 805575 Opened 12 years ago Closed 12 years ago

Privacy notification UX for stub installer

Categories

(Firefox :: Installer, defect)

Product:
Firefox
Component:
Installer
Version:
17 Branch
Platform:
x86_64
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: robert.strong.bugs, Assigned: lco)

References

Details

(Whiteboard: [stub+] [qa-])

Attachments

(2 files, 1 obsolete file)

schematic for several phases of Firefox instance life cycle
111.80 KB, application/pdf
Details
Adding Privacy Item to Options Pane
417.00 KB, image/png
Details 
Per bug 804231 we will need a user experience design for the stub installer to send a ping.
Whiteboard: [stub+]
I suggest that the stub ping only happens for funnelcake builds and that this privacy notification UI is only displayed for these builds.
For non funnelcake we could always notify on failure. For the notification I would prefer to use a messagebox and keep it dirt simple as I suggested in email last night.
Design directions we're looking into, if we need this feature (relatively in the order of desirability, in my opinion)

1. Show a dialog + message when stub installer fails only. Use it as an opportunity to direct the user to the regular installer, and ask the user to send data so that we can fix it in the future.

2. Add a checkbox to the first screen (possibly under "make Firefox my default browser") to ask for permission to send data. This requires some UI rework, and it's going to be pretty crammed, but it's not the end of the world.

Note that in both cases, I agree with the idea that this shows up only on funnelcake builds.

Is engineering ok with idea #1? If not, Stephen, want to take a crack at the layout for #2? I can help with the text or ideas
(In reply to Larissa Co from comment #3)
> Design directions we're looking into, if we need this feature (relatively in
> the order of desirability, in my opinion)
> 
> 1. Show a dialog + message when stub installer fails only. Use it as an
> opportunity to direct the user to the regular installer, and ask the user to
> send data so that we can fix it in the future.
> 
> 2. Add a checkbox to the first screen (possibly under "make Firefox my
> default browser") to ask for permission to send data. This requires some UI
> rework, and it's going to be pretty crammed, but it's not the end of the
> world.
> 
> Note that in both cases, I agree with the idea that this shows up only on
> funnelcake builds.
> 
> Is engineering ok with idea #1? If not, Stephen, want to take a crack at the
> layout for #2? I can help with the text or ideas
Engineering is ok with idea #1 though metrics is another stake holder that needs to provide input.

For the second case, can we get away with a similar experience where we prompt the user after the installation since it will only be for a small percentage of people?
For clarity, I am proposing that:

with funnelcake builds we always show the privacy notification and we always ping for both failure and success. Since this only applies to a small percentage of our users I would prefer to use a simple notification messagebox after the installation has completed.

with non-funnelcake builds we might choose to gather only failure data and would show the privacy notification to send the ping data.
Actually, I don't believe funnelcake should read on our decision here.

This week's funnelcake test will go out without the metrics ping. When we do get the ping built, reviewed, approved and delivered, we're going to want it for all installs, so that we can identify issues with our install flow or infrastructure for years to come. We need a general solution here, not an FC-specific one.
(In reply to Johnathan Nightingale [:johnath] from comment #6)
> Actually, I don't believe funnelcake should read on our decision here.
> 
> This week's funnelcake test will go out without the metrics ping. When we do
> get the ping built, reviewed, approved and delivered, we're going to want it
> for all installs, so that we can identify issues with our install flow or
> infrastructure for years to come. We need a general solution here, not an
> FC-specific one.
OK... I am concerned though about adding more and more stuff to the stub as is what happened to the full installer. :(

With this being the case then the very first page (Intro) will need another control that is also shown on the 2nd page (Options) to allow the user to decide whether to send the ping along the lines of the 2nd option.
Also, keep in mind when designing this that this is an installer and it is not a web page. If it is decided that we need to have a hover popup it will increase the stub's size. If it is decided that we need to have a link it will increase the stub's size. etc.
Also also, keep in mind that this is localized and we will need plenty of space to handle all locales so the checkbox text must be relatively short. With this and comment #8 in mind it might be best to display a "new" page in order to display the privacy statement though how to initiate that I have no idea.
I'm concerned about that, too.

I think metrics, ux, and privacy have their work cut out for them. The data's useful, but we either need to find an excellent and acceptable way of collecting it, or we don't get to have it. Given that this is our first contact with our users, I'd like the design to be very excellent indeed. Most excellent might well be to devise ways to gather it without the installer at all (server side, CDN metrics, &c). I'm not ruling out installer-side collection, that's the whole point, but I don't want people's first contact with us to be a UI conversation about installer telemetry. It would be far better not to have the data, than to end up there.

Larissa, Sid, Tom, Anurag, Gilbert - who's gonna go get some people together and figure stuff out?
(In reply to Johnathan Nightingale [:johnath] from comment #10)

> Larissa, Sid, Tom, Anurag, Gilbert - who's gonna go get some people together
> and figure stuff out?

It would help to cc those people. Done.
It appears to me that all the stakeholders on the cc for this bug.

http ://cl. ly /image/2N0X1J3z3l1U/o 

the foregoing URL is what I have to hand as mockups for the SI screens. At one point Tom and I discussed having the check-box for SI ping submission on the second screen - near the pre-check that makes FF your default browser.

I am assuming that the need for an intimate measure of the efficacy of SI still exists.
That second screen is encountered only when users choose to view the "options" and I understand Tom to be asserting that that is insufficient. I'm not entirely sure I'm persuaded of that, but I wouldn't *need* to be persuaded if we had an elegant and fluid design that accomplished all of our goals without user discomfort.

When we were debating the question of opt-in vs. opt-out for search suggestions on Android, we spent a great deal of time arguing about the hypothetical merits of the situation before actually trying out some designs. The opt-in design we ended up with was attractive and minimally disruptive and made it much easier to accept the cost, as weighed against the benefit of better adherence to the privacy principles.

What I take from that lesson is that I should avoid reacting to the privacy concerns (which, tbh, seem pretty minor, though worth considering) without also understanding the best we can do to address them without user grief. The difference between this situation and the android search suggestions one is that we have an additional variable that might read on the discussion, namely: the set of metrics we collect.

What I'm asking then, is for privacy, UX, and metrics to meet (maybe here, but maybe in some higher-bandwidth medium) and discuss what is needed, and what is possible to deliver that. I would be dismayed to learn that the answer is, "We can't have nice things." 

I fervently hope that something like

[X] Send information about this install process to mozilla

in the secondary-flow options panel will be deemed sufficient, perhaps for a more restricted set of data (is the presence of an existing Firefox install really something that Firefox users would be upset about Firefox knowing?) but I am regularly amazed by the things our design team can do so as far as I'm concerned, the sky is the limit.

> I am assuming that the need for an intimate measure of the efficacy of SI
> still exists.

As am I - certainly the need to improve and optimize our install experience still exists; urgently. (That failures in that experience result in people remaining on other browsers whose privacy record is worse goes, I suspect, without saying.)
(In reply to Johnathan Nightingale [:johnath] from comment #10)
> Given that this is our first
> contact with our users, I'd like the design to be very excellent indeed.
> Most excellent might well be to devise ways to gather it without the
> installer at all (server side, CDN metrics, &c). I'm not ruling out
> installer-side collection, that's the whole point, but I don't want people's
> first contact with us to be a UI conversation about installer telemetry. It
> would be far better not to have the data, than to end up there.
> 
> Larissa, Sid, Tom, Anurag, Gilbert - who's gonna go get some people together
> and figure stuff out?

I think I understand Tom's POV right now, but I would like to understand the Metrics team's POV and try to find ways to answer the questions we want to answer with minimal user interruption. Metrics team, can we set up a small meeting when I get back to MV on Monday?

In the meantime, I'll work on a couple of ideas for a scenario in which we do have to ask user permission to collect data (which I'm suspecting we'll end up having to do for some portion of the time).
Data points for the ping that have been discussed previously for reference

Result code for the installation - success or failure code
Server url that the stub downloaded the install from (so we can evaluate problematic servers and it might be possible to get this info in another manner)
Download duration - seconds (so we can evaluate issues with downloading including different IP networks)
Download amount - bytes (so we can pair with download duration when people cancel the download before it finishes)
Firefox launched code - cancelled before install, failure due to already running, or success (interesting data point to understand if we should do something different about already running instances of Firefox)
Pre-existing Firefox profile - 1 or 0 (allows us to know if this is or was an existing Firefox user)
Install over an existing install - 1 or 0 (allows us to know if this is or was an existing Firefox user)
Install into default location - 1 or 0 (not that important but interesting to understand how much effort we should put into supporting non default install locations or remove it as many other apps do)
Firefox locale installed (not terribly interesting for the stub itself but it could help us understand differences in the data)
Windows version (not terribly interesting for the stub itself but it could help us understand differences in the data)
32 or 64 bit OS (not terribly interesting for the stub itself but it could help us understand differences in the data)
This attached schematic shows how the proposed SI ping sits in the overall funnel for an instance of FF from acguisition, activation, retention, and update/upgrade. It shows the succession of phases for the life cycle of an active instance of the desktop product.

Our ability to measure the dynamics of the life cycle currently depends on relatively indirect methods using server logs. This approach needs to be augmented by more direct methods such as using the existing Block List Ping and proposed Firefox Health Report as shown.  

Having a direct method to understand the dynamics of the SI phase has motivated the new 'completion ping' from the SI phase.  You may have notices that some additional signals from within SI that have been mentioned in this and other bugs. This line of discussion has been deferred for future action.

I hope it is clear that we are tackling the measurement problems here in a comprehensive and more thorough manner.  This approach also makes the benefits and risks more apparent.  In this instance of the SI ping, the need for some considered version of informed user consent was surfaced very quickly.

Hope this helps, happy to provide any further clarifications.
(In reply to Gilbert FitzGerald :GF from comment #16)
> Created attachment 675471 [details]
> schematic for several phases of Firefox instance life cycle
> 
> 
> This attached schematic shows how the proposed SI ping sits in the overall
> funnel for an instance of FF from acguisition, activation, retention, and
> update/upgrade. It shows the succession of phases for the life cycle of an
> active instance of the desktop product.

Thanks for the information!

> Our ability to measure the dynamics of the life cycle currently depends on
> relatively indirect methods using server logs. This approach needs to be
> augmented by more direct methods such as using the existing Block List Ping
> and proposed Firefox Health Report as shown.  

Based on your schematic, there are several pings. Which of these, besides the SI ping and the proposed FX Health Report ping require separate user consent? And which of these requests for data can be combined (from a practical perspective, and also from a privacy perspective?) I want to be very sensitive about asking the user for permission multiple times in the first few interactions they have with the product.

Only the SI ping is addressed by this bug.  

All others are dealt with in separate work tracks.

The data from each ping and in particular the SI ping are devised to be minimal in weight and independent in purpose - overlapping only in a few essentially unavoidable ways.

Thus these pings cannot be combined cross-sectionally except in very aggregate form.  There is no longitudinal structure to any ping except the upcoming FHR data which draws much of its reason to exist from being the first data collect as time series. Thus FHR arises from the need for Mozilla to move away from the position of having zero longitudinal information from sources other than from instances of FF opting in to the self-selecting data submission modalities (such as Telemetry, TestPilot, etc...)
Larissa, should this be assigned to you?
(In reply to Tom Lowenthal [:StrangeCharm] from comment #19)
> Larissa, should this be assigned to you?

OK, I'll take it.
Assignee: nobody → lco
Just had a conversation with Asa, Jim, and Anurag; we had a thought for an alternative way to offer notice and choice:

1. we put the (prechecked) checkbox on the options screen, and 
2. put a little notice on the web page, under the download button (where it currently says "Systems & Languages | What’s New | Privacy").

The notice on the download page can have a link to a SUMO article explaining the stats sent by the installer, and how to turn that off. There are even string suggestions!

Checkbox on the options page:
[ ] Tell Mozilla how this install goes.

Notice under the download button on the web:
Mozilla collects stats about Firefox installs. [Learn more.]

If this works for everyone, we'll need a SUMO page about the ping, including a screenshot of the options page (and probably an explanation about how useful and also not invasive this info is).

Have we missed something?
Just spoke with Tom, and I like the suggestion. Lightweight for the user but still there for those who care. 

(In reply to Tom Lowenthal [:StrangeCharm] from comment #21)
> Just had a conversation with Asa, Jim, and Anurag; we had a thought for an
> alternative way to offer notice and choice:
> 
> 1. we put the (prechecked) checkbox on the options screen, and 
> 2. put a little notice on the web page, under the download button (where it
> currently says "Systems & Languages | What’s New | Privacy").
> 
> The notice on the download page can have a link to a SUMO article explaining
> the stats sent by the installer, and how to turn that off. There are even
> string suggestions!
> 
> Checkbox on the options page:
> [ ] Tell Mozilla how this install goes.
> 

Let's put this as the last element on that screen (after the Destination Folder section. Tom (despite my fear of you saying yes), do we need a "Learn More" link after this choice?

The sentence construction is a little awkward and I think "Tell Mozilla..." just sounds like we're asking for user feedback instead of collecting their data. But I don't have any robust suggestions that are also short, so I'm willing to go with it. A couple of other directions:

"Share installation stats with Mozilla"
"Help improve Mozilla's install experience by sharing stats about this one"


> Notice under the download button on the web:
> Mozilla collects stats about Firefox installs. [Learn more.]

Is "Stats" too informal? 

> 
> If this works for everyone, we'll need a SUMO page about the ping, including
> a screenshot of the options page (and probably an explanation about how
> useful and also not invasive this info is).
> 
> Have we missed something?
This sounds good to me, fwiw. Thanks.
Brian - given what you know of Robert's patch in bug 802734, how difficult would it be to implement what Tom describes in comment 21?
Should be pretty easy to add.  Jonath, would you like me to add a new bug and implement it? The options page checkbox is the same as I suggested in bug 802734 on 10/24.
 
A couple of additional questions:
- Should it be defaulted to unchecked?
- Should it be left aligned with the text "Destination folder" or left aligned with the checkboxes for taskbar, start menu, and desktop? If it is left aligned with the checkboxes, we'll need a new title for the new section.
- I think we could probably just add it to bug 802734 since it's essential to the ping code which hasn't otherwise landed.

- It should be checked by default - that is, the ping should send by default (see comment 21, bullet 1)

- I defer to Larissa on the design question

As for whether you should pick it up and go, I know that Robert warned you would have a lot of b2g stuff keeping you busy, and loathe though I am to say so, that's a higher priority. If you find yourself in a lull, though, I'd welcome your attention here as priority 2.

As a matter of housekeeping, Tom, can we close bug 804231 at this point, and then maybe close this one too once we have a UX agreed to, so that the work itself can go live in bug 802734?
(In reply to Johnathan Nightingale [:johnath] from comment #26)
> - It should be checked by default - that is, the ping should send by default
> (see comment 21, bullet 1)

Thanks, I thought so but wanted to make sure given that the picture of the checkbox below was unchecked.

> Checkbox on the options page:
> [ ] Tell Mozilla how this install goes.



> As for whether you should pick it up and go, I know that Robert warned you
> would have a lot of b2g stuff keeping you busy, and loathe though I am to
> say so, that's a higher priority. If you find yourself in a lull, though,
> I'd welcome your attention here as priority 2.

I'm actually more on Metro as of this week than B2G, so I'll pick this up.  B2G doesn't have a concrete plans on how updating will work yet and all of the blockers are done. Ditto all the signing work they wanted me to do is done.  So I'm just doing work on blockers as they come up.
Re: The alignment question, there is a checkbox for:
[X] Install the Nightly background update service

which is already present under the Destination folder, and left aligned with it, when the Maintenance Service is not already installed.  So I'll just add this checkbox above it. 

Destination Folder
  [                                   ] [Browse]
  Space Required: X
  Space Available: Y

[X] Tell Mozilla how this install goes

[X] Install the Nightly background update service
I agree with Larissa, the sentence is a bit awkward and isn't clear that we're collecting data rather than asking for feedback. I also wonder if it will be clear to everyone at this point in the installation who or what Mozilla is. Could we say "us" instead? Here are a few variations, both with and without Mozilla:

Share information about this installation.

Send us information about this installation.

Send information about this installation to Mozilla.
(In reply to Larissa Co from comment #22)
> 
> > Notice under the download button on the web:
> > Mozilla collects stats about Firefox installs. [Learn more.]
> 
> Is "Stats" too informal? 

Oops, forgot to comment on this. I think it is. And as much as I like familiar language where possible, "installs" doesn't sound right here either. We could be referring to completed installations rather than the process of installing. I'd say something like:

Mozilla collects data about installing Firefox. [Learn more.]
(In reply to Matej Novak [:matej] from comment #30)
> (In reply to Larissa Co from comment #22)
> > 
> > > Notice under the download button on the web:
> > > Mozilla collects stats about Firefox installs. [Learn more.]
> > 
> > Is "Stats" too informal? 
> 
> Oops, forgot to comment on this. I think it is. And as much as I like
> familiar language where possible, "installs" doesn't sound right here
> either. We could be referring to completed installations rather than the
> process of installing. I'd say something like:
> 
> Mozilla collects data about installing Firefox. [Learn more.]

I want to add an obvious flag that this may hurt download conversion rate, which ultimately, will effect Firefox use and our mission at large. A message like this may scare consumers. 

I don't want to block since we need to deploy the stub, but we should run a test with and without this message to see how it effects download conversion rate. I hope that it will prove me wrong. If not, we will get a clearer sense of the cost of doing this.
The wording should be as precise as possible but that comes at a cost of word count in the UX.  

We are endeavoring to male our product better by improving the ease of installations and the following is the key idea:  

"We would like to ascertain that this install COMPLETED SUCCESSFULLY FOR YOU. Please allow us to know that your install went well, so we can improve this aspect of FF for others"

-------------------------------

Running a test with an without has undoubted value but demands that we test one arm of the A/B test without informed consent.
Shorlander says he'll do a quick change to the mockups for the checkbox alignment, but the location Brian suggested is fine with me. And yes, it should be on by default.

Matej's variation of the message that I prefer is:
[x] Send us information about this installation.

And for the webpage, how about something like: 
Mozilla collects data to improve our installation experience. [Learn more.]
(In reply to Larissa Co from comment #33)
> Shorlander says he'll do a quick change to the mockups for the checkbox
> alignment, but the location Brian suggested is fine with me. And yes, it
> should be on by default.
> 
> Matej's variation of the message that I prefer is:
> [x] Send us information about this installation.
> 
> And for the webpage, how about something like: 
> Mozilla collects data to improve our installation experience. [Learn more.]

I like that, but maybe this makes it even clearer:

Mozilla collects installation data to improve the experience. [Learn more.]
> Mozilla collects installation data to improve the experience. [Learn more.]

"...to improve YOUR experience", maybe?

"the experience" seems a little generic.
(In reply to Larissa Co from comment #35)
> > Mozilla collects installation data to improve the experience. [Learn more.]
> 
> "...to improve YOUR experience", maybe?
> 
> "the experience" seems a little generic.

I like it, but I wonder if it's actually accurate. How many times will a user download and install Firefox (i.e. how many more experiences with the installer will they have after this one)?
Attached image Adding Privacy Item to Options Pane (obsolete) — 
Mockup showing added option.
Removed the period for consistency
Attachment #677129 - Attachment is obsolete: true
Howdy folks. Brian needs a call on a final string to use here.

Given shorlander's mockup in comment 38, I don't think we can use "us" since we use "my" in the same page to refer to the user, and the target of first person pronouns shouldn't change within the course of the UI flow. I therefore assert that the checkbox text should be:

Send information about this installation to Mozilla

I just got Matej to over the shoulder it and while he wonders if the Mozilla brand will be foreign to some people, he seems good with it, too.

Gonna call this settled unless someone feels like they need to overrule it?
Works for me! Ship it! :)
Marking RESOLVED. Anyone who disagrees can reopen; otherwise, onward! Thanks, all.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Whiteboard: [stub+] → [stub+] [qa-]
(In reply to Johnathan Nightingale [:johnath] from comment #26)
> As a matter of housekeeping, Tom, can we close bug 804231 at this point, and
> then maybe close this one too once we have a UX agreed to, so that the work
> itself can go live in bug 802734?

Done, noting the conclusions we reached here.

(In reply to Laura Forrest from comment #31)
> I want to add an obvious flag that this may hurt download conversion rate,
> which ultimately, will effect Firefox use and our mission at large. A
> message like this may scare consumers. 

As noted in #804231 comment 12, I think that we need to have a notice *somewhere* before we send the ping, whether that's on the download page or in the installer. Wherever it is, some of the people who see it will wonder what info we're collecting, some won't like it, and some folks will just get spooked.

I think that we make this tradeoff with *every* measure like this that we take. Noting what works and doesn't helps us improve, better serve users, and get more people using Firefox; but the counting or telling-about-counting may concern some users. I see it as a small short-term loss for a large medium- and long-term gain. 

However, measuring people without informing them about it is even worse. We're confident that what we're doing helps users, and is a good deal. We should not be afraid to tell them about it.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: