1181295 - FTE should allow User to enable collection of some metrics, and disable others

Status: VERIFIED FIXED

(Firefox OS Graveyard :: Gaia::First Time Experience, defect)

Description

[Ravi Dandu]

Metric collection on the device has various components, some of which are default on, and some default off. FTE should give the user the ability to opt-out/in out of different kinds of information. Here is a proposal to converge the User experience of opt-in/opt-out process. 

1) FirefoxOS Health Report (on by default): eg: Collects App Usage info, Num of searches
          1a) Telemetry (off by default): Collects App Startup time, Bootup time  

2) Dogfooding (developer) flag will be a different setting, not in the FTE which will enable more detailed information.  

This will enable the various metrics collection in privacy centric method.

Comment 1

[Ravi Dandu]

Marshall, Can you weigh in on terminology 
Option 1: 
      - FxOS HealthReport (opt-out) eg: App usage
         - Telemetry  (opt-in) ex - number of calls made, amount of data use
Option 2:
      - Info about Usage (opt-out)
      - Info about Device (opt-in)

Comment 2

[Ravi Dandu]

Option 3:
      - Basic (opt-out): App Usage
      - Enhanced (opt-in):  Includes Basic + num of calls/data + warnings/errors + performance data like app/device startup time  
      - Full(opt-in): Includes Basic + Enhanced + collecting engg performance data like memory usage, async anumation failures, jank, time spent on a panel.

Comment 3

[Tiffanie Shakespeare [:tif]]

Adding NI to myself

Comment 4

[Tiffanie Shakespeare [:tif]]

Sorry - I don't know why bugzilla didn't do it. Trying again.

Comment 5

[Tiffanie Shakespeare [:tif]]

In this latest WIP spec, I've added in the IA (page 6) and current screens (page 8) for FTE for reference.

On page 9, I've removed the Full Opt-in option. As you will see, while scrolling was eliminated in option 2 and 3, option 1 still scrolls. Keep in mind this is English. I'm confident that localization will create a scrolling screen even for option 2 and 3.

I have come up with additional two options that keep the "welcome" text but alleviate the clutter of the telemetry page (pages 11 and 12).

The purpose of Alternative A is to prevent scrolling on the telemetry screen by moving the welcome paragraph to the screen asking for the email and moving it so that it appears first. The issue is that we have overloaded this screen and it runs the risk of scrolling.

The purpose of Alternative B is to keep both the telemetry and email screen clean and simple by creating an additional screen for the "welcome" paragraph that is shown directly after the language selection. It makes more sense to welcome the user at this point than wait until the 2nd to last screen of FTE. The issue though is we are adding an additional screen to an already very long FTE (see IA on page 6).

Because of this, I would still suggest cutting the "welcome" paragraph (see page 10) and putting both privacy and FxOS links on the telemetry page (and/or adding to email page too). Users don't read and if we overwhelm them with text they will be more likely to opt-out and not provide their email. I know you think it's valuable to keep, but I also feel we shouldn't add another screen to FTE.

NI to Ravi for comment on the proposals. Thanks!

Comment 6

[Sam Foster [:sfoster] (he/him)]

Having a whole screen for the welcome message with no actions or choices (Alternative B) doesnt seem to add much value IMO. I do like moving it up to the front of the sequence, but as we don't have any connection at that point, we cant offer FxA signup or newsletter there - which makes it just a wall of text. 

There's another option for Alternative A - though its a bit more involved implementation-wise. We could combine the Firefox Accounts and Welcome screen (most of the FxA screen is the logo) I.e. the action you can take after reading our nice greeting is to sign up for a firefox account. Then, rather than toggling on/off the whole FxA screen if we don't have a network connection, we would just hide that block. I guess we run into the same risk here of having the sign-up button pushed below the fold in some locales... 

I've always been a bit sad that our welcome page only offers a newsletter sign-up as a "welcome present".

Comment 7

[Ravi Dandu]

I'm fine option 7 which is the simple paragraph description. 
Another important aspect of it is the description of the  "basic", "enhanced" will add more lines. Currently it is a single line which does not convey much. Can you add the lines in the column 2 ("wording") and see how it looks. https://docs.google.com/document/d/1iEDhJQS0ymaWdOe1OuVMuB31qTDl9tDZekqTsmKS37E

Comment 8

[Tiffanie Shakespeare [:tif]]

Hey Ravi - as we discussed over IRC, the copy is too long to fit into the UI. The supporting message for the two options should be kept to no more than two lines (preferably one line).

Ravi, mentioned that he would take a stab at reducing copy. I would strongly recommend keeping the copy strictly to stating what type(s) of data will be sent - e.g. "Send App and Search use information (such as number of times I use an app)." but obviously in a more elegant way. :)

Please don't hesitate to include me in the process, I will be happy to help you. Katie and I have gotten quite good at distilling complicated ideas down to a single sentence (SSL work) :)

Until we finalize copy, spec will be incomplete. Thanks!

Comment 9

[Tamara Hills [:thills]]

Ravi,

I am reading all the comments here and just want to clarify that this is now actionable.  The way I'm reading it is that we are replacing the current screen which says "About B2G OS" to Option 7 in Tiffanie's spec.  

I also want to make sure that there are no other changes to the flow (e.g. we are not changing the order in which this screen shows.)

Thanks,

-tamara

Comment 10

[Merwin]

Ravi, I made a few suggestions in the Google Doc for how to shorten these.

Comment 11

[Ravi Dandu]

   The order of showing the screen should not change. 

   The radio options in Tiffanie's document are currently in the order "basic, enhanced, none". Order "none, basic, enhanced" will denote increasing information collection. The latter order might be easier to understand for user.

Comment 12

[:Dominik Strohmeier [:dstrohmeier]]

>    The radio options in Tiffanie's document are currently in the order
> "basic, enhanced, none". Order "none, basic, enhanced" will denote
> increasing information collection. The latter order might be easier to
> understand for user.

I have a question regarding the order of the radio buttons. I assume that we will have more people not changing the settings if the first option is enabled which would be "basic". I have no evidence that this is correct, just an assumption based on my behavior. So, "basic, enhanced, none" might actually a good option to go with.

Tiffanie, what do you think? My goal would be to maximize the number of people that don't change their option to "none".

Comment 13

[Tiffanie Shakespeare [:tif]]

By listing basic first, we are showing preference towards that option. As long as 'none' is visible on the screen it will not be seen as nefarious and the supporting text makes it very clear what none means.

Comment 14

[Ravi Dandu]

ok, Tiffanie, if UX thinks it is not confusing, that is fine.
Lets go with "basic, enhanced, none" order.

Comment 15

[Tiffanie Shakespeare [:tif]]

What is the status of the supporting text? So far I see this in the document:

Basic - Basic data includes app usage data and data that is vital to the operation of FirefoxOS.

Enhanced - Enhanced data includes basic data + diagnostic information that can be used to understand and improve FxOS. 

"data that is vital to the operation of FxOS" doesn't really tell me what that means - it also seems to imply that not sending data may impact device operation. 

The same goes for "can be used to understand and improve FxOS" - why does FxOS need to be understood? How does this data improve things?

I would make these much simpler and reduce further to make it easier to understand for the user:
Basic - Send app usage and error report.

Enhanced - Send Basic Report and diagnostic information. 

Still needs a bit more wordsmithing but you get the gist. I've reviewed with UX so if you agree with this proposal then I can get them reviewed by the copywriters and they can help clean it up.

Thanks!!

Comment 16

[Tamara Hills [:thills]]

Ravi,

We need to talk about dogfooders.  

Current behavior for dogfooders is for them to have the single checkbox checked and the checkbox disabled.

For dogfooders, we will disable the radio buttons, but the question is whether we want them to have Basic or Advanced as the pre-selected option.

thanks,

-tamara

Comment 17

[Tiffanie Shakespeare [:tif]]

I created a quick spreadsheet for the strings on the screen.

https://docs.google.com/a/mozilla.com/spreadsheets/d/1yimxKEeZYWaas7qbTLcEhmejpRJGp3Z6pQri8doe1jw/edit?usp=sharing

Tamara, for dogfooders can we tweak the UI? I would be pretty confused if all the options were disabled and there was no information as to why I'm unable to change something. Even just adding in something about sending data is required as part of foxfooding program would be better than nothing.

Comment 18

[Sam Foster [:sfoster] (he/him)]

Implementation-wise, I'm not crazy about baking in dogfooding-specific logic into the FTU.. We already have the disable-checkbox-if-dogfooder logic. Branching the UI further for a condition that will *never* be met in a standard production release seems like we're going about this the wrong way. Either we need a more generic (i.e. not just for foxfooders, but for any carrier/vendor) way to preset this value - and present UI to inform the user what that preset is - or we need a way to replace the normal telemetry opt-in panel for the dogfooding builds.

Comment 19

[Ravi Dandu]

(In reply to Tiffanie Shakespeare [:tif] UX from comment #15)
> "data that is vital to the operation of FxOS" doesn't really tell me what
> that means - it also seems to imply that not sending data may impact device
> operation. 
The intent for string "data that.." is not have to list each item in the FTE screen. The "privacy link" would have details. I'm ok with your suggestion "Basic data includes app usage and error data" 

> The same goes for "can be used to understand and improve FxOS" - why does
> FxOS need to be understood? How does this data improve things?

"Usage" and "Performance" needs to be understood. This is used to fix performance degradation, make data driver decisions on fixing issues. 

The latest proposal is in column 2 "wording" of https://docs.google.com/document/d/1iEDhJQS0ymaWdOe1OuVMuB31qTDl9tDZekqTsmKS37E/edit#heading=h.jw91r6h9m4pf

Comment 20

[Ravi Dandu]

About the dogfooding discussion: collection for dogfooding is more than commercial devices (specifically IMEI is collected in dogfooding and extremely data intensive items will not be in production).

The priority for this screen is the standard production release.  Lets use wiki's and other methods to give info on foxfooding, and not complicate this too much. For now, I'm ok with just disable this screen completely for foxfooding.

Participation team (William Quiviger) have a phased approach of how to evolve foxfooding.  We do need to converge foxfooding plan into the commercial devices plan in the longer run. William is going to setup a meeting for that. Lets keep 2.5 FTE metrics separate from foxfooding for now.

Comment 21

[Tiffanie Shakespeare [:tif]]

Excellent! We are coming to a consensus for Basic. I would still argue for using "report" instead of "data". Not only is report more accurate but it's also an easier concept. 

For Enhanced option, the copy needs to be reduced.

Since we can't say all that we want to say in a single sentence, we need to focus on what is being sent to Mozilla. We need a super simple sentence to convey the idea as the privacy link will have a lot more details about what is being collected and why. 

Enhanced: Send diagnostic information plus Basic report.
OR if more accurate
Enhanced: Send performance information plus Basic report.

Comment 22

[Ravi Dandu]

Hi Tiffanie, is the screen size not enough to display the current copy for enhanced option? 
If UX insists on reducing, I prefer "Send diagnostic information plus Basic report" rather than "Send performance information", as diagnostic is more generic. 
 
Marshall, comments from legal perspective?

Comment 23

[Merwin]

I think both of these are fine.  Adding Elvin in case he wants to comment. Regardless of the language we use on the device, we will want to be more clear in the PN.

Comment 24

[Tiffanie Shakespeare [:tif]]

Excellent! I will send over to our copywriter for final approval. Thanks!

Comment 25

[Elvin Lee [:ellee] (no longer with Mozilla)]

Just want to clarify where you came out on the dogfood UX. From my reading of the above, this particular screen won't be presented to dogfooders at all based on the idea that the choice doesn't apply to them.

Is that correct?

Comment 26

[Tiffanie Shakespeare [:tif]]

That point is still under discussion. I believe we are discussing this later this afternoon.

Comment 27

[Tamara Hills [:thills]]

In terms of settings we decided that we will have one setting that is the source of truth and it's value is the the value of the 3-state radio button.

We will do the following:
1.  Remove the telemetry checkbox in settings|Developer|developerHUD
2.  Replace the submit data checkbox in settings|B2g OS|submit data with the three state radio button.
3.  Working assumption is to disable the radio buttons for dogfooders with 'Enhanced' selected.  We will continue discussion with :sfoster on this and any changes would be in another bug.
4.  Open a new bug for the settings UI changes as those need to be reviewed by separate peers from the FTE app.

Thanks,

-tamara

Comment 28

[Tiffanie Shakespeare [:tif]]

Hey guys - I was taking a look at the settings -> Firefox OS screen. It's super strange to have the three radio buttons above the App Usage line item. 

Previously, the send data was a single item so having app usage under wasn't a big deal. But now it feels like app usage is being lumped in with the three radio buttons and it's confusing. It makes one's brain work harder to understand what is going on in the UI.

Can we do one of these (in order of preference):
a) move app usage to under send mozilla feedback
b) move app usage to its own header
c) move app usage to above the "help improve paragraph"

Thanks

Comment 29

[Russ Nicoletti [:russn]]

I like option b). I believe that would be the cleanest approach, would help avoid the screen from becoming to cluttered.

Comment 30

[GitHub Autolander Bot]

Attachment: [gaia] tamarahills:bugfix/1181295-fte-allow-user-metrics-choice > mozilla-b2g:master

Comment 31

[Tamara Hills [:thills]]

My .02 is also option b.  I think option a could be confusing as sending statistics is different than sending feedback.  To me, feedback implies something I type in about my sentiment vs. something I do or something which is collected as a result of my actions on the phone.

(In reply to Tiffanie Shakespeare [:tif] UX from comment #28)
> Hey guys - I was taking a look at the settings -> Firefox OS screen. It's
> super strange to have the three radio buttons above the App Usage line item. 
> 
> Previously, the send data was a single item so having app usage under wasn't
> a big deal. But now it feels like app usage is being lumped in with the
> three radio buttons and it's confusing. It makes one's brain work harder to
> understand what is going on in the UI.
> 
> Can we do one of these (in order of preference):
> a) move app usage to under send mozilla feedback
> b) move app usage to its own header
> c) move app usage to above the "help improve paragraph"
> 
> Thanks

Comment 32

[Tamara Hills [:thills]]

Comment on attachment 8666389 [details] [review]
[gaia] tamarahills:bugfix/1181295-fte-allow-user-metrics-choice > mozilla-b2g:master

Hi Russ,

This is a very rough patch so you can see where it's going.  I have not put any effort into the tests as of yet.

So, I made the following changes in the FTE based on this:
1.  created a new setting called 'metrics.selectedMetrics.level' which has three different settings:  'Basic|Enhanced|None'.
2.  Removed the other settings and just now have this one setting
3.  Changed all the Gaia references to the old setting to now depend on what the new setting is.
4.  I had to add some styles to the style.css for the FTU to get the gaia-radio button to work.... also the .disabled styles were needed.  

I have not done the Gecko patch as of yet.  It's fairly simple and just looks at the new value.

Let me know if you see any issues with the approach, obviously lot of cleanup to do and fix tests.

thanks for your feedback.

-tamara

Comment 33

[Russ Nicoletti [:russn]]

Comment on attachment 8666389 [details] [review]
[gaia] tamarahills:bugfix/1181295-fte-allow-user-metrics-choice > mozilla-b2g:master

Looks good to me. I left some nits in the PR.

Comment 34

[Tiffanie Shakespeare [:tif]]

LMK when it's ready for UI-review. Thanks guys! :)

Comment 35

[Tamara Hills [:thills]]

Comment on attachment 8666389 [details] [review]
[gaia] tamarahills:bugfix/1181295-fte-allow-user-metrics-choice > mozilla-b2g:master

Hi Sam,

Would you mind feedback'ing this for general direction?  I still have some work to do on the UX based on latest from Tiffanie,

Thanks,

-tamara

Comment 36

[Sam Foster [:sfoster] (he/him)]

Comment on attachment 8666389 [details] [review]
[gaia] tamarahills:bugfix/1181295-fte-allow-user-metrics-choice > mozilla-b2g:master

Left a note in the PR about all the UI code added to navigation.js, but I'm not too concerned about addressing it in this patch - I have others in-progress where I can move this stuff around if necessary. Functionally lgtm.

Comment 37

[Tiffanie Shakespeare [:tif]]

FYI - I fixed a typo in the spec. Also finalized Settings layout per feedback. Updated version posted. Thanks!

Comment 38

[Tamara Hills [:thills]]

Comment on attachment 8666389 [details] [review]
[gaia] tamarahills:bugfix/1181295-fte-allow-user-metrics-choice > mozilla-b2g:master

Hi Sam,

This updated version has added the l10n ids as well as some additional styling for the text on the radio buttons.

Thanks,

-tamara

Comment 39

[Sam Foster [:sfoster] (he/him)]

Comment on attachment 8666389 [details] [review]
[gaia] tamarahills:bugfix/1181295-fte-allow-user-metrics-choice > mozilla-b2g:master

Looks good, apologies for the delay.

Comment 40

[Tamara Hills [:thills]]

Try run: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=dd19eaacdc60983fd4dab2c37f48faff82c56a1b

Comment 41

[Tamara Hills [:thills]]

https://github.com/mozilla-b2g/gaia/commit/3159ee2b39e7eaa3a215ba58acdd194e53bd75e4

Comment 42

[Iris Hsiao [:ihsiao]]

Verified@
Build ID               20151106013953
Gaia Revision          1fb7e4b7904d577a1859b91269a85b363ddc2836
Gaia Date              2015-11-06 02:30:49
Gecko Revision         https://hg.mozilla.org/integration/mozilla-inbound/rev/f073181a927d63ae7abac86e47cc3872ccac7275
Gecko Version          45.0a1
Device Name            aries
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.worker.20151106.005858
Firmware Date          Fri Nov  6 00:59:06 UTC 2015
Bootloader             s1

Verify result:
Users are able to choose if they want to give Mozilla information about their device and usage or not during FTE.