1306348 - Instrument non-default application usage

Status: NEW

(Firefox :: Downloads Panel, defect, P5)

Description

[:Paolo Amadini]

We want to know how often an application that is not the system default is used to open each file type, versus the default application. We can differentiate between a non-default application provided by the operating system, and a non-default application whose path was previously chosen by the user.

Comment 1

[Morpheus Chen [:morpheus] UX]

Hi Paolo, 

Thanks for highlighting the topic. 

If we understand correctly, it seems the instrument here tries to build Telemetry to collect usage data to validate if the handler choice dialog is needed or not. However, from UX perspective, we would suggest validating the assumption by User Testing after implementing the spec in Nightly. Here we try to list the reasons of why we don’t think Telemetry is appropriate for this project, and why User Testing might be a better alternative to bridge the gap.

While Telemetry, an quantitative research tool, is generally great for getting statistically significant data on the usage of a feature. It can lead to many biases due to its natural limitations of the probe subject.

1. Most participants will be limited to only Nightly users (Almost all users are tech savvy), which is unable to represent our general users.
2. It’s difficult to set a convincing threshold to validate the assumption before implementing the probes.
3. Cost time and resources to implement the probes, which may delay the schedule.
4. The last and the most important point, the amount of the participants who choose to open downloaded files with the non-default application can only reflect whether users has the need to open a file with OS app (whether default or non-default), but it can’t reflect where the right place/timing is to provide the option to our users. Also, there is no way to know the user’s mental model by simply examining the usage data of the dialog.

According to the limitations above, we would recommend having user testing instead of Telemetry. We haven’t had the entire research plan, but we are working on the draft with Firefox UR team to use usertesting.com to compare the new design and the current design on Nightly. We’ll ensure it’s a more reliable and validated method to fulfill the original goal of this instrument.

Comment 2

[:Paolo Amadini]

(In reply to Morpheus Chen [:morpheus]  UX from comment #1)
> If we understand correctly, it seems the instrument here tries to build
> Telemetry to collect usage data to validate if the handler choice dialog is
> needed or not.

As I've said previously, we know we would like to remove the handler choice dialog, so we don't need the data for that purpose. The purpose is to understand if, for certain content types, there is a prevalence of non-default application usage, as this may point us to a direction that we didn't think to investigate before.

> 1. Most participants will be limited to only Nightly users (Almost all users
> are tech savvy), which is unable to represent our general users.

This is incorrect. Telemetry data is collected on Nightly, Developer Edition, and Beta by default. With proper review and in case of necessity, we can also collect data from users on Release who have not opted out.

> 2. It’s difficult to set a convincing threshold to validate the assumption
> before implementing the probes.

In some cases, and I'm not saying this applies to this case because the purpose of the collection is different, we use the 80-20-2 rule. Obviously the problem of having to think about thresholds in advance does not apply to, say, user testing videos because of their qualitative nature. It does apply to mass user testing like A/B studies though.

> 3. Cost time and resources to implement the probes, which may delay the
> schedule.

We probably have different perceptions about the complexity of implementing a telemetry probe, but if schedule is an issue you can seek help and have Telemetry implemented by an external developer.

> 4. The last and the most important point, the amount of the participants who
> choose to open downloaded files with the non-default application can only
> reflect whether users has the need to open a file with OS app (whether
> default or non-default), but it can’t reflect where the right place/timing
> is to provide the option to our users. Also, there is no way to know the
> user’s mental model by simply examining the usage data of the dialog.

I totally agree, that's why qualitative user testing is also important.

> According to the limitations above, we would recommend having user testing
> instead of Telemetry. We haven’t had the entire research plan, but we are
> working on the draft with Firefox UR team to use usertesting.com to compare
> the new design and the current design on Nightly. We’ll ensure it’s a more
> reliable and validated method to fulfill the original goal of this
> instrument.

If after reading the considerations above you still think you do not need these telemetry probes, whether implemented by the team or by an external developer, feel free to close this bug.

Comment 3

[Morpheus Chen [:morpheus] UX]

Thanks for your feedback, please find my comments inline.

(In reply to :Paolo Amadini from comment #2)
> (In reply to Morpheus Chen [:morpheus]  UX from comment #1)
> > If we understand correctly, it seems the instrument here tries to build
> > Telemetry to collect usage data to validate if the handler choice dialog is
> > needed or not.
> 
> As I've said previously, we know we would like to remove the handler choice
> dialog, so we don't need the data for that purpose. The purpose is to
> understand if, for certain content types, there is a prevalence of
> non-default application usage, as this may point us to a direction that we
> didn't think to investigate before.
> 
> > 1. Most participants will be limited to only Nightly users (Almost all users
> > are tech savvy), which is unable to represent our general users.
> 
> This is incorrect. Telemetry data is collected on Nightly, Developer
> Edition, and Beta by default. With proper review and in case of necessity,
> we can also collect data from users on Release who have not opted out.

Thanks for the clarification. Actually, we consulted with the Telemetry team on early August, including Matthew Grimes and Ilana Segall. What we heard from them was that currently there are only 1% general users in Firefox that is opted in to extended telemetry by default, and the Telemetry team are working with legal and privacy to have more measures added to the default data set. We were wondering that "collect data from users on Release who have not opted out" which you meant is the same as the 1% sample?

> > 2. It’s difficult to set a convincing threshold to validate the assumption
> > before implementing the probes.
> 
> In some cases, and I'm not saying this applies to this case because the
> purpose of the collection is different, we use the 80-20-2 rule. Obviously
> the problem of having to think about thresholds in advance does not apply
> to, say, user testing videos because of their qualitative nature. It does
> apply to mass user testing like A/B studies though.

For the 80-20-2 rule, we are very interested in the details and how this can apply to the telemetry data, could you elaborate more for our understanding? Thanks. 

> > 3. Cost time and resources to implement the probes, which may delay the
> > schedule.
> 
> We probably have different perceptions about the complexity of implementing
> a telemetry probe, but if schedule is an issue you can seek help and have
> Telemetry implemented by an external developer.

Our concern is the engineers of this project don't have prior experience on telemetry. So it might take a while to familiar with the implementation. However, it would wipe out the concern if we can have external developer help.

> > 4. The last and the most important point, the amount of the participants who
> > choose to open downloaded files with the non-default application can only
> > reflect whether users has the need to open a file with OS app (whether
> > default or non-default), but it can’t reflect where the right place/timing
> > is to provide the option to our users. Also, there is no way to know the
> > user’s mental model by simply examining the usage data of the dialog.
> 
> I totally agree, that's why qualitative user testing is also important.
> 
> > According to the limitations above, we would recommend having user testing
> > instead of Telemetry. We haven’t had the entire research plan, but we are
> > working on the draft with Firefox UR team to use usertesting.com to compare
> > the new design and the current design on Nightly. We’ll ensure it’s a more
> > reliable and validated method to fulfill the original goal of this
> > instrument.
> 
> If after reading the considerations above you still think you do not need
> these telemetry probes, whether implemented by the team or by an external
> developer, feel free to close this bug.

Since the purpose is just to understand the non-default application usage for reference in the future and this won't impact our current design decision, we don't think UX is entitled to make the final call. We would suggest EPM or tech lead to chime in because this bug would probably result extra efforts and delay of schedule.

Comment 4

[Ben Tian (inactive)]

(In reply to Morpheus Chen [:morpheus]  UX from comment #3)
> Since the purpose is just to understand the non-default application usage
> for reference in the future and this won't impact our current design
> decision, we don't think UX is entitled to make the final call.

As the telemetry result won't impact UX design, this bug and bug 1306349 should not block the implementation meta bug 1306338. Remove the dependencies accordingly.

If any developer is interested in this bug or bug 1306349, feel free to take and work on them.

Comment 5

[:Paolo Amadini]

(In reply to Morpheus Chen [:morpheus]  UX from comment #3)
> Thanks for the clarification. Actually, we consulted with the Telemetry team
> on early August, including Matthew Grimes and Ilana Segall. What we heard
> from them was that currently there are only 1% general users in Firefox that
> is opted in to extended telemetry by default, and the Telemetry team are
> working with legal and privacy to have more measures added to the default
> data set. We were wondering that "collect data from users on Release who
> have not opted out" which you meant is the same as the 1% sample?

You can set the "releaseChannelCollection" property to "opt-out" in "Histograms.json" so that data from the specific probe will be collected from all Release users even if they don't take a specific action to opt in to the collection. So, release users only send a limited quantity of data by default.

From the "data choices" preferences panel, Release users can choose not to send any data at all, or to send additional data that users from other channels would normally send by default.

I'm not sure about the 1% you're talking about. It might refer either to the number of Release users that opt in to sending information for all probes, and not only those explicitly marked as opt-out, or to the fact that we limit how many requests are sent by Release users to reduce server load.

> For the 80-20-2 rule, we are very interested in the details and how this can
> apply to the telemetry data, could you elaborate more for our understanding?

An example could be a simple UI feature like "undo close tab". We measure which percentage of users actually use this feature, applying specific UI telemetry. If 80% of users or more use this feature, then it definitely needs to be prominent in primary UI. If 20% or less use this feature, then it belongs to secondary UI, for example only accessible through a context menu. But if it's used by less than 2% of users, then it's probably something that doesn't need to be in the product and an add-on could do. As you can imagine, this is just a rule of thumb and might vary from case to case, and specific design considerations still need to apply. It's worth keeping this rule in mind when in doubt about setting thresholds.