Closed Bug 1659157 Opened 7 months ago Closed 6 months ago

Add telemetry to track Win 10 installs in related to the system's MSFT verified app setting.

Categories

(Toolkit :: General, task, P1)

task

Tracking

()

RESOLVED FIXED
82 Branch
Tracking Status
firefox82 --- fixed

People

(Reporter: rachel, Assigned: bytesized)

References

(Depends on 1 open bug, )

Details

(Whiteboard: [iu_tracking])

Attachments

(5 files)

Ie, for all Firefox windows users: is their machine setup to install apps from:

"Anywhere", "Recommendations", "PreferStore", or "StoreOnly"?

We should track this both for new installations and for existing installs.

Relevant registry entry here: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\AicEnabled

Group: mozilla-employee-confidential
Assignee: nobody → ksteuber

(In reply to Rachel Tublitz [:rachel] from comment #0)

Relevant registry entry here: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\AicEnabled

This is for Win 10 Pro build 19041, but I suspect there may have been relevant changes made during the Windows 10 lifecycle; the setting itself was added in 15063, and I think it may have at least gained more possible values more recently than that (I think at first it was just Anywhere and StoreOnly). We should look at some older builds before we land anything, to make sure we won't send misleading indicators. I'd also like to check that nothing is significantly different on Win 10 Home; a lot of OEM machines with Win 10 Home seem to ship with the setting on either Recommendations or PreferStore (not certain which), but I don't know whether they're being configured that way specifically or if the default when the registry value is absent is different on Home (for Pro we know it's Anywhere).

(If it winds up being really complicated to get this working on both Win7 AND 10, we can start with Win10 only)

The entire OS feature and the infrastructure around it is all new in Win 10.

(In reply to Molly Howell (she/her) [:mhowell] from comment #1)

The entire OS feature and the infrastructure around it is all new in Win 10.

Aha! Well good at least that simplifies this a bit then. Thanks.

Summary: Add telemetry to track Win 7/10 installs in related to the system's MSFT verified app setting. → Add telemetry to track Win 10 installs in related to the system's MSFT verified app setting.

Comment on attachment 9170089 [details]
Warning screen from Windows 10

Screen encountered while downloading from Chrome on lenovo ideapad S340 15"
windows 10 home

Thanks Tyler, that fills in a couple holes I was wondering about (the model of this machine and whether it was indeed running Home).

(for context, I was shown the above screenshot, and replied with a link to this bug in the interest of gathering all relevant info)

FWIW I have windows 10 pro and can still get it into a state where I see this screen as well by adjusting settings. So it's not limited to Home.

(In reply to Rachel Tublitz [:rachel] from comment #6)

FWIW I have windows 10 pro and can still get it into a state where I see this screen as well by adjusting settings. So it's not limited to Home.

Same here, there is a Windows setting allowing to choose custom behavior per applicatiuon origin with 4 states:
1 Run regardless of origin
2 Run regardless of origin but notify me if a comparable app exists on the store
3 Run regardless of origin but warn me if the app does not come from thre store (this is the case here)
4 Store apps only

For info we hit a somehow similar case with 10S where we initially attempted to quantify such cases in terms of downloads: https://github.com/mozilla/bedrock/issues/8139

https://docs.microsoft.com/en-us/windows/win32/devnotes/wldp-windows-lockdown-mode was the API used to identify browsers running in locked mode on moz.org - see all details on https://github.com/mozilla/bedrock/issues/6367

It would be super valuable to determine if Edge or Edgium exposes similar "locked" information that would reflect one of the 4 states above (so we could then quantify the size of the issue on moz.org), osmode or ossku on https://docs.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/platform-apis/mt795399%28v%3dvs.85%29 may be reflecting this value?

I've run some testing to examine how reliable it is to use the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\AicEnabled registry entry to determine which setting is selected. I tested on 3 Windows builds:

Version 1703, Build 15063.0:

  • I believe that this is the build that introduced this feature.
  • Default setting is "Allow apps from anywhere".
  • Registry entry does not exist by default, but is set to "Anywhere" if the setting is changed and changed back.
  • Cycling through the available options shows that they correspond to these possible values in the registry: "Anywhere", "PreferStore", and "StoreOnly".

Version 1709, Build 16299.19:

  • Default setting is "Allow apps from anywhere".
  • Registry entry does not exist by default, but is set to "Anywhere" if the setting is changed and changed back.
  • Cycling through the available options shows that they correspond to these possible values in the registry: "Anywhere", "PreferStore", and "StoreOnly".

Version 2004, Build 19041.450:

  • I believe that this is the most recent release build.
  • Default setting is the same, though the text has changed to simply "Anywhere"
  • Registry entry does not exist by default, but is set to "Anywhere" if the setting is changed and changed back.
  • Cycling through the available options shows that they correspond to these possible values in the registry: "Anywhere", "Recommendations", "PreferStore", and "StoreOnly"

I believe that this is sufficient to verify that the suggested registry entry and its data are stable and can be used for this purpose.

It is very possible that this doesn't correctly identify the restrictions of Windows 10S. However, I'm not too concerned with that since there shouldn't be any way for Firefox to be installed on Windows 10S anyways, so this code will never run there.

It does seem relevant to the implementation of this probe that the registry entry may not exist.
@Romain- Given that, how would you like the probe to work?

Option 1: Just report the value of the registry entry, as-is.
Option 2: Determine if the build we are running on has this feature (i.e. is Windows build 15063 or later), and report "Anywhere" if the registry entry is missing. On earlier versions report something like "NoSuchFeature", or nothing at all.

Option 2 is somewhat more effort, but I'm happy to do it either way.

Flags: needinfo?(rtestard)

@Romain- One additional question. We are going to run into the same problem here that we ran into for the profile count. When do we want this collected? Do we want to use that same Idle callback again? If not, when would be better?

@Romain- I'm planning to put you as the person responsible for this probe. Does that sound ok? Also, do you want this to expire and, if so, in what version?

Hi Kirk, here are my comments:

  • OK for being the person responsible for the probe
  • Option2 would be ideal I think since it may be that MS changes behavior on next Win10 version so even if documentation is good now, it may become irrelevant later when MS ships a change.
  • Regarding timing of the collection - I may be missing something but I don't feel like this is critical to get early at startup the way profile count was so it seems ok to collect this with the main telemetry ping?
Flags: needinfo?(rtestard)

(In reply to Romain Testard [:RT] from comment #11)

  • Option2 would be ideal I think since it may be that MS changes behavior on next Win10 version so even if documentation is good now, it may become irrelevant later when MS ships a change.

I'll go ahead and implement Option 2. Just to be clear though, all I'm doing is checking if this is Windows build 15063 or later. As far as I know, there's no other way of detecting if the feature exists. Since you didn't specify, I'll report "NoSuchFeature" on earlier versions.

  • Regarding timing of the collection - I may be missing something but I don't feel like this is critical to get early at startup the way profile count was so it seems ok to collect this with the main telemetry ping?

Most of the collection happens whenever the thing being recorded happens. For example, update-related main-ping telemetry gets recorded when we are updating; feature usage telemetry generally gets collected when the feature is used. There is no specific time when main-ping telemetry data is collected. Unfortunately, there is no natural time to collect this new probe, which is why I plan to put it in an idle callback.

Attachment #9173737 - Flags: data-review?(chutten)

Comment on attachment 9173737 [details]
app_source_data_review_form.md

Load-balancing to Megan.

Attachment #9173737 - Flags: data-review?(chutten) → data-review?(mmccorquodale)

Comment on attachment 9173737 [details]
app_source_data_review_form.md

  1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?
    This is documented in the probe dictionary.

  2. Is there a control mechanism that allows the user to turn the data collection on and off?
    Yes, users can opt out of telemetry.

  3. If the request is for permanent data collection, is there someone who will monitor the data over time?
    Romain Testard.

  4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
    Category 2, Interaction data.

  5. Is the data collection request for default-on or default-off?
    Default on.

  6. Does the instrumentation include the addition of any new identifiers?
    No new identifiers.

  7. Is the data collection covered by the existing Firefox privacy notice?
    Yes.

  8. Does there need to be a check-in in the future to determine whether to renew the data?
    No.

  9. Does the data collection use a third-party collection tool?
    No.


data-review +

Attachment #9173737 - Flags: data-review?(mmccorquodale) → data-review+
Pushed by ksteuber@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f92c03cd4278
Add scalar to track allowed app installation sources on Windows r=RT
https://hg.mozilla.org/integration/autoland/rev/7bd416f59f37
Collect allowed app install source data and submit to telemetry r=Gijs,chutten,mhowell
https://hg.mozilla.org/integration/autoland/rev/ec08e998d0c4
Add testing for allowed app sources telemetry r=chutten
Blocks: 1665386
Depends on: 1666420
You need to log in before you can comment on or make changes to this bug.