899363 - Implement stub installer data ping version 6

Status: RESOLVED FIXED

(Firefox :: Installer, defect)

Description

[Robert Strong (they/them - no direct email)]

Changes that should be made

Removal:
IntroPageShownCount (removed back button)

OptionsPageShownCount (removed back button)

Incorrect data points:
The values from WinVerGetMajor, WinVerGetMinor, and WinVerGetBuild aren't always correct likely due to bug 444664 which we have a workaround for when checking the minimum supported windows version. We should also report the value from the registry to supplement this info when it is correct.

The Downloaded Bytes data point has an incorrect value (empty string) for 0.95% of the data pings.

When Downloaded Bytes has an incorrect value (empty string) the Last Download Seconds data point has an incorrect value of 4294966 for 0.94% of the data pings.

The Last Downloaded Seconds data point also has an incorrect value for 0.03% of the data pings that isn't accounted for by the Downloaded Bytes has an incorrect value (empty string) case.

Anurag, are there any new stub installer data points you would like to get?

Comment 1

[Robert Strong (they/them - no direct email)]

(In reply to Robert Strong [:rstrong] (do not email) from comment #0)
>...
> When Downloaded Bytes has an incorrect value (empty string) the Last
> Download Seconds data point has an incorrect value of 4294966 for 0.94% of
> the data pings.
s/Last Download Seconds data point has an incorrect value/Last Download Seconds data point usually has an incorrect value/

Comment 2

[Robert Strong (they/them - no direct email)]

As can be seen below the Windows major version info is seldom incorrect since the sample size this is from is 7290650

Ver | Count
----+------
 0  |   7
 4  |  42

Comment 3

[Robert Strong (they/them - no direct email)]

(In reply to Robert Strong [:rstrong] (do not email) from comment #2)
> As can be seen below the Windows major version info is seldom incorrect
> since the sample size this is from is 7290650
> 
> Ver | Count
> ----+------
>  0  |   7
>  4  |  42
Given this happens so seldom I don't think it is worth reporting the value from the registry

Comment 4

[Robert Strong (they/them - no direct email)]

We should also report service pack level

!include "WinVer.nsh"
...
${WinVerGetServicePackLevel} $1

Comment 5

[Robert Strong (they/them - no direct email)]

Perhaps IsServerOS as well

Comment 6

[Robert Strong (they/them - no direct email)]

The seconds spent on the Option Page can actually be zero (likely the same for the Install Page). It should be changed to always round up.
Option Page Seconds > 0 : 347386
Option Page Count != 0  : 347632

So, 347386 times Options was clicked and 246 times it was seen for under a second before clicking install.

Comment 7

[Robert Strong (they/them - no direct email)]

Any additional data points either of you think we should add?

Comment 8

[Brian R. Bondy [:bbondy]]

Maybe we can detect and report if wininet is using a proxy. I  remember there was a flag for specifying how to handle proxy (whether or not to use direct) and I don't remember if we implemented a fallback.  If we did we could indicate that a fallback was used, if not, we could add that and then report it (probably in a different bug).

Comment 9

[Robert Strong (they/them - no direct email)]

(In reply to Brian R. Bondy [:bbondy] from comment #8)
> Maybe we can detect and report if wininet is using a proxy. I  remember
> there was a flag for specifying how to handle proxy (whether or not to use
> direct) and I don't remember if we implemented a fallback.  If we did we
> could indicate that a fallback was used, if not, we could add that and then
> report it (probably in a different bug).
Awesome data point to add!

Comment 10

[Brian R. Bondy [:bbondy]]

Possibly also the failure point (or success) inside InetBgDl.  And what the last error code was for that failure point. See the Trace commands in InetBgDl.cpp

Comment 11

[Anurag Phadke[:[e-mail]]]

(In reply to Robert Strong [:rstrong] (do not email) from comment #7)
> Any additional data points either of you think we should add?

:rstrong - This is well instrumented, I don't have any more suggestion in terms of adding more data points.

Comment 12

[Robert Strong (they/them - no direct email)]

Note to self: Add total download size so we can know this info for cancelled downloads.

Comment 13

[Robert Strong (they/them - no direct email)]

Note to self: add whether the install is already default for pave over and if the check box to set as default is checked or not.

Comment 14

[Robert Strong (they/them - no direct email)]

Also, whether there is an existing Firefox install other than the current one that is default

Comment 15

[Robert Strong (they/them - no direct email)]

Attachment: Patch in progress

Notes

;DONE - removed IntroPageShownCount (removed back button)
;DONE - removed OptionsPageShownCount (removed back button)

;DONE - The seconds spent on the Option Page can actually be zero (likely
; the same for the Install Page). It should be changed to always round up.
; Option Page Seconds > 0 : 347386
; Option Page Count != 0  : 347632

; So, 347386 times Options was clicked and 246 times it was seen for under a
; second before clicking install.

;WONTFIX and ADDED COMMENT -The values from WinVerGetMajor, WinVerGetMinor, and
; WinVerGetBuild aren't always correct likely due to bug 444664 which we have a
; workaround for when checking the minimum supported windows version. We should
; also report the value from the registry to supplement this info when it is
; correct.
; As can be seen below the Windows major version info is seldom incorrect since
; the sample size this is from is 7290650
; Ver | Count
; ----+------
;  0  |   7
;  4  |  42

;DONE - We should also report service pack level

;STILL NEED TO CHECK OUT VALUE - Perhaps IsServerOS as well

; DONE Add whether the install is already default for pave over and if the check box
; to set as default is checked or not. Also, whether there is an existing
; Firefox install other than the current one that is default.
; Is default metrics
; 0 - Not set as default
; 1 - This Firefox install is set as default
; 2 - Another Firefox install is set as default

; DONE Add set as default metrics
; Set as default metrics
; 0 - Can't set as default
; 1 - Set as default
; 2 - Not set as default

; The Downloaded Bytes data point has an incorrect value (empty string) for
; 0.95% of the data pings.

: When Downloaded Bytes has an incorrect value (empty string) the Last
; Download Seconds data point usually has an incorrect value of 4294966 for
; 0.94% of the data pings.

; The Last Downloaded Seconds data point also has an incorrect value for 0.03%
; of the data pings that isn't accounted for by the Downloaded Bytes has an
; incorrect value (empty string) case.

Comment 16

[Robert Strong (they/them - no direct email)]

Attachment: Patch in progress

Comment 17

[Robert Strong (they/them - no direct email)]

Attachment: patch rev1

I didn't try to fix a couple of the cases where it incorrectly reports. I'll do those as time permits.

Comment 18

[Robert Strong (they/them - no direct email)]

Note: because of http://msdn.microsoft.com/en-us/library/windows/desktop/dn302074%28v=vs.85%29.aspx Win 8.1 won't be reported properly. NSIS will likely need to be upgraded to get stats on Win 8.1

Comment 19

[Robert Strong (they/them - no direct email)]

Anurag, below are the new set of parameters sent. The Stub_URL_Version will be v6.

http://download-stats.mozilla.org/stub/Stub_URL_Version/Build_Channel/Update_Channel/Locale/Firefox_x64/Running_x64_Windows/Windows_Major_Version/Window__Minor_Version/Windows_Build_Version/Windows_Service_Pack/Is_Windows_Server/Exit_Code/Firefox_Launch_Code/Download_Retry_Count/Downloaded_Bytes/Introduction_Phase_Seconds/Options_Phase_Seconds/Download_Phase_Seconds/Last_Download_Seconds/Download_First_Transfer_Seconds/PreInstall_Phase_Seconds/Install_Phase_Seconds/Finish_Phase_Seconds/Initial_Install_Requirements_Code/Opened_Download_Page/Existing_Profile/Existing_Firefox_Version/Existing_Firefox_Build_ID/New_Firefox_Version/New_Firefox_Build_ID/Default_Install_Dir/Has_Admin/Default_Status_Code/SetAsDefault_Code/Download_Server_IP

Stub_URL_Version = a unique ID used to associate the parameters used in the metrics ping as they are modified.

Build_Channel = The build channel. Expected values for our builds:
nightly
aurora
beta
release

Update_Channel = The application update channel. Expected values for our builds:
nightly
aurora
beta
release

Locale = the Firefox locale that was chosen to install (e.g. en-US)

Firefox_x64 = 1 if installing Firefox 64 bit and 0 otherwise (e.g. 32 bit Firefox)

Running_x64_Windows = 1 if running Windows 64 bit and 0 otherwise (e.g. 32 bit Windows)

Windows_Major_Version = self-explanatory (e.g. 6 = Windows 7)

Windows_Minor_Version = self-explanatory (e.g. 1 = current Windows 7)

Windows_Build_Version = self-explanatory (e.g. 7601 = current Windows 7)

Windows_Service_Pack = self-explanatory (e.g. Service Pack 2 = 2)

Is_Windows_Server = 1 if server, 0 if not server

Exit_Code = the stub installer's exit code. Possible values:
0 = successful install

The following apply to the download phase:
10 = download cancelled by the user
11 = too many attempts to download (current max retries = 10)

The following apply to the pre-install check phase:
20 = unable to acquire a file handle to the downloaded file
21 = downloaded file's certificate is not trusted by the certificate store
22 = downloaded file's certificate attribute values were incorrect
23 = downloaded file's certificate is not trusted by the certificate store and certificate attribute values were incorrect

The following apply to the install phase:
30 = the installation timed out (current install timeout = 120 seconds)

Firefox_Launch_Code = Possible values:
0 = if no attempt was made to launch Firefox
1 = if Firefox was running
2 = if Firefox was launched

Download_Retry_Count = The number of download retries. If there are no retries this value will be 0 and the current maximum value is 9 (e.g. maximum attempts = 10 and maximum retries = 9).

Downloaded_Bytes = The amount downloaded in bytes for the last attempted download.

Download_Size_Bytes = The size of the download in bytes.

Introduction_Phase_Seconds = The total number of seconds the introduction page was displayed (this page can be displayed multiple times).

Options_Phase_Seconds = The total number of seconds the options page was displayed (this page can be displayed multiple times).

Download_Phase_Seconds = The number of seconds to complete the download phase which includes all download attempts. It is possible for this to be 0 if the user cancelled the download really quickly.

Download_First_Transfer_Seconds = The number of seconds from the start of the download phase until the first bytes are received. This is only recorded for first request so it is possible to determine connection issues for the first request.

Last_Download_Seconds = The number of seconds to complete the last download which can be different from the Download Phase Seconds if the previous download had to be restarted or if it took a second or more to initiate the download. It is possible for this to be 0 if the user cancelled the download really quickly.

Preinstall_Phase_Seconds = The number of seconds to complete the pre-installation check phase. It is possible for this phase to complete in 0 seconds and if this phase was never entered it will be also 0 (e.g. cancelled download).

Install_Phase_Seconds = The number of seconds to complete the installation phase. If this phase was never entered it will be 0 (e.g. cancelled download) and I highly doubt it will ever be 0 if the phase was entered.

Finish_Phase_Seconds = The number of seconds after the installation has finished for the installer to close. It is possible for this phase to complete in 0 seconds and if this phase was never entered it will be 0 (e.g. cancelled download).

Initial_Install_Requirements_Code = if the installation requirements were not initially met. Possible values:
0 = if the installation requirements were met
1 = if both the write check for the installation directory and the free space check failed
2 = if only the write check for the installation directory failed
3 = if only the free space check failed

Opened_Download_Page = 1 if the page to download the complete installer was opened when a failure occurs and 0 otherwise.

Existing_Profile = 1 if the %LOCALAPPDATA%\Mozilla\Firefox directory exists and 0 otherwise.

Existing_Firefox_Version = Existing Firefox version when installing on top of an existing install and 0 otherwise.

Existing_Firefox_Build_ID = Existing Firefox build ID when installing on top of an existing install and 0 otherwise.

New_Firefox_Version = The new install's Firefox version when installation was successful and 0 otherwise.

New_Firefox_Build_ID = The new install's Firefox build ID when installation was successful and 0 otherwise.

Default_Install_Dir = 1 when installing into the default installation directory and 0 otherwise.

Has_Admin = 1 when installing with administrator privileges and 0 otherwise.

Default_Status_Code = 0 - Not set as default, 1 - This Firefox install is set as default, 2 - Another Firefox install is set as default

SetAsDefault_Code = 0 - Can't set as default due to permissions, 1 - Can't set as default Win 8 or above, 2 - Set as default, 3 - Not set as default

Download_Server_IP = The IP address of the download server.

Comment 20

[Robert Strong (they/them - no direct email)]

Anurag, heads up regarding comment #19

Comment 21

[Robert Strong (they/them - no direct email)]

Pushed to mozilla-inbound
https://hg.mozilla.org/integration/mozilla-inbound/rev/7fc260be35da

Comment 22

[Anurag Phadke[:[e-mail]]]

rob - thx for the headsup. do we plan to spin a dashboard for this data?

Comment 23

[Robert Strong (they/them - no direct email)]

I have no plans or needs to do so.

Comment 24

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/7fc260be35da

Comment 25

[Robert Strong (they/them - no direct email)]

Comment on attachment 8393803 [details] [diff] [review]
patch rev1

[Approval Request Comment]
Bug caused by (feature/regressing bug #): None. This patch improves the metrics reported by the stub installer data ping and I'd like to get it on the more heavily used channels sooner rather than later to assist with the metrics for bug 899352 and bug 799215.
User impact if declined: None
Testing completed (on m-c, etc.): on m-c and manual testing
Risk to taking this patch (and alternatives if risky): small since it has been on m-c for several days and is fairly straightforward
String or IDL/UUID changes made by this patch: None

Comment 26

[Robert Strong (they/them - no direct email)]

Pushed to mozilla-aurora
https://hg.mozilla.org/releases/mozilla-aurora/rev/1e065473e9c8

Comment 27

[Robert Strong (they/them - no direct email)]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #19)
> Anurag, below are the new set of parameters sent. The Stub_URL_Version will
> be v6.
> 
> http://download-stats.mozilla.org/stub/Stub_URL_Version/Build_Channel/
> Update_Channel/Locale/Firefox_x64/Running_x64_Windows/Windows_Major_Version/
> Window__Minor_Version/Windows_Build_Version/Windows_Service_Pack/
> Is_Windows_Server/Exit_Code/Firefox_Launch_Code/Download_Retry_Count/
> Downloaded_Bytes/Introduction_Phase_Seconds/Options_Phase_Seconds/
> Download_Phase_Seconds/Last_Download_Seconds/Download_First_Transfer_Seconds/
> PreInstall_Phase_Seconds/Install_Phase_Seconds/Finish_Phase_Seconds/
> Initial_Install_Requirements_Code/Opened_Download_Page/Existing_Profile/
> Existing_Firefox_Version/Existing_Firefox_Build_ID/New_Firefox_Version/
> New_Firefox_Build_ID/Default_Install_Dir/Has_Admin/Default_Status_Code/
> SetAsDefault_Code/Download_Server_IP
Missed one field
http://download-stats.mozilla.org/stub/Stub_URL_Version/Build_Channel/Update_Channel/Locale/Firefox_x64/Running_x64_Windows/Windows_Major_Version/Window__Minor_Version/Windows_Build_Version/Windows_Service_Pack/Is_Windows_Server/Exit_Code/Firefox_Launch_Code/Download_Retry_Count/Downloaded_Bytes/Download_Size_Bytes/Introduction_Phase_Seconds/Options_Phase_Seconds/Download_Phase_Seconds/Last_Download_Seconds/Download_First_Transfer_Seconds/PreInstall_Phase_Seconds/Install_Phase_Seconds/Finish_Phase_Seconds/Initial_Install_Requirements_Code/Opened_Download_Page/Existing_Profile/Existing_Firefox_Version/Existing_Firefox_Build_ID/New_Firefox_Version/New_Firefox_Build_ID/Default_Install_Dir/Has_Admin/Default_Status_Code/SetAsDefault_Code/Download_Server_IP

Comment 28

[u279076]

Robert, do you need QA attention on these changes?

Comment 29

[Robert Strong (they/them - no direct email)]

Anthony, no QA needed for this.