1584442 - Investigate retention drop in July/August of 2019

Status: RESOLVED FIXED

(Data Science :: Investigation, task)

Description

[Jesse McCrosky]

Brief description of the request:

There was a retention drop in both absolute and year-over-year terms around the end of July. The retention drop appears to be primarily explained by Mac OS users. The drop coincides approximately with the release of the beta 5 release of Catalina, which caused a crash bug in the browser preventing use entirely.

We need to determine if it is possible that a first run of the browser under Catalina beta 5 could result in the successful creation of a profile and transmission of a main telemetry ping before crash. If so, this crash may explain the retention drop, although it is strange that retention has not recovered, since the bug was fixed in beta 6, released 8-19.

Link to any assets:

There is some analysis here: https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/176635

Is there a specific data scientist you would like or someone who has helped to triage this request:

Jesse has been working on this so far. I don't really have bandwidth to put much time into it, so if anyone else is available, I'm happy to hand it off.

Comment 1

[Romain Testard [:RT]]

This is high priority for product, the retention drop is significant (-25% YoY on MacOS) and does not show signs of slowing-down. Thanks for helping assign someone to help progress analysis:

• Identify user characteristics affected by retention drop (OS version sounds like the first candidate to evaluate)
• Identify Firefox changes or OS changes that may be the source of this issue. Windows retention dropped too following the same trend (-5% YoY) - there may be a similar source to both drops even though the impact is larger on MacOS

Comment 2

[Romain Testard [:RT]]

Darwin 19.0 is Catalina - 0.3% of MacOS users seem to run Catalina: https://sql.telemetry.mozilla.org/queries/62336/source#160168

Comment 3

[Jesse McCrosky]

For reference, bugs relating to the Catalina crash

https://bugzilla.mozilla.org/show_bug.cgi?id=1579086
https://bugzilla.mozilla.org/show_bug.cgi?id=1570451

Comment 4

[Romain Testard [:RT]]

Hi Roland, did you hear of anything around the end of July that could explain a retention drop for new users installing Firefox on MacOS?

Comment 5

[Jesse McCrosky]

Eyeballing the graph, we lost around 2% DAU around when beta 5 came out, but it probably mostly recovered. This seems inconsistent with the 0.3% of Mac OS users on Catalina. Maybe we didn't see a bunch of them though. And maybe the beta at certain points reported something different than we expect for OS version....

Will keep looking. Still happy to have this taken over by anyone that has time though.

Comment 6

[Jesse McCrosky]

The plot thickens. There was a pretty significant increase in new profiles being generated, with a spike around the beta 5 release.

See graph here: https://docs.google.com/document/d/1rtQ3zmhOIZD-GW3O-F3gF92Pl7pQZTXgJkScUxsbC9k/edit

Comment 7

[Romain Testard [:RT]]

A few avenues for investigations:

• Measure retention change per MacOS version (discussing with Pascal, he had the idea that Armagaddon fix may not be compatible with older MacOS versions)
• Measure retention change per Firefox version, per macOS version (67 may be somehow incompatible with older MacOS versions)
• Did we make changes in how we get MacOS users to download Firefox (source traffic, download UI?)

Other comments:

• July 9th Chinese installs started to be processed/reported through the stub installer (2nd largest country per install numbers). This although does not seem related since retention per countries seen in the US, DE or FR dropped
• The activity stream browser migration message was removed in 68 (https://bugzilla.mozilla.org/show_bug.cgi?id=1545487) although per https://sql.telemetry.mozilla.org/queries/51285/source#137524 only 1% of new profiles used it so unlikely to be the source of the issue
• The MacOS retention drop seems slightly more pronounced in DE and FR than US per
  https://docs.google.com/spreadsheets/d/1M-6Dfub-b50kA86Sw-s2wCfWDbbu0Wvq0dzdggPB-zE/edit#gid=1907201843

Comment 8

[Romain Testard [:RT]]

(In reply to Jesse McCrosky from comment #5)

Eyeballing the graph, we lost around 2% DAU around when beta 5 came out, but it probably mostly recovered. This seems inconsistent with the 0.3% of Mac OS users on Catalina. Maybe we didn't see a bunch of them though. And maybe the beta at certain points reported something different than we expect for OS version....

Will keep looking. Still happy to have this taken over by anyone that has time though.

Over the last 3 months we get a total of 53,289 downloads on moz.org from MacOS 10.15 for a little less than 3M MacOS downloads overall - this is about 2% of new downloads

https://analytics.google.com/analytics/web/#/my-reports/TjGzl9yxSHWHnKjuEP5oTg/a36116321w64087799p134840501/_u.date00=20190701&_u.date01=20191001&_.goalOption=2&_.useg=&4357-table.secSegmentId=analytics.operatingSystemVersion&4357-table.plotKeys=%5B%5D&4357-table.rowStart=0&4357-table.rowCount=500&4357-table-tableMode.selected=pie/

Comment 9

[Romain Testard [:RT]]

Attachment: image.png

(In reply to Jesse McCrosky from comment #6)

The plot thickens. There was a pretty significant increase in new profiles being generated, with a spike around the beta 5 release.

See graph here: https://docs.google.com/document/d/1rtQ3zmhOIZD-GW3O-F3gF92Pl7pQZTXgJkScUxsbC9k/edit

I graphed "moz.org unique downloads / non distribution new profiles" and this is trending in a similar way: New profiles increase in the absolute but also compared to downloads and also compared to last year
For some reason we're getting more new profiles getting created whereas downloads are stable.
I checked on new profiles created as a result of profile per channel (although this shipped with Firefox 67) and it shows no signs of MacOS bias: https://sql.telemetry.mozilla.org/queries/65113/source#165828

NI to mossop in case he can think of a reason why new profiles would get created in large amounts starting in July on MacOS. I am aware of https://bugzilla.mozilla.org/show_bug.cgi?id=1557401 that could have had an impact although the timing (profile per install was shipped with 67 on May 21st) does not align and telemetry shows no signs of odd MacOS profile creations as a result of profile per install ( https://sql.telemetry.mozilla.org/queries/65113/source#165828)

Comment 10

[Jesse McCrosky]

Is it possible that launching a new install of Firefox under Catalina 5 would create a profile and send a main ping before crashing but fail to save the profile to disk, resulting on the next launch having the same effect?

Comment 11

[Jesse McCrosky]

As discussed via slack, hoping for help understanding what may or may not have happened in terms of telemetry pings and profile creation before the crash.

Comment 12

[Chris H-C :chutten]

Given that this is a startup crash, it is exceedingly unlikely that Telemetry has had a chance to init in any way. If this crash is recoverable it is possible that some storage in the profile directory will result in some odd data for clients coming out the other side of this (:haik came up with what the profile directory could look like at this stage of a new startup and it didn't have a times.json which means the profile creation date and first use date and the scalar tracking when the profile directory was scanned will not report identical days, for example).

In terms of retention, if this is an unrecoverable startup crash then this will act as a wall in a client's individual retention. Anyone who previously had Firefox installed and then stepped into this crash will not be sending us any more pings and will look as though they churned out (because they have, at least for now).

Comment 13

[Romain Testard [:RT]]

Pascal pointed me to Bug 1585522 that sounds like it could be related (user unable to update with 10.14)

Comment 14

[Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland]

(In reply to Romain Testard [:RT] from comment #4)

Hi Roland, did you hear of anything around the end of July that could explain a retention drop for new users installing Firefox on MacOS?

i don't see anything in the july or august SUMO questions but i'll double check and comment if i find anything

for future reference and in the unlikely event :-) anybody has time here are the firefox desktop en-us questions from support.mozilla.org for august and july 2019

1. july 2019 (github csv july 2019; google spreadsheet for mozilla staff for those who love spreadsheets :-) more than CSVs)
2. august 2019 (github csv august 2019 ; google spreadsheet for mozilla staff for those who love spreadsheets :-) more than CSVs)

Comment 15

[Romain Testard [:RT]]

(In reply to Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland from comment #14)

(In reply to Romain Testard [:RT] from comment #4)

Hi Roland, did you hear of anything around the end of July that could explain a retention drop for new users installing Firefox on MacOS?

i don't see anything in the july or august SUMO questions but i'll double check and comment if i find anything

for future reference and in the unlikely event :-) anybody has time here are the firefox desktop en-us questions from support.mozilla.org for august and july 2019

1. july 2019 (github csv july 2019; google spreadsheet for mozilla staff for those who love spreadsheets :-) more than CSVs)
2. august 2019 (github csv august 2019 ; google spreadsheet for mozilla staff for those who love spreadsheets :-) more than CSVs)

Thanks Roland. I've been through the July issues but nothing stands out as recurring. Most issues that could explain churn happen on Catalina but these don't seem recurrent enough to imapact numerous users:
https://support.mozilla.org/en-US/questions/1265442
https://support.mozilla.org/en-US/questions/1264590
https://support.mozilla.org/en-US/questions/1264475
https://support.mozilla.org/en-US/questions/1264403

Comment 16

[Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland]

(In reply to Romain Testard [:RT] from comment #15)

(In reply to Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland from comment #14)

(In reply to Romain Testard [:RT] from comment #4)

Hi Roland, did you hear of anything around the end of July that could explain a retention drop for new users installing Firefox on MacOS?

i don't see anything in the july or august SUMO questions but i'll double check and comment if i find anything

for future reference and in the unlikely event :-) anybody has time here are the firefox desktop en-us questions from support.mozilla.org for august and july 2019

1. july 2019 (github csv july 2019; google spreadsheet for mozilla staff for those who love spreadsheets :-) more than CSVs)
2. august 2019 (github csv august 2019 ; google spreadsheet for mozilla staff for those who love spreadsheets :-) more than CSVs)

Thanks Roland. I've been through the July issues but nothing stands out as recurring. Most issues that could explain churn happen on Catalina but these don't seem recurrent enough to imapact numerous users:
https://support.mozilla.org/en-US/questions/1265442
https://support.mozilla.org/en-US/questions/1264590
https://support.mozilla.org/en-US/questions/1264475
https://support.mozilla.org/en-US/questions/1264403

Yes, I believe this indicates that people don't seem to complain on SUMO before they churn on OS X.

Comment 17

[George Kaberere [:gkaberere]]

Analysis results can be found below:
https://docs.google.com/document/d/1OMA5nZx3JVwE5Y7_q0wdOvtTXkzG-aJnhG6_K9oeQ98/edit