1591247 - Review Dark Mode Probes for iOS

Status: RESOLVED FIXED

(Data Science :: General, task)

Description

[athomas]

Brief Description of the request (required):

We're adding new probes in to track the usage of Dark Mode in iOS. Probes are already in, but I'm requesting to review to ensure they can answer the below questions & keep the bigger picture of tracking feature usage in mobile in mind

Business purpose for this request (required):

To answer 2 questions:

• What percent of overall users use dark theme?
• What percent of users have their theme set to "use system theme"?

Requested timelines for the request or how this fits into roadmaps or critical decisions (required):

Probes have just been added and will start release process Dec 2nd. So need to know before then to ensure we have time to change

Links to any assets (e.g Start of a PHD, BRD; any document that helps describe the project):

Ticket for probes: https://github.com/mozilla-mobile/firefox-ios/issues/5569
Dark mode feature brief & mocks: https://docs.google.com/document/d/1ZB7JX4GkI6RoNE0OxKMsSeWRJuwdjr4eqehuT1jfjxc/edit#

Name of Data Scientist (If Applicable):

Please note if it is found that not enough information has been given this will delay the triage of this request.

Comment 1

[Felix Lawrence]

I think you want to know:

1. For each user, is their firefox theme set to:
   a) default (or is this not distinct from (b)?)
   b) always light
   c) always dark
   d) based on screen brightness
2. For each user, is their OS set to light mode or dark mode, at some moment in time or across some period of time?

You want to know (1) to see whether our users are discovering these features, and (2) to see whether there is demand at the OS level for this feature. You would like to monitor (1) over time once we start defaulting to respecting the OS's default to see whether there is demand for firefox to override the OS default. Does this summarise what you want to know, or is there something else?

Comment 2

[athomas]

Hi Felix!
Thanks for picking this up. That's pretty close, but I may be able to reduce the scope a little. I want to know:

• Do we need options outside of the default (match system setting)? I don't need to know which other option is chosen for this, just to monitor "over time once we start defaulting to respecting the OS's default to see whether there is demand for firefox to override the OS default."
• Is dark mode used enough to prioritize bugs specific to it? #2 in your suggested tracking could be used to answer this. It's worth noting that there are 3 situations where a user would see FF in dark theme:
  a) Setting is set to match system and system is dark
  b) Setting is set to manual > dark
  c) Setting is set to based on screen brightness and they're over the threshold

Does that make sense? Let me know if you want me to set up a quick sync on this.

Comment 3

[athomas]

• How often are people adjusting settings? This could be a hint that something's wrong, and/or that there's a need for settings separate from the OS level. Probes:
  • Record event telemetry for adjustments (brightness setting, manual-light, manual-dark, match system)
• What fraction of users have the OS set to dark mode may be enough to answer the prioritization settings. It's preferred to cut the scope to look specifically at OS setting because it will last beyond any changes to the UI. This doesn't necessarily mean that the Firefox theme is changing along with it (if user isn't using default setting for FF theme)
  • Record state of OS setting
• How many users currently have their theme set to something other than the default? This information will be useful to know how many users will be effected if we decide to remove the other options
  • Record state of the Firefox setting

Note: when are the events recorded? If it's only once a day, there's potential to miss out on tracking for users who change their setting & then changed it back within the day. Sounds like every time the app is backgrounded though, so that should be safer.