1480242 - Shield study enrollment is lower than it should be

Status: RESOLVED DUPLICATE of bug 1495152

(Core :: Graphics: WebRender, defect, P4)

Description

[Jeff Muizelaar [:jrmuizel]]

For example, of 1621 submissions that are enrolled in the treatment group 1007 end up with d3d11 and only 561 get WebRender

Comment 1

[Darkspirit]

526 - Pref Flip Study: WebRender V1.1 Experiment [Bug 1474484]
> Filter
> (
>  normandy.channel == 'nightly'
>  && normandy.version >= '63' 
>  && normandy.telemetry.main.environment.build.buildId >= '20180725103029'
>  && normandy.os.windowsVersion == 10.0 
>  && normandy.telemetry.main.environment.system.gfx.adapters[.vendorID == '0x10de'][0]
> )

https://crash-stats.mozilla.com/search/?telemetry_environment=~%22prefflip-webrender-v1-1-1474484%22%3A%7B%22branch%22%3A%22treatment%22&app_notes=~WR-&build_id=%3E%3D20180731105217&product=Firefox&version=63.0a1&platform=Windows&date=%3E%3D2018-08-01T01%3A19%3A19.000Z&date=%3C2018-08-02T01%3A19%3A19.000Z&_sort=-date&_facets=signature&_facets=telemetry_environment&_facets=app_notes&_facets=build_id&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports

Unqualified in treatment:
https://crash-stats.mozilla.com/report/index/30f7e5b6-354e-47cc-b22d-e0e310180801#tab-metadata
> {"compositor":"d3d11","gpuProcess":{"status":"available"},"wrQualified":{"status":"blocked"},"webrender":{"status":"opt-in"}
> {"prefflip-webrender-v1-1-1474484":{"branch":"treatment"

= Battery or old Nvidia?
https://searchfox.org/mozilla-central/rev/d0a0ae30d534dc55e712bd3eedc1b4553ba56323/gfx/thebes/gfxPlatform.cpp#2585,2605

---

Qualified, but either pref not set to true or manually disabled?
https://crash-stats.mozilla.com/report/index/f201e3a7-63fa-4f3e-849c-d334a0180801#tab-metadata
> {"compositor":"d3d11","gpuProcess":{"status":"available"},"wrQualified":{"status":"available"},"webrender":{"status":"opt-in"}
> {"prefflip-webrender-v1-1-1474484":{"branch":"treatment"

Comment 2

[Darkspirit]

Intel ("GPUActive":true) + Nvidia ("GPUActive":false) with d3d11 in treatment (the problem?):
https://crash-stats.mozilla.com/report/index/4c6f365f-0a13-4a47-8db8-292710180802#tab-metadata

For comparison: qualified Nvidia ("GPUActive":true) + Intel ("GPUActive":false):
https://crash-stats.mozilla.com/report/index/1ae679af-ed93-42ef-8902-c7cfd0180802#tab-metadata

Comment 3

[Darkspirit]

Isn't this possible? (about:telemetry > bottom left "Raw JSON")
normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == 'available'

Comment 4

[Jeff Muizelaar [:jrmuizel]]

I don't understand your question.

Comment 5

[Darkspirit]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #0)
> For example, of 1621 submissions that are enrolled in the treatment group 1007 end up with d3d11 and only 561 get WebRender

How I interpret the current study and your numbers:
You focus on all users somehow having Nvidia, but they doen't need to be wrQualified, and want to compare WR with d3d11. That could be why you have so many d3d11 users in treatment:
>  normandy.channel == 'nightly'
>  && normandy.version >= '63' 
>  && normandy.telemetry.main.environment.build.buildId >= '20180725103029'
>  && normandy.os.windowsVersion == 10.0 
>  && normandy.telemetry.main.environment.system.gfx.adapters[.vendorID == '0x10de'][0]
- 50% treatment: 1621 submissions
  - 561 webrender
    - wrQualified enabled by study
    - manually enabled: gfx.webrender.all.qualified;true
    - manually enabled: gfx.webrender.all;true (qualified + unqualified Nvidia)
  - 1007 d3d11
    - unqualified Nvidia: Battery
    - unqualified Nvidia: old Nvidia device
    - unqualified Nvidia: dual GPUs: Intel is in use (bp-4c6f365f-0a13-4a47-8db8-292710180802)
    - gfx.webrender.all.qualified manually disabled (bp-f201e3a7-63fa-4f3e-849c-d334a0180801 ?)
  - other
- 50% control
  - d3d11/other by default
  - WR
    - manually enabled: gfx.webrender.all.qualified;true
    - manually enabled: gfx.webrender.all;true (qualified + unqualified Nvidia)

Don't you want to focus only on wrQualified users (Win10, modern Nvidia, no battery) and compare WR and d3d11 based on them?
>  normandy.channel == 'nightly'
>  && normandy.version >= '63' 
>  && normandy.telemetry.main.environment.build.buildId >= '20180725103029'
>  && normandy.os.windowsVersion == 10.0 
>  && normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == 'available'
- 50% treatment
  - WR (most)
    - by default: gfx.webrender.all.qualified;true
    - (wrQualified users who already have manually enabled gfx.webrender.all.qualified)
    - (wrQualified users who already have manually enabled gfx.webrender.all (unqualified wouldn't get the study))
  - d3d11 (few)
    - manually disabled: gfx.webrender.all.qualified;false
  - other (few)
- 50% control
  - d3d11 (most)
  - other (few)
  - WR (few)
    - some wrQualified users saw gfx.webrender.all.qualified;false and enabled it manually
    - wrQualified users who have manually enabled gfx.webrender.all (unqualified wouldn't get the study)

Comment 6

[Thomas Elin [:relaas]]

What does this mean? Are we planning to extend the time of the study? Do we need to tweak and start over?

Comment 7

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Attachment: power_curve.png

Sorry if this is redundant but, just to state the analytics perspective, the fewer users that qualify, the harder it is to make a robust claim about small changes in the observed metrics; additionally, if the users that are not qualified for the experiment are biased in an unexpected way, that can make the results less generalizable, so it's interesting to understand why the numbers are lower than expected. I've attached the power curve from the pre-experiment power analysis illustrating the fraction change in the median that's observable as a function of the number of users in the treatment arm.

Comment 8

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

(With respect to remedies, extending the study duration is fine if we don't think the disqualifications are for an interesting reason.)

Comment 9

[Darkspirit]

> jrmuizel: darkspirit: I noticed you marked your comments in bug 1480242 as obsolete. Why?
I noticed a mistake. I didn't want to spam. Then I thought you might be filtering for the exact wrQualified and webrender statuses anyway afterwards, and that you might have forgotten it only in comment 0. It was hot, I had headaches and felt unsure, so I wanted to bring it out of sight.

(In reply to Jan Andre Ikenmeyer [:darkspirit] from comment #5)
>     - unqualified Nvidia: dual GPUs: Intel is in use (bp-4c6f365f-0a13-4a47-8db8-292710180802)

bp-4c6f365f-0a13-4a47-8db8-292710180802
> {"compositor":"d3d11","gpuProcess":{"status":"available"},"wrQualified":{"status":"blocked"},"webrender":{"status":"opt-in"}
> {"prefflip-webrender-v1-1-1474484":{"branch":"treatment","type":"normandy-exp"}

This example of "unqualified Nvidia: dual GPUs: Intel is in use" looks wrong to me because webrender has status "opt-in"[1] instead of "blocked"[2], so it might be more an example of >unqualified manually disabled in treatment<.

Why would an unqualified user in treatment manually disable gfx.webrender.all.qualified which has no practical function?
Is there still an app.normandy.startupExperimentPrefs.gfx.webrender.all.qualified;false pref around? (bug 1479649 comment 2)

An example of "unqualified Nvidia: dual GPUs: Intel is in use" (d3d11 in treatment) might be rather bp-644068d6-a700-43b9-b652-cc07d0180807 because webrender is blocked [2]: 
> {"compositor":"d3d11","gpuProcess":{"status":"available"},"wrQualified":{"status":"blocked"},"webrender":{"status":"blocked"}
> {"prefflip-webrender-v1-1-1474484":{"branch":"treatment","type":"normandy-exp"}

[1] https://searchfox.org/mozilla-central/rev/51268dcbdff0f6f4a5cff7986df0f616efc5bcfd/gfx/thebes/gfxPlatform.cpp#2619
[2] https://searchfox.org/mozilla-central/rev/51268dcbdff0f6f4a5cff7986df0f616efc5bcfd/gfx/thebes/gfxPlatform.cpp#2640

Comment 10

[Jeff Muizelaar [:jrmuizel]]

So I looked some more at this. The majority of wrQualified users in the treatment group that have a d3d11 compositor report "opt-in" for webrender.status. My reading of the code suggests that this should only happen if the webrender pref is not set.

Is there perhaps another reason why the pref is not being set as widely as we hoped?

Comment 11

[Jeff Muizelaar [:jrmuizel]]

Yesterday I tried to reproduce not being enrolled locally with a sample study from the staging server and was not able to. Michael is going to look at some alternative data to see if he sees any anomalies there.

Comment 12

[Michael Cooper [:mythmon]]

Oh, sorry I wasn't clear. I wasn't planning on doing that analysis myself. It is beyond my skills with Spark and Telemetry analysis.

I can summarize the data I mentioned though: Normandy emits Telemetry Events [0] about when it enrolls and unenrolls a user. My theory is that because the feature requires a restart before Webrender becomes active, the data between when Normandy enrolls a user and when they restart for the first time is polluting the data. In theory this noise could be removed by figuring out if the ping being analyzed was generated during the same Firefox session as when the user enrolled in the study.

Comment 13

[Michael Cooper [:mythmon]]

Oops, forgot my link from comment 12: [0] should be https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html

Comment 14

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

I can help with that but I'm at onboarding this week so it'll be delayed; please let me know if you'd like me to take a look the week after next.

Comment 15

[Maire Reavy [:mreavy]]

Jeff -- is this bug still active or has it been overtaken by the newer study?

Comment 16

[Jeff Muizelaar [:jrmuizel]]

Yeah. We can replace this bug with bug 1495152.