Run A/B test for special painting



Mozilla QA
2 years ago
2 years ago


(Reporter: billm, Unassigned)


Version 4


In bug 1279086 we introduced some changes that allow us to paint content while JS may be running in that content. This is something we've never done before. We know that it reduces jank. But we also know that there's a possibility that the paints we're doing are "incorrect" in some way. They may show content in an intermediate state, in the wrong place, or maybe worse. We don't know how likely this is.

We would like to run a kind of A/B test where we introduce jank in the content process and then turn these special paints either on or off. With the paints off, the user will see the jank more clearly. With the paints on, they may see weird drawing artifacts, although they'll be temporary. The idea is to ask the user subjectively which one they prefer.

I don't think we would want to run this sort of experiment against any of our channels since it would likely be a pretty bad experience either way. Ryan, is this the sort of experiment that your QA team could run? I could develop an add-on they could install that would introduce the jank and then switch the special paints on or off. I'm not sure exactly how we would ask people which one is better.
Flags: needinfo?(ryanvm)
Yeah, I think we could help with this.
Flags: needinfo?(ryanvm) → needinfo?(rares.bologa)
Out of curiosity, how does Treeherder react to this option?
So far we've had two assertions but no crashes. I haven't yet tried it with a version that introduces artificial jank. That would cause the special paints to trigger much more often.
Bill, SV-Vegas can assist with this


2 years ago
Flags: needinfo?(rares.bologa) → needinfo?(mfunches)
Thanks Michelle. I'll try to get a build to you by Monday/Tuesday.
Can we close this out?
Flags: needinfo?(mfunches) → needinfo?(wmccloskey)
No, I still want to do it. I just haven't made any progress on it.
Flags: needinfo?(wmccloskey)


2 years ago
QA Contact: amasresha
You need to log in before you can comment on or make changes to this bug.