1198904 - Create two custom Try builds for User Diary Study

Reporter

Description

•

9 years ago

We are conducting a user research diary study from 9/2 to 9/4, with followup interviews afterward. The primary objectives are to 1. Compare the "A" list to the "B" list, and 2. see whether users find the new PBM (with TP) to be an improvement on the old. We will need 2 custom Try builds to provide to study participants. 1. One build will have the existing blocklist, the "A" list (the standard Disconnect list, which does not include Content category) 2. One build will have the "B" list (the full Disconnect list including Content category) For each of the two builds above, we also need: * The entity whitelist working on both. * To manually turn on the UI tour in both builds. This is accomplished by (comment 2) in bug 1190447. https://bugzilla.mozilla.org/show_bug.cgi?id=1190447#c2 * Each test build must create a new user profile. The participants will be existing Fx users. We need to make sure they don't have interfering add-ons (i.e. adblockers) running, and we also don't want to interfere with their existing profiles. * We need to test all of the above for both builds (A/B): each has the correct blocklist, working entity whitelisting, the UI tour activated, and that it creates a clean profile.

Javaun Moradi [:javaun]

Reporter

Comment 1

•

9 years ago

I'm awaiting from Bill a date on which the study recruiter needs the builds to give to participants. We'll back up our due date from there.

Flags: needinfo?(wselman)

Javaun Moradi [:javaun]

Reporter

Updated

•

9 years ago

Priority: -- → P1

Whiteboard: [fxprivacy]

Brian Grinstead [:bgrins]

Comment 2

•

9 years ago

Is it OK if the builds don't 'install' the custom Firefox? In other words, users will need to remember to run the particular executable they downloaded every time they close the browser (as opposed to selecting it from the start menu). Seems like if it lasts 2 days that should be OK, but just want to make sure the build approach we've used for UX studies will work here.

Panos Astithas (he/him) [:past] (please ni?)

Assignee

Comment 3

•

9 years ago

The necessary changes in Nightly to target the staging server's lists are: pref("browser.trackingprotection.updateURL", "https://tracking.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); pref("browser.trackingprotection.gethashURL", "https://tracking.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); (only the domain name changes) The above should happen to both "A" and "B" profiles. The additional required changes for the "B" list are: pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,mozpub-track-digest256,mozpub-trackwhite-digest256,moztestpub-track-digest256"); (add the last mozpub entry) pref("urlclassifier.trackingTable", "test-track-simple,moztestpub-track-digest256"); (modify the last entry to be moztestpub)

Panos Astithas (he/him) [:past] (please ni?)

Assignee

Comment 4

•

9 years ago

(In reply to Panos Astithas [:past] from comment #3) > digest256,mozpub-trackwhite-digest256,moztestpub-track-digest256"); > > (add the last mozpub entry) Grrrr, that should have been "add the last *moztestpub* entry".

Javaun Moradi [:javaun]

Reporter

Comment 5

•

9 years ago

(In reply to Brian Grinstead [:bgrins] from comment #2) > Is it OK if the builds don't 'install' the custom Firefox? Is the custom icon going to be Nightly or DevEdition? That would be better, since we could in that case give users the instructions to "Click the blue icon" (etc) as opposed to the Fx they're used to launching. Let's ask Bill and Gemma, this may be a barrier. We've had users forget special instructions and invalidate their test in the past.

François Marier [:francois]

Comment 6

•

9 years ago

Note that moztestpub is not actually the real list you're going to use during the study. The real one hasn't made it to stage yet, it's waiting for the new stack to be deployed. Once the new stack is in stage, then you'll need to use "mozpub2-track-digest256" which is the full Disconnect list including the "content" category. The other list (moztestpub-track-digest256) is not what you want since it's a copy of the normal Disconnect list, not the full one. If you use https://github.com/fmarier/safebrowsing-tools/blob/master/get-lists.py you can see the lists available on stage currently. Once you see mozpub2-track-digest256 in that list, that means the new stack is running on stage and you can switch to mozpub2.

Brian Grinstead [:bgrins]

Comment 7

•

9 years ago

(In reply to Javaun Moradi [:javaun] from comment #5) > (In reply to Brian Grinstead [:bgrins] from comment #2) > > Is it OK if the builds don't 'install' the custom Firefox? > > Is the custom icon going to be Nightly or DevEdition? That would be better, > since we could in that case give users the instructions to "Click the blue > icon" (etc) as opposed to the Fx they're used to launching. It's going to be the Nightly icon

Bill Selman

Comment 8

•

9 years ago

(In reply to Javaun Moradi [:javaun] from comment #5) > (In reply to Brian Grinstead [:bgrins] from comment #2) > > Is it OK if the builds don't 'install' the custom Firefox? > > Is the custom icon going to be Nightly or DevEdition? That would be better, > since we could in that case give users the instructions to "Click the blue > icon" (etc) as opposed to the Fx they're used to launching. > > Let's ask Bill and Gemma, this may be a barrier. We've had users forget > special instructions and invalidate their test in the past. Regardless of the duration of the study, participants have varying levels of comfort and expertise. I'm certain we will have at least some participants with a good deal of anxiety about installing software. We can walk them through this process. In terms of participants' usage, we need to distribute a build to participants that behaves as much like a typical Firefox browser install as possible.

Flags: needinfo?(wselman)

Javaun Moradi [:javaun]

Reporter

Comment 9

•

9 years ago

1. We made a call in standup today to not force a new profile, that's no longer a build requirement. We want users to have their history/bookmarks since that's going to affect their enjoyment and productivity of the browser and affect the test. We'll use their existing profile. 2. No complex engineering changes. We don't want to spend too much time or introduce unexpected things. Prefs are all we're doing. We want this to be as easy as possible to create the builds, quickly, and we want to spend time QA'ing the builds since there are already so many things that can go wrong (wrong lists, UI tour doesn't work, etc.) 3. Javaun/Ash will talk to Bill/Gemma. We'll make a call on whether to give users manual instructions turn off add-ons that may affect the test (i.e. ABP) or whether it's acceptable to leave them on. We'll also work on the overall instructions we give users to make sure they don't accidentally launch normal fx and stick to our Nightly build.

Javaun Moradi [:javaun]

Reporter

•

9 years ago

Attachment #8654117 - Flags: feedback?(francois)

François Marier [:francois]

•

9 years ago

Flags: needinfo?(past)

Flags: needinfo?(francois)

Javaun Moradi [:javaun]

Reporter

•

9 years ago

(In reply to Javaun Moradi [:javaun] from comment #14) > The entity whitelist is not working for me in the B build above for Mac. > (haven't tried any other builds yet). Unfortunately I don't know how to debug the whitelist, I think we need François for that. I wonder if the network/server configurations work that IT had planned for this weekend could be related. (In reply to Javaun Moradi [:javaun] from comment #16) > One more: I've tried two macs. When I launch the UI tour from the built-in > page, it's not showing the lock in the URL bar (doesn't appear that TP is > actually on), and then when I get to step 3 and the Control Center is open, > It says "No tracking Elements detected on this page." There is no "Disable" > button, and there is no callout coming off it, so there is no way to click > "Got it" to go to step 4. Screenshot attached. Could be my profile is screwy. I tried build "B" on Linux and I can't reproduce this. I see both the lock and the shield (I assume you meant that you don't see the shield above).

Flags: needinfo?(past)

Panos Astithas (he/him) [:past] (please ni?)

Assignee

Comment 20

•

9 years ago

Comment on attachment 8654111 [details] [diff] [review] build A Hey Alex, could you take a look at the changes I did for enabling the new UI Tour and see if everything looks OK?

Attachment #8654111 - Flags: feedback?(agibson)

Panos Astithas (he/him) [:past] (please ni?)

Assignee

Comment 21

•

9 years ago

Comment on attachment 8654117 [details] [diff] [review] build B Same here, and also take a note of comment 16. If you could also take one of the test builds (comment 11, comment 12) for a spin to see if they work as expected as far as the UI Tour is concerned, that would be fantastic.

Attachment #8654117 - Flags: feedback?(agibson)

Panos Astithas (he/him) [:past] (please ni?)

Assignee

Comment 22

•

9 years ago

(In reply to Panos Astithas [:past] from comment #19) > I tried build "B" on Linux and I can't reproduce this. I see both the lock > and the shield (I assume you meant that you don't see the shield above). Hmm, I managed to reproduce this only once, on build A, in my first attempt after running it on a new profile. It may be related to me doing things very quickly and not letting the page settle down, I don't know. I couldn't reproduce it afterwards.

Panos Astithas (he/him) [:past] (please ni?)

Assignee

Comment 23

•

9 years ago

Oh, shoot, I noticed in the QA script that you must be using the debug builds. Sorry Javaun, this is my mistake for not excluding them in the first place. Debug builds are slower than regular optimized builds and not what we ship to end users. There is a chance that timing differences due to the above could be affecting what you observe. Could you try again with the regular builds (the ones without -debug)? I've updated the QA script to that effect.

Alex Gibson [:agibson]

Comment 24

•

9 years ago

I'm not really the best person to be reviewing these changes. As far as I can tell, the preferences look ok - but that's just a guess. I'm afraid I have limited knowledge here. I downloaded build A for Mac OSX, and I still see the same thing Javaun as when we tested yesterday. The tracking protection feature does not seem to be working correctly using the block list, which is why the UITour is not working correctly (since that is a dependency). Everything seems fine in Nightly, so it must be something specific to the changes here. I can't seem to download build 'B' for OSX as it seem to be incomplete looking in the FTP directory? Matt gave some useful advice here that may be worth investigating: "Looking at the log after enabling browser.safebrowsing.debug may give clues. Without testing the build or anything, I'm wondering if the problem is just that the tracking list hasn't been downloaded yet or there is an error during download (which would show in the Browser Console [not Web Console] when logs are on) Looking at the network requests from the Browser Console should also show the list being downloaded for a new profile."

Alex Gibson [:agibson]

Updated

•

9 years ago

Attachment #8654117 - Flags: feedback?(agibson) → feedback+

Alex Gibson [:agibson]

Updated

•

9 years ago

Attachment #8654111 - Flags: feedback?(agibson) → feedback+

Panos Astithas (he/him) [:past] (please ni?)

•

9 years ago

Great, thanks!

Status: ASSIGNED → RESOLVED

Closed: 9 years ago

Resolution: --- → FIXED

build A 9 years ago Panos Astithas (he/him) [:past] (please ni?) 8.12 KB, patch	francois : feedback+ agibson : feedback+	Details \| Diff \| Splinter Review
build B 9 years ago Panos Astithas (he/him) [:past] (please ni?) 2.02 KB, patch	francois : feedback+ agibson : feedback+	Details \| Diff \| Splinter Review
TP_Build_B_UIerror.png 9 years ago Javaun Moradi [:javaun] 244.98 KB, image/png		Details