Closed Bug 1198904 Opened 9 years ago Closed 9 years ago

Create two custom Try builds for User Diary Study

Categories

(Firefox :: General, defect, P1)

defect

Tracking

()

RESOLVED FIXED
Iteration:
43.2 - Sep 7

People

(Reporter: javaun, Assigned: past)

Details

(Whiteboard: [fxprivacy])

Attachments

(3 files)

We are conducting a user research diary study from 9/2 to 9/4, with followup interviews afterward. The primary objectives are to 1. Compare the "A" list to the "B" list, and 2. see whether users find the new PBM (with TP) to be an improvement on the old.

We will need 2 custom Try builds to provide to study participants.
1.  One build will have the existing blocklist, the "A" list (the standard Disconnect list, which does not include Content category)
2.  One build will have the "B" list (the full Disconnect list including Content category)

For each of the two builds above, we also need: 
* The entity whitelist working on both. 
* To manually turn on the UI tour in both builds. This is accomplished by (comment 2) in bug 1190447. https://bugzilla.mozilla.org/show_bug.cgi?id=1190447#c2
* Each test build must create a new user profile. The participants will be existing Fx users. We need to make sure they don't have interfering add-ons (i.e. adblockers) running, and we also don't want to interfere with their existing profiles.
* We need to test all of the above for both builds (A/B): each has the correct blocklist, working entity whitelisting, the UI tour activated, and that it creates a clean profile.
I'm awaiting from Bill a date on which the study recruiter needs the builds to give to participants. We'll back up our due date from there.
Flags: needinfo?(wselman)
Priority: -- → P1
Whiteboard: [fxprivacy]
Is it OK if the builds don't 'install' the custom Firefox?  In other words, users will need to remember to run the particular executable they downloaded every time they close the browser (as opposed to selecting it from the start menu).  Seems like if it lasts 2 days that should be OK, but just want to make sure the build approach we've used for UX studies will work here.
The necessary changes in Nightly to target the staging server's lists are:

pref("browser.trackingprotection.updateURL", "https://tracking.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
pref("browser.trackingprotection.gethashURL", "https://tracking.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");

(only the domain name changes)

The above should happen to both "A" and "B" profiles. The additional required changes for the "B" list are:

pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,mozpub-track-digest256,mozpub-trackwhite-digest256,moztestpub-track-digest256");

(add the last mozpub entry)

pref("urlclassifier.trackingTable", "test-track-simple,moztestpub-track-digest256");

(modify the last entry to be moztestpub)
(In reply to Panos Astithas [:past] from comment #3)
> digest256,mozpub-trackwhite-digest256,moztestpub-track-digest256");
> 
> (add the last mozpub entry)

Grrrr, that should have been "add the last *moztestpub* entry".
(In reply to Brian Grinstead [:bgrins] from comment #2)
> Is it OK if the builds don't 'install' the custom Firefox? 

Is the custom icon going to be Nightly or DevEdition? That would be better, since we could in that case give users the instructions to "Click the blue icon" (etc) as opposed to the Fx they're used to launching.

Let's ask Bill and Gemma, this may be a barrier. We've had users forget special instructions and invalidate their test in the past.
Note that moztestpub is not actually the real list you're going to use during the study. The real one hasn't made it to stage yet, it's waiting for the new stack to be deployed.

Once the new stack is in stage, then you'll need to use "mozpub2-track-digest256" which is the full Disconnect list including the "content" category. The other list (moztestpub-track-digest256) is not what you want since it's a copy of the normal Disconnect list, not the full one.

If you use https://github.com/fmarier/safebrowsing-tools/blob/master/get-lists.py you can see the lists available on stage currently. Once you see mozpub2-track-digest256 in that list, that means the new stack is running on stage and you can switch to mozpub2.
(In reply to Javaun Moradi [:javaun] from comment #5)
> (In reply to Brian Grinstead [:bgrins] from comment #2)
> > Is it OK if the builds don't 'install' the custom Firefox? 
> 
> Is the custom icon going to be Nightly or DevEdition? That would be better,
> since we could in that case give users the instructions to "Click the blue
> icon" (etc) as opposed to the Fx they're used to launching.

It's going to be the Nightly icon
(In reply to Javaun Moradi [:javaun] from comment #5)
> (In reply to Brian Grinstead [:bgrins] from comment #2)
> > Is it OK if the builds don't 'install' the custom Firefox? 
> 
> Is the custom icon going to be Nightly or DevEdition? That would be better,
> since we could in that case give users the instructions to "Click the blue
> icon" (etc) as opposed to the Fx they're used to launching.
> 
> Let's ask Bill and Gemma, this may be a barrier. We've had users forget
> special instructions and invalidate their test in the past.

Regardless of the duration of the study, participants have varying levels of comfort and expertise. I'm certain we will have at least some participants with a good deal of anxiety about installing software. We can walk them through this process. In terms of participants' usage, we need to distribute a build to participants that behaves as much like a typical Firefox browser install as possible.
Flags: needinfo?(wselman)
1. We made a call in standup today to not force a new profile, that's no longer a build requirement. We want users to have their history/bookmarks since that's going to affect their enjoyment and productivity of the browser and affect the test. We'll use their existing profile. 

2. No complex engineering changes. We don't want to spend too much time or introduce unexpected things. Prefs are all we're doing. We want this to be as easy as possible to create the builds, quickly, and we want to spend time QA'ing the builds since there are already so many things that can go wrong (wrong lists, UI tour doesn't work, etc.) 

3. Javaun/Ash will talk to Bill/Gemma. We'll make a call on whether to give users manual instructions turn off add-ons that may affect the test (i.e. ABP) or whether it's acceptable to leave them on. We'll also work on the overall instructions we give users to make sure they don't accidentally launch normal fx and stick to our Nightly build.
My #1 in comment 9 is unclear (double negative). To be clear: 

Our test builds will not create a new profile, we'll use their existing Fx profile, allowing them access to prior history, PWs, bookmarks, etc.
Assignee: nobody → past
Status: NEW → ASSIGNED
Attached patch build ASplinter Review
This is the patch that I've used for the "A" build.

Once completed, builds for all platforms will be available at:
https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/pastithas@mozilla.com-5b97c5a95055
Attachment #8654111 - Flags: feedback?(francois)
Iteration: --- → 43.2 - Sep 7
Flags: qe-verify-
Attached patch build BSplinter Review
This is the patch that I've used for the "B" build, on top of the previous one.

Once completed, builds for all platforms will be available at:
https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/pastithas@mozilla.com-b1420b664298
Attachment #8654117 - Flags: feedback?(francois)
Attachment #8654111 - Flags: feedback?(francois) → feedback+
Attachment #8654117 - Flags: feedback?(francois) → feedback+
Thanks, I'm going to test them Panos. I assume for win32 all I need is the .exe and for MacOSX the .dmg?
The entity whitelist is not working for me in the B build above for Mac. (haven't tried any other builds yet).

On Yahoo mail, I see a bunch of blocked yimg loads that should be entity whitelisted:

The resource at "https://s.yimg.com/zz/combo?kx/yucs/uh3s/atomic/63/css/atomic-min.css&kx/yucs/uh_common/meta/3/css/meta-min.css&kx/yucs/uh3s/uh/394/css/uh-center-aligned-min.css" was blocked because tracking protection is enabled.[Learn More] launch
The resource at "https://s.yimg.com/zz/combo?nq/s/a/style_5_2986.css&nq/s/a/messagelist_5_2986.css&nq/s/a/basepane_5_2986.css&nq/s/a/pane_5_2986.css&nq/s/a/notifications_5_2986.css&nq/s/a/notifications-v2_5_2986.css&nq/s/a/ads_5_2986.css&nq/s/a/pc-overrides_5_2986.css&nq/s/a/compose_5_2986.css&nq/s/a/lozenge_5_2986.css&nq/s/a/search_5_2986.css&nq/s/a/boss_5_2986.css&nq/s/a/navigation_5_2986.css&nq/s/a/days_5_2986.css&nq/s/a/predictive-search_5_2986.css&nq/s/a/compose-inlineoverlay_5_2986.css&nq/s/a/nav-tabs_5_2986.css&nq/s/a/federated-autocomplete_5_2986.css&nq/s/a/transitions_5_2986.css&nq/s/a/conversations_5_2986.css&nq/s/a/multiple-participants_5_2986.css&nq/s/a/theme/purple_5_2986.css" was blocked because tracking protection is enabled.[Learn More] launch
The resource at "https://s.yimg.com/zz/combo?nq/s/php/mail/js/launch_5_2991.js&nq/s/php/mail/js/promises_5_2991.js&nq/s/php/mail/js/mousetrap_5_2991.js&nq/s/php/mail/js/reactlibs_5_2991.js" was blocked because tracking protection is enabled.[Learn More] launch
The resource at "https://s.yimg.com/zz/combo?os/stencil/3.0.1/desktop/styles-ltr.css" was blocked because tracking protection is enabled.[Learn More] launch
The resource at "https://s.yimg.com/kx/yucs/uh3s/promo-ff/1/images/ff_icon-compressed.png" was blocked because tracking protection is enabled.[Learn More] launch
The resource at "https://s.yimg.com/os/mit/media/m/base/images/transparent-95031.png" was blocked because tracking protection is enabled.[Learn More] launch
The resource at "https://s.yimg.com/zz/combo?nq/2991/yui/yui-min.js&nq/2991/oop/oop-min.js&nq/2991/attribute/attribute-min.js&nq/2991/event-custom/event-custom-min.js&nq/2991/event-base/event-base-min.js&nq/2991/event-delegate/event-delegate-min.js&nq/2991/event-synthetic/event-synthetic-min.js&nq/2991/event-resize/event-resize-min.js&nq/2991/event-focus/event-focus-min.js&nq/2991/event-key/event-key-min.js&nq/2991/base/base-base-min.js&nq/2991/pluginhost/pluginhost-min.js&nq/2991/dom/dom-min.js&nq/2991/node/node-min.js&nq/2991/json/json-min.js&nq/2991/querystring-parse-simple/querystring-parse-simple-min.js&nq/2991/querystring-stringify-simple/querystring-stringify-simple-min.js&nq/2991/cookie/cookie-min.js&nq/2991/plugin/plugin-min.js" was blocked because tracking protection is enabled.[Learn More] launch
The resource at "https://s.yimg.com/zz/combo?nq/s/intl/js/launch/lang_en-US_5_2991.js&nq/s/intl/js/launch/conf_us_5_2991.js" was blocked because tracking protection is enabled.[Learn More] launch
The resource at "https://s.yimg.com/zz/combo?nq/s/launch/common-neo-base_5_2991.js&nq/s/launch/inbox-conversations_5_2991.js&nq/s/launch/inbox-mail-new_5_2991.js&nq/s/launch/inbox-base_5_2991.js" was blocked because tracking protection is enabled.[Learn More] launch
The resource at "https://s.yimg.com/rz/d/yahoo_mail_en-US_s_f_pw_351x40_mail.png" was blocked because tracking protection is enabled.[Learn More] launch
Flags: needinfo?(past)
Flags: needinfo?(francois)
Similarly on the Youtube.com homepage, I see a ton of blocked loads. I didn't see all of these domains whitelisted, but Doubleclick is on the list and being blocked: 

The resource at "https://yt3.ggpht.com/-ugxFMqj2ljU/AAAAAAAAAAI/AAAAAAAAAAA/F0UMjiGsYJE/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-FPW52uVh3I0/AAAAAAAAAAI/AAAAAAAAAAA/Y-drd_btCtQ/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-T9rEqYD-3SQ/AAAAAAAAAAI/AAAAAAAAAAA/A5w3XlKK_hA/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-4kK186DpGLw/AAAAAAAAAAI/AAAAAAAAAAA/Sml0b01jsQY/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/jMgqgWNnSbj-GL8f3OyAJw_j9KYUGTEJT6axyB-OyjQbyvXwI010nla5ugPRUrAPsrQmbatMVTIeelJ7Nw=s88-nd" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-ios1OK1LQyc/AAAAAAAAAAI/AAAAAAAAAAA/1lJALaYer2c/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://pubads.g.doubleclick.net/gampad/ads?ad_rule=0&gdfp_req=1&iu=%2F6762%2Fmkt.ythome_1x1&scp=k18%3D1%26k21%3D1%26kage%3D41%26kar%3D5%26kauth%3D1%26kga%3D1003%26kgender%3Dm%26kgg%3D1%26klg%3Den%26kmyd%3Dad_creative_3%26ssl%3D1&sz=1x1&correlator=4407410600160204" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://apis.google.com/_/scs/abc-static/_/js/k=gapi.gapi.en.vwFhyajD8zA.O/m=iframes/rt=j/sv=1/d=1/ed=1/am=AAI/rs=AItRSTPOWGI_kROKlKX9NGLO-MOVVH8_gQ/cb=gapi.loaded_0" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://apis.google.com/_/scs/abc-static/_/js/k=gapi.gapi.en.vwFhyajD8zA.O/m=client/exm=iframes/rt=j/sv=1/d=1/ed=1/am=AAI/rs=AItRSTPOWGI_kROKlKX9NGLO-MOVVH8_gQ/cb=gapi.loaded_1" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://apis.google.com/_/scs/abc-static/_/js/k=gapi.gapi.en.vwFhyajD8zA.O/m=card,gapi_iframes/exm=client,iframes/rt=j/sv=1/d=1/ed=1/am=AAI/rs=AItRSTPOWGI_kROKlKX9NGLO-MOVVH8_gQ/cb=gapi.loaded_2" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-3xTchkWeEDY/AAAAAAAAAAI/AAAAAAAAAAA/BgbhtqDeE6A/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-VxZgZkhnb74/AAAAAAAAAAI/AAAAAAAAAAA/mwYsteEti-8/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-OojyL2VZrjg/AAAAAAAAAAI/AAAAAAAAAAA/KookuDiU9Hc/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-OH2bXZESFYA/AAAAAAAAAAI/AAAAAAAAAAA/PkM8GEwnuB8/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-Uc4InMop_JE/AAAAAAAAAAI/AAAAAAAAAAA/tegzwid6a7s/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-inWvjylQAzM/AAAAAAAAAAI/AAAAAAAAAAA/6eiImQd3vq0/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-q15PcbNg4cw/AAAAAAAAAAI/AAAAAAAAAAA/E7D7QZnDBIo/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-sp0YiR_yyR0/AAAAAAAAAAI/AAAAAAAAAAA/kXU4u1ny2T4/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-H0mi-PXThxM/AAAAAAAAAAI/AAAAAAAAAAA/Iq-E3PEJ1Is/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-0FtRHbyiQoQ/AAAAAAAAAAI/AAAAAAAAAAA/bKKVkoaXMg8/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-6h1R3UqbjE4/AAAAAAAAAAI/AAAAAAAAAAA/GzBJZ8gN4DA/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-4XZ2aFHsGpo/AAAAAAAAAAI/AAAAAAAAAAA/WzvWe1mgKVc/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-ERhFcb82qcI/AAAAAAAAAAI/AAAAAAAAAAA/OOW4LsB40_k/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/--Hqpsmv756s/AAAAAAAAAAI/AAAAAAAAAAA/leEsNCuKbFY/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-fXTbviunvO4/AAAAAAAAAAI/AAAAAAAAAAA/5PBevKp-IZ4/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-xGE6OuB5BrM/AAAAAAAAAAI/AAAAAAAAAAA/qX8NFYzRJ-Y/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-Mtgb7PDyOSo/AAAAAAAAAAI/AAAAAAAAAAA/sI9Y6MYZhvg/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://yt3.ggpht.com/-eIWIWbBTofk/AAAAAAAAAAI/AAAAAAAAAAA/ODF_JboKWqc/s88-c-k-no/photo.jpg" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
The resource at "https://pagead2.googlesyndication.com/pagead/show_ads.js" was blocked because tracking protection is enabled.[Learn More] www.youtube.com
One more: I've tried two macs. When I launch the UI tour from the built-in page, it's not showing the lock in the URL bar (doesn't appear that TP is actually on), and then when I get to step 3 and the Control Center is open, It says "No tracking Elements detected on this page." There is no "Disable" button, and there is no callout coming off it, so there is no way to click "Got it" to go to step 4. Screenshot attached. Could be my profile is screwy.
Attached image TP_Build_B_UIerror.png
I've created a QA test script here. We'd like to have the builds QA'ed, working properly and to Bill Selman by Monday Afternoon 8/31. (Bill is PT USA, so the European and USA east coast teams have some time).

https://docs.google.com/document/d/1Q-CI8zqjWu0MROGLRo0Afr0UQ1SPh3KrVzl0KZpnUR0/edit#
(In reply to Javaun Moradi [:javaun] from comment #14)
> The entity whitelist is not working for me in the B build above for Mac.
> (haven't tried any other builds yet).

Unfortunately I don't know how to debug the whitelist, I think we need François for that.

I wonder if the network/server configurations work that IT had planned for this weekend could be related.

(In reply to Javaun Moradi [:javaun] from comment #16)
> One more: I've tried two macs. When I launch the UI tour from the built-in
> page, it's not showing the lock in the URL bar (doesn't appear that TP is
> actually on), and then when I get to step 3 and the Control Center is open,
> It says "No tracking Elements detected on this page." There is no "Disable"
> button, and there is no callout coming off it, so there is no way to click
> "Got it" to go to step 4. Screenshot attached. Could be my profile is screwy.

I tried build "B" on Linux and I can't reproduce this. I see both the lock and the shield (I assume you meant that you don't see the shield above).
Flags: needinfo?(past)
Comment on attachment 8654111 [details] [diff] [review]
build A

Hey Alex, could you take a look at the changes I did for enabling the new UI Tour and see if everything looks OK?
Attachment #8654111 - Flags: feedback?(agibson)
Comment on attachment 8654117 [details] [diff] [review]
build B

Same here, and also take a note of comment 16. If you could also take one of the test builds (comment 11, comment 12) for a spin to see if they work as expected as far as the UI Tour is concerned, that would be fantastic.
Attachment #8654117 - Flags: feedback?(agibson)
(In reply to Panos Astithas [:past] from comment #19)
> I tried build "B" on Linux and I can't reproduce this. I see both the lock
> and the shield (I assume you meant that you don't see the shield above).

Hmm, I managed to reproduce this only once, on build A, in my first attempt after running it on a new profile. It may be related to me doing things very quickly and not letting the page settle down, I don't know. I couldn't reproduce it afterwards.
Oh, shoot, I noticed in the QA script that you must be using the debug builds. Sorry Javaun, this is my mistake for not excluding them in the first place. Debug builds are slower than regular optimized builds and not what we ship to end users. There is a chance that timing differences due to the above could be affecting what you observe.

Could you try again with the regular builds (the ones without -debug)? I've updated the QA script to that effect.
I'm not really the best person to be reviewing these changes. As far as I can tell, the preferences look ok - but that's just a guess. I'm afraid I have limited knowledge here.

I downloaded build A for Mac OSX, and I still see the same thing Javaun as when we tested yesterday. The tracking protection feature does not seem to be working correctly using the block list, which is why the UITour is not working correctly (since that is a dependency). Everything seems fine in Nightly, so it must be something specific to the changes here.

I can't seem to download build 'B' for OSX as it seem to be incomplete looking in the FTP directory?

Matt gave some useful advice here that may be worth investigating:

"Looking at the log after enabling browser.safebrowsing.debug may give clues.

Without testing the build or anything, I'm wondering if the problem is just that the tracking list hasn't been downloaded yet or there is an error during download (which would show in the Browser Console [not Web Console] when logs are on)

Looking at the network requests from the Browser Console should also show the list being downloaded for a new profile."
Attachment #8654117 - Flags: feedback?(agibson) → feedback+
Attachment #8654111 - Flags: feedback?(agibson) → feedback+
(In reply to Alex Gibson [:agibson] from comment #24)
> I can't seem to download build 'B' for OSX as it seem to be incomplete
> looking in the FTP directory?

Thanks, the original build job had failed, but I restarted it and now it's there.
The entity whitelist not working has nothing to do with these builds BTW. It's currently broken on Nightly too. I'm investigating.
Flags: needinfo?(francois)
The entity whitelist was fixed yesterday. Javaun, can we close this bug or is there more we want to track here?
Flags: needinfo?(jmoradi)
Yes, let's close this bug!
Flags: needinfo?(jmoradi)
Great, thanks!
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: