Closed Bug 999784 Opened 10 years ago Closed 10 years ago

[amo] Test new AMO blocklist endpoint with current and older Fx Desktop browsers

Categories

(Cloud Services :: Operations: Marketplace, task)

Product:
Cloud Services
Component:
Operations: Marketplace
Platform:
x86_64
Windows 8.1
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jjensen, Assigned: u279076)

References

Details

Attachments

(3 files)

Screenshot_2014-06-04-16-38-20.png
94.41 KB, image/png
Details
console log.txt
3.36 KB, text/plain
Details
FF android vs desktop.png
547.02 KB, image/png
Details 
The IT and Cloud Services team are working to change the endpoint to which the blocklist ping is directed in bug 991843.

Browsers requesting the "old" address will receive a 301 redirect. We need to make sure that this works correctly with older versions of the browser. If it doesn't, then browsers may not receive updated blocklists and we may not receive correct ADI information.

Ideally we would do some analysis of the relevant Fx client code, see when it has changed in the past, and select appropriate Fx versions in the past.

I'm not sure how feasible that would be, so for now am suggesting the following:
the following versions:

1) current Nightly
2) current release
3) Firefox 10
3) Firefox 3.6
4) Firefox 4.0

Anthony Hughes has kindly offered to help out here.

Presumably what would need to happen is that

a) the browser version would be started
b) the ping would either be activated or the tester would wait for it to occur
c) the tester would see if the correct xml file was downloaded

A possible extra step could be someone server-side watching the relevant logs to make sure that the request was seen. But I don't think that's necessary.
Which platforms does this need testing on? In particular, Firefox 3.6 supported Windows 2000 and Mac OSX PPC which QA no longer has access to. I could probably scrounge up an old Windows 2000 VM but a PPC Mac will be much harder to come by.
Also, are we expecting this to impact Android, Android (XUL), and Firefox OS builds?
This will affect anything that hits the addon blocklist.
Blocks: 981663
Blocks: 1006615
(In reply to Jason Thomas [:jason] from comment #4)
> I've created a test domain https://addons-blocklist-redirect.allizom.org
> which redirects /blocklist/* to https://blocklist.addons.mozilla.org.
> 
> ex: https://addons-blocklist-redirect.allizom.org/blocklist/3/x/x/

Hi Jason,

I'm working on a draft test plan. How can we be sure these redirects are working properly? Are these URLs logged somewhere? And how do we know the blocklist.xml is the "correct" one?
The redirect is configured in nginx and a rewrite of al /blocklist/ path to the new domain [1]. The redirects will be logged to nginx web logs.

Since blocklist.addons.mozilla.org will be using the same webapp as addons.mozilla.org the data from blocklist.xml should be the same. [2][3] shows how the blocklist.xml is determined for a client.

Also can test to make sure blocklist.xml is being served correctly by comparing previous url data to a new url data.

cc'ing krupa as she is the QA for AMO.

λ kenshin ~ → curl https://blocklist.addons.mozilla.org/blocklist/3/x/x/ 2>/dev/null| md5sum
8d4c4429035c3ddd7ebe4b63e79ad5e3  -
λ kenshin ~ → curl https://addons.mozilla.org/blocklist/3/x/x/ 2>/dev/null| md5sum
8d4c4429035c3ddd7ebe4b63e79ad5e3  -

[1] https://github.com/mozilla-services/svcops-puppet/blob/master/modules/marketplace/templates/nginx/redirect_blocklist.conf

[2] https://github.com/mozilla/olympia/blob/813700f59ef954ba61582d8a37b52ff607a9628a/lib/urls_base.py#L27
[3] https://github.com/mozilla/olympia/blob/813700f59ef954ba61582d8a37b52ff607a9628a/apps/blocklist/views.py
(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #1)
> Which platforms does this need testing on? In particular, Firefox 3.6
> supported Windows 2000 and Mac OSX PPC which QA no longer has access to. I
> could probably scrounge up an old Windows 2000 VM but a PPC Mac will be much
> harder to come by.

Hi Anthony,

Sorry, I missed this comment earlier. Windows XP is fine. Don't worry about Mac PPC.
:ashughes please let me know if you require additional information from me to proceed with testing. Thanks!
Flags: needinfo?(anthony.s.hughes)
Assignee: server-ops-amo → anthony.s.hughes
Sorry for taking so long to respond. I'm working out a few remaining details for the test plan. I'll post more information in this bug soon.
Flags: needinfo?(anthony.s.hughes)
I've confirmed that this should only need testing on Desktop and Android but I'm still a bit unclear about A) confirming we've hit the right server and B) confirming we have the right blocklist.xml. Needinfo to Krupa who can hopefully create some clarity around this. 

Krupa, see also comment 6, thanks.
Flags: needinfo?(krupa.mozbugs)
FYI, preliminary documentation has been posted here:
https://wiki.mozilla.org/QA/Desktop_Firefox/Test_Plans/Bug_999784
My suggestion to test this would be as follows:

1. Point blocklist.addons.mozilla.org point to addons-dev for testing
2. Have an add-on blocked on addons-dev which is not blocked on addons.mozilla.org
3. On an older version of Firefox (3.6?), have this blocked add-on installed and see if the add-on gets blocklisted

If the add-on gets blocklisted, it will be a fair assumption that the redirect worked.

Jason and Anthony can coordinate on when to have blocklist.addons.mozilla.org point to addons-dev for testing

Does this seem like a reasonable testcase?
Flags: needinfo?(krupa.mozbugs)
Krupa's plan looks good, but let's redirect from addons-dev.allizom.org to addons.allizom.org, so we don't have to alter production.
I have removed the test domain in comment 4 and setup the following based on comment 12 and comment 13.

1. Configured http://blocklist-dev.allizom.org to represent the dev version of https://blocklist.addons.mozilla.org. This blocklist domain is connected to the same database as https://addons-dev.allizom.org/

i.e. https://blocklist-dev.allizom.org/blocklist/3/x/x/

2. https://addons-dev.allizom.org/blocklist/* traffic is configured to 301 to http://blocklist-dev.allizom.org/

i.e 
[root@addonsadm.private.phx1 ~]# curl -L https://addons-dev.allizom.org/blocklist/3/x/x/ -so /dev/null -D-
HTTP/1.1 301 Moved Permanently
Server: nginx
X-Backend-Server: dev2
Content-Type: text/html
Date: Fri, 16 May 2014 13:51:50 GMT
Location: https://blocklist-dev.allizom.org/blocklist/3/x/x/
Via: Moz-pp-zlb09
Connection: keep-alive
Content-Length: 178

HTTP/1.1 200 OK
Content-Security-Policy-Report-Only: script-src 'self' https://www.google.com https://mozorg.cdn.mozilla.net https://www.paypalobjects.com https://ssl.google-analytics.com https://addons-dev-cdn.allizom.org; default-src * data:; frame-src https://ssl.google-analytics.com https://sandbox.paypal.com; object-src 'none'; report-uri /services/csp/report
Server: nginx
X-Backend-Server: dev2
Cache-Control: max-age=3600
Content-Type: text/xml; charset=utf-8
Strict-Transport-Security: max-age=31536000
Date: Fri, 16 May 2014 13:43:56 GMT
Transfer-Encoding: chunked
Via: Moz-pp-zlb09
X-Frame-Options: DENY
Connection: Keep-Alive
X-Cache-Info: cached
(In reply to krupa raj[:krupa] from comment #12)
> Does this seem like a reasonable testcase?

Is there an add-on that is currently blocked in dev but not production that I can use for testing? Note, this needs to work for Desktop Firefox 3.6 through 32 and Firefox for Android 21 through 32.
NEEDINFOing Krupa for test addon.
Flags: needinfo?(krupa.mozbugs)
(In reply to krupa raj[:krupa] from comment #17)
> https://addons-dev.allizom.org/en-US/firefox/addon/test-add-on-for-
> blacklisting/

I confirm that I'm able to install this add-on in Firefox 29.0.1 on Linux and Firefox 32.0a1 on Android. I'll go ahead and update the testplan with the details we talked about last week. I will try to have the testing completed by the end of this week.
Summary: Test new AMO endpoint with current and older Fx Desktop browsers → [amo] Test new AMO blocklist endpoint with current and older Fx Desktop browsers
Has the test add-on been blocked on the test server? I'm able to successfully ping blocklist-dev.allizom.org but the add-on in comment 17 remains enabled.
(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #19)
> Has the test add-on been blocked on the test server? I'm able to
> successfully ping blocklist-dev.allizom.org but the add-on in comment 17
> remains enabled.

By the way, this is the blocklist.xml I get:
https://blocklist-dev.allizom.org/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/29.0.1/Firefox/20140514131124/Linux_x86_64-gcc3/en-US/default/Linux%203.14.4-200.fc20.x86_64%20(GTK%202.24.22)/default/default/invalid/invalid/0/
I just added it to the -dev blocklist. It's possible it was removed during a recent migration, or maybe it wasn't added.
Thanks Jorge, I just retested this and it blocks now. We'll resume our testing now.
Desktop testing has been completed and found no issues. I still need to spotcheck a couple of Android versions but I'm at a work week this week. The soonest I can check is early next week.
:ashughes - any update? Can we have it ready so CAB can approve the change on Wednesday, 6/4?
Flags: needinfo?(anthony.s.hughes)
(In reply to Anurag Phadke[:aphadke@mozilla.com] from comment #24)
> :ashughes - any update? Can we have it ready so CAB can approve the change
> on Wednesday, 6/4?

I will do my best to meet that deadline. As stated in comment 23, the soonest I will be able to test Android is on Monday.
Flags: needinfo?(anthony.s.hughes)
(In reply to Anurag Phadke[:aphadke@mozilla.com] from comment #24)
> Can we have it ready so CAB can approve the change on Wednesday, 6/4?

Does that mean you plan to do the 301 redirect right away after that or is that just the technical green light for that?
I'd like that to only happen once we have confirmed we get the BAMO data into Socorro correctly and tested the setup with Nightly (and then we'd ideally switch on the 301 pretty much exactly at a UTC midnight so we can cut off using the AMO data with that date switch).
kairo - my plan is that we deploy 301 on all channels, collect + compare logs on BAMO and AMO and switch off AMO completely end of Q2.
I've been having trouble forcing a blocklist ping in Firefox for Android. Until I figure this out I won't be able to test Android blocklist redirects. The alternative will be to just let my phone sit there until it naturally pings addons-dev. 

If I can't get this figured out in the next 24 hours we might need to just go ahead without Android testing. I'm not familiar at all with Android blocklisting so I can't speak to how risky it is to go live untested.
we don't want to go live w/o testing and without getting a sign-off from you/your team. Let's wait until you are comfortable before we deploy it.
I just sent an email to the Firefox for Android QA team for help as I've been unable to figure this out.
I managed to figure this out. However the add-on does not appear to be blocked.

Here is the blocklist I get:
https://blocklist-dev.allizom.org/blocklist/3/%7Baa3c5121-dab2-40e2-81ca-7ea25febc110%7D/32.0a1/

It seems like the redirect is working properly but not the blocklisting.
:krupa is this something you can help with?
Flags: needinfo?(krupa.mozbugs)
Blocks: 1020320
(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #31)
> I managed to figure this out. However the add-on does not appear to be
> blocked.
> 
> Here is the blocklist I get:
> https://blocklist-dev.allizom.org/blocklist/3/%7Baa3c5121-dab2-40e2-81ca-
> 7ea25febc110%7D/32.0a1/
> 
> It seems like the redirect is working properly but not the blocklisting.

Can you try now?
Flags: needinfo?(krupa.mozbugs)
I see the GUID of the test add-on in the blocklist set 

<emItem blockID="i568" id="krupa@krupa.com">
<versionRange minVersion="0" maxVersion="*" severity="1">
                    </versionRange><prefs>
              </prefs></emItem>
Here's a screenshot showing the about:addons page for the test addon after pinging the blocklist.
Jason, the redirected BAMO URL in comment #34 doesn't look right, is there a bug in the redirect?
Flags: needinfo?(jthomas)
This should now be fixed.

λ master ~ → curl -I https://addons-dev.allizom.org/blocklist/3/%7Baa3c5121-dab2-40e2-81ca-7ea25febc110%7D/32.0a1/Fennec/20140602072051/Android_arm-eabi-gcc3/en-US/nightly/Linux%2019/default/default/2/2/1/
HTTP/1.1 301 Moved Permanently
Server: nginx
X-Backend-Server: dev1
Content-Type: text/html
Date: Thu, 05 Jun 2014 18:24:21 GMT
Location: https://blocklist-dev.allizom.org/blocklist/3/%7Baa3c5121-dab2-40e2-81ca-7ea25febc110%7D/32.0a1/Fennec/20140602072051/Android_arm-eabi-gcc3/en-US/nightly/Linux%2019/default/default/2/2/1/
Via: Moz-zlb10
Connection: keep-alive
Content-Length: 178
Flags: needinfo?(jthomas)
Anthony, please try again now.


Also, I think we should run the same tests again (possibly with fewer builds though, as the testing here verifies the clients do the right thing) with the prod servers after bug 1020320 is fixed, just to be sure we do have the correct redirects on prod.
This doesn't appear to be working at all anymore with a new profile.

1. Start Firefox with a new profile
2. Install the test add-on from comment 17
3. Open about:config and change "addons.mozilla.org" to "addons-dev.allizom.org" in extensions.blocklist.url
4. Change extensions.blocklist.interval to 10
5. Change devtools.debugger.remote-enabled to true
5. Restart Firefox
6. Connect the phone to the computer via USB and run "adb devices" from terminal to ensure it's listed
7. Run "adb forward tcp:6000 tcp:6000" from terminal
8. Go to the Open menu in Firefox on the computer
9. Select Developer > Connect...
10. Tap OK on the "Incoming Connection" prompt on the phone
11. Click "Main Process" in the Connect tab in Firefox on the computer
12. When the debugger opens, execute the following code
> Components.classes["@mozilla.org/extensions/blocklist;1"].getService(Components.interfaces.nsITimerCallback).notify(null);

Result:
The following is output in the Console and the add-on remains enabled.
> Components.classes["@mozilla.org/extensions/blocklist;1"].getService(Components.interfaces.nsITimerCallback).notify(null);
> undefined

I'm also getting an error in Console on startup:
> WARN	Exception running bootstrap method startup on krupa@krupa.com: Error opening input stream (invalid filename?)

I'm wondering if there might be something wrong with the add-on for Android?
Flags: needinfo?(krupa.mozbugs)
The test add-on is just Adblock Plus with GUIDs changed. I can probably make an Android-specific test add-on if that helps. But, I'd be surprised if Adblock Plus is busted on Android.
Flags: needinfo?(krupa.mozbugs)
If we can try testing with a Android-specific test addon-on that would be great. Krupa could you create one for us?
Flags: needinfo?(krupa.mozbugs)
I made https://addons-dev.allizom.org/en-US/android/addon/android-add-on-for-blocklistin/

Let me know if that works.
Flags: needinfo?(krupa.mozbugs)
Krupa - is this something Jason/myself need to check in the logs?
Flags: needinfo?(krupa.mozbugs)
(In reply to Anurag Phadke[:aphadke@mozilla.com] from comment #44)
> Krupa - is this something Jason/myself need to check in the logs?

I'd expect Anthony Hughes & team to do the testing on Android. Thanks!
Flags: needinfo?(krupa.mozbugs)
(In reply to krupa raj[:krupa] from comment #43)
> I made
> https://addons-dev.allizom.org/en-US/android/addon/android-add-on-for-
> blocklistin/
> 
> Let me know if that works.

I didn't have time to check this today. Paul, can you give it a try using any Android device and a few recent Firefox versions? I've put some rough instructions here: https://wiki.mozilla.org/QA/Desktop_Firefox/Test_Plans/Bug_999784#Steps
Flags: needinfo?(paul.silaghi)
Attached file console log.txt — 
I tested this using the STR in comment 40 and the addon in comment 43 on 
Firefox for Android Nightly 33.0a1(2014-06-16), Aurora 32.0a2(2014-06-16)
Samsung Galaxy Tab (Android 4.0.4)
After executing the ping, the addon gets disabled (grayed out) in addons manager, but there is no message that the addon is blocklisted (like on FF Desktop: "Test addon for blocklisting is known to cause security or stability issues").
Also, see attached the console log.

So, what do you think?
Flags: needinfo?(paul.silaghi) → needinfo?(anthony.s.hughes)
Flags: needinfo?(anthony.s.hughes) → needinfo?(krupa.mozbugs)
Paul, can you attach a screenshot comparing what you see on Android to what you see on Desktop? I'm wondering if what you're seeing is just a difference in implementation of the UI for each platform and not necessarily a result of the changes here. To confirm, you could check this with a known blocked add-on and pinging the production blocklist (addons.mozilla.org) and see what the UI shows.

Kevin, when an add-on is blocked on Android, does it typically display the reason it was blocked in the UI?
Flags: needinfo?(kbrosnan)
(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #48)
> Paul, can you attach a screenshot comparing what you see on Android to what
> you see on Desktop?
Attached
I feel like that's probably expected but I'd like Kevin Brosnan to confirm. He's more familiar with how Android looks than I.
I don't think there is a defined behaivor for alerting the user for unstable extensions on Android. We have not used blocklisting on Android. Filing a bug under the Firefox for Android product makes sense to see if Android wants to copy the desktop behavior.
Flags: needinfo?(kbrosnan)
Thanks Kevin.

Paul, please go ahead and file a bug if you think Android should include this type of information in our UI.

Jason, I think we can go ahead with a push to production whenever you're ready. We'll retest this once live. Note, please need-info me when bug 1020320 is resolved so we can test this again.
Flags: needinfo?(krupa.mozbugs)
Fantastic! Thanks everyone for your help with testing!
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Is this pushed now or will that happen in bug 1020320? also, how long will it take the push to cache?
(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #54)
> Is this pushed now or will that happen in bug 1020320? also, how long will
> it take the push to cache?

As I said over in the other bug, we're not ready for pushing it yet - though the testing here is bringing this one step closer. Thanks for that!
Okay, thanks!
(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #52)
> Paul, please go ahead and file a bug if you think Android should include
> this type of information in our UI.
bug 1027535
Component: Server Operations: AMO Operations → Operations: Marketplace
Product: mozilla.org → Mozilla Services
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: