1511071 - QA test plan for Return to AMO

Status: RESOLVED FIXED

(Firefox :: Messaging System, enhancement, P1)

Description

[Ursula Sarracini (:ursula)]

This bug is to discuss how we plan on end-to-end test the new Return to AMO feature. Right now we can test the UI portion of Return to AMO by "faking" some attribution in Activity Stream dev tools. The instructions on how to test that are found here: https://github.com/mozilla/activity-stream/pull/4578#issue-233701010. This sets the attribution and allows us to see the UI.

We need a plan for testing the entire flow, start to finish:
1. Without Firefox installed,
2. Navigate to https://addons.mozilla.org via another browser (Chrome for example)
3. Find an add-on and click the "Only with Firefox - Get Firefox Now" button
4. Install Firefox and start it up
5. Observe that we see the Return to AMO flow with the correct add-on that we were trying to install in the other browser

Shane mentioned there are some scripts that we can drop to add attribution to a Nightly build, and then we can start that Nightly and observe that the attribution got picked up from the build, and therefore we see the Return to AMO UI. There are also ways to test that the attribution gets set from AMO into a build.

Testing these two pieces separately may be enough to get GREEN sign-off from QA if we don't have a way to end-to-end test the entire flow.

Shane and David, feel free to add any other context you have to this bug.

Comment 1

[David Durst [:ddurst]]

NI to pmac, whom we spoke to about how attribution works. (Paul, this supersedes my email.) If there's already a way to test attribution end-to-end (from initiating installation from a mozilla property to verifying the attribution is there in Nightly), we could use that here.

If there isn't already a way to do that, then recommendations are welcome. We're looking to land this in 65 and QA is blocked on this.

Comment 2

[Paul [:pmac] McLanahan]

If all that is required on the www.m.o side is for query params to exist on the download page, then you should just manually be able to add them to the download URL. e.g.

https://www.mozilla.org/en-US/firefox/download/thanks/?utm_source=return-to-amo

I know that doesn't test the whole flow, but that should test that the data from the URL makes it into the installed Firefox. If the AMO button links directly to that URL then no further website changes are necessary. If you'll be linking to /firefox/new/ instead (which I do recommend btw. I feel it's a jarring UX to land on a new website and immediately start downloading) then a full test will need to wait until we pass the parameters over to the /thanks/ page.

If you just need to test windows however, it should work now. It's the Mac attribution that doesn't fully work yet.

Comment 3

[Shane Caraveo (:mixedpuppy)]

Attachment: attibutionScratchpad

scratchpad script to output attribution codes to console, both windows and osx.

Comment 4

[Shane Caraveo (:mixedpuppy)]

Attachment: showattrs

osx command line script to dump the attributes for any given downloaded app

Comment 5

[Shane Caraveo (:mixedpuppy)]

Attachment: attributionReleaseTest.63.txt

Outlines how to test a new download for attribution data, requires other scripts attached.

Comment 6

[Shane Caraveo (:mixedpuppy)]

Attachment: attributionScratchpad

Comment 7

[Shane Caraveo (:mixedpuppy)]

Attachment: attributionNightlyTest.txt

Instructions to test nightly on osx and see the attribution UI in about:welcome.

Comment 8

[Shane Caraveo (:mixedpuppy)]

Attachment: attributionNightlyTest.txt

adds windows test

Comment 9

[Peter deHaan [:pdehaan]]

I tried downloading FF65 Beta channel via https://download.mozilla.org/?product=firefox-beta-latest-ssl&os=osx&lang=en-US&utm_campaign=non-fx-button&utm_content=%7BEF522540-89F5-46b9-B6FE-1829E2B572C6%7D&utm_medium=referral&utm_source=addons.mozilla.org

And then after unDMG-ing the archive and running ./showattrs.sh /Applications/Firefox65.0b9.app, I get the following:

$ ./showattrs.sh /Applications/Firefox65.0b9.app
D8405CA2-AF9D-4326-85DD-F6845A6ADA5F|568748566.0|com.google.Chrome|Chrome|https://download-installer.cdn.mozilla.net/pub/firefox/releases/65.0b9/mac/en-US/Firefox%2065.0b9.dmg|||0|||

But after running the scratchpad code in the Browser context, I get the following output in my Browser Console log:

*** failed getting attribution data  Object {  }    Scratchpad/1:16:7
*** addon is null    Scratchpad/1:28:3 

Any ideas what I'm doing wrong? I'm using Beta 65 and not Nightly, but wasn't sure if I still needed to follow the xattr steps at https://bug1511071.bmoattachments.org/attachment.cgi?id=9028763

Comment 10

[David Durst [:ddurst]]

(I assume that you should be testing with 66, based on the changes landed in bug 1511165.)

Comment 11

[Peter deHaan [:pdehaan]]

I am unable to get the steps working on the latest macOS (Mojave) or on my older laptop with High Sierra using the following steps: https://gist.github.com/pdehaan/72ea2ccf30678c1a7ee34dfc1ffeb034

Has anybody else gotten this working and can see where my steps are wrong or why I'm not seeing the expected output in the Scratchpad.

Or is it possible to set up some dev/stage version of AMO+downloads that passes the correct utm_ parameters and attribution values so we don't need to copy over attribution values from Release to Nightly and fake a lot of the values?

Comment 12

[Peter deHaan [:pdehaan]]

I have a WIP E2E test plan at https://github.com/pdehaan/return-to-amo-e2e-test which covers downloading Firefox Nightly (v67) and using mixedpuppy's https://pastebin.com/kWZ3DhdZ script to copy attributes and then launch Firefox Nightly. I'm not seeing the about:welcome page open in the new profile, but if I open about:welcome in a new tab it will prompt me to install the SearchPreview add-on. :+1:

Comment 13

[Vlad Jiman]

We've had a few discussions with Andrei who tried to help get the end to end testing working on Windows, but to no success, the steps we used were the ones (that apply) in https://gist.github.com/piatra/f0a2b9f9f700cbe19afd2ebf1561e6ee and https://bug1511071.bmoattachments.org/attachment.cgi?id=9028763 for which we also tried using the encoded URL instead of the one shown in the steps but this did not work either

Comment 14

[David Durst [:ddurst]]

Bug 1525076 prevents RTAMO from working with MacOS, which is not blocking.

Only Windows QA paths apply at this time.

Comment 15

[Tim Spurway [:tspurway]]

closing this as i believe we've done what we can :-)