Closed Bug 1511071 Opened Last year Closed 9 months ago

QA test plan for Return to AMO

Categories

(Firefox :: Messaging System, enhancement, P1)

57 Branch
enhancement

Tracking

()

RESOLVED FIXED
Iteration:
67.2 - Feb 11 - 24

People

(Reporter: ursula, Assigned: tspurway)

References

(Blocks 1 open bug)

Details

Attachments

(4 files, 2 obsolete files)

This bug is to discuss how we plan on end-to-end test the new Return to AMO feature. Right now we can test the UI portion of Return to AMO by "faking" some attribution in Activity Stream dev tools. The instructions on how to test that are found here: https://github.com/mozilla/activity-stream/pull/4578#issue-233701010. This sets the attribution and allows us to see the UI.

We need a plan for testing the entire flow, start to finish:
1. Without Firefox installed,
2. Navigate to https://addons.mozilla.org via another browser (Chrome for example)
3. Find an add-on and click the "Only with Firefox - Get Firefox Now" button
4. Install Firefox and start it up
5. Observe that we see the Return to AMO flow with the correct add-on that we were trying to install in the other browser

Shane mentioned there are some scripts that we can drop to add attribution to a Nightly build, and then we can start that Nightly and observe that the attribution got picked up from the build, and therefore we see the Return to AMO UI. There are also ways to test that the attribution gets set from AMO into a build.

Testing these two pieces separately may be enough to get GREEN sign-off from QA if we don't have a way to end-to-end test the entire flow.

Shane and David, feel free to add any other context you have to this bug.
NI to pmac, whom we spoke to about how attribution works. (Paul, this supersedes my email.) If there's already a way to test attribution end-to-end (from initiating installation from a mozilla property to verifying the attribution is there in Nightly), we could use that here.

If there isn't already a way to do that, then recommendations are welcome. We're looking to land this in 65 and QA is blocked on this.
Flags: needinfo?(pmac)
Depends on: 1511104
If all that is required on the www.m.o side is for query params to exist on the download page, then you should just manually be able to add them to the download URL. e.g.

https://www.mozilla.org/en-US/firefox/download/thanks/?utm_source=return-to-amo

I know that doesn't test the whole flow, but that should test that the data from the URL makes it into the installed Firefox. If the AMO button links directly to that URL then no further website changes are necessary. If you'll be linking to /firefox/new/ instead (which I do recommend btw. I feel it's a jarring UX to land on a new website and immediately start downloading) then a full test will need to wait until we pass the parameters over to the /thanks/ page.

If you just need to test windows however, it should work now. It's the Mac attribution that doesn't fully work yet.
Flags: needinfo?(pmac)
Attached file attibutionScratchpad (obsolete) —
scratchpad script to output attribution codes to console, both windows and osx.
Attached file showattrs
osx command line script to dump the attributes for any given downloaded app
Outlines how to test a new download for attribution data, requires other scripts attached.
Attached file attributionScratchpad
Attachment #9028738 - Attachment is obsolete: true
Attached file attributionNightlyTest.txt (obsolete) —
Instructions to test nightly on osx and see the attribution UI in about:welcome.
Depends on: 1511156
adds windows test
Attachment #9028750 - Attachment is obsolete: true
Depends on: 1511165
Iteration: --- → 66.1 - Dec 10-23
Priority: -- → P2
Iteration: 66.1 - Dec 10-23 → 66.2 - Dec 24 - Jan 6
Assignee: nobody → andrei.br92
Iteration: 66.2 - Dec 24 - Jan 6 → 66.3 - Jan 7 - 20
Priority: P2 → P1

I tried downloading FF65 Beta channel via https://download.mozilla.org/?product=firefox-beta-latest-ssl&os=osx&lang=en-US&utm_campaign=non-fx-button&utm_content=%7BEF522540-89F5-46b9-B6FE-1829E2B572C6%7D&utm_medium=referral&utm_source=addons.mozilla.org

And then after unDMG-ing the archive and running ./showattrs.sh /Applications/Firefox65.0b9.app, I get the following:

$ ./showattrs.sh /Applications/Firefox65.0b9.app
D8405CA2-AF9D-4326-85DD-F6845A6ADA5F|568748566.0|com.google.Chrome|Chrome|https://download-installer.cdn.mozilla.net/pub/firefox/releases/65.0b9/mac/en-US/Firefox%2065.0b9.dmg|||0|||

But after running the scratchpad code in the Browser context, I get the following output in my Browser Console log:

*** failed getting attribution data  Object {  }    Scratchpad/1:16:7
*** addon is null    Scratchpad/1:28:3 

Any ideas what I'm doing wrong? I'm using Beta 65 and not Nightly, but wasn't sure if I still needed to follow the xattr steps at https://bug1511071.bmoattachments.org/attachment.cgi?id=9028763

(I assume that you should be testing with 66, based on the changes landed in bug 1511165.)

Severity: normal → enhancement

I am unable to get the steps working on the latest macOS (Mojave) or on my older laptop with High Sierra using the following steps: https://gist.github.com/pdehaan/72ea2ccf30678c1a7ee34dfc1ffeb034

Has anybody else gotten this working and can see where my steps are wrong or why I'm not seeing the expected output in the Scratchpad.

Or is it possible to set up some dev/stage version of AMO+downloads that passes the correct utm_ parameters and attribution values so we don't need to copy over attribution values from Release to Nightly and fake a lot of the values?

Iteration: 66.3 - Jan 7 - 20 → 66.4 - Jan 21 - 27
Assignee: andrei.br92 → tspurway
Iteration: 66.4 - Jan 21 - 27 → 67.1 - Jan 28 - Feb 10
Iteration: 67.1 - Jan 28 - Feb 10 → 67.2 - Feb 11 - 24

I have a WIP E2E test plan at https://github.com/pdehaan/return-to-amo-e2e-test which covers downloading Firefox Nightly (v67) and using mixedpuppy's https://pastebin.com/kWZ3DhdZ script to copy attributes and then launch Firefox Nightly. I'm not seeing the about:welcome page open in the new profile, but if I open about:welcome in a new tab it will prompt me to install the SearchPreview add-on. :+1:

We've had a few discussions with Andrei who tried to help get the end to end testing working on Windows, but to no success, the steps we used were the ones (that apply) in https://gist.github.com/piatra/f0a2b9f9f700cbe19afd2ebf1561e6ee and https://bug1511071.bmoattachments.org/attachment.cgi?id=9028763 for which we also tried using the encoded URL instead of the one shown in the steps but this did not work either

Status: NEW → RESOLVED
Closed: 9 months ago
Resolution: --- → FIXED
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Bug 1525076 prevents RTAMO from working with MacOS, which is not blocking.

Only Windows QA paths apply at this time.

See Also: → 1525076

closing this as i believe we've done what we can :-)

Status: REOPENED → RESOLVED
Closed: 9 months ago9 months ago
Resolution: --- → FIXED
Component: Activity Streams: Newtab → Messaging System
You need to log in before you can comment on or make changes to this bug.