718937 - Selenium automated tests for free add-on/app installation flows

Status: RESOLVED WONTFIX

(addons.mozilla.org Graveyard :: Code Quality, defect)

Description

[Christopher Van Wiemeersch [:cvan]]

This may already be done somewhere, but we should at a minimum get tests that check the installation flows for the following:

* free add-on
* free app
* premium add-on
* premium app

We should also try to make these tests very visible, such that when CI fails everyone on AMO (in #amo) knows ASAP.

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

Thanks for filing this, cvan; is this something anyone from AMO-dev would be interesting in partnering on?

Comment 2

[Christopher Van Wiemeersch [:cvan]]

(In reply to Stephen Donner [:stephend] from comment #1)
> Thanks for filing this, cvan; is this something anyone from AMO-dev would be
> interesting in partnering on?

I'm game - Selenium folks, holler at me and we can make something happen.

Comment 3

[Zac C (:zac)]

Suggested solution:

Copy the Mcom firefox download tests framework using packages requests and BeautifulSoup (not Selenium).

1. Load the details page of an internal Mozilla addon and extract the download link.
2. Perform a HEAD request, follow the redirects and assert that the Status is 200 and the Content-Type is application/x-xpinstall.

Anything beyond this is out of the scope of the website.

The 4 test cases suggested are restrained by not being able to use the public's addons for testing against. Start with just one Mozilla/internal add-on for now.

Test could be expanded to cover install links from other pages (eg Homepage) or using different user-agents to spoof install validity.

Comment 4

[Christopher Van Wiemeersch [:cvan]]

I like the idea of that test. If there was a regression in the JS or CSS that causes the add-on/app installation trigger to not be initialized - is it possible to catch that case? (In the case of a purchased add-on/app a modal would appear on click of an "Install" button, so that should be easier to test I'd guess.)

Comment 5

[Zac C (:zac)]

Is it website side javascript that causes the add-on or app to install? It appears to be all in the browser window chrome to me - once the download is initiated it's out of Selenium's hands.

The link to the addon is a straight download link with no js masking that I can see.

Unless we want to check clicking against addons that are incompatible with the current brwoser version - that is a different test case.

Comment 6

[Christopher Van Wiemeersch [:cvan]]

(In reply to Zac C from comment #5)
> Is it website side javascript that causes the add-on or app to install? It
> appears to be all in the browser window chrome to me - once the download is
> initiated it's out of Selenium's hands.

That's unfortunate, yeah. Can Selenium listen for an event? If so, that might work. But anyway here's a breakdown of what I think we can test:

* Free Add-on: there's an installation doorhanger upon click -> test that event was fired
* Paid Add-on (Future): modal popups -> that that modal appears (and eventually we'll want to test that payment can be completed)
* Free App (Firefox 13+): there's an installation doorhanger upon click -> test that event was fired
* Free App (Firefox 12 and earlier & non-Firefox): a JavaScript modal appears upon click -> that that modal appears
* Paid Add-on (Future): modal popups -> modal appears and can be completed

To keep things sane, we should probably leave the purchase tests for a separate bug if you'd like.

> The link to the addon is a straight download link with no js masking that I
> can see.

For non-Firefox/Thunderbird/Seamonkey user agents this is true. But for those browsers we do some fancy JS (https://github.com/mozilla/zamboni/blob/master/media/js/zamboni/buttons.js#L481) to call `InstallTrigger.install`.

As a fallback you'll always be able to download the XPI (in Chrome or if JS is disabled). The test should check that.

Historically, there have been several cases where borked styles or JS would cause this JS to not get initialized and the buttons were not initiating add-on installation.

Comment 7

[Zac C (:zac)]

If that doorhanger changes something in the DOM then we can definitely read and check that.

The modal popup we can check easily enough as that's HTML but does that eventually trigger the doorhanger too?

Comment 8

[Zac C (:zac)]

Chris, Bebe and I have spent some time looking at this and we don't think we can passively catch the event. We could only write js to attach our own eventhandler to it which would write to the DOM from which WebDriver can easily read.

But this interferes with the page a bit and isn't the favoured solution.

Comment 9

[Marlena Compton [:marlenac]]

We're actually working on install tests and are taking a divide and conquer approach since selenium won't currently allow us to manipulate browser chrome.  The two pieces are 1) selenium navigates to the correct page and finds the "install" button 2) mozmill checks that clicking the install button opens the browser's install dialog

The case of installing a paid addon isn't one we've looked at yet.  The mozmill guys have committed to 3 of our tests for this quarter, I can talk with them about working on this test for the next quarter.

As for the apps ones...it'll be easier if y'all have stable pages up for the marketplace :oP  I'm sure we'll get to these in the next quarter as well.

Comment 10

[Wil Clouser [:clouserw]]

is this bug still being worked on?

Comment 11

[Florin Strugariu [:Bebe]]

I don't think so... 
Selenium can't automate the install flow. Only the web page content.


This is partially solved in:

https://github.com/mozilla/Addon-Tests/blob/master/tests/desktop/test_installs.py

We are automating this until pressing the install button. After that is out of Selenium's reach.

Comment 12

[Wil Clouser [:clouserw]]

ok.  closing based on comment 11. thanks.