1129079 - Evaluate "The Intern" As Test Runner

Status: RESOLVED INVALID

(www.mozilla.org :: Project Tracking, defect)

Description

[Ben (:bensternthal)]

We would like to evaluate the intern as a possible integrated test runner.

This test should include ~2 of the existing python tests so we can see how this works in the 'real world'.

Comment 1

[Alex Gibson [:agibson]]

As mentioned in our previous meeting, I have managed to get the intern running locally on bedrock following kuma's example.

It's worth noting here that there is an upstream Selenium bug [1] effecting Firefox 35 and greater, which is a blocker to running the Intern (at least locally):

[1] https://code.google.com/p/selenium/issues/detail?id=8387

Hopefully this should be fixed soon.

Comment 2

[Alex Gibson [:agibson]]

I have a working branch I'm using for evaluation here: https://github.com/alexgibson/bedrock/commits/blame-the-intern

So far I have it running locally and successfully passing a test on the home page.

I'm still a bit slowed down by the fact I need to re-download Firefox 34 each time I play with the branch, just to get Selenium to work. Hopefully the bug in Comment 1 will be fixed soon

Comment 3

[Alex Gibson [:agibson]]

I seem to be making decent progress finding my way and writing various tests for the homepage. Once I have enough together, I may need to seek Matt's help in how to take this further (i.e. past just running tests locally).

Comment 4

[Alex Gibson [:agibson]]

Selenium bug has been merged into this one: https://code.google.com/p/selenium/issues/detail?id=8390

Looks like pressure is mounting for a fix

Comment 5

[Matt Brandt [:mbrandt]]

The work looks like it's coming along amazing -- dare I say -- smooth. Alex, would it be helpful if I invested some time into how it might look to refactor towards a page object model analogue? Something like, https://github.com/angular/protractor/blob/master/docs/page-objects.md.

Comment 6

[Alex Gibson [:agibson]]

(In reply to Matt Brandt [:mbrandt] from comment #5)
> The work looks like it's coming along amazing -- dare I say -- smooth. Alex,
> would it be helpful if I invested some time into how it might look to
> refactor towards a page object model analogue? Something like,
> https://github.com/angular/protractor/blob/master/docs/page-objects.md.

This sounds like a great suggestion - thanks Matt. I think this approach makes a lot of sense when it comes to writing generic test for multiple pages, and would also save time for those writing said tests +1

I've pushed this to a branch on the mozilla repo if you need to get at the code: 

https://github.com/mozilla/bedrock/tree/blame-the-intern

I'm hoping to pick this up again in the next week or so and finish writing a few more tests for the homepage. It would be good to also add a few tests for another bedrock page, so we can make use of page objects you suggest here.

Comment 7

[Alex Gibson [:agibson]]

Selenium v2.45 is out, which fixes the bug in Comment 1. Seems to be working well again in Firefox 36

http://selenium-release.storage.googleapis.com/index.html?path=2.45/

Comment 8

[Alex Gibson [:agibson]]

Noting here that running the latest version of Selenium together with the Intern against bedrock locally, crashed my entire machine twice yesterday :(

Comment 9

[Alex Gibson [:agibson]]

Matt, just wondering if you still have some availability to work on this. I'd like to try and wrap this evaluation up in the next week or so, before the end of quarter craziness kicks in.

If you don't have any capacity atm, no worries. I can just take what we have here and open it up for the rest of the bedrock team to try it out.

Thanks!

Comment 10

[Matt Brandt [:mbrandt]]

Alex, comment 8, ugg :( Can you give me through the end of next week to work on refactoring a few tests to use the page object model? I completely understand the desire to open this up to the rest of the team sooner-than-later. End-of-quarter cray-cray has already kicked in :-)

Comment 11

[Alex Gibson [:agibson]]

(In reply to Matt Brandt [:mbrandt] from comment #10)
> Alex, comment 8, ugg :( Can you give me through the end of next week to work
> on refactoring a few tests to use the page object model? I completely
> understand the desire to open this up to the rest of the team
> sooner-than-later. End-of-quarter cray-cray has already kicked in :-)

Sounds fine to me Matt, thanks! 

Yeah I still have some reservations about the reliability and overall stability of the Selenium JS bindings with Firefox. It makes me feel uneasy, but I think we should still open this up to the rest of the bedrock team to get their opinions before making a final evaluation.

Comment 12

[Alex Gibson [:agibson]]

I spoke with Matt yesterday about my thoughts on the stability and reliability of using the Intern, and we both agree that it is probably best to not invest much more time in this and instead make some conclusions here, and let other people try it out if still interested [1].

My overall experience has been somewhat mixed. While it seems like a nice enough tool to allow folks to write tests in JS and automate the browser, it's current reliability and stability leave me with some serious concerns about moving to it:

1.) The Selenium Webdriver JS bindings broke in Firefox 35 [2]. It took over an entire release cycle, until after Firefox 36 was released, for the issue to get fixed. Having this happen for such a long period would have been a serious blocker for us.

2.) Once the issue was fixed, using the latest release I have seen it completely crash my machine on more than one occasion when trying to exit the process and running just a small number of tests. =/

From a technical perspective, when the Intern works it seems pretty capable and could certainly replace most of the mcom tests. One area that may prove difficult in moving to JS however, is testing for valid response codes in links. I'm quite not sure how we would do it in JS as easily as other languages. Because JS only has XHR requests, and not all external links can or will support CORS, testing external link responses becomes more tricky. This is pretty vital for bedrock, especially when it comes to download buttons. It may require some sort of server-side proxy, but then it might be easier just to keep those tests in Python, where it is much more trivial.

If anyone would like to try the Intern out, the branch is below [1]. I would be happy to assist anyone in getting it up and running. If people would like to avoid the possibility of crashing their machine however, I totally understand :P

I'm happy to have a meeting to discuss this evaluation further if people think it may be desirable, else I think we can close this for now. Perhaps in the future it may be more stable and give me more confidence, but for now sadly I cannot really recommend moving to it :(

[1] https://github.com/mozilla/bedrock/tree/blame-the-intern
[2] https://code.google.com/p/selenium/issues/detail?id=8390

Comment 13

[Alex Gibson [:agibson]]

Closing this, as we're going to move on to look at CasperJS, which hopefully may be a more reliable and more suitable alternative.

Comment 14

[Matt Brandt [:mbrandt]]

(In reply to Alex Gibson [:agibson] from comment #13)
> Closing this, as we're going to move on to look at CasperJS, which hopefully
> may be a more reliable and more suitable alternative.

Alex, a genuine thank you for the amount of work and insight you put into this experiment. I learned quite a bit.

Comment 15

[Alex Gibson [:agibson]]

Thanks Matt, will keep you in the loop on how we get on.

This is all relatively new to me too, so I certainly learned a lot also!