Closed Bug 848467 Opened 9 years ago Closed 9 years ago

[research] look into liveservertestcase selenium tests

Categories

(Input Graveyard :: Code Quality, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: willkg, Assigned: mythmon)

References

Details

(Whiteboard: u=dev c=general p=3 s=2013.11)

Currently, QA writes a bunch of selenium tests and they exist in a separate repository from the code base. It'd be better if that wasn't the case for a variety of reasons.

This bug is for looking into writing test scaffolding that allow us to have the selenium tests with our other tests and making that work with developer instances, jenkins, and all that.

Stephen was thinking we could grab the QA test suite and start with that. He also said Peter has done this with l10n-dashboard already, so he could be a good resource.

https://docs.djangoproject.com/en/1.4/topics/testing/#django.test.LiveServerTestCase
This sounds great; thanks -- once we have a prod instance of this up and running, I'll happily delete https://github.com/mozilla/input-tests and the associated runs.

Let me/the team (mozwebqa@mozilla.org), #mozwebqa know if you need any help w/WebDriver.
I am interested in getting involved in this effort. I will look for you in IRC to discuss.
Putting these in my queue for this quarter.
Assignee: nobody → willkg
Whiteboard: u=dev c=general p= s=input.2013q2
Making this a P2 so it gets done this quarter.

Right now this is a research bug and if things look good, this could cover building the infrastructure/documentation, too.


Bob: I was hoping to look at it this week, but didn't get that far. Ricky is interested in this, too, for Kitsune. I'm passing this off to him.
Assignee: willkg → rrosario
Priority: -- → P2
Thanks willkg. I was hoping to devote some time to this in the next few weeks. Has anything been started, or is this a research project from the ground up? r1cky, please feel free to ping me if and when you plan to get started on this and I'd be happy to help out (if I haven't started on it already).
Letting this go so :mythmon can run with it. Making it 3pts at this point, but it's pretty unknown.
Assignee: rrosario → nobody
Whiteboard: u=dev c=general p= s=input.2013q2 → u=dev c=general p=3 s=2013.11
I'll take a look at this later in the sprint.
Assignee: nobody → mcooper
mythmon, please ping me in #mozwebqa when you get started on this as I'd like to help/be involved. I've been trying to squeeze out some time for it, but gaia-ui-tests keep getting in the way. :(
Working on this now.
Status: NEW → ASSIGNED
Right now, there are two kinds of tests (maybe more, but lets just look at these two:

* Unit/Integration tests in the repository that we run locally on our laptops, and in our CI builds on ephemeral test environments.
* Selenium jobs that are run by QA that get run across a grid of systems, across many browsers, across the three permanent environments (dev, stage, and prod).

The main difference, besides that one uses selenium, is that the QA tests run against dev, stage, and prod, where as the unit/integration tests run against ephemeral instances started specifically for tests on laptops or Jenkins.

I think this separation is good. I think having two test suites is bad. I propose that we make a single test suite that can be run either against test instances or against real environments. Developers can use this by running it on laptops and on ci.mozilla.org, and QA can use this against dev, stage, and prod.

I'm not sure what the test runner for this looks like yet, but I think the idea is sound.

:stephend, :willkg, :bsilverberg: What do you think about all this?
(In reply to Mike Cooper [:mythmon] from comment #10)
> Right now, there are two kinds of tests (maybe more, but lets just look at
> these two:
> 
> * Unit/Integration tests in the repository that we run locally on our
> laptops, and in our CI builds on ephemeral test environments.
> * Selenium jobs that are run by QA that get run across a grid of systems,
> across many browsers, across the three permanent environments (dev, stage,
> and prod).
> 
> The main difference, besides that one uses selenium, is that the QA tests
> run against dev, stage, and prod, where as the unit/integration tests run
> against ephemeral instances started specifically for tests on laptops or
> Jenkins.
> 
> I think this separation is good. I think having two test suites is bad. I
> propose that we make a single test suite that can be run either against test
> instances or against real environments. Developers can use this by running
> it on laptops and on ci.mozilla.org, and QA can use this against dev, stage,
> and prod.
> 
> I'm not sure what the test runner for this looks like yet, but I think the
> idea is sound.
> 
> :stephend, :willkg, :bsilverberg: What do you think about all this?

That's what this bug is for: figuring out the infrastructure and scaffolding for fjord to support this and whether LiveServerTestCase makes that easier/possible.
(In reply to Mike Cooper [:mythmon] from comment #10)
> 
> I think this separation is good. I think having two test suites is bad. I
> propose that we make a single test suite that can be run either against test
> instances or against real environments. Developers can use this by running
> it on laptops and on ci.mozilla.org, and QA can use this against dev, stage,
> and prod.
> 
> I'm not sure what the test runner for this looks like yet, but I think the
> idea is sound.
> 
> :stephend, :willkg, :bsilverberg: What do you think about all this?

When you talk about a single test suite, does that mean that all unit/integration tests *and* all functional (Selenium) tests will get executed on every run? Is that what the goal of this bug is? Or is it to find a way to allow developers to run the functional tests, as needed, in the same environments as they are currently running unit tests?

If the latter it might make sense to start with the framework that Web QA uses for functional tests (pytest-mozwebqa) and see if it will work, rather than looking for an entirely different solution from the outset.
Depends on: 879548
Ok, I was off in the wrong rabbit hole in my last comment. This bug is to just research Selenium and live server test cases. The rest of the work (writing tests, collaborating with QA, etc) is out of scope, and can wait until later.

What I have figured out so far:

On my laptop, getting selenium tests working was really simple. I added selenium to vendor-local, and made sure I had Firefox running. Then I added a new TestCase class that extended LiveServerTestCase, and wrote two tests with it. This was done on this branch: https://github.com/mythmon/fjord/commits/live-server-848467 .  If you check out that branch, and have Firefox installed, it should just work. I was pleasantly surprised that for day to day tests, LiveServerTestCase made working with Selenium very easy, and only added a couple seconds to my test runs.

On Jenkins is another story. First, Firefox isn't installed, nor any other browser. Second, Firefox can't run headless. It would be wasteful to run an entire X session on CI. Instead we can use a tool called xvfb, which provides a virtual X frame buffer for applications, including Firefox, to run. Towards this end, I have filed bug #879548.

Note: xvfb is useful for local testing as well. For me, having Firefox windows flash open and closed during testing is annoying, and xvfb helps that.

My conclusion is that if we can get the dependencies installed on CI, this is a pretty easy way we can extend our tests, and we should do it. I will follow up as I learn more from IT.
The way forward is clear to me. For full implementation, we need bug #879548 to land, but the research is done here.

I landed code changes that do some selenium testing here: https://github.com/mozilla/fjord/commit/357ec16cace33398e8bc68c57de28309ce0569e3
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: Input → Input Graveyard
You need to log in before you can comment on or make changes to this bug.