Install Firefox and xvfb on ci.mozilla.org for Selenium tests?

RESOLVED WONTFIX

Status

RESOLVED WONTFIX
6 years ago
2 years ago

People

(Reporter: mythmon, Unassigned)

Tracking

Details

(Reporter)

Description

6 years ago
For bug #848467 I am researching integrating Selenium based tests into the test suite for Input. I've got this working for developer laptops, but on Jenkins the tests fail because they try and use Firefox to run the tests in, and Firefox isn't installed. Not surprising.

Based on [this blog post][1], I believe that to get this working we could install two packages on the jenkins server: Firefox and xvfb. Since Firefox cannot be run headless, xvfb provides a virtual framebuffer for Firefox to run inside. I believe with these two tools, we could do selenium based testing as part of our normal Jenkins jobs, which would be a boon for our CI workflow.


[1] http://www.labelmedia.co.uk/blog/setting-up-selenium-server-on-a-headless-jenkins-ci-build-machine.html
(Reporter)

Comment 1

6 years ago
Any news on this? If this isn't possible, or is going to take a long time, that would be good to know so we can look into other alternatives.
Assignee: server-ops-devservices → server-ops-webops
Component: Server Operations: Developer Services → Server Operations: Web Operations
QA Contact: shyam → nmaul
Mike, sorry...this bug was in the wrong queue, I've moved it. I'll try to get someone to give you a time period.
Talk to :stephend about how to possibly use SauceLabs for this, I know I don't have time to even look into this until next month
(Reporter)

Comment 4

6 years ago
Sorry this was in the wrong queue, I wasn't sure where it should be.

I'm uncomfortable running these on saucelabs for two reasons.

1) I want to make this as uniform as possible between developer instances (laptops) and CI tests. I don't think we can do this if we go with saucelabs.

2) I want to keep this available to non-employee contributors, and I don't think we can do that with SauceLabs.

The point of this is not to just run selenium tests. We already have that for SUMO thanks to QA (thanks!). The point is to add a new kind of tests that tighten the feedback loop while developers work on things, so we can find problems sooner.

I understand not having time for this. I'll talk to the team and see if we can wait, or if we need to pursue alternate solutions in the mean time.
We have a Selenium hub with two nodes running in PHX1 (see bug 727353) that could be used here. If you use Sauce Labs then you'll most likely need to configure Sauce Connect [1] so access to the local instance can be granted.

[1] https://saucelabs.com/docs/connect
(Reporter)

Comment 6

6 years ago
The goal here is two fold. First, I want Selenium tests to be run as part of the everyday test suite that developers run on their laptops. Second, I want the tests that run on Jenkins to be as similar as possible to the test that run everyday on my laptop. Because of this, I don't feel that things like Selenium servers and Sauce Labs, both of which would likely require authentication, are appropriate since we a) can no longer run tests purely on our development systems (laptops), or b) Jenkins and our local tests will diverge.

I think there is still a place for more comprehensive Selenium testing, such as Sauce Labs, but that is not what I want in this case.
If Selenium tests are running locally, there's often no need to use the Java server. That said, I would still recommend that you use the remote server and keep this separate from the Jenkins instance.

Disclaimer: I'm not in a position to help with the request in this bug, I'm just familiar with Selenium. :)
(Reporter)

Comment 8

6 years ago
:solarce, friendly ping. It is next month now, do you have spare cycles to look into this, or is your schedule still super busy?
Component: Server Operations: Web Operations → WebOps: IT-Managed Tools
Product: mozilla.org → Infrastructure & Operations
(In reply to Mike Cooper [:mythmon] from comment #8)
> :solarce, friendly ping. It is next month now, do you have spare cycles to
> look into this, or is your schedule still super busy?

Basically Jenkins is in maintenance mode as this point while we develop a roadmap to outsource our CI, so we're trying to avoid any additional major packages/services that we might need to provide ongoing support for. 

In this case it seems like getting Firefox versions setup would require a lot of maintenance as the Firefox rpms for RHEL are still on 17.0.7

If we gave you sudo would you be willing to manage installing and managing Firefox versions, and troubleshooting things related to this kind of Selenium testing?

Updated

6 years ago
Flags: needinfo?(mcooper)
(Reporter)

Comment 10

6 years ago
I'm sad to hear that Jenkins isn't getting any more attention. I'm not a sysadmin (anymore), and I don't feel comfortable keeping Firefox and other packages up to date on a RHEL box.

In regards to outsourcing CI, how far along is this? Is this something that will be usable this quarter? next? Sometime in 2014? CI is an important part of our development process, and if we can't get the kind of environment we need in the current system, and the next system is too far out, we may need to look into alternatives.
Flags: needinfo?(mcooper)
Brandon, can you kindly comment on this issue as to the time-line for alternate products?

With eternal gratitude
jd :)
Flags: needinfo?(bburton)
Our fine Labs webops folks have built an interesting Jenkins self-service mechanism using the Labs Eucalyptus cloud, which let's you get a dedicated Jenkins instance you can customize at the OS level.

I think that this would be the best option for you to look at at this time
Assignee: server-ops-webops → webops-labs
Component: WebOps: IT-Managed Tools → WebOps: Labs
Flags: needinfo?(bburton)
We gave up waiting on finding out what the deal is here and got all our stuff running on Travis-CI. SUMO is no longer using ci.mozilla.org.

I don't think we need to do anything here anymore.

Mike: Can you think of anything or can we just close this out?
Flags: needinfo?(mcooper)
(Reporter)

Comment 14

5 years ago
SUMO doesn't need this since we switched to Travis, and I don't know of any other teams that want this (but they might exist). I think this should be RESOLVED WONTFIX, since ci.mozilla.org isn't going to get this feature.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Flags: needinfo?(mcooper)
Resolution: --- → WONTFIX
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.