686727 - Run JS tests in latest Firefox

Status: RESOLVED WONTFIX

(addons.mozilla.org Graveyard :: Code Quality, defect)

Description

[Kumar McMillan [:kumar]]

In CI, let's run JavaScript tests in the latest version of Firefox.  Maybe we need a cron/upgrade script to manage this?

https://jenkins.mozilla.org/job/amo-master-js/

Comment 1

[Wil Clouser [:clouserw]]

I think the plan here is to start using a QA managed box for this since they already have scripts for this stuff in place.  Stephend:  This is what we talked about where the browser just needs to run 24/7 and have a single URL loaded - it'll automatically check for jobs every 30s or so.  Do you want to talk on IRC about next steps for this or do you already have a server planned out?

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Wil Clouser [:clouserw] from comment #1)
> I think the plan here is to start using a QA managed box for this since they
> already have scripts for this stuff in place.  Stephend:  This is what we
> talked about where the browser just needs to run 24/7 and have a single URL
> loaded - it'll automatically check for jobs every 30s or so.  Do you want to
> talk on IRC about next steps for this or do you already have a server
> planned out?

Wil/Kumar: if all you need/want is the latest Firefox build, this is pretty much ready to go.  If you give us the job information, we can transplant it (or you guys can just feel free to add a job -- we can take the "constant-running browser tests" hit quite easily with our current CI, and we can always also expand capacity pretty easily, too.

Comment 3

[Wil Clouser [:clouserw]]

The only thing you need to do is open this URL in a browser:  http://jstestnet.farmdev.com/work/


Right now we're running firefox 3.6, so anything newer than that is already an improvement.  Does "latest Firefox build" mean nightlies?  If so, that's fine - just have that url ^^ come up when it restarts.

What else do you need from us?

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Wil Clouser [:clouserw] from comment #3)
> The only thing you need to do is open this URL in a browser: 
> http://jstestnet.farmdev.com/work/
> 
> 
> Right now we're running firefox 3.6, so anything newer than that is already
> an improvement.  Does "latest Firefox build" mean nightlies?  If so, that's
> fine - just have that url ^^ come up when it restarts.
> 
> What else do you need from us?

1) How do we report pass/fail, from this test?  It's not working for me right now, so I can't tell: "Status: [worker=117] No commands from server. Checking again in 5 seconds."
2) We can run on either/both Aurora/Beta builds -- as well as shipping/stable, of course.
3) Looking at your CI output, you use nose to run the tests -- can we just get that job copied over to our CI?  I'm still not clear on how you actually run tests, apart from "./scripts/run_jstests.sh jstests"

I've cc:ed a few WebQA folk, in hopes that one of them will kindly work with you to figure this out :-)

Comment 5

[Wil Clouser [:clouserw]]

> 1) How do we report pass/fail, from this test?  It's not working for me
> right now, so I can't tell: "Status: [worker=117] No commands from server.
> Checking again in 5 seconds."

You don't.  The browser is merely working and reporting back to jstestnet.  It's a dumb slave.

> 2) We can run on either/both Aurora/Beta builds -- as well as
> shipping/stable, of course.

Aurora sounds good for now.  We just need 1 browser, we can work on others at some other time.

> 3) Looking at your CI output, you use nose to run the tests -- can we just
> get that job copied over to our CI?  I'm still not clear on how you actually
> run tests, apart from "./scripts/run_jstests.sh jstests"

We could, but I thought the long term plan was to shut your CI down and move to webdev's jenkins.  In that case, why would you want to run these tests on your end?  Ours is already set up to report the results into our IRC channels.

Comment 6

[Wil Clouser [:clouserw]]

Any news on this?  We can talk on IRC if it doesn't make sense.

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Wil Clouser [:clouserw] from comment #6)
> Any news on this?  We can talk on IRC if it doesn't make sense.

Wil, Potch, and I worked through this last Friday (iirc), and are now awaiting bug 692912; setting the dependency.

Comment 8

[Wil Clouser [:clouserw]]

We have a new slave up running the latest Firefox stable (10, right now) but it won't auto-update.  Stephen: any news?

Comment 9

[Kumar McMillan [:kumar]]

I chatted with stephend. They do not auto-update their browsers in the grid either. They upgrade manually.

Comment 10

[Wil Clouser [:clouserw]]

(In reply to Kumar McMillan [:kumar] from comment #9)
> I chatted with stephend. They do not auto-update their browsers in the grid
> either. They upgrade manually.

Will they add ours to the list of things they upgrade manually? :D

Comment 11

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Wil Clouser [:clouserw] from comment #10)
> (In reply to Kumar McMillan [:kumar] from comment #9)
> > I chatted with stephend. They do not auto-update their browsers in the grid
> > either. They upgrade manually.
> 
> Will they add ours to the list of things they upgrade manually? :D

Only for the meantime; we really don't want a resident browser left open -- we have browsers firing up, running tests, and shutting down all the time, so there's a good possibility that we'd accidentally close it.  I understand you also have a VM that's running these tests, so it doesn't feel like there's a need for two.

We'd be happy to brainstorm ways to auto-update browsers, as that definitely seems to be a shared problem.

Comment 12

[Wil Clouser [:clouserw]]

We don't use the jstestnet anymore.  Pretty sure headless webkit is what all the cool kids are using these days.