Run JS tests in latest Firefox

RESOLVED WONTFIX

Status

addons.mozilla.org Graveyard
Code Quality
RESOLVED WONTFIX
7 years ago
3 years ago

People

(Reporter: kumar, Assigned: clouserw)

Tracking

Details

In CI, let's run JavaScript tests in the latest version of Firefox.  Maybe we need a cron/upgrade script to manage this?

https://jenkins.mozilla.org/job/amo-master-js/
Assignee: nobody → clouserw
(Assignee)

Comment 1

7 years ago
I think the plan here is to start using a QA managed box for this since they already have scripts for this stuff in place.  Stephend:  This is what we talked about where the browser just needs to run 24/7 and have a single URL loaded - it'll automatically check for jobs every 30s or so.  Do you want to talk on IRC about next steps for this or do you already have a server planned out?
(In reply to Wil Clouser [:clouserw] from comment #1)
> I think the plan here is to start using a QA managed box for this since they
> already have scripts for this stuff in place.  Stephend:  This is what we
> talked about where the browser just needs to run 24/7 and have a single URL
> loaded - it'll automatically check for jobs every 30s or so.  Do you want to
> talk on IRC about next steps for this or do you already have a server
> planned out?

Wil/Kumar: if all you need/want is the latest Firefox build, this is pretty much ready to go.  If you give us the job information, we can transplant it (or you guys can just feel free to add a job -- we can take the "constant-running browser tests" hit quite easily with our current CI, and we can always also expand capacity pretty easily, too.
(Assignee)

Comment 3

7 years ago
The only thing you need to do is open this URL in a browser:  http://jstestnet.farmdev.com/work/


Right now we're running firefox 3.6, so anything newer than that is already an improvement.  Does "latest Firefox build" mean nightlies?  If so, that's fine - just have that url ^^ come up when it restarts.

What else do you need from us?
(In reply to Wil Clouser [:clouserw] from comment #3)
> The only thing you need to do is open this URL in a browser: 
> http://jstestnet.farmdev.com/work/
> 
> 
> Right now we're running firefox 3.6, so anything newer than that is already
> an improvement.  Does "latest Firefox build" mean nightlies?  If so, that's
> fine - just have that url ^^ come up when it restarts.
> 
> What else do you need from us?

1) How do we report pass/fail, from this test?  It's not working for me right now, so I can't tell: "Status: [worker=117] No commands from server. Checking again in 5 seconds."
2) We can run on either/both Aurora/Beta builds -- as well as shipping/stable, of course.
3) Looking at your CI output, you use nose to run the tests -- can we just get that job copied over to our CI?  I'm still not clear on how you actually run tests, apart from "./scripts/run_jstests.sh jstests"

I've cc:ed a few WebQA folk, in hopes that one of them will kindly work with you to figure this out :-)
(Assignee)

Comment 5

7 years ago
> 1) How do we report pass/fail, from this test?  It's not working for me
> right now, so I can't tell: "Status: [worker=117] No commands from server.
> Checking again in 5 seconds."

You don't.  The browser is merely working and reporting back to jstestnet.  It's a dumb slave.

> 2) We can run on either/both Aurora/Beta builds -- as well as
> shipping/stable, of course.

Aurora sounds good for now.  We just need 1 browser, we can work on others at some other time.

> 3) Looking at your CI output, you use nose to run the tests -- can we just
> get that job copied over to our CI?  I'm still not clear on how you actually
> run tests, apart from "./scripts/run_jstests.sh jstests"

We could, but I thought the long term plan was to shut your CI down and move to webdev's jenkins.  In that case, why would you want to run these tests on your end?  Ours is already set up to report the results into our IRC channels.
(Assignee)

Comment 6

7 years ago
Any news on this?  We can talk on IRC if it doesn't make sense.
(In reply to Wil Clouser [:clouserw] from comment #6)
> Any news on this?  We can talk on IRC if it doesn't make sense.

Wil, Potch, and I worked through this last Friday (iirc), and are now awaiting bug 692912; setting the dependency.
Depends on: 692912
(Assignee)

Comment 8

7 years ago
We have a new slave up running the latest Firefox stable (10, right now) but it won't auto-update.  Stephen: any news?
I chatted with stephend. They do not auto-update their browsers in the grid either. They upgrade manually.
(Assignee)

Comment 10

7 years ago
(In reply to Kumar McMillan [:kumar] from comment #9)
> I chatted with stephend. They do not auto-update their browsers in the grid
> either. They upgrade manually.

Will they add ours to the list of things they upgrade manually? :D
(In reply to Wil Clouser [:clouserw] from comment #10)
> (In reply to Kumar McMillan [:kumar] from comment #9)
> > I chatted with stephend. They do not auto-update their browsers in the grid
> > either. They upgrade manually.
> 
> Will they add ours to the list of things they upgrade manually? :D

Only for the meantime; we really don't want a resident browser left open -- we have browsers firing up, running tests, and shutting down all the time, so there's a good possibility that we'd accidentally close it.  I understand you also have a VM that's running these tests, so it doesn't feel like there's a need for two.

We'd be happy to brainstorm ways to auto-update browsers, as that definitely seems to be a shared problem.
(Assignee)

Comment 12

5 years ago
We don't use the jstestnet anymore.  Pretty sure headless webkit is what all the cool kids are using these days.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → WONTFIX
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.