Closed Bug 1068809 Opened 7 years ago Closed 6 years ago

Use Firefox instead of B2G Desktop in Gu tests

Categories

(Release Engineering :: General, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: julienw, Unassigned)

References

Details

Attachments

(2 files)

We're using B2G Desktop to run unit tests in TBPL for some time now, and it didn't bring any advantage, and rather some inconveniences.

Therefore we'd like to change this job and use Firefox Nightly instead of B2G Desktop, as we used to do in Travis, and as most developers do.

Things to take care:
* use a profile generated with the environment variables "DEBUG=1 DESKTOP=0"
* launch the test-agent using the URL "app://test-agent.gaiamobile.org" (simply as a parameter to the Firefox launching command line)

I think all the other things are the same.

See [1] for a reference. BTW I wonder if it's not possible to use [1] directly in the Gu build?

[1] https://github.com/mozilla-b2g/gaia/blob/master/bin/gaia-test
Blocks: 1030936
Blocks: 1043562
There are several things that need to happen here:

1 - the runner needs to be modified to work with Firefox.  (We don't use gaia-test because it doesn't produce TBPL-compatible output and doesn't handle crashes.)
2 - schedule the jobs against Firefox builds instead of b2gdesktop builds
3 - modify gaia-try to pick up these jobs (right now it only picks up things run against b2gdesktop) and modify it to use the latest Firefox in this context

needinfo'ing jhford for #3.

We also want/need to do this soon for Mulet...does it make sense to switch these jobs to Firefox now and then Mulet again soon-ish?
Flags: needinfo?(jhford)
(In reply to Jonathan Griffin (:jgriffin) from comment #1)
> There are several things that need to happen here:
> 
> 1 - the runner needs to be modified to work with Firefox.  (We don't use
> gaia-test because it doesn't produce TBPL-compatible output and doesn't
> handle crashes.)

The output is a separate process; gaia-test simply launches the server and firefox, but does not launch the tests. 
For output, I suppose we could use the same mocha reporter we already use for Gij.
About the crash handling, let's discuss later on IRC, I'd be glad if we can make TBPL use the same tools we use as developer.

But all this is for a separate bug, I'll file it later.

> 2 - schedule the jobs against Firefox builds instead of b2gdesktop builds
> 3 - modify gaia-try to pick up these jobs (right now it only picks up things
> run against b2gdesktop) and modify it to use the latest Firefox in this
> context
> 
> needinfo'ing jhford for #3.
> 
> We also want/need to do this soon for Mulet...does it make sense to switch
> these jobs to Firefox now and then Mulet again soon-ish?

As far as unit tests are concerned, we don't need Mulet and they can continue to run on Firefox.
For Gaia-Try, there are two parts involved:

1) find the latest Firefox build and include it in commits as a new JSON config file
2) configure buildbot to invoke Gu mozharness runs with the new JSON config files


I'm definately the right person for 1), but I'm not the right person for 2).

Chris, who from Releng can take on 2) above and when do you think it'll be ready to land?

We currently have:

* linux32.json
* linux64.json
* linux64-debug.json
* macosx64.json

It seems that we're only running Gu for Linux 64bit optimized builds.  If that's true, I'll add 'firefox-linux64.json' to the pushes.
Flags: needinfo?(jhford) → needinfo?(catlee)
(In reply to Julien Wajsberg [:julienw] from comment #2)
> As far as unit tests are concerned, we don't need Mulet and they can
> continue to run on Firefox.

Just a note:  using different Gecko products on Gaia-Try means that because of how the files are found and uploaded to FTP, the Gu tests could be completed against a different revision of Firefox/Mulet/B2G Desktop.  In the current system, the only way to ensure that all of the tests are run against the same version of Gecko is to use the same product.

This is not to say that it's impossible or that I'm unwilling to do it, just that I want to make sure you understand the consequences of using Firefox instead of B2G Desktop or Mulet for one test suite.
Gaia-Try is now creating the firefox-linux64.json file, here's an example:

http://hg.mozilla.org/integration/gaia-try/rev/7d918be5b246

The remaining part of this from the Gaia-Try perspective is changing Buildbot to pass firefox-linux64.json as the configuration file for Gu runs.
Here's an idea for the changes that need to be made to Gaia-Try to use firefox-linux64.json.  This does not change non-Gaia-Try runs because I don't know how that works.

I don't really have the setup to test this more fully than making sure 'python b2g_config.py' works and has the changes that I'd expect.  Those changes are:

http://pastebin.mozilla.org/6550215
(In reply to John Ford [:jhford] -- please use 'needinfo?' instead of a CC from comment #4)
> (In reply to Julien Wajsberg [:julienw] from comment #2)
> > As far as unit tests are concerned, we don't need Mulet and they can
> > continue to run on Firefox.
> 
> Just a note:  using different Gecko products on Gaia-Try means that because
> of how the files are found and uploaded to FTP, the Gu tests could be
> completed against a different revision of Firefox/Mulet/B2G Desktop.  In the
> current system, the only way to ensure that all of the tests are run against
> the same version of Gecko is to use the same product.
> 
> This is not to say that it's impossible or that I'm unwilling to do it, just
> that I want to make sure you understand the consequences of using Firefox
> instead of B2G Desktop or Mulet for one test suite.

Do you know when Mulet will be ready to be used to run tests? I don't want to make people work twice to do the same thing.

I don't think this is that important for unit tests, actually. It's important that Gij and Gip run on the same underlying engine, but for Gu I don't think it is.
I can get Gu running against Mulet next week on Linux; we don't have OSX Mulet builds running in buildbot yet, so that will take another week or two, most likely.
The 6 hardcode is less than ideal.  Instead of having to bump this each time we add a new platform to gaia-try, let's just set some upper bound that's highly unlikely to be reached.
Attachment #8492466 - Flags: review?(jlund)
Attachment #8492466 - Flags: review?(jlund) → review+
Comment on attachment 8492466 [details] [diff] [review]
Allow an essentially arbitrary number of files in Gaia-Try changes

https://hg.mozilla.org/build/mozharness/rev/3ff10908c0f2
Attachment #8492466 - Flags: checked-in+
Hey Jonathan, what's the status here? Are we now waiting for the Mulet?
Flags: needinfo?(jgriffin)
We have Mulet builds now, but I think this ties into what's going on with TaskCluster.  James, do we want to switch Gu in buildbot to use Mulet?
Flags: needinfo?(jgriffin) → needinfo?(jlal)
Flags: needinfo?(jlal)
Still wanted?
Flags: needinfo?(catlee)
I think we want to use the Mulet now but not a lot of people really care.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.