Closed Bug 826111 Opened 7 years ago Closed 7 years ago

Support mochitests on Desktop b2g

Categories

(Testing :: Mochitest, defect)

x86
macOS
defect
Not set

Tracking

(firefox19 wontfix, firefox20 wontfix, firefox21 fixed, b2g18 fixed)

RESOLVED FIXED
mozilla21
Tracking Status
firefox19 --- wontfix
firefox20 --- wontfix
firefox21 --- fixed
b2g18 --- fixed

People

(Reporter: bholley, Assigned: jgriffin)

References

Details

Attachments

(1 file, 6 obsolete files)

Apparently this isn't supported, and jgriffin told me to file a bug about it.

Emulators and devices are a pain in general, and I think we should make things work for desktop b2g wherever possible. For example, I'm currently writing mochitests for packaged apps. These don't need to run on a device per se, they just need to run on a b2g build (since that's the only place where packaged apps exist).

If this hasn't be tried, might be worth just taking a half-hour crack at it and seeing if it turns out to be really easy.
Without having looked at it, I think this would be fairly easy...probably some modifications of the desktop mochitest testrunner to handle some B2Gisms.
jgriffin, what specifically do you think needs to be done to turn this on?  It would be a major productivity boost to a lot of folks.
The current B2G mochitest runner assumes a remote device that can be manipulated with adb.  This is obviously not appropriate for the b2g desktop build.

Instead, we'll probably need to modified version of the standard desktop test runner to know how to launch the B2G desktop build and to know how to deal with the Gaia profile, since mochitest runs inside of a Gaia app for permissions reasons.
Cool.  As I recall we have a b2g command-line option for launching particular apps, so that should be relatively easy.

Does the mochitest app depend on any magical DNS/port forwarding that's set up through adb or is it self-enclosed?  (Except for the testing extensions, of course.)
All of the network proxying occurs through a PAC set up in the test profile.  In the case of desktop b2g, we'll have to modify the Gaia profile before we launch the b2g binary.
Assignee: nobody → jgriffin
(In reply to Jonathan Griffin (:jgriffin) from comment #6)
> Created attachment 698976 [details] [diff] [review]
> Support mochitests on b2g desktop build
> 
> Currently running this patch on try:
> https://tbpl.mozilla.org/?tree=Try&rev=7a043d221b43

Nice! Is it running well enough locally for me to try it out?

Will the normal |TEST_PATH=foo make mochitest-plain| work here, or does it require something different?
You can try it out, but it doesn't have a make target.  To use:

python runtestsb2g.py --desktop --profile /path/to/gaia/profile

You'll need to use a build that has Marionette enabled (ENABLE_MARIONETTE=1 in mozconfig).
I get:

Traceback (most recent call last):
  File "/files/mozilla/build/u/_tests/testing/mochitest/runtests.py", line 739, in runTests
    onLaunch=onLaunch)
  File "/files/mozilla/build/u/_tests/testing/mochitest/automation.py", line 1104, in runApp
    onLaunch()
  File "runtestsb2g.py", line 495, in startTests
    assert(self.automation.marionette.wait_for_port())
AttributeError: 'Marionette' object has no attribute 'wait_for_port'
This patch includes a change to Marionette.  You'll need to go to testing/marionette/client after applying the patch and then run python setup.py develop to pick up that change.
Hm, ok. Now B2GDebug beachballs on startup.
This is backtrace of the eventual Timeout:

args: ['/files/mozilla/build/u/dist/B2GDebug.app/Contents/MacOS/b2g', '-foreground', '-profile', '/var/folders/dr/l_l5sx1x2w3dw7kwhlf1yg2r0000gn/T/tmpWYMofa/']
INFO | automation.py | Application pid: 34999
Server listening on port 4443 with cert pgo server certificate INFO | runtests.py | Received unexpected exception while running application
Traceback (most recent call last):
  File "/files/mozilla/build/u/_tests/testing/mochitest/runtests.py", line 739, in runTests
    onLaunch=onLaunch)
  File "/files/mozilla/build/u/_tests/testing/mochitest/automation.py", line 1104, in runApp
    onLaunch()
  File "runtestsb2g.py", line 500, in startTests
    self.automation.test_script)
  File "/files/mozilla/repos/u/testing/marionette/client/marionette/marionette.py", line 476, in execute_script
    specialPowers=special_powers)
  File "/files/mozilla/repos/u/testing/marionette/client/marionette/marionette.py", line 231, in _send_message
    raise TimeoutException(message='socket.timeout', status=ErrorCodes.TIMEOUT, stacktrace=None)
TimeoutException: socket.timeout
TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | missing output line for total leaks!
TEST-UNEXPECTED-FAIL | tab process 35000 | automationutils.processLeakLog() | missing output line for total leaks!
I have been testing on linux; unfortunately I can't seem to get a build on my mac, I get compile errors:

/Users/jgriffin/gaia-hack/src/gfx/thebes/gfxPlatformMac.cpp:391:88: error: expected a class or namespace ? gfxImageFormat::ImageFormatRGB16_565
How did you generate your b2g desktop build?  Did you use GAIADIR in your mozconfig or not?

And, how are you generating your gaia profile?  Are you using DEBUG=1 make, or just make?
You should just be able to delete the gfxImageFormat:: prefixes there.
(In reply to Jonathan Griffin (:jgriffin) from comment #14)
> How did you generate your b2g desktop build?  Did you use GAIADIR in your
> mozconfig or not?

I did not. Should I?

> And, how are you generating your gaia profile?  Are you using DEBUG=1 make,
> or just make?

Just make.
(In reply to Bobby Holley (:bholley) from comment #16)
> (In reply to Jonathan Griffin (:jgriffin) from comment #14)
> > How did you generate your b2g desktop build?  Did you use GAIADIR in your
> > mozconfig or not?
> 
> I did not. Should I?

Not necessarily; I just wanted to test using the same build process you're using.

> 
> > And, how are you generating your gaia profile?  Are you using DEBUG=1 make,
> > or just make?
> 
> Just make.

Thanks, I'll try to reproduce this problem on a mac.
I can successfully run the mochitests on my osx10.7 laptop with a build made from this mozconfig:  http://pastebin.mozilla.org/2045772

After updating Marionette (python setup.py develop from testing/marionette/client), I use this command-line:

python runtestsb2g.py --desktop --profile /path/to/gaia/profile

If you still can't get it to run, I can add some debugging statements into this patch so that we might see what's going wrong.
(In reply to Jonathan Griffin (:jgriffin) from comment #18)
> I can successfully run the mochitests on my osx10.7 laptop with a build made
> from this mozconfig:  http://pastebin.mozilla.org/2045772

The only difference in our mozconfigs is that I was doing a debug build (--enable-debug, --disable-optimize). I'll try your mozconfig and see if that fixes it.
Painfully, this patch fails on try due to lack of both json and simplejson.  :(
(In reply to Josh Matthews [:jdm] from comment #15)
> You should just be able to delete the gfxImageFormat:: prefixes there.

For the record, that didn't work either, I had to change gfxImageFormat:: to gfxASurface::
It works (sort of), when I use a release mozconfig. Might be worth looking into why it fails for debug builds?
(In reply to Bobby Holley (:bholley) from comment #22)
> It works (sort of), when I use a release mozconfig. Might be worth looking
> into why it fails for debug builds?

Yes, I'll take a look.
New version which doesn't require json/simplejson; https://tbpl.mozilla.org/?tree=Try&rev=9bbf00445d71
Attachment #698976 - Attachment is obsolete: true
Attachment #699455 - Attachment is obsolete: true
(In reply to Bobby Holley (:bholley) from comment #22)
> It works (sort of), when I use a release mozconfig. Might be worth looking
> into why it fails for debug builds?

It's failing due to stdout buffering that the mochitest testrunner does, combined with some of the additional startup code needed to support b2g desktop.  I'll try to move this into a separate thread...
(In reply to Jonathan Griffin (:jgriffin) from comment #27)
> (In reply to Bobby Holley (:bholley) from comment #22)
> > It works (sort of), when I use a release mozconfig. Might be worth looking
> > into why it fails for debug builds?
> 
> It's failing due to stdout buffering that the mochitest testrunner does,
> combined with some of the additional startup code needed to support b2g
> desktop.  I'll try to move this into a separate thread...

That would be great. At this point I'm blocked on not being able to use gdb on this stuff.
(In reply to Bobby Holley (:bholley) from comment #28)
> (In reply to Jonathan Griffin (:jgriffin) from comment #27)
> > (In reply to Bobby Holley (:bholley) from comment #22)
> > > It works (sort of), when I use a release mozconfig. Might be worth looking
> > > into why it fails for debug builds?
> > 
> > It's failing due to stdout buffering that the mochitest testrunner does,
> > combined with some of the additional startup code needed to support b2g
> > desktop.  I'll try to move this into a separate thread...
> 
> That would be great. At this point I'm blocked on not being able to use gdb
> on this stuff.

Turns out to be quite a bit worse than that.  The test runner uses Marionette to bootstrap the tests by executing this batch of code:

http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtestsb2g.py#398

On a debug b2g desktop build (but not a regular debug b2g build), Marionette is completely broken due to a failure to load a frame script here:

http://mxr.mozilla.org/mozilla-central/source/testing/marionette/marionette-actors.js#2074

I have no idea why this occurs.  I can try turning off IPC to see if that makes the problem go away.
Yeah, in my experimentation, IPC is broken on Desktop in a number of ways in a number of ways, in addition to the painting issues. I've turned it off locally. :-(
The debug problem appears to be some kind of timing problem, in that Marionette attaches to the wrong window if start_session() is called too early.  The problem goes away if I add a sleep at a particular place.  I'm trying to track down a better way to handle it.
This version works on debug b2g desktop builds.  I had to disable IPC, since with that enabled there is a constant stream of assertions (see bug 788866) that makes it basically unusable.
Attachment #699495 - Attachment is obsolete: true
Some cleanup.  Last patch was all green on try, so moving over to r? status
Attachment #700602 - Flags: review?(ahalberstadt)
Attachment #699955 - Attachment is obsolete: true
I applied this patch and tried following the directions from comment 18, but I'm 
just getting a white b2g-desktop window, and nothing else seems to be happening. I can't see the mozconfig that was linked because pastebin deleted it, but I am using a debug build, so maybe that's causing the problem.

(I'm using OS X 10.8.2)
Also didn't work with an opt build :( I'm probably doing something wrong.
Can you launch the build manually and get a functional gaia?  Or do you also get a white screen?

There are some tips on dealing with the white screen problem here:  https://wiki.mozilla.org/Gaia/Hacking#Blank_screen_when_B2G_Desktop_starts
Comment on attachment 700602 [details] [diff] [review]
Support mochitests on b2g desktop build,

Review of attachment 700602 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good! Most of my comments stem from not understanding how things work, I'll leave them to your judgement.

::: build/automation.py.in
@@ +285,5 @@
> +         appId INTEGER,
> +         isInBrowserElement INTEGER)""")
> +    except sqlite3.OperationalError:
> +      # this table can alreay exist if we're seeding an existing profile
> +      pass

This will hide other errors too (e.g permissions problems), but I'm guessing this is quite stable so probably not worth worrying about.

@@ +301,5 @@
>    def setupTestApps(self, profileDir, apps):
> +    webappJSONTemplate = Template(""""$id": {
> +  "origin": "$origin",
> +  "installOrigin": "$origin",
> +  "receipt": null,

Can't say I understand why these changes are needed, but I guess if it works...

@@ +353,5 @@
> +
> +      for app_content in contents.split('},'):
> +        app = {}
> +        # ghetto json parser needed due to lack of json/simplejson on test slaves
> +        for line in app_content.split('\n'):

Ouch, we should file a bug to replace this once the test slaves have it. Though I'm very surprised that there are some that don't, we'll have to follow up with releng about it.

::: testing/mochitest/runtestsb2g.py
@@ +174,5 @@
> +        self.add_option("--profile", action="store",
> +                        type="string", dest="profile",
> +                        help="for desktop testing, the path to the "
> +                        "gaia profile to use")
> +        defaults["profile"] = None

Is there a difference between "gaia profile" and "gecko profile"? If so we should change the options to --gaia-profile or something since --profile is used everywhere for gecko. If not, then I think s/gaia/gecko would be less confusing.

@@ +508,5 @@
> +    def runMarionetteScript(self, marionette, test_script):
> +        assert(marionette.wait_for_port())
> +        marionette.start_session()
> +        marionette.set_context(marionette.CONTEXT_CHROME)
> +        marionette.execute_script(test_script)

Can we detect if a session is already running and only start a new one if there isn't?

@@ +522,5 @@
> +
> +    def buildURLOptions(self, options, env):
> +        retVal = Mochitest.buildURLOptions(self, options, env)
> +
> +        self.setupCommonOptions(options, OOP=False)

Might be nice to have an option for OOP so in future it will be easier to turn it on to test why it doesn't work. Then again, maybe we don't care about that case.

@@ +641,5 @@
> +    if options == None:
> +        sys.exit(1)
> +
> +    if options.desktop and not options.profile:
> +        raise Exception("must specify --profile when specifying --desktop")

Why is profile required? Can't we just create a profile if it isn't specified?
Attachment #700602 - Flags: review?(ahalberstadt) → review+
(In reply to Jonathan Griffin (:jgriffin) from comment #37)
> Can you launch the build manually and get a functional gaia?

Yes, when I launch the build manually it's fully functional.
(In reply to :Margaret Leibovic from comment #39)
> (In reply to Jonathan Griffin (:jgriffin) from comment #37)
> > Can you launch the build manually and get a functional gaia?
> 
> Yes, when I launch the build manually it's fully functional.

Seems like the something bad is happening with the profile.  Which command did you use to generate the gaia profile, and which to launch mochitest?
(In reply to Jonathan Griffin (:jgriffin) from comment #40)
> (In reply to :Margaret Leibovic from comment #39)
> > (In reply to Jonathan Griffin (:jgriffin) from comment #37)
> > > Can you launch the build manually and get a functional gaia?
> > 
> > Yes, when I launch the build manually it's fully functional.
> 
> Seems like the something bad is happening with the profile.  Which command
> did you use to generate the gaia profile, and which to launch mochitest?

I was just using |make| in my gaia directory (I tried blowing away the old profile to see if this would fix the issue, but it didn't).

To launch the tests, I was doing |python runtestsb2g.py --desktop --profile /Users/leibovic/code/gaia/profile/| from inside my obj-b2g/_tests/testing/mochitest directory.

Here's the log of the test run (before I killed it) if it helps:
http://www.pastebin.mozilla.org/2058977
Mochitest launches a webserver on port 8888; could you possibly have something else running on that port?
(In reply to Andrew Halberstadt [:ahal] from comment #38)
> Comment on attachment 700602 [details] [diff] [review]
> Support mochitests on b2g desktop build,
> 
> Review of attachment 700602 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Looks good! Most of my comments stem from not understanding how things work,
> I'll leave them to your judgement.
> 
> ::: build/automation.py.in
> @@ +285,5 @@
> > +         appId INTEGER,
> > +         isInBrowserElement INTEGER)""")
> > +    except sqlite3.OperationalError:
> > +      # this table can alreay exist if we're seeding an existing profile
> > +      pass
> 
> This will hide other errors too (e.g permissions problems), but I'm guessing
> this is quite stable so probably not worth worrying about.

You're right, I think I should add "if not exists" to the table creation sql, which then won't fail if it exists, and I can drop the try/except.

> 
> @@ +301,5 @@
> >    def setupTestApps(self, profileDir, apps):
> > +    webappJSONTemplate = Template(""""$id": {
> > +  "origin": "$origin",
> > +  "installOrigin": "$origin",
> > +  "receipt": null,
> 
> Can't say I understand why these changes are needed, but I guess if it
> works...
> 

Basically it's needed because I'm trying to squash together the existing webapp definitions from the gaia profile together with the static ones defined by mochitest, and they weren't using same format.

> @@ +353,5 @@
> > +
> > +      for app_content in contents.split('},'):
> > +        app = {}
> > +        # ghetto json parser needed due to lack of json/simplejson on test slaves
> > +        for line in app_content.split('\n'):
> 
> Ouch, we should file a bug to replace this once the test slaves have it.
> Though I'm very surprised that there are some that don't, we'll have to
> follow up with releng about it.
> 

Yes, this is admittedly gross.  The problem is that this code is used by *all* the mochitest test runners.  Once we know that Python 2.7 is available everywhere, we can use json, but that will involve updating all the relevant buildbot and mozharness configs.

simplejson is also available in-tree, but our mochitest testrunner doesn't have access to it on the slaves right now.

> ::: testing/mochitest/runtestsb2g.py
> @@ +174,5 @@
> > +        self.add_option("--profile", action="store",
> > +                        type="string", dest="profile",
> > +                        help="for desktop testing, the path to the "
> > +                        "gaia profile to use")
> > +        defaults["profile"] = None
> 
> Is there a difference between "gaia profile" and "gecko profile"? If so we
> should change the options to --gaia-profile or something since --profile is
> used everywhere for gecko. If not, then I think s/gaia/gecko would be less
> confusing.
> 

The two are the same thing, I only use "gaia profile" since developers who work on the b2g desktop build are used to that terminology.

> @@ +508,5 @@
> > +    def runMarionetteScript(self, marionette, test_script):
> > +        assert(marionette.wait_for_port())
> > +        marionette.start_session()
> > +        marionette.set_context(marionette.CONTEXT_CHROME)
> > +        marionette.execute_script(test_script)
> 
> Can we detect if a session is already running and only start a new one if
> there isn't?
> 

This code only operates on a b2g desktop build that the harness has launched itself, so there's no danger of a marionette session already existing.

> @@ +522,5 @@
> > +
> > +    def buildURLOptions(self, options, env):
> > +        retVal = Mochitest.buildURLOptions(self, options, env)
> > +
> > +        self.setupCommonOptions(options, OOP=False)
> 
> Might be nice to have an option for OOP so in future it will be easier to
> turn it on to test why it doesn't work. Then again, maybe we don't care
> about that case.
> 

I don't want to add an option for it until it actually works, as otherwise it's inevitable that someone would try it and think it a bug in the harness that it blew up.

> @@ +641,5 @@
> > +    if options == None:
> > +        sys.exit(1)
> > +
> > +    if options.desktop and not options.profile:
> > +        raise Exception("must specify --profile when specifying --desktop")
> 
> Why is profile required? Can't we just create a profile if it isn't
> specified?

Mochitests are run inside a gaia test application for B2G builds.  This test app is part of the gaia profile.  Without the gaia profile, the code that tries to run this test app will fail.
> > > +      for app_content in contents.split('},'):
> > > +        app = {}
> > > +        # ghetto json parser needed due to lack of json/simplejson on test slaves
> > > +        for line in app_content.split('\n'):
> > 
> > Ouch, we should file a bug to replace this once the test slaves have it.
> > Though I'm very surprised that there are some that don't, we'll have to
> > follow up with releng about it.
> > 
> 
> Yes, this is admittedly gross.  The problem is that this code is used by *all* the mochitest test runners.  Once we know that Python 2.7 is available everywhere, we can use json, but that will involve updating all the relevant buildbot and mozharness configs.

Yes, please a file a bug for this (and CC me, plz)
Update per review comments.
Attachment #700602 - Attachment is obsolete: true
Comment on attachment 701898 [details] [diff] [review]
Support mochitests on b2g desktop build,

Carry r+ forward.
Attachment #701898 - Flags: review+
Let's run it on try one more time: https://tbpl.mozilla.org/?tree=Try&rev=b4a86775c3e2
https://hg.mozilla.org/integration/mozilla-inbound/rev/a5b75feea6dd
Whiteboard: [automation-needed-in-aurora][automation-needed-in-b2g18]
Blocks: 830538
I believe this may have caused permaorange on linux/linux64 marionette:
https://tbpl.mozilla.org/php/getParsedLog.php?id=18799310&tree=Mozilla-Inbound

Retriggers are still pending (there were high levels of coalescing last night, so a large range to retrospectively cover).

Unfortunately it hadn't been spotted for several hours, since a similar intermittent had been filed and this had been mis-starred as it - likely due to the low SnR in the Marionette failure output (bug 823090 and friends) :-(
Retriggers confirmed this was the cause. Backed out:
https://hg.mozilla.org/integration/mozilla-inbound/rev/85ccfb160eb0
Depends on: 830622
The problem with desktop Marionette tests apparently was caused by changing a time.sleep(5) to time.sleep(5)!  I'll file a separate bug to investigate this, but I've restored the '5' in this version, and it passed a linux64 try run.
Attachment #701898 - Attachment is obsolete: true
Comment on attachment 702525 [details] [diff] [review]
Support mochitests on b2g desktop build,

Carry r+ forward.
Attachment #702525 - Flags: review+
(In reply to Jonathan Griffin (:jgriffin) from comment #52)
> Created attachment 702525 [details] [diff] [review]
> Support mochitests on b2g desktop build,
> 
> The problem with desktop Marionette tests apparently was caused by changing
> a time.sleep(5) to time.sleep(5)!  I'll file a separate bug to investigate
> this, but I've restored the '5' in this version, and it passed a linux64 try
> run.

/s/time.sleep(5) to time.sleep(5)/time.sleep(5) to time.sleep(1)/
https://hg.mozilla.org/mozilla-central/rev/ef40e265cbcd
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla21
https://hg.mozilla.org/releases/mozilla-b2g18/rev/74dd2860f9eb
Whiteboard: [automation-needed-in-aurora][automation-needed-in-b2g18]
(In reply to Jonathan Griffin (:jgriffin) from comment #42)
> Mochitest launches a webserver on port 8888; could you possibly have
> something else running on that port?

I have the same problem as what Margaret described. A white screen where you should be seeing running mochitests on Desktop b2g. This is on MacOSX 10.7.5.

Andrea Marchesini (irc:baku) helped me with this. He checked with telnet localhost 8888 if something else was running on port 8888, but nothing was there.
We also checked that the copied profile was correct, it seemed to be.
And nothing in the logs seem to indicate what the cause of this problem is.
I made sure I could run ./b2g -profile /Users/mwargers/B2G_desktop/gaia/profile.

Basically I'm out of ideas on how to get b2g Desktop mochitests working, now.
Is this a downloaded build, or one you built yourself?
This is one I built myself.
Did you add ENABLE_MARIONETTE=1 to your mozconfig?

I had the same issue and adding the ENABLE_MARIONETTE=1 fixed it.
Thanks! That was indeed the issue, apparently.
So I alrady added ENABLE_MARIONETTE=1 in my .mozconfig and I think I did rebuild, but apparently that wasn't picked up or something.
This time I just removed the whole object directory and rebuilt from scrath and now it's working!
You need to log in before you can comment on or make changes to this bug.