861015 - Marionette (Mn) tbpl test should be easier to run on desktop

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Steve Fink [:sfink] [:s:]]

I had a test failure in Marionette ( https://tbpl.mozilla.org/?tree=Try&rev=6bd87c010171 ). So I attempted to reproduce the test run. Sadly, the commands in the tbpl log file weren't of much help. mdas on #developers assisted me in coming up with this set of instructions for a local build:

1. add ENABLE_MARIONETTE=1 to mozconfig
2. rebuild
3. source obj/_virtualenv/bin/activate
4. cd testing/marionette/client
5. python setup.py develop
6. cd marionette
7. python runtests.py --binary=$obj/dist/bin/firefox --type=browser tests/unit-test.ini

It should be very straightforward to run these tests, since they're considered tier 1 and so require developers to fix them if they break. I encountered some friction while doing so. I'll record my issues here for now. Feel free to file separate bugs for these if I don't get around to it.

Problems:

1. ENABLE_MARIONETTE not on by default. (Does it slow the build substantially?)
2. setting ENABLE_MARIONETTE=1 requires a clobber, but the build system doesn't know that. It'll happily rebuild without adding in marionette support.
3. if you don't clobber (or, presumably, don't have it enabled at all), it runs with no error message, it just doesn't run the tests
4. no mach command for running the Mn marionette test suite that shows up on tbpl
5. I couldn't find the above steps on the wiki anywhere.

Comment 1

[Jonathan Griffin (:jgriffin)]

Hey Steve,

Sorry you had troubles with this.  We're aware of this problem and hope to have it fixed quite soon, or at least most of it.  Making a mach target for running Mn tests is ticketed as bug 799308.

Marionette isn't enabled by default at a request from the Security team, since it could theoretically allow an attacker to take control of the browser.

But, I think we could make the mach target smart enough so that it would detect whether or not Marionette was enabled in the build, and let you know what to do (i.e., modify mozconfig and clobber...) if it isn't.

Comment 2

[Malini Das [:mdas] - Away, not checking bugmail]

(In reply to Steve Fink [:sfink] from comment #0)
> 5. I couldn't find the above steps on the wiki anywhere.

I've just updated the wiki (https://developer.mozilla.org/en-US/docs/Marionette/Running_Tests) so it contains relevant instructions

Comment 3

[Steve Fink [:sfink] [:s:]]

(In reply to Jonathan Griffin (:jgriffin) from comment #1)
> Sorry you had troubles with this.  We're aware of this problem and hope to
> have it fixed quite soon, or at least most of it.  Making a mach target for
> running Mn tests is ticketed as bug 799308.

Sorry, I wasn't really complaining, more just recording my experience with the current state.

> Marionette isn't enabled by default at a request from the Security team,
> since it could theoretically allow an attacker to take control of the
> browser.

Makes sense. Though it's on in Nightly, right? It almost seems like you want an "available but protected by a key" state or something, not that it would add all that much security.

> But, I think we could make the mach target smart enough so that it would
> detect whether or not Marionette was enabled in the build, and let you know
> what to do (i.e., modify mozconfig and clobber...) if it isn't.

Why not the runtests script? Seems like whatever piece tries to connect to the browser to control it should get a connection refused or something. (Note that I don't know anything about how Marionette works, so I'm really just talking out of my... uh, orifice that not used for eating.)

Comment 4

[Jonathan Griffin (:jgriffin)]

(In reply to Steve Fink [:sfink] from comment #3)
> 
> > Marionette isn't enabled by default at a request from the Security team,
> > since it could theoretically allow an attacker to take control of the
> > browser.
> 
> Makes sense. Though it's on in Nightly, right? It almost seems like you want
> an "available but protected by a key" state or something, not that it would
> add all that much security.

No, it isn't enabled in Nightly, only per-commit debug builds.

> 
> > But, I think we could make the mach target smart enough so that it would
> > detect whether or not Marionette was enabled in the build, and let you know
> > what to do (i.e., modify mozconfig and clobber...) if it isn't.
> 
> Why not the runtests script? Seems like whatever piece tries to connect to
> the browser to control it should get a connection refused or something.
> (Note that I don't know anything about how Marionette works, so I'm really
> just talking out of my... uh, orifice that not used for eating.)

Yes, we get a connection refused error.  Unfortunately, this can be caused by things other than not having Marionette enabled in the build.  For example, in B2G, it can be caused by failing to initiate adb port forwarding.  And we've seen bugs in B2G where networking problems have caused the same symptom.  It can also occur when B2G crashes (which obviously takes Marionette down).  I'm afraid having runtests try to figure this out would lead to misleading error messages.

Comment 5

[Jonathan Griffin (:jgriffin)]

Marionette is now enabled by default in desktop Firefox builds, and we have a mach command for running tests, so I think this is done.  Re-open if you see something we missed.