Closed Bug 1411658 Opened 2 years ago Closed 2 years ago

[geckoview] Enable Marionette to drive GeckoView instances

Categories

(GeckoView :: General, enhancement)

enhancement
Not set

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: nalexander, Unassigned)

References

Details

Attachments

(1 obsolete file)

This ticket tracks working through the story to let Marionette to drive GeckoView instances.  The use case that I am really interested in is for Bug 1291391, although there may be others: I want to write Java/JUnit tests that run on-device and use the Selenium WebDriver API, appropriately modified/shimmed, to talk to Marionette running in the Gecko process that is rendering the GeckoView instances in the test App.

What I see locally is that the Marionette start-up process is built around Gecko observer notifications, not all of which are implemented (or appropriate!) for GeckoView.  In particular, Marionette needs "sessionstore-windows-restored" at http://searchfox.org/mozilla-central/rev/d30462037ffea383e74c42542c820cf65b2b144e/testing/marionette/components/marionette.js#233-256 to start listening, and GeckoView doesn't provide that.  (Maybe it should.)

I'd also like to discuss what the correct policy for enabling Marionette is for GeckoView consumers.  Can we always include Marionette, but only enable it (via an environment variable, say) when the consuming App asks for it explicitly?  (I think Marionette watches its enabled pref to allow this, but I'm not sure that it couldn't be enabled all the time by accident.)
(In reply to Nick Alexander :nalexander from comment #0)
> This ticket tracks working through the story to let Marionette to drive
> GeckoView instances.  The use case that I am really interested in is for Bug
> 1291391, although there may be others: I want to write Java/JUnit tests that
> run on-device and use the Selenium WebDriver API, appropriately
> modified/shimmed, to talk to Marionette running in the Gecko process that is
> rendering the GeckoView instances in the test App.
> 
> What I see locally is that the Marionette start-up process is built around
> Gecko observer notifications, not all of which are implemented (or
> appropriate!) for GeckoView.  In particular, Marionette needs
> "sessionstore-windows-restored" at
> http://searchfox.org/mozilla-central/rev/
> d30462037ffea383e74c42542c820cf65b2b144e/testing/marionette/components/
> marionette.js#233-256 to start listening, and GeckoView doesn't provide
> that.  (Maybe it should.)

snorp: jchen: can you comment on whether sessionstore-windows-restored makes sense for GeckoView?

> I'd also like to discuss what the correct policy for enabling Marionette is
> for GeckoView consumers.  Can we always include Marionette, but only enable
> it (via an environment variable, say) when the consuming App asks for it
> explicitly?  (I think Marionette watches its enabled pref to allow this, but
> I'm not sure that it couldn't be enabled all the time by accident.)

ato: dburns: GeckoView is an Android system WebView-like thing.  Is there a precedent for remote controlling such views in the automated testing community?  Is there prior art for enabling/disabling/etc that we should be following, or can we take our own path?
Flags: needinfo?(snorp)
Flags: needinfo?(nchen)
Flags: needinfo?(dburns)
Flags: needinfo?(ato)
(In reply to Nick Alexander :nalexander from comment #1)
> (In reply to Nick Alexander :nalexander from comment #0)
> > This ticket tracks working through the story to let Marionette to drive
> > GeckoView instances.  The use case that I am really interested in is for Bug
> > 1291391, although there may be others: I want to write Java/JUnit tests that
> > run on-device and use the Selenium WebDriver API, appropriately
> > modified/shimmed, to talk to Marionette running in the Gecko process that is
> > rendering the GeckoView instances in the test App.
> > 
> > What I see locally is that the Marionette start-up process is built around
> > Gecko observer notifications, not all of which are implemented (or
> > appropriate!) for GeckoView.  In particular, Marionette needs
> > "sessionstore-windows-restored" at
> > http://searchfox.org/mozilla-central/rev/
> > d30462037ffea383e74c42542c820cf65b2b144e/testing/marionette/components/
> > marionette.js#233-256 to start listening, and GeckoView doesn't provide
> > that.  (Maybe it should.)
> 
> snorp: jchen: can you comment on whether sessionstore-windows-restored makes
> sense for GeckoView?

We can fire the event if we need to, but actually using sessionstore doesn't makes sense IMO. We'll provide a GeckoView API for apps to save/restore session data, but we won't be using the Gecko SessionStore.js[m] stuff.
Flags: needinfo?(snorp)
(In reply to James Willcox (:snorp) (jwillcox@mozilla.com) from comment #2)
> (In reply to Nick Alexander :nalexander from comment #1)
> > (In reply to Nick Alexander :nalexander from comment #0)
> > > This ticket tracks working through the story to let Marionette to drive
> > > GeckoView instances.  The use case that I am really interested in is for Bug
> > > 1291391, although there may be others: I want to write Java/JUnit tests that
> > > run on-device and use the Selenium WebDriver API, appropriately
> > > modified/shimmed, to talk to Marionette running in the Gecko process that is
> > > rendering the GeckoView instances in the test App.
> > > 
> > > What I see locally is that the Marionette start-up process is built around
> > > Gecko observer notifications, not all of which are implemented (or
> > > appropriate!) for GeckoView.  In particular, Marionette needs
> > > "sessionstore-windows-restored" at
> > > http://searchfox.org/mozilla-central/rev/
> > > d30462037ffea383e74c42542c820cf65b2b144e/testing/marionette/components/
> > > marionette.js#233-256 to start listening, and GeckoView doesn't provide
> > > that.  (Maybe it should.)
> > 
> > snorp: jchen: can you comment on whether sessionstore-windows-restored makes
> > sense for GeckoView?
> 
> We can fire the event if we need to, but actually using sessionstore doesn't
> makes sense IMO. We'll provide a GeckoView API for apps to save/restore
> session data, but we won't be using the Gecko SessionStore.js[m] stuff.

Yeah, I'm worried about firing the event, since _so much_ of Gecko listens for it, and it's very likely to subtly invoke things that may break/be broken.

I'd prefer to factor the Marionette code so that sessionstore-windows-restored fires a marionette-* notification which GeckoView can also fire.  That preserves the decoupling while not having GeckoView opt-in to things it might not want.
(In reply to Nick Alexander :nalexander from comment #3)
> 
> I'd prefer to factor the Marionette code so that
> sessionstore-windows-restored fires a marionette-* notification which
> GeckoView can also fire.  That preserves the decoupling while not having
> GeckoView opt-in to things it might not want.

I agree.
Flags: needinfo?(nchen)
(In reply to Nick Alexander :nalexander from comment #1)

> 
> > I'd also like to discuss what the correct policy for enabling Marionette is
> > for GeckoView consumers.  Can we always include Marionette, but only enable
> > it (via an environment variable, say) when the consuming App asks for it
> > explicitly?  (I think Marionette watches its enabled pref to allow this, but
> > I'm not sure that it couldn't be enabled all the time by accident.)
> 
> ato: dburns: GeckoView is an Android system WebView-like thing.  Is there a
> precedent for remote controlling such views in the automated testing
> community?  Is there prior art for enabling/disabling/etc that we should be
> following, or can we take our own path?

chrome have done the following in the past https://sites.google.com/a/chromium.org/chromedriver/getting-started/getting-started---android

the current working model with companies is to try reuse their 'remote' tests as much as possible speaking into a real device or emulator
Flags: needinfo?(dburns)
(In reply to David Burns :automatedtester from comment #5)
> (In reply to Nick Alexander :nalexander from comment #1)
> 
> > 
> > > I'd also like to discuss what the correct policy for enabling Marionette is
> > > for GeckoView consumers.  Can we always include Marionette, but only enable
> > > it (via an environment variable, say) when the consuming App asks for it
> > > explicitly?  (I think Marionette watches its enabled pref to allow this, but
> > > I'm not sure that it couldn't be enabled all the time by accident.)
> > 
> > ato: dburns: GeckoView is an Android system WebView-like thing.  Is there a
> > precedent for remote controlling such views in the automated testing
> > community?  Is there prior art for enabling/disabling/etc that we should be
> > following, or can we take our own path?
> 
> chrome have done the following in the past
> https://sites.google.com/a/chromium.org/chromedriver/getting-started/getting-
> started---android

Thanks, dburns.  From that doc:

"""
ChromeDriver supports running tests on ... WebView-based apps ... that have enabled web debugging and JavaScript.
"""

We can agree with that: if web debugging is on, then Marionette is enabled.
Getting WebDriver to run in GeckoView all sounds very exciting!

First of all I have to say that I’m not at all familiar with
GeckoView, what it is, or how it functions.  For that reason you
should take what I say with a grain of salt because I will be making
some entirely uneducated assumptions.

The following is a bit of a brain dump, but I hope some of the
information will be useful to you! (-:

(In reply to Nick Alexander :nalexander from comment #0)

> I think Marionette watches its enabled pref to allow this, but I'm
> not sure that it couldn't be enabled all the time by accident.

It’s currently impossible to enable Marionette at runtime through
a preference.  The reason for this is that addons can arbitrarily
set preferences, potentially opening up Gecko to a remote attacker.

With Firefox, the only way to start the remote server is by passing
the -marionette flag or by setting the MOZ_MARIONETTE output
variable prior to launching the process.

(In reply to Nick Alexander :nalexander from comment #1)

> GeckoView is an Android system WebView-like thing.  Is there a
> precedent for remote controlling such views in the automated
> testing community?  Is there prior art for enabling/disabling/etc
> that we should be following, or can we take our own path?

A much younger me did the very first implementation of WebDriver on
Android back in 2012 for Presto-based Opera:

	https://dev.opera.com/articles/mobile-browser-automation/

If you want to control GeckoView using the Selenium WebDriver Java
client, you will have to speak to Marionette through geckodriver
[1], which is our implementation of the W3C WebDriver HTTP wire
protocol [2].  Marionette does not itself directly implement the
WebDriver protocol.

Depending how GeckoView works, we can either put the geckodriver
Rust binary on the host machine or cross-compile it to run on the
target Android device.  Which you choose will impact how easy it is
to make it work with Selenium.

At its core, a WebDriver session is created via an HTTP request
to an HTTPD server.  Selenium clients, such as the Java client,
typically has built-in support for launching this server, and once
the service has been started, it falls back to making the session
using what is called, in Java, the RemoteWebDriver class.

The RemoteWebDriver class connects you to any HTTP remote end, so
if you have a WebDriver-capable HTTPD, such as geckodriver, already
running it can speak directly to this.  This means you are faced
with two options:

  (1) Make the assumption that geckodriver is a host-system service
  responsible for setting up the adb debug bridge and ensuring
  Marionette is enabled in GeckoView.

  (2) Use a RemoteWebDriver-esque approach to talk directly to a
  geckodriver HTTPD that you have manually uploaded to and started
  on the target device, after you’ve manually set up the adb
  bridge yourself.

In both cases modifications will have to be made to geckodriver so
it recognises it shouldn’t create Firefox profiles and try launch
the system Firefox when a new session is requested.  My gut feeling
is that option 1 would be the more user friendly approach because it
entails fewer manual steps.

When the New Session [1] request is made, you can pass along a
set of configuration capabilities.  Two of these are platformName
and browserName, and they sound like they might be useful for
determining which code path geckodriver should take.  Given for
example "android" it could check that adb is available on the
system, open an adb bridge, somehow start a GeckoView process with
MOZ_MARIONETTE set and wait for the TCP server to bind to a given
port.  Once this instrumentation phase is complete, we return to the
(Selenium) client with a new WebDriver session. geckodriver will
then hold a TCP connection to the Marionette remote server.

What is not clear to be is how GeckoView is different to Fennec?
Marionette already has preliminary support for Fennec, which I
assume means that sessionstore-windows-restores is fired correctly.
We could obviously listen for something else in the case of
GeckoView, but the nature of GeckoView eludes me.

I am also making the assumption that Marionette is built in to
GeckoView?  If it is we will not need a custom Android package like
Chrome does, like it says in the documentation David linked to.

  [1] https://searchfox.org/mozilla-central/source/testing/geckodriver/README.md
  [2] https://w3c.github.io/webdriver/webdriver-spec.html
  [3] https://w3c.github.io/webdriver/webdriver-spec.html#new-session
Flags: needinfo?(ato)
Triage consensus: we don't think we'll do this.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → WONTFIX
(In reply to David Bolter [:davidb] (NeedInfo me for attention) from comment #8)
> Triage consensus: we don't think we'll do this.

Really?  What's the alternative for this very specific type of content interaction?
Flags: needinfo?(dbolter)
Flags: needinfo?(dbolter) → needinfo?(snorp)
(In reply to Nick Alexander :nalexander from comment #9)
> (In reply to David Bolter [:davidb] (NeedInfo me for attention) from comment
> #8)
> > Triage consensus: we don't think we'll do this.
> 
> Really?  What's the alternative for this very specific type of content
> interaction?

For testing the GeckoView API, we may not need something so involved as WebDriver/Marionette. However, we do have a need for simple messaging between the GeckoSession test runner and the test page, in order to verify for instance that enabling tracking protection actually prevents the loading of some blocked resource. Getting the WebDriver/Marionette stuff working for testing will be a substantial amount of work, and I'm not sure what the value will really be.

The a11y stuff seems like it would be a better path to expose element-level detail to the app, since I believe we would also get UiAutomator working for "free". AFAICT, this is also the idiomatic way of getting visibility inside the Android WebView.
Flags: needinfo?(snorp)
since marionette works against fennec currently, what are the potential blockers that block it for geckoview? Can the Marionette team help?

Do we have a plan for making sure that we our expectations for it being an interoperable browser? Originally I thought we would want something like WPT to work on GeckoView so we know what we are delivering is still correct. WPT on mobile is on my team's radar for H1
Flags: needinfo?(snorp)
(In reply to David Burns :automatedtester from comment #11)
> since marionette works against fennec currently, what are the potential
> blockers that block it for geckoview? Can the Marionette team help?

It might work, but AFAIK there is not a good way of consuming it from the Java (Kotlin, actually) tests we have.

> 
> Do we have a plan for making sure that we our expectations for it being an
> interoperable browser? Originally I thought we would want something like WPT
> to work on GeckoView so we know what we are delivering is still correct. WPT
> on mobile is on my team's radar for H1

We do want to run WPT. There are also plans to run existing mochitests/reftests/etc on GeckoView (which works now, just needs hooked up to automation).
Flags: needinfo?(snorp)
Where are we on this bug? Particularly interested in using this to automate pageload tests for Mobile.
Flags: needinfo?(dbolter)
(In reply to Vicky Chin [:vchin] from comment #13)
> Where are we on this bug? Particularly interested in using this to automate
> pageload tests for Mobile.

eeejay recently did a bunch of work to allow driving the DOM from UI Automator.  I'm hoping that his write-up will be made public sometime soon, but in the interim I've CCed you on that email.  Perhaps that will be useful for you?

eeejay: perhaps you can comment directly?
Flags: needinfo?(eitan)
Yeah, we should have UIAutomator support now, or are at least close to it actually working.

I don't think it is relevant for this bug, WebDriver is a completely different beast.
Flags: needinfo?(eitan)
> I don't think it is relevant for this bug, WebDriver is a completely
> different beast.

Right -- not relevant to this ticket, but relevant to vchin's desire to "automate pageload tests for Mobile".  Perhaps.  Anyway, I leave it to vchin to see if UI Automator helps.
Vicky UI Automator is probably the way to go?
Flags: needinfo?(dbolter) → needinfo?(vchin)
We essentially want to be able to use the sitespeed tools (specifically, https://github.com/sitespeedio/browsertime#test-on-your-mobile-device) on a GV product like Focus.  This uses selenium webdriver to not only control the browser, but also collect performance metrics such as the ones recorded in window.performance.timing.  Essentially, I think this boils down to supporting geckodriver for GV/Focus/Fenix.
What is our current outlook for WebDriver?
Flags: needinfo?(vchin) → needinfo?(eitan)
(In reply to David Bolter [:davidb] (NeedInfo me for attention) from comment #17)
> Vicky UI Automator is probably the way to go?

I would avoid using UIAutomator here since it initializes accessibility and it will affect your perf numbers.

(In reply to Denis Palmeiro [:denispal] from comment #18)
> We essentially want to be able to use the sitespeed tools (specifically,
> https://github.com/sitespeedio/browsertime#test-on-your-mobile-device) on a
> GV product like Focus.  This uses selenium webdriver to not only control the
> browser, but also collect performance metrics such as the ones recorded in
> window.performance.timing.  Essentially, I think this boils down to
> supporting geckodriver for GV/Focus/Fenix.

So, if I understand there are two types of automation needed:
(1) Browser 'chrome' control to feed a URI and load a page.
(2) DOM in-page interactions.

Is that correct?

For the first, if you just want to load pages but don't need to do anything else, you can send certain parameters to `am start` via `adb shell` that would launch or use a current browsing session to load a URI.

For the second, yeah, you will need GeckoDriver.

(In reply to Vicky Chin [:vchin] from comment #19)
> What is our current outlook for WebDriver?

I don't know, I think that's a question for the core GeckoView team. I just did the accessibility bits which indirectly introduced UIAutomator support.
Flags: needinfo?(eitan)
(In reply to Eitan Isaacson [:eeejay] from comment #20)
> (In reply to David Bolter [:davidb] (NeedInfo me for attention) from comment
> #17)
> > Vicky UI Automator is probably the way to go?
> 
> I would avoid using UIAutomator here since it initializes accessibility and
> it will affect your perf numbers.
> 
> (In reply to Denis Palmeiro [:denispal] from comment #18)
> > We essentially want to be able to use the sitespeed tools (specifically,
> > https://github.com/sitespeedio/browsertime#test-on-your-mobile-device) on a
> > GV product like Focus.  This uses selenium webdriver to not only control the
> > browser, but also collect performance metrics such as the ones recorded in
> > window.performance.timing.  Essentially, I think this boils down to
> > supporting geckodriver for GV/Focus/Fenix.
> 
> So, if I understand there are two types of automation needed:
> (1) Browser 'chrome' control to feed a URI and load a page.
> (2) DOM in-page interactions.
> 
> Is that correct?

Yes, correct.  Also the ability to execute in-page JS snippets, which I guess also falls under 2.
I am happy to meet with the geckoview team and anyone else interested next week to discuss/hack support for this.
Attachment #9032528 - Attachment is obsolete: true
See Also: → 1496773
Attachment #9032528 - Attachment description: Bug 1411658 - More WIP on enabling Marionette. → Bug 1411658 - WIP on a GeckoView API for enabling Marionette in code.
Attachment #9032528 - Attachment is obsolete: false
gbrown: just FYI -- https://phabricator.services.mozilla.com/D15016 is some GeckoView-side plumbing (and a little garbage that reveals issues like you're addressing in Bug 1496773) that might be of interest.

Really pleased to see this moving forward!
Flags: needinfo?(gbrown)
Product: Firefox for Android → GeckoView
Thanks!
Flags: needinfo?(gbrown)
Attachment #9032528 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.