1524607 - [remote-dbg-next] Setup semi-automated testing solution with Android devices

Status: NEW

(DevTools :: about:debugging, enhancement, P3)

Description

[Julian Descottes [:jdescottes]]

Daisuke mentioned he would like to investigate a solution to run automated tests using both Firefox desktop and Firefox for android. Such tests would not run on CI and would require the developer to probably plug in a phone, but we could have a set of smoke tests that would run like that. It would be semi automated, but it would be better than having to manually test everything related to mobile.

Comment 1

[Julian Descottes [:jdescottes]]

Daisuke: maybe you already have some ideas you want to share here so that we don't forget about this?

Comment 2

[Daisuke Akatsuka (:daisuke)]

Thank you for filing this, Julian!

I have not investigated enough though, I was thinking hazily like below:
At first, makes tests that are using usb mockup to be enabled to test on actual device. Perhaps, we need something config file which write runtime information of actual environment, then when specifies that as environmental variables, the test will be enabled. Next phase, ideally, want to run tests that add/remove tabs/ adding on "this-firefox" though, this might take a time. Because remote runtime does not have such as ability yet (I think?). However, it looks tests of such as runtime info etc are easy.
So, basically, I thought that don't want to add the special tests for USB for maintainability. Ideally, if we write a test once, wants to make the test can run on other runtime as well.

Comment 3

[Julian Descottes [:jdescottes]]

filter on: remote-debugging-m2-reserve-cleanup

Comment 4

[Soledad Penades [:sole] [:spenades]]

Hey GBrown, Andreas (Ato) has told me that you might have some insights here that can help us :-)

Comment 5

[Geoff Brown [:gbrown]]

All of our python-based test harnesses supporting Android in continuous integration (mochitest, reftest, etc) use mozdevice [1] [2] to communicate with Android devices (physical phones or emulators). The harnesses use some combination of if/else and/or subclassing to separate desktop from Android implementations of device-specific operations. For example:

if options.android:
self.device.push(src, remote_dst)
else:
shutil.copy(src, dst)

See [3] for a real-world example.

mozdevice lets you copy files to/from an android device, launch and monitor processes, etc, but it doesn't really interact with the test app (firefox) in any way. Years ago we wrote UI interaction tests for Fennec with robotium [4] (the "robocop" test suite); that's poorly maintained now and was always problematic -- not recommended. More recently, I heard that No-Jun Park did something with Espresso [5], but I don't know how well that worked out. And of course there's also marionette [6].

I hope that helps?

[1] https://searchfox.org/mozilla-central/source/testing/mozbase/docs/mozdevice.rst
[2] https://searchfox.org/mozilla-central/source/testing/mozbase/mozdevice/mozdevice
[3] https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#491-492
[4] https://github.com/RobotiumTech/robotium
[5] https://developer.android.com/training/testing/espresso
[6] https://firefox-source-docs.mozilla.org/testing/marionette/marionette/index.html

Comment 6

[Julian Descottes [:jdescottes]]

Thanks for the info! I will flag again to make sure we are talking about the same thing :)

The tests we want to write here should use simultaneously:

• a Firefox desktop instance that runs the remote debugging client
• a Firefox mobile instance that runs the remote debugging server
  because what we really want to test is the interaction between the two.

Which one of the options suggested above would be the most appropriate for such tests?

We would also be interested in running tests with 2 Firefox desktop instances (one as client, one as server).

And finally the best would be if we could run different versions of the client and the server, say run Firefox Nightly as the client and run Beta or Release as the server.

Comment 7

[Geoff Brown [:gbrown]]

Oh, thanks for the clarification -- I had no idea!

Would you need to run any test code on the mobile instance? I'm thinking maybe not. In that case, suppose you have a local desktop build and a local mobile build. From the mobile build context, something like 'mach build && mach package && mach android-emulator && mach install && mach run' gets an android emulator running with Firefox for Android installed and running. Now from your desktop build context, build and run Firefox and start your remote debugging test: Maybe just a mochitest that connects to and interacts with the server (I suppose you just need a network connection between them? emulator port forwarding, or ip lookup on a phone should do it).

If you need more control over the device (automated install and run?), I'd write a quick python script using mozdevice. I'm not sure it needs to be any more complicated.