Open Bug 1064253 (luciddream) Opened 10 years ago Updated 2 years ago

Developer tools tests should be run against a variety of platforms

Categories

(Testing :: General, defect)

Product:
Testing
Component:
General
Platform:
All
Other
Type:
defect

Tracking

(Not tracked)

People

(Reporter: jwalker, Unassigned)

References

Details

Attachments

(2 files, 1 obsolete file)

Integrate luciddream into m-c
24.40 KB, patch
jgriffin
: feedback+
Details | Diff | Splinter Review
taskcluster patch to run luciddream
5.28 KB, patch
Details | Diff | Splinter Review 
>>Problem:
Developer tools currently run against themselves for browser debugging (by spawning a separate process). However this only tests that Firefox desktop version X can debug Firefox desktop version X.

This is more of a problem for B2G where a version mismatch is virtually guaranteed. FeverDream will enable testing against other back-ends, making the complexity of the problem much greater by introducing debug targets that are themselves moving targets. (In general B2G targets are of a few known versions)

We would like a way for automated tests to run not only against Firefox Desktop, but also against a number of other platforms.

>>Solution:
Initially we don't need 'automatic' backout on test failure, so having a somewhat separate platform for this testing is acceptable, but we should aim for the ability to be fully integrated, which means we should have a goal of working with the current infrastructure as much as possible.


>>Mozilla Top Level Goal:
Developer Tools

>>Existing Bug:
No bug

>>Per-Commit:
Can be run less frequently initially

>>Data other than Pass/Fail:
Pass/fail with output capture as for existing tests

>>Prototype Date:
Not provided

>>Production Date:
Not provided

>>Most Valuable Piece:
Not provided

>>Responsible Engineer:
Not provided

>>Manager:
jwalker@mozilla.com

>>Other Teams/External Dependencies:
Not provided

>>Additional Info:
Not provided
Are you looking to reuse the existing mochitest-browser-chrome devtools tests, or are you open to something different?

> We would like a way for automated tests to run not only against Firefox Desktop, but also against a 
> number of other platforms.

Which other platforms, just various versions of B2G?
Flags: needinfo?(jwalker)
(In reply to Joe Walker [:jwalker] (overloaded - needinfo me or ping on irc) from comment #0)
> >>Problem:
> Developer tools currently run against themselves for browser debugging (by
> spawning a separate process). However this only tests that Firefox desktop
> version X can debug Firefox desktop version X.

Can you link to the tests that you're talking about here?
(In reply to Jonathan Griffin (:jgriffin) from comment #1)
> Are you looking to reuse the existing mochitest-browser-chrome devtools
> tests, or are you open to something different?
> 
> > We would like a way for automated tests to run not only against Firefox Desktop, but also against a 
> > number of other platforms.
> 
> Which other platforms, just various versions of B2G?

Somewhat sorted by the order I'd tackle them in:

* Current B2G
* Older sizable B2G releases
* Latest Chrome/Chromium on same platform
* Latest Chrome/Chromium on Android
* Latest Safari on iOS
Flags: needinfo?(jwalker)
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #2)
> (In reply to Joe Walker [:jwalker] (overloaded - needinfo me or ping on irc)
> from comment #0)
> > >>Problem:
> > Developer tools currently run against themselves for browser debugging (by
> > spawning a separate process). However this only tests that Firefox desktop
> > version X can debug Firefox desktop version X.
> 
> Can you link to the tests that you're talking about here?

Devtools already has a decent set of mochitests and xpcshell tests. See browser/devtools/*/test and toolkit/devtools/*/tests

But we're not against trying something new at all.

Quick background: Devtools have run against B2G for a while, and we have a new project to target things that use the webkit devtools protocol.

We need to up our testing game. Testing is currently manual, but we're starting to automate it. We could build a mini test infrastructure by attaching a bunch of devices to some computer somewhere and running some custom scripts to see what the current state is.

But we'd like this to be part of the official infrastructure, included in try, with sheriff backout, etc, etc. It's going to take a while to get there for sure, but I'd rather not have a time when we have to throw away a hacked up mess to port to the Official Infrastructure.

So the short term question is: What should we do now in order to make using the Official Infrastructure easier in the future.
(In reply to Joe Walker [:jwalker] (overloaded - needinfo me or ping on irc) from comment #3)
> (In reply to Jonathan Griffin (:jgriffin) from comment #1)
> > 
> > Which other platforms, just various versions of B2G?
> 
> Somewhat sorted by the order I'd tackle them in:
> 
> * Current B2G
> * Older sizable B2G releases
> * Latest Chrome/Chromium on same platform
> * Latest Chrome/Chromium on Android
> * Latest Safari on iOS

Do these priorities change based on Firefox Developer Edition plans?
Flags: needinfo?(jwalker)
(In reply to Jonathan Griffin (:jgriffin) from comment #5)
> Do these priorities change based on Firefox Developer Edition plans?

No.
Flags: needinfo?(jwalker)
Ted is going to work on this in Q4, and will follow up with jwalker to determine exact requirements.
Assignee: jgriffin → ted
CC Alex, who expressed and interest in this problem.
Alex and I had a chat today about the scope of the work here. He indicated that the highest-value would be getting some sort of testing of the Firefox->device codepaths, as that has no test coverage right now and is prone to breakage.

We mutually agreed that a test harness built on Marionette is likely to give us the most testing leverage, since it would allow tests to control the device environment (and hopefully keep the door open to using webdriver to test against other browsers/devices in the future). We both agreed that it would be nice to ensure that we can write complex, powerful tests but still write simple tests without requiring a ton of overhead.

Alex said they've written a handful of Gaia tests that test devtools functionality, which should be a great starting point for imagining what tests in this harness will look like.

We discussed the possibility of refactoring some of the existing devtools tests to run in the new harness, it sounds like that's a definite possibility but the tests across different devtools features vary widely in their approach since they've been written by a number of different people. We were reasonably successful with that approach in Steeplechase, so I'm hopeful we can make it work in some capacity with this harness (which we need a codename for).

For the Q4 goal I'm going to focus on writing a Python harness that can run tests against a combination of a desktop Firefox build and a B2G emulator build. I'll be looking to get something that we can run locally for this quarter, with an eye towards getting it in some sort of automation next quarter. The primary focus will be to define what the tests will look like, and build the harness to accommodate that.

I'll be creating a stub harness in a Github repo this week to get started, and Alex agreed to provide feedback as I make progress.

One other thing we discussed that's out of scope this quarter but important for the future is how to manage cross-branch testing. When we get these tests in automation we'd like to test a matrix of cross-branch compatibility, i.e. Firefox from mozilla-inbound against B2G 1.1. We'll need to sort out the details of how to figure out what builds to run against, and what the reporting looks like when we get to that point.
Here is the gaia tests I mentioned:
  https://github.com/mozilla-b2g/gaia/tree/master/tests/integration/devtools
And another test that never landed because of pushback from gaia devs:
  https://github.com/mozilla-b2g/gaia/pull/21498/files
Sorry for the delay, I had a few issues getting B2G emulators working on my desktop, and then I wound up scrapping a bunch of work I had to leverage the existing Marionette test harness instead of rolling my own. Anyway, here's the brand new not very full-featured test harness which I have code-named "Lucid dream":
https://github.com/luser/luciddream

The only quirk of running it right now is that it relies on a bug fix in marionette-client that I haven't even landed yet (bug 1101539), but I should be rectifying that shortly.

There are lots of things that need work here, but the skeleton of what a test might look like can be seen in test_sample.py:
https://github.com/luser/luciddream/blob/master/example-tests/test_sample.py

Right now this is just a MarionetteTestCase (so you can use self.marionette to control the emulator), but you also have a self.browser that is a Marionette instance that can control the Firefox browser.

I suspect the minimal thing that would make this more useful would be to have a DevtoolsTestCase on top of that that instantiated a remote devtools connection from the browser to the device.
Alias: luciddream
I'm pretty happy with where this has wound up. Alex and I had a face-to-face at the all-hands in Portland the other week, and he was generally pleased with the state of things but needed to sit down and write some tests with it to really figure out if it's fully capable. I don't think he's completely explored that yet, but the framework as-written is solid and I've fixed up a few things from his initial testing: B2G desktop support, a method for executing whole JS scripts as tests (see bug 1114695).

I don't see any huge roadblocks to accomplishing what we want to do with this harness, so I'm going to call the prototype finished. I have a Q1 goal of getting this harness stood up in CI running tests with a Firefox Linux desktop build + B2G emulator build per-checkin, with a stretch goal of running tests on the per-checkin desktop build against stable B2G release emulator builds. Once we get that up and get some tests written by the devtools folks we'll be in a good position to figure out what else the harness needs to do, and we can prioritize support for other platforms.
Depends on: 1121696
Depends on: 1128099
I'm wondering if that regressed?
I'm seeing the following exceptions now:

$ runluciddream --b2g-desktop ~/gaia/b2g_sdk/34.0a1-2014-08-12-04-02-01/b2g/b2g-bin --gaia-profile ~/gaia/profile --browser-path ~/firefox64/firefox example-tests/luciddream.ini 
Traceback (most recent call last):
  File "/home/alex/luciddream/ve/bin/runluciddream", line 9, in <module>
    load_entry_point('luciddream==0.1', 'console_scripts', 'runluciddream')()
  File "/home/alex/luciddream/luciddream/runluciddream.py", line 147, in main
    runner.run_tests([args.manifest])
  File "/home/alex/luciddream/ve/local/lib/python2.7/site-packages/marionette_client-0.8.5-py2.7.egg/marionette/runner/base.py", line 715, in run_tests
    self.capabilities
  File "/home/alex/luciddream/ve/local/lib/python2.7/site-packages/marionette_client-0.8.5-py2.7.egg/marionette/runner/base.py", line 560, in capabilities
    self._capabilities = self.marionette.session_capabilities
  File "/home/alex/luciddream/ve/local/lib/python2.7/site-packages/marionette_client-0.8.5-py2.7.egg/marionette/marionette.py", line 839, in session_capabilities
    response = self._send_message('getSessionCapabilities', 'value')
  File "/home/alex/luciddream/ve/local/lib/python2.7/site-packages/marionette_client-0.8.5-py2.7.egg/marionette/decorators.py", line 36, in _
    return func(*args, **kwargs)
  File "/home/alex/luciddream/ve/local/lib/python2.7/site-packages/marionette_client-0.8.5-py2.7.egg/marionette/marionette.py", line 596, in _send_message
    raise errors.MarionetteException("Please start a session")
marionette.errors.MarionetteException: MarionetteException: Please start a session

$ runluciddream --b2g-desktop /mnt/desktop/gecko/obj-b2g/dist/bin/b2g-bin --gaia-profile ~/gaia/profile --browser-path ~/firefox64/firefox example-tests/luciddream.ini 
Traceback (most recent call last):
  File "/home/alex/luciddream/ve/bin/runluciddream", line 9, in <module>
    load_entry_point('luciddream==0.1', 'console_scripts', 'runluciddream')()
  File "/home/alex/luciddream/luciddream/runluciddream.py", line 147, in main
    runner.run_tests([args.manifest])
  File "/home/alex/luciddream/ve/local/lib/python2.7/site-packages/marionette_client-0.8.5-py2.7.egg/marionette/runner/base.py", line 710, in run_tests
    self.start_marionette()
  File "/home/alex/luciddream/ve/local/lib/python2.7/site-packages/marionette_client-0.8.5-py2.7.egg/marionette/runner/base.py", line 655, in start_marionette
    self.marionette = Marionette(**self._build_kwargs())
  File "/home/alex/luciddream/ve/local/lib/python2.7/site-packages/marionette_client-0.8.5-py2.7.egg/marionette/marionette.py", line 520, in __init__
    assert(self.wait_for_port()), "Timed out waiting for port!"
AssertionError: Timed out waiting for port!

I do see firefox opening, but not b2g. Could it be an issue with the profile being passed to b2g desktop??
Ted, I'm making good progress on writing first test scripts against WebIDE.
Do you have any update about running lucciddream on CI?
(In reply to Alexandre Poirot [:ochameau] from comment #13)
> I'm wondering if that regressed?
> I'm seeing the following exceptions now:
> 
Are you still having this problem?
(In reply to Jonathan Griffin (:jgriffin) from comment #15)
> Are you still having this problem?

No, I got it to work by running against a custom, up-to-date b2g desktop.
Ted, I filled various bugs about lucciddream https://github.com/luser/luciddream/issues
 - in js tests: assertion failure are not displayed until finish() is called
 - in js tests: successful assertions are not displayed at all
 - in js tests: line reported when an exception happen is wrong
 - I had to patch __init__.py in order to prevent an exception about logger being undefined
I can contribute if you tell me where the related code lives.

So far, I've had a good experience with the test harness as-is.
I'm currently looking at trying to use luciddream to run existing mochitest-devtools (mochitest browser ones),
but against devtools connected to a b2g runtime! It may require some of your help in order to find and filter such tests via browser.ini (or similar) manifest files... but I'm still evaluating this option. I already have very useful new test scripts that doesn't rely on mochitests.

Also I was wondering if you are still planning to run luciddream on CI this quarter?
Flags: needinfo?(ted)
Yes, we still plan on getting this running in CI this quarter, although I or someone else will likely be doing most of the work.
(In reply to Alexandre Poirot [:ochameau] from comment #17)
> Ted, I filled various bugs about lucciddream
> https://github.com/luser/luciddream/issues
>  - in js tests: assertion failure are not displayed until finish() is called
>  - in js tests: successful assertions are not displayed at all
>  - in js tests: line reported when an exception happen is wrong
>  - I had to patch __init__.py in order to prevent an exception about logger
> being undefined
> I can contribute if you tell me where the related code lives.

Alex: these sound mostly like bugs in the marionette-client code, which lives here:
https://dxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/

You can file bugs in Testing: Marionette for issues in that code. The run_js_test code is here:
https://hg.mozilla.org/mozilla-central/annotate/2cb22c058add/testing/marionette/client/marionette/marionette_test.py#l495

but we haven't released a version of the marionette-client package that has that function yet, so Luciddream is still using its own copy and pasted version:
https://github.com/luser/luciddream/blob/c6412d90186f32f705ed23af4d7d8a6b5a25170b/luciddream/__init__.py#L17

I just pushed a tiny fix to make that method use the base class method if it exists so that when we release a new version of marionette-client it will use it. (We can remove that once a release exists if we require the latest version of marionette-client in requirements.txt.)

If you want to hack on the marionette-client code and test it with Luciddream, you can do the following (assuming you've activated a virtualenv for Luciddream in the current shell):
(cd /path/to/mozilla-central/testing/marionette/client; python setup.py develop)

That will install the marionette-client from m-c as symlinks in your virtualenv so you can make changes to the code and re-run Luciddream to test them.

We have folks on the a-team with way more marionette-client experience than me, so you should be able to find help in #ateam or by CCing the right people on bugs.
Flags: needinfo?(ted)
Lucciddream running with a poor's man bash script on taskcluster:
  https://treeherder.allizom.org/#/jobs?repo=try&revision=27e863e8d9d9
\o/
very cool!  Is there a list of todo items that you know of left to do with this?  The more we can offer up the opportunity for others to help out a little here and there the faster we can get this done!
I think we probably want to use a mozharness script here, rather than a bash script...see bug 1128099.  It may not be an absolute requirement, but will make things much much easier to maintain.
TBH, I don't really care how we run it, as soon as we run it sooner than later.
If you think using mozharness is better, fine, it is just easier for me to use bash script to get something running today and see what could go wrong in running this new test suite.
I've been working on a new mach command and also landing luciddream on m-c as it would make it much easier to run!
But I wish I could reuse this mach command from taskcluster, but I'm not sure that is the right way to process.
I would like to avoid having multiple cli interfaces for luciddream... and above all, avoid having subtle differences in how to run it!
Attached patch Integrate luciddream into m-c (obsolete) — — Splinter Review 
And provide ./mach luciddream
Speaking about todo, I haven't had time to look into marionettes issues mentioned in comment 17.
Depends on: 1137285
We can't currently use mach commands in test jobs, whether TaskCluster or other, since we don't have a an objdir in that environment.  In Q2, we'll likely modify our infrastructure jobs to use mach via mozharness, so a mozharness script is still probably the best way to go here.
I improved the mach command in order to work without any objdir,
and also be compatible with try environment (fix various unexpected python import issue on try...).
So that running tests on CI and locally are similar.
I really want to strip any difference between running tests locally and on CI.

As you said, that would be great if mozharness was only downloading the builds and calling mach.
Is there more value in mozharness than just downloading the right os-specific packages?
Is there something about chunks, *.ini files used to whitelist/blacklist tests (I still miss that), or the way we log?
May be we could plug this patch into yours and make mozharness call this mach command?

Ted, would you agree on landing luciddream in m-c and fade it behind such mach command?
The main drawback is that luciddream doesn't run independently, it becomes a feature of mach.
Attachment #8571898 - Flags: feedback?
https://treeherder.allizom.org/#/jobs?repo=try&revision=a1cfd7987376
luciddream, running on TC, using the in-mozilla-central luciddream, via ./mach luciddream.
Attachment #8569945 - Attachment is obsolete: true
Oops that was on old, broken TC link, here is the good one:
https://treeherder.allizom.org/#/jobs?repo=try&revision=037703198199
Comment on attachment 8571898 [details] [diff] [review]
Integrate luciddream into m-c

Review of attachment 8571898 [details] [diff] [review]:
-----------------------------------------------------------------

f+, but please add the missing __init__.py to this patch.  I think it makes sense to add the example-tests as well.

Note, I had to update __init__.py due to some recent Marionette changes, so use the version from https://github.com/mozilla/luciddream.

Re: mach and mozharness.  mozharness is basically a bootstrapper.  Besides downloading appropriate bits, it also sets up the correct Python environment, and intelligently handles things like timeouts in a way that our infrastructure is able to handle.  

Mozharness is going to be moved into the tree, and mach and mozharness entry points unified, probably done in Q2.  I'd rather not try to do that here as a one-off, since it would likely be inconsistent with the architecture being developed.
Attachment #8571898 - Flags: feedback? → feedback+
Regarding TaskCluster vs buildbot, I'll leave the choice up to you.  It wouldn't be hard to get this running in buildbot, and then these would be visible in Treeherder per-push to all our usual integration branches.

If you go the TaskCluster route (which is very cool you were able to hook this up), your tests won't be showing up in the primary Treeherder views and so won't be sheriffed, until some additional work is done on the Treeherder and TaskCluster projects to unify views for jobs coming in from those sources.  This likely will happen in Q2.  It also means you won't get OSX or Windows tests until TaskCluster is updated to handle those, and I have no ETA on that.
Depends on: 1141632
Depends on: 1142833
Depends on: 1145873
Comment on attachment 8571898 [details] [diff] [review]
Integrate luciddream into m-c

I'm going to split this patch into two parts, and update it.
Depends on: 1147029
Depends on: 1147031
Depends on: 1148569
Depends on: 1153060
Assignee: ted → nobody
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: