1121696 - Experience luciddream to test WebIDE connected to simulators/emulators

Status: RESOLVED WONTFIX

(DevTools Graveyard :: WebIDE, defect)

Description

[Alexandre Poirot [:ochameau]]

Ted provided a new test harness (https://github.com/luser/luciddream) that allows writing test scripts involving ( Fx Desktop and ( b2g-desktop or emulator ) ) so that we can finally cover all WebIDE <-> b2g codepaths that are used to regress.
We can finally cover connecting to a simulator and see if WebIDE correctly detect runtime apps, install an app with the regular production codepath, open a toolbox in WebIDE in a test, ...

Now it is time to write such tests and contribute to luciddream in order to shape it for our particular needs. For example, right now, we don't control b2g lifecycle, b2g is being run before the test script starts, also, gaia profile is being built by the automation tool. That may be something that prevent us from testing particular scenarios. Writing tests is going to allow us seeing what we could possibly miss or need to tweak.

Comment 1

[Alexandre Poirot [:ochameau]]

Here is a WIP test showing that we can already cover quite a lot of exciting stuff we were dreaming covering:
https://github.com/ochameau/luciddream/commit/303836c58516d637ea67adbe71fff1e01a3b8654
- register a fake runtime in WebIDE in order to be able to connect to b2g desktop on a arbitrary test port,
- click on this fake runtime to connect to it
- select main process app
- toolbox automatically opens, now select console panel and eval something

This WIP confirms that we can cover way more, but it doesn't involve complex scenarios where you want to call code from both WebIDE *and* b2g. Also, I'm most likely trying to re-write a basic webconsole test here, we may want to leverage existing test scripts on luciddream??

Comment 2

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Setting ni? to look at this in more depth.

Comment 3

[Alexandre Poirot [:ochameau]]

Improved test script testing the inspector:
https://github.com/ochameau/luciddream/blob/test-webide/alex-tests/test_chrome.js

Comment 4

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Looking at Alex's tests so far, everything looks good from what I can see.

Leveraging existing tests would be amazing, but even being able to write these simple cases to check for the tools at all is great too.

This setup looks like it allows us to perform the type of testing we're looking for.  Good progress Alex and Ted!

Comment 5

[Alexandre Poirot [:ochameau]]

I pushed yet another test script, still in the same file:
https://github.com/ochameau/luciddream/blob/test-webide/alex-tests/test_chrome.js
Where I'm using existing mochitests, but against b2g runtime!
Here I'm only testing one inspector test that is known to work, but more are working fine.
I think this is an interesting idea as most tests e10s compatible should be abstracted enough to not play with DOM nodes. Actually they do, by using CPOW, but I think it is reasonable to say most tests shouldn't.
I already discussed with Patrick, just about Inspector mochitests and it is acceptable to make tests stop using getNode/CPOW and pass selector strings to the child process.
Also various tests do: content.location = ..., but they should be using TabActor.navigateTo instead.

Comment 6

[Alexandre Poirot [:ochameau]]

I think I got a quite good picture about converting inspector tests in order to be remote friendly.
It is quite substantial work, but I don't think it pollutes the tests themself while adding quite a lot of value.
I opened bug 1137285 in order to move forward on this.

Comment 7

[Alexandre Poirot [:ochameau]]

Attachment: wip patch

Here is a work in progress patch, converting just a few tests.
I can already split this patch in meaningful subset,
but I would like some feedback from inspector peers in order 
to ensure I'm sane and that this contribution will be accepted ;)

Comment 8

[Alexandre Poirot [:ochameau]]

Attachment: Run inspector test on luciddream

Here is the luciddream magic to make it run.
I'm not really satisfied by this as it contains
a list of whitelisted files to run in the Javascript helper file.

Comment 9

[Alexandre Poirot [:ochameau]]

Attachment: Introduce first luciddream test in m-c

But before introducing these complex inspector tests, 
I think it would be great to run such simple test first. 
Just to catch the big regressions we used to face.

Comment 10

[Alexandre Poirot [:ochameau]]

Comment on attachment 8569951 [details] [diff] [review]
wip patch

Oops attached to the wrong bug...

Comment 11

[Alexandre Poirot [:ochameau]]

Attachment: patch v1

Updated patch to work against current master, and hooked on new buildbot integration of luciddream.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=11c076c136c4