916652 - [marionette-debug] Post screenshot images to a server

Status: RESOLVED FIXED

(Testing Graveyard :: JSMarionette, defect)

Description

[Evan Tseng [:evanxd]]

We would like to implement the "marionette-debug" plug-in for debugging purpose when write integration tests.

And we could post the screenshot images to a server with following script:

client.plugin('debug', require('marionette-debug'), { server: 'http://192.168.1.63' });
// catch a screenshot
client.debug.screenshot();
// send the images to the server
client.debug.sendScreenshot();

Comment 1

[Evan Tseng [:evanxd]]

Hi James,

So we should implement a server-side to receive the images.

Will the server save the images as files in some folder in disk after it received images?
Or the server should create HTML files contained the screenshot images?

I think the second one(create HTML files) is better.
How do you think?

And we also should host a server with public IP to do the above things in CI.
Where could we host the server?

Comment 2

[James Lal [:lightsofapollo]]

So I think the client should be very dumb and just post the raw information it gets from marionette to a server. This server can do magic things (like create the html file [which I think is a good idea]).

For our current use case we can write a very dumb node server which can create an HTML file and use travis's http://about.travis-ci.org/blog/2012-12-18-travis-artifacts/ artifact system to give us some nice html results... Dave Hunt gave me a nice demo of what webqa did with this for gaia-ui-tests already.

---

Long term some things to keep in mind:

- we may move away from a strictly travis model so we need to post this very generically [and have some gaia/travis only steps]

- we might need to "tag" the screenshot data with some other data to identify where it came from. Maybe this is another thing which a proxy server could do to keep our current concerns very simple/easy.

Comment 3

[Evan Tseng [:evanxd]]

Attachment: Patch for server side

Hi James,

Please help me review the patch.
Thanks. :)

Comment 4

[Evan Tseng [:evanxd]]

Attachment: Patch for API side.

Hi James,

Please help me review the patch.

And We also need to create two new repository in mozilla-b2g in GitHub:
1. marionette-debug: for APIs side.
2. marionette-debug-server: for server side.

Then I could send the PR to these repositories.

Thanks. :)

Comment 5

[James Lal [:lightsofapollo]]

Comment on attachment 808494 [details]
Patch for server side

this is close but see my next set of comments

Comment 6

[James Lal [:lightsofapollo]]

Comment on attachment 808499 [details]
Patch for API side.

Hey Evan- the api looks sane but we are doing some things here that (at first) seem sane but are complicated by the fact that we assume all marionette calls are blocking but we are doing non-blocking operations.

How will the tests know when sending information is complete if we are always blocking the outgoing http connection ? A perhaps (more icky) but better choice would be to use sockit-to-me and json-wire-protocol OR implement some magic in marionette-js-runner so we can do normal non-blocking operations in a more sane way in the parent process while the child blocks.

Comment 7

[Evan Tseng [:evanxd]]

Attachment: Patch for server side V2

Hi James,

I updated the patch.
Please help me review the patch again.

Thanks.

Comment 8

[Evan Tseng [:evanxd]]

Attachment: Patch for API side V2

Hi James,

I updated the patch.
Please help review the patch.

Thanks. :)

Comment 9

[Evan Tseng [:evanxd]]

Hi James,

For comment 6, that HTTP request is asynchronous.
So it it non blocking, right?

See it in https://github.com/evanxd/marionette-debug/commit/d8a8e9770611b9422bafede8dc15fa19289ecbd1#diff-168726dbe96b3ce427e7fedce31bb0bcR54.

Have fun in the Summit. :D

Comment 10

[James Lal [:lightsofapollo]]

Comment on attachment 813000 [details]
Patch for server side V2

aside from one nit this is a good start... There are lots of optimizations I have in my head but we can go down that road in the future.

Comment 11

[Evan Tseng [:evanxd]]

Comment on attachment 813000 [details]
Patch for server side V2

><!DOCTYPE html>
><meta charset="utf-8">
><meta http-equiv="refresh" content="5;https://github.com/evanxd/marionette-debug-server/commit/0c7b45dbddd74bb89bd4042fd8a67afdbf0fd381">
><title>Bugzilla Code Review</title>
><p>You can review this patch at <a href="https://github.com/evanxd/marionette-debug-server/commit/0c7b45dbddd74bb89bd4042fd8a67afdbf0fd381">https://github.com/evanxd/marionette-debug-server/commit/0c7b45dbddd74bb89bd4042fd8a67afdbf0fd381</a>, or wait 5 seconds to be redirected there automatically.</p>

Comment 12

[Evan Tseng [:evanxd]]

Comment on attachment 813000 [details]
Patch for server side V2

><!DOCTYPE html>
><meta charset="utf-8">
><meta http-equiv="refresh" content="5;https://github.com/evanxd/marionette-debug-server/commit/0c7b45dbddd74bb89bd4042fd8a67afdbf0fd381">
><title>Bugzilla Code Review</title>
><p>You can review this patch at <a href="https://github.com/evanxd/marionette-debug-server/commit/0c7b45dbddd74bb89bd4042fd8a67afdbf0fd381">https://github.com/evanxd/marionette-debug-server/commit/0c7b45dbddd74bb89bd4042fd8a67afdbf0fd381</a>, or wait 5 seconds to be redirected there automatically.</p>

Comment 13

[Evan Tseng [:evanxd]]

Comment on attachment 813000 [details]
Patch for server side V2

><!DOCTYPE html>
><meta charset="utf-8">
><meta http-equiv="refresh" content="5;https://github.com/evanxd/marionette-debug-server/commit/0c7b45dbddd74bb89bd4042fd8a67afdbf0fd381">
><title>Bugzilla Code Review</title>
><p>You can review this patch at <a href="https://github.com/evanxd/marionette-debug-server/commit/0c7b45dbddd74bb89bd4042fd8a67afdbf0fd381">https://github.com/evanxd/marionette-debug-server/commit/0c7b45dbddd74bb89bd4042fd8a67afdbf0fd381</a>, or wait 5 seconds to be redirected there automatically.</p>

Comment 14

[Evan Tseng [:evanxd]]

Attachment: Patch for server side V3

Update for little nit in V2.

Comment 15

[Evan Tseng [:evanxd]]

Attachment: Patch for API side V3

Update the callback function.

Comment 16

[Evan Tseng [:evanxd]]

Hi James,

Could we add the below two new repositories in mozilla-b2g?
1. marionette-debug
2. marionette-debug-server

And the patch for API side is OK, or what should we update for it?
I just updated the callback function in the " Patch for API side V3" attachment.

Thanks. :)

Comment 17

[James Lal [:lightsofapollo]]

Comment on attachment 813002 [details]
Patch for API side V2

r+ (mostly because I would like to land something now)
--

I think we can be much more efficient by:

Overall: using a length + bytes format (like the json wire protocol) encoding an entire image in json and then reading it out over http is slow for various reasons. (we don't begin writing to disk until after the json is parsed)

On the client: Given we have a main / child process setup we can probably begin sending the image data immediately rather then buffering it all in memory until the end of the test (or when sendScreenshot is called)

Comment 18

[Evan Tseng [:evanxd]]

Hi James,

For sending the image data immediately,
how do we handle the below situation in server side?

There are two testing runners run test almost at same time.
Then the server will receive image data from these two runners separately.

How do we generate report correctly?
For example, insert right image data in a report.

So I the we need a database or a smart file manager in server side to resolve this?
Or other smart and simple solution?

Comment 19

[Evan Tseng [:evanxd]]

And could we add new repositories in mozilla-b2g(GitHub) for these two patch?
Then we could send PR to fix the issues more easily.

Comment 20

[Evan Tseng [:evanxd]]

Hi James,

So could we add two new repositories in mozilla-b2g for the patches?
Or what else should we do before we land it in mozilla-b2g? And we could do that.

Thanks.

Comment 21

[James Lal [:lightsofapollo]]

Ah! go for it Evan r+ means please land if I didn't for you already

Comment 22

[Evan Tseng [:evanxd]]

Hi James,

Sorry, but I don't have permission for creating a new repository in mozilla-b2g.
So could you help me for that.
Or could I apply the permission from somewhere?

Comment 23

[James Lal [:lightsofapollo]]

Hey Evan,

The right flow is to move your repo from your user to the org. Anyone can do this then someone who is an owner can set permissions (Tim is an owner, I am an owner, and there is a list under teams for the org)

Comment 24

[Evan Tseng [:evanxd]]

Hi James,

Cool, got it.
Thanks, I learned a new things and done. :D
https://help.github.com/articles/how-to-transfer-a-repository

Comment 25

[Evan Tseng [:evanxd]]

marionetee-debug master: https://github.com/mozilla-b2g/marionette-debug/commit/e2af9c66fc0cf3b5d2eef6fffc4fbf107462bcab

marionette-debug-server master: https://github.com/mozilla-b2g/marionette-debug-server/commit/0c7b45dbddd74bb89bd4042fd8a67afdbf0fd381