825913 - Integrate code coverage library in the test agent

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia, defect)

Description

[Julien Wajsberg [:julienw]]

It would be useful to have a test coverage library :

For exeample blanket [1] could be a good match.

[1] http://migrii.github.com/blanket/

Comment 1

[[e-mail]]

I've setup a working browser(client-side) coverage(via blanket.js + mocha) in webapplate project to prove it works
http://github.com/gasolin/webapplate

But there are several difference between browser coverage and test-agent build.
We'd applied a program for Taiwan Summer of Code(TSoC) project that fund an project to help us implement code coverage in the test agent.

Yuren, Alive, and I will help mentor Ricky to

Comment 2

[[e-mail]]

Yuren, Alive, and I will mentor `Ricky` to integrate code coverage library in the test agent, and try to increase test coverage/improve code quality (start from system app).

It will be helpful if Julien/james/kevin can provide some suggestion about the test-agent integration.

Comment 3

[Julien Wajsberg [:julienw]]

note there is a bug to integrate code coverage information inside SpiderMonkey: bug 651621. Not much progress there lately though. There is also bug 750364 (even if I don't get where it is right now).

Comment 4

[James Lal [:lightsofapollo]]

test-agent itself is basically one big proxy so we can push results out into your terminal. This also handles sandboxing and aggregating results so it all looks like one test run (rather then 1*apps).

My suggestion would be to look at the various code coverage tools and look at https://github.com/lightsofapollo/js-test-agent and see how they fit together.

Most of the tooling I have seen directly modifies the JS under test but there might be something fancier at the gecko level I don't know about... In theory we could extend the server in test-agent to serve modified JS files that report their test-coverage which we could then export out to disk after the tests run per-app.

Comment 5

[Julien Wajsberg [:julienw]]

The coverage library that are present in Mocha by default use an instrumented javascript engine (so that they don't need to modify the JS files, and they can be a lot more precise about the actual covered lines and branches vs the covered methods)

Comment 6

[[e-mail]]

Thanks for your valuable comments, I think we all(ricky, yuren, alive, me) will learned more about testing during the process of improving the test-agent.


In my previous research, blanket.js is pure js, has both browser test and server test support, and have mocha coverge reporter integration

(demo site) blanket.js browser test with mocha
http://alex-seville.github.io/blanket/test/mocha-browser/adapter.html


test-agent use some kind of browser test, so the first take might not that complicate (may not need gecko-side patch?)

Comment 7

[Julien Wajsberg [:julienw]]

I'd be happy with a js-only solution for now (better than nothing :) ), but long-term we need to do real instrumentation to know the real coverage.

Comment 8

[Ricky Chien [:rickychien] (inactive)]

Hello everybody, I'm Ricky, a master student in NCU CSIE.
Studying in Object-oriented programming and automatic testing.
I'm glad to work with mozilla :)

Forgive me I was a little busy in last semester.
But so far still have some progress to report

1. Building gaia (Reported and fixed 2 bug).
2. Getting started with gaia test and try to understand how gaia test work.
3. Review javascript tutorial in MDN.
4. For practice javascript, I wrote a nodejs project and push to github.
5. Wrote another simple nodejs server and import mocha to unit testing.
6. Add blanket.js (as Fred Lin said) to my nodejs server project and know how does it work with       mocha.

I'm not familiar with javascript and also some of the technology firefox os used.
So, I always get stuck and hope everybody can help me :)

Comment 9

[[e-mail]]

Thanks Ricky to take this task :)


There will be `test-agent/config.json` file. It gathered all gaia app unittest cases which are generated by `make install-gaia` command.


The next step is to figure out how those testcase are tested / learn how coverage tool works, then patch test-agent app/server with some test coverage tools.

Comment 10

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

FYI: I found Chai[1] assertion library running a coverage service "coveralls"[2], we may consider to integte it just like Travis :D

[1] https://github.com/chaijs/chai
[2] https://coveralls.io/r/chaijs/chai?branch=master

Comment 11

[Ricky Chien [:rickychien] (inactive)]

Hello!

I'm beginning to survey how to integrate blanket.js to test-agent, but have some questions.

Blanket.js can launch by using Mocha and run test coverage directly. So, the simplest way to use blanket.js is to use mocha --require blanket if we launch mocha in normal way.
Instead of launching mocha in normal way, test-agent wrap the mocha and only invoke mocha reporter to test. I have no idea why test-agent doesn't launch mocha directly.
So, when integrate blanket.js, I also need to survey how mocha invoke blanket.js and wrap some part of code like the way of wrap mocha in test-agent right? Thanks :)

Comment 12

[James Lal [:lightsofapollo]]

@Ricky

Are you sure your not mixing terms? "mocha -- require blanket" is only for node style mocha tests.. test-agent runs code in the browser and reports to a terminal which is not quite the same thing.

Comment 13

[Ricky Chien [:rickychien] (inactive)]

Here is my new progress from 2 weeks ago:

1. Understood the architecture of test-agent.
2. Getting started with node.js.
3. I found some part of javascript in test-agent is not easy to understand, so I started to read a book call javascript patterns.
4. Studied the implemention of test-agent(include client side and server side).

Now, I'm starting to integrate blanket.js to test-agent, but encounter some obstacles I posted above.

Comment 14

[Ricky Chien [:rickychien] (inactive)]

@James Lal

Thanks your reply :)

What is test-agent runs code in the browser? I know test-agent will check the environment and then choose to run in node or browser. Are they different?

Comment 15

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Hi Ricky,

you should figure out how do we setup coverage report correctly on mocha in browser environment first, and then find a point to add coverage support in our test-agent environment.

Comment 16

[James Lal [:lightsofapollo]]

A quick look at https://github.com/alex-seville/blanket/blob/master/docs/getting_started_browser.md implies that we can easily integrate this tool by making a small change to how "require" is implemented in test agent so we can add data-* options via some configuration. From there we can just alter our sandbox files to include blanket based on some env variable ?

I did not look at how to bubble up the reporting information after that but that is more where the TA integration comes in I think.

Comment 17

[James Lal [:lightsofapollo]]

test-agent never actually runs test in node- the node component is simply for reporting/aggregation of test results which are run in the browser.

You can see the gaia specific configuration here: https://github.com/mozilla-b2g/gaia/blob/master/test_apps/test-agent/common/test/agent.js

blanket.js would go here: https://github.com/mozilla-b2g/gaia/blob/master/test_apps/test-agent/agent.js

The "require" changes I mentioned would probably go here https://github.com/mozilla-b2g/js-test-agent/blob/master/lib/test-agent/loader.js. I would like a configuration option rather then just adding data-* attributes to all requires.

Comment 18

[Ricky Chien [:rickychien] (inactive)]

@James Lal

I'm trying to integrate blanket.js which needs to load source code out of the test-agent (such as browser app, calendar app...etc),  so I need to do some cross domain request in blanket.js for reading these sources.

My question is how can I do this in gaia? (Is gaia has built-in function to send cross domain request?)

Comment 19

[James Lal [:lightsofapollo]]

Cross domain request like XHR or cross domain request like a script tag?

Comment 20

[[e-mail]]

per offline discussion, I think ricky means XHR. 

He traced blanket.js and find the `getFile` function is doing XMLHTTPRequest
https://github.com/alex-seville/blanket/blob/master/dist/qunit/blanket.js#L5280

and will raise showManualLoader while can't send xhr (#L5313)

Comment 21

[James Lal [:lightsofapollo]]

So why do we need to run cross domain requests? IIRC all of our code is mapped to $DOMAIN_NAME/shared in the shared case and runs inside of the correct domain in other cases? So in theory all the requests should be in the same domain as the iframe running the tests.

Comment 22

[Ricky Chien [:rickychien] (inactive)]

@James Lal

Because blanket.js will integrate into test-agent app where its domain is in "test-agent.gaiamobile.org:8080".
And blanket also need to load source file such as "browser.gaiamobile.org:8080/js/browser.js", cause the cross domain request issue.

Blanket provides showManualLoader to upload local file when xhr fail, but it uses html input file approach to let user choose upload file manually. I think this is a stupid way.

Comment 23

[Ricky Chien [:rickychien] (inactive)]

@James Lal

After discussion today, we determined to integrate blanket like integrating sinon or mocha.
Using test-agent sandbox to cover sources for each APP and then retrieve the result from each sandbox. Finally, we show up the coverage result like mocha show its test result and also calculate coverage percentage of all tests.

We thought this is an ideal way to integrate blanket.

Could you give me some suggestion because I want to know where is the entry point that mocha be launched in each sandbox and how mocha sends the test result to outside html?

Thanks :)

Comment 24

[James Lal [:lightsofapollo]]

Right now test-agent uses a custom mocha reporter [https://github.com/lightsofapollo/js-test-agent/blob/master/lib/test-agent/mocha/json-stream-reporter.js](its ugly right now) to export mocha's internal test format out to node via websocket (which then uses mocha's tooling to display results after we aggregate the data).

I don't have much context into how blanket works or on what level it reports information? I expect that it exports something that can be sent over websocket which could then be processed somehow.

If its helpful we could have a chat (on irc or vidyo).

Comment 25

[Ricky Chien [:rickychien] (inactive)]

Attachment: github_pullrequest.txt

Comment 26

[(inactive after 6/18) Alive Kuo [:alive]]

Comment on attachment 785625 [details] [review]
github_pullrequest.txt

I left some coding style comments on github.
Ricky said this is a WIP but blanket just works.
Need Feedback from James.

Comment 27

[[e-mail]]

WIP: Ricky has make blanket.js work with test-agent to show coverage results 'in browser'.

here's a short video
https://dl.dropboxusercontent.com/u/18302699/blanket-demo.avi

The result is within the app iframe.

The next step might be find a way to redirect results to console.

Comment 28

[Ricky Chien [:rickychien] (inactive)]

Attachment: 825913.html

Comment 29

[Ricky Chien [:rickychien] (inactive)]

Attachment: 825913.html

Comment 30

[Ricky Chien [:rickychien] (inactive)]

Attachment: Pull request url

Comment 31

[Ricky Chien [:rickychien] (inactive)]

Attachment: Pull request url for js-test-agent

Comment 32

[Ricky Chien [:rickychien] (inactive)]

https://dl.dropboxusercontent.com/u/18302699/blanket-system.png

Here is a demo picture that shows the result of testing some of system test cases.
Now, I have got mocha assertion errors in some of test cases. So I need to against to each test case and find why integrating blanekt cause these errors.

Comment 33

[(inactive after 6/18) Alive Kuo [:alive]]

(In reply to Ricky Chien from comment #32)
> https://dl.dropboxusercontent.com/u/18302699/blanket-system.png
> 
> Here is a demo picture that shows the result of testing some of system test
> cases.
> Now, I have got mocha assertion errors in some of test cases. So I need to
> against to each test case and find why integrating blanekt cause these
> errors.

\O/
Nice to have request: show all the modules without any test 0/line#
and flag a special style of 0% ;)

Comment 34

[Ricky Chien [:rickychien] (inactive)]

Here is my current test case pass/fail status:
https://www.evernote.com/shard/s104/sh/8932605d-0f3d-46c6-9d07-84f1a449ac60/c3f78cbd70d7b38d9f70ba3732ee9a48

I found a lot of issue when I testing all of the test cases. Sometimes testing encountered fail but sometimes pass(Maybe there are dependence). I fixed most of the fails and I discovered the major reason is blanket will run the source script again that cause some script be executed twice.
So far still have some fails in testing need someone help me to get rid of it.

Comment 35

[Ricky Chien [:rickychien] (inactive)]

As above link mentioned. I found the point where calendar/test/unit/app_test.js will block later test scripts because at app_test.js line:239 "window.dispatchEvent(...)" cause this blocking behavior(but I don't know why).

Comment 36

[Julien Wajsberg [:julienw]]

Events are synchronous, you should have a look to the event handler (app.js line 564). I don't know this code at all so I don't know what it exactly does ;)

Comment 37

[Julien Wajsberg [:julienw]]

Also, don't forget that some of the tests already fail in the current repository :)

Comment 38

[Ricky Chien [:rickychien] (inactive)]

I have added options to Makefile for switching coverage report by adding COVERAGE=1 in 'make test-agent-test' command.

Comment 39

[Ricky Chien [:rickychien] (inactive)]

Comment on attachment 788043 [details]
Pull request url for js-test-agent

><html>
>  <head>
>    <meta http-equiv="Refresh"
>    content="2; url=https://github.com/mozilla-b2g/js-test-agent/pull/40" />
>  </head>
>  <body>
>    Redirect to pull request #11452
>  </body>
></html>

Comment 40

[Ricky Chien [:rickychien] (inactive)]

Comment on attachment 788043 [details]
Pull request url for js-test-agent

><html>
>  <head>
>    <meta http-equiv="Refresh"
>    content="2; url=https://github.com/mozilla-b2g/js-test-agent/pull/40" />
>  </head>
>  <body>
>    Redirect to pull request #11452
>  </body>
></html>

Comment 41

[Ricky Chien [:rickychien] (inactive)]

Attachment: Pull request link for js-test-agent

Comment 42

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Ricky, your commit break unit test for js-test-agent, please check it first.

https://travis-ci.org/mozilla-b2g/js-test-agent/jobs/10658378

Comment 43

[Ricky Chien [:rickychien] (inactive)]

Attachment: Pull request link for gaia

Comment 44

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Ricky, please provide steps to verify your work.

Comment 45

[Ricky Chien [:rickychien] (inactive)]

1. Apply this gaia patch at https://bugzilla.mozilla.org/attachment.cgi?id=796495
2. Clone js-test-agent (https://bugzilla.mozilla.org/attachment.cgi?id=795121) into gaia/tools
3. Apply js-test-agent patch.
4. If you want to watch blanket coverage report, add COVERAGE=1 flag such as "make test-agent-test APP=xxx COVERAGE=1".

P.S. There still have some unit test fails when running calendar with blanket.

Comment 46

[James Lal [:lightsofapollo]]

Comment on attachment 795121 [details]
Pull request link for js-test-agent

I tried not to get nit-picky here but there is one piece I am concerned about (the hardcoding of files to not run)

Comment 47

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Comment on attachment 796495 [details]
Pull request link for gaia

Ricky, the first step can by applied to master branch. set r? to me if you solved the conflict.

Comment 48

[James Lal [:lightsofapollo]]

Comment on attachment 795121 [details]
Pull request link for js-test-agent

https://github.com/mozilla-b2g/js-test-agent/commit/d8b0d7d0714181d26b3ad52e8830d582aef39284

Comment 49

[James Lal [:lightsofapollo]]

Comment on attachment 796495 [details]
Pull request link for gaia

Going to un-mark myself for review here on the gaia side... Yuren knows whats going on and aside from my comment on the config this looks pretty good... I do have a request to file a follow up bug:

- this only seems to work for content that is not lazily loaded (code that ends up on window that was not included via requireApp). For calendar this means it looks like zero coverage since in most cases we use our lazy loading utils in tests too.

This is very exciting and gives us lots of new things we can try to do in the future thanks for the hard work!!!

Comment 50

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Comment on attachment 796495 [details]
Pull request link for gaia

looks great for Makefile with nits addressed, we are ready to land this change ;-)

Comment 51

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

merged.

https://github.com/mozilla-b2g/gaia/commit/5ddf140f909ec3b57dc67b30738f59e6e6ebcf31

Ricky, James, Gaoslin & Julien thanks your contribution for this issue!

Comment 52

[[e-mail]]

During this issue's dev phase, gaia have increased test cases from 3258 to 4930. 
Test-coverage will be a good helper tool for developers to evaluate their unittest case coverage at early stage.

Congrats Ricky for your great work. Thanks James, Julien, Yuren, Alive for kindly mentoring and review this feature.

Comment 53

[Ricky Chien [:rickychien] (inactive)]

I'm very delighted this feature can greatly help a lot of gaia developer and improve the stability of system. Thanks James, Julien and my mentors Yuren, Gasolin and Alive help me for contribute gaia. During this summer I learned a lot of skill such as JavaScript, contributing open source, software testing, quality code, code review, teamwork process and experience greatly enthusiasm from Mozilla guys.

Thanks to give me this change and let me acquired greatly experience during the summer.