837581 - Add travis support for gaia unit testing

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia, defect)

Description

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

travis-ci.org is a continuous integration web service for project hosted by github, and it's free for open source project.

travis build process will be triggered by someone send a pull request to gaia github repository, so we can get test result before code landed.

for example Tim send a pull request to me and build process was triggered automatic and shown result on pull request page.
https://github.com/yurenju/gaia/pull/1

and there is the build history for every pull request:
https://travis-ci.org/yurenju/gaia/builds

it can be help us for improving code quality and ensure every commit passed unit tests before landed.

However, I would like to get more feedback for travis support. please let me know if you have any problems.

Comment 1

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

WIP: https://github.com/yurenju/gaia/commit/8514eebfbf9217edee31d588ec2db4097a3636f4

Comment 2

[Julien Wajsberg [:julienw] (PTO -> Aug 14)]

Mocha is supposed to be able to output junit-like log file, that should be understandable by any CI software.

Comment 3

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

I really feel like we need bug 802317 to be fixed but since I have no idea how long it will took let's try to understand what this would provide.

What type of tests will it allow us to run? Unit tests only? or is it possible to run integration tests on a desktop build as well?

Comment 4

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Julien,
travis-ci doesn't support junit integration[1] now, but we still can upload junit format test report to somewhere or use API to get the result.

Vivien,
Yes we can run gaia-ui-tests or other tests on travis, but travis has build time limitation about 25 minutes[2]. I will add gaia-ui-tests on travis and measure how much time we spend on gaia-ui-tests.


[1] https://github.com/travis-ci/travis-ci/issues/239
[2] http://about.travis-ci.org/docs/user/build-configuration/#Build-Timeouts

Comment 5

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(In reply to Yuren Ju [:yurenju] from comment #4)
> Yes we can run gaia-ui-tests or other tests on travis, but travis has build
> time limitation about 25 minutes[2]. I will add gaia-ui-tests on travis and
> measure how much time we spend on gaia-ui-tests.

With the coming long holiday in Taipei, if this take more than 1 day, I would tend to think that we should enable travis-ci for Gaia unit test first, and modify test script afterwards.

Additionally, in order to keep the badge green (so that the future offender would know s/he breaks something), we should land another commit that disable all the out-dated failed tests, and file bugs against them to have them fixed.

If that's fine with everyone, Yuren can wrap his WIP work into patch review (who should review it?)

Comment 6

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Attachment: pull request

as Tim said, we have a long holiday in Taipei, so I would like to submit the pull request for initial travis integration and we can improve it afterward.

Comment 7

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Comment on attachment 710104 [details]
pull request

Redirect the review request to Vivien.

The only concern of mine is that some of the scripts are dups of what James had done for botio, but I am not sure if we are even going to maintain that. I am sure we could figure out how things should put together afterward.

Comment 8

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

Comment on attachment 710104 [details]
pull request

r+ if you fix the script to make sure we don't fall into and endless loop. My wish for this script is that it will be removed soon but that a good way to start having green tests imho. Thanks for the work.

Comment 9

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Merged.
https://github.com/mozilla-b2g/gaia/commit/0f31fd5abf3f67c1ba4d826d50ad3fddb95bf801

Comment 10

[Julien Wajsberg [:julienw] (PTO -> Aug 14)]

Comment on attachment 710104 [details]
pull request

I'd like this to land on v1-train and v1.0.1 as well, so that unit tests run on these branches too.

Now we have error notices from Travis for each merge because we don't have this there. Also, it makes a lot of sense to run unit tests on these branches too.

Comment 11

[Julien Wajsberg [:julienw] (PTO -> Aug 14)]

Marking tef? so that this lands on v1.0.1.

No risk whatsoever, we only add new no-code files.

Comment 12

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 710104 [details]
pull request

Patch only affects travis support so marking npotb and blocking so no need for approval.

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

Uplifting commit 0f31fd5abf3f67c1ba4d826d50ad3fddb95bf801:
v1-train: ccc560eb6722daa32772d7fa0494201abd6a9962
v1.0.1: 05b77466975143469d62875a0fc703c264768c18

Comment 14

[amiller]

Does not make sense to create a regression issue.

Comment 15

[Darren]

Cannot verify, need steps to blackbox test this issue.

Comment 16

[Julien Wajsberg [:julienw] (PTO -> Aug 14)]

check that https://travis-ci.org/mozilla-b2g/gaia/builds has builds for all branches.