1312823 - Run mocha tests with TaskCluster

Status: RESOLVED FIXED

(DevTools :: Console, defect, P2)

Description

[Lin Clark [:linclark]]

We have unit tests that require Node so that we can use mocha and Enzyme. We should run these in CI.

How to run them:

- cd devtools/client/webconsole
- npm install
- npm test

Comment 1

[Lin Clark [:linclark]]

Alex, if you can help us get started with TaskCluster, that would be great. The test commands are above.

Comment 2

[Alexandre Poirot [:ochameau]]

I imagine there is an additional parameters to specify where firefox binary lives?

Could we get them running via ./mach somehow?
I don't know about node/npm I don't see any existing taskcluster task using that.
I'm not sure that's available in existing docker images.
We can surely come up with our own docker image, but I never did that and don't known what it takes to create and use a new one.

Taskcluster is somewhat easy. All the scripts live here:
http://searchfox.org/mozilla-central/source/taskcluster/ci/

So. To introduce a new test runner, it is as simple as editing:
http://searchfox.org/mozilla-central/source/taskcluster/ci/desktop-test/tests.yml
And copy for example the "mochitest" section into a "devtools-console" one.
Here is a tweaked example, where I replaced the usage of the magic "mozharness" thing, which is a super helper made in python to run stuff around firefox. Instead it uses a "run-task" rule that allows arbitrary command execution.

devtools-console:
    description: "Devtools console unit tests"
    suite: devtools/console
    treeherder-symbol: tc-Dt-c()
    loopback-video: true
    max-run-time: 5400
    run:
        using: run-task
        command: >
            cd /home/worker/checkouts/gecko/devtools/client/webconsole &&
            wget -O build.tar.bz2 {{build_url}} &&
            tar jxvf build.tar.bz2 && # if that {{build_url}} thing still works, you should have firefox here: ./firefox/firefox-bin
            npm install &&
            npm test
    instance-size: legacy
    allow-software-gl-layers: false

(I don't know what some fields are about and just keep them from mochitest rule)

Then you would have to register this new test rule in this file:
http://searchfox.org/mozilla-central/source/taskcluster/ci/desktop-test/test-sets.yml#12

There is some docs over there:
http://searchfox.org/mozilla-central/source/taskcluster/docs

You may find valuable help on #taskcluster if the right people are online when you are asking for help.

Comment 3

[Nicolas Chevobbe [:nchevobbe]]

Attachment: Bug 1312823 - Add devtools console mocha tests

Review commit: https://reviewboard.mozilla.org/r/128856/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/128856/

Comment 4

[Nicolas Chevobbe [:nchevobbe]]

Comment on attachment 8856915 [details]
Bug 1312823 - Add devtools console mocha tests

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/128856/diff/1-2/

Comment 5

[Nicolas Chevobbe [:nchevobbe]]

Comment on attachment 8856915 [details]
Bug 1312823 - Add devtools console mocha tests

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/128856/diff/2-3/

Comment 7

[Nicolas Chevobbe [:nchevobbe]]

Attachment: Bug 1312823 - Fix console mocha tests. r=jdescottes.

This adds missing preferences to mocha-test-setup.js.

Comment 8

[Nicolas Chevobbe [:nchevobbe]]

Attachment: Bug 1312823 - Run console mocha tests on TRY. r=jdescottes.

To make the test pass, we need to define URLSearchParams
if it doesn't exist.

Depends on D25044

Comment 9

[Alexandre Poirot [:ochameau]]

• What about trying to run each test suite individually?
  It looks like we could do a better at running only the webconsole tests when only the webconsole folder is changed, only framework when framework is changed and so on.

• Is there any plan to integrate these node tests into *.ini files and ./mach test?
  Supporting ini file would help being tier 1/2 and be "sheriffable".
  While supporting ./mach test would help running them when you are not used to DevTools specifics.

Comment 10

[Julian Descottes [:jdescottes]]

It looks like we could do a better at running only the webconsole tests when only the webconsole folder is changed, only framework when framework is changed and so on.

It is an option. Felt like an overkill for "framework" and "aboutdebugging" right now, because they both have just one test. My idea was to wait until we had more tests written, but if you feel strongly about this we can change it here.

Is there any plan to integrate these node tests into *.ini files and ./mach test?
Supporting ini file would help being tier 1/2 and be "sheriffable".
While supporting ./mach test would help running them when you are not used to DevTools specifics.

Would be nice, I don't know if this is being worked on. Note that this is why we added READMEs in the jest test folders for framework & aboutdebugging. I think the list of things to improve for those tests is long: error reporting, mach support, sheriffing.

Comment 11

[Alexandre Poirot [:ochameau]]

(In reply to Julian Descottes [:jdescottes] from comment #10)

It looks like we could do a better at running only the webconsole tests when only the webconsole folder is changed, only framework when framework is changed and so on.

It is an option. Felt like an overkill for "framework" and "aboutdebugging" right now, because they both have just one test. My idea was to wait until we had more tests written, but if you feel strongly about this we can change it here.

TBH I'm fine with the current in-flight patch. I mostly wanted to raise these two points so that they are being addressed at some point. Followups and proper staffing on this would be great!

Comment 12

[Jason Laster [:jlast]]

Here is a meta for improving the node tooling bug 1535113. I'll add two bugs for *.ini and mach integration

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4fe8d3976d58
Fix console mocha tests. r=jdescottes.
https://hg.mozilla.org/integration/autoland/rev/4338bf79beb2
Run console mocha tests on TRY. r=jdescottes.

Comment 14

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/4fe8d3976d58
https://hg.mozilla.org/mozilla-central/rev/4338bf79beb2