1021567 - Coverage threshold check in test agent file watcher

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::TestAgent, defect)

Description

[Ricky Chien [:rickychien] (inactive)]

After bug 991442 was landed, we could enable coverage threshold check while modifying source code or test code. It would be a good way to notify programmer about any coverage change when their add new code but forget to write tests. Through coverage threshold, we will output a warning message if the current coverage rate is lower than threshold.

Comment 1

[Ricky Chien [:rickychien] (inactive)]

Attachment: Gaia PR

Comment 2

[Ricky Chien [:rickychien] (inactive)]

Attachment: Js-test-agent PR

Comment 3

[Ricky Chien [:rickychien] (inactive)]

Attachment: Gaia-node-modules PR

Comment 4

[Ricky Chien [:rickychien] (inactive)]

Attachment: Gaia PR

Comment 5

[Ricky Chien [:rickychien] (inactive)]

Comment on attachment 8439083 [details] [review]
Js-test-agent PR

Patch works properly after rebasing to master and testing.

Watcher will re-run unit test and coverage per file you change, So coverage threshold checker also will test that file every time according to your module threshold value. I think it's reasonable.

Comment 6

[Julien Wajsberg [:julienw]]

Comment on attachment 8439083 [details] [review]
Js-test-agent PR

r=me

It works, but I see the same issue I outlined for /shared, but for other files now.

For example, I tried saving "apps/sms/test/unit/localization_helper_test.js". The tests run, and I have the following output:

  -- Test Coverage for Sms --
  
  File Name - Covered/Total Smts - Coverage (%)
  
    http://sms.gaiamobile.org:8080/js/utils.js - 12/256 - 4.69 %
    http://sms.gaiamobile.org:8080/js/localization_helper.js - 33/33 - 100 %
  
  Global Total - 45/289 - 15.57 %
  
  Threshold = 0%, COVERAGE PASS
  
    8 passing (145ms)
  
  SUMMARY COVERAGE PASS: 3/3 files passed coverage


See the line for "utils": we should _not_ test the coverage for this file, because it's not the tested file :/

See also the line "3/3 files": it's not reset after each run, so it's always incremented. If I save the file once more, I get "5/5".

A last suggestion:
after "SUMMARY COVERAGE PASS: x/x files passed coverage" we should have a link to a documentation explaining how to configure the threshold.


So, that means you can land this, but we need 3 more bugs for the 3 issues outlined above.

Comment 7

[Ricky Chien [:rickychien] (inactive)]

Since localization_helper_test.js require:

require('/js/utils.js');
require('/js/localization_helper.js');

require('/shared/test/unit/mocks/mock_l10n.js');

All shared/ has been skipped now so that you saw utils.js was included in report. Maybe we should just track source_name.js if someone executes source_name_test.js, and make sure all of these source_name.js correspond/map perfectly to source_name_test.js. If that's right, I'm going to patch it.:)

Comment 8

[Ricky Chien [:rickychien] (inactive)]

PR updated for addressing incorrect "3/3 files" numbers and explaining how to configure the threshold.

Comment 9

[Ricky Chien [:rickychien] (inactive)]

Attachment: Gaia-node-modules PR

Comment 10

[Ricky Chien [:rickychien] (inactive)]

Merged.

https://github.com/mozilla-b2g/gaia-node-modules/commit/5d3091a0c3bb50ea51c753806cff7edcc184ab6e

https://github.com/mozilla-b2g/js-test-agent/commit/3c8e8a26048d86fd24e3ef9d0095c1d12b446420

https://github.com/mozilla-b2g/gaia/commit/155f9c1b27a0603703675e79fe79e7e2cc0807a2