1343216 - Run every blocklist tests easily

Status: RESOLVED FIXED

(Toolkit :: Blocklist Implementation, defect)

Description

[Mathieu Leplatre [:leplatrem]]

The blocklist tests are spread in many files.

It would be handy to have a simple way to run every `nsBlocklist` service tests. 

I propose that their name would at least be prefixed with `test_blocklist_` (for example):

    $ ack -l "@mozilla.org/extensions/blocklist" toolkit/mozapps/extensions/test/xpcshell/*.js

```
toolkit/mozapps/extensions/test/xpcshell/test_asyncBlocklistLoad.js
toolkit/mozapps/extensions/test/xpcshell/test_blocklist_change.js
toolkit/mozapps/extensions/test/xpcshell/test_blocklist_gfx.js
toolkit/mozapps/extensions/test/xpcshell/test_blocklist_json.js
toolkit/mozapps/extensions/test/xpcshell/test_blocklist_metadata_filters.js
toolkit/mozapps/extensions/test/xpcshell/test_blocklist_override.js
toolkit/mozapps/extensions/test/xpcshell/test_blocklist_prefs.js
toolkit/mozapps/extensions/test/xpcshell/test_blocklist_regexp.js
toolkit/mozapps/extensions/test/xpcshell/test_bug335238.js
toolkit/mozapps/extensions/test/xpcshell/test_bug393285.js
toolkit/mozapps/extensions/test/xpcshell/test_bug406118.js
toolkit/mozapps/extensions/test/xpcshell/test_bug430120.js
toolkit/mozapps/extensions/test/xpcshell/test_bug449027.js
toolkit/mozapps/extensions/test/xpcshell/test_bug455906.js
toolkit/mozapps/extensions/test/xpcshell/test_bug468528.js
toolkit/mozapps/extensions/test/xpcshell/test_bug514327_1.js
toolkit/mozapps/extensions/test/xpcshell/test_bug514327_2.js
toolkit/mozapps/extensions/test/xpcshell/test_bug514327_3.js
toolkit/mozapps/extensions/test/xpcshell/test_bug619730.js
toolkit/mozapps/extensions/test/xpcshell/test_bug620837.js
toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_Device.js
toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_DriverNew.js
toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_Equal_DriverNew.js
toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_Equal_DriverOld.js
toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_Equal_OK.js
toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_GTE_DriverOld.js
toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_GTE_OK.js
toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_No_Comparison.js
toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_OK.js
toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_OS.js
toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_OSVersion_match.js
toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_OSVersion_mismatch_DriverVersion.js
toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_OSVersion_mismatch_OSVersion.js
toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_prefs.js
toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_Vendor.js
toolkit/mozapps/extensions/test/xpcshell/test_gfxBlacklist_Version.js
toolkit/mozapps/extensions/test/xpcshell/test_pluginBlocklistCtp.js
toolkit/mozapps/extensions/test/xpcshell/test_softblocked.js
```

Comment 1

[Mathieu Leplatre [:leplatrem]]

Attachment: Bug 1343216 - Add blocklist tags in xpcshell tests

Review commit: https://reviewboard.mozilla.org/r/116032/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/116032/

Comment 2

[Matthew N. [:MattN]]

Comment on attachment 8842001 [details]
Bug 1343216 - Add blocklist tags in xpcshell tests

https://reviewboard.mozilla.org/r/116032/#review117644

Hello,

(In reply to Mathieu Leplatre (:leplatrem) from comment #0)
> It would be handy to have a simple way to run every `nsBlocklist` service
> tests. 

Note that `tags = blocklist` or similar in xpcshell.ini would also solve this.

It seems like in this case a subdirectory would be a better idea. You should be able to reference `support-files` and `head` with a relative path from the subdirectory though I'm not sure if that requires changing the path used in tests. If this doesn't work well then I think `tags` is the next best option but since I don't have any ownership of this code I think you should ask people like :kmag or :Mossop to review.

Comment 3

[Mathieu Leplatre [:leplatrem]]

Comment on attachment 8842001 [details]
Bug 1343216 - Add blocklist tags in xpcshell tests

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116032/diff/1-2/

Comment 4

[Mathieu Leplatre [:leplatrem]]

Thanks for your help!

The tags make a lot sense! It seems to be the common way of solving this indeed (I saw `tags = webextensions` here and there). 
Plus, it allows us to run tests from different folders — like `services/common` and `extensions/mozapps/extensions` in our use case.

Comment 5

[Kris Maglione [:kmag]]

Comment on attachment 8842001 [details]
Bug 1343216 - Add blocklist tags in xpcshell tests

https://reviewboard.mozilla.org/r/116032/#review118850

This seems like a good idea, but it looks like you're missing a bunch of tests, including:

browser/base/content/test/plugins/browser_CTP_context_menu.js
browser/base/content/test/plugins/browser_CTP_crashreporting.js
browser/base/content/test/plugins/browser_CTP_data_urls.js
browser/base/content/test/plugins/browser_CTP_drag_drop.js
browser/base/content/test/plugins/browser_CTP_hide_overlay.js
browser/base/content/test/plugins/browser_CTP_iframe.js
browser/base/content/test/plugins/browser_CTP_multi_allow.js
browser/base/content/test/plugins/browser_CTP_nonplugins.js
browser/base/content/test/plugins/browser_CTP_notificationBar.js
browser/base/content/test/plugins/browser_CTP_outsideScrollArea.js
browser/base/content/test/plugins/browser_CTP_remove_navigate.js
browser/base/content/test/plugins/browser_CTP_resize.js
browser/base/content/test/plugins/browser_CTP_zoom.js
browser/base/content/test/plugins/browser_blocking.js
browser/base/content/test/plugins/browser_blocking.js
browser/base/content/test/plugins/browser_blocklist_content.js
browser/base/content/test/plugins/browser_blocklist_content.js
browser/base/content/test/plugins/browser_bug743421.js
browser/base/content/test/plugins/browser_bug743421.js
browser/base/content/test/plugins/browser_bug812562.js
browser/base/content/test/plugins/browser_clearplugindata.js
browser/base/content/test/plugins/browser_plugin_reloading.js
browser/base/content/test/plugins/browser_plugin_reloading.js
browser/base/content/test/plugins/browser_pluginnotification.js
browser/base/content/test/plugins/browser_pluginnotification.js
browser/base/content/test/plugins/browser_plugins_added_dynamically.js
browser/base/content/test/plugins/browser_plugins_added_dynamically.js
browser/base/content/test/social/browser_blocklist.js
security/manager/ssl/tests/unit/test_cert_blocklist.js
toolkit/mozapps/extensions/test/browser/browser_CTP_plugins.js

And for the new kinto-based blocklist, at least:

security/manager/ssl/tests/unit/test_ev_certs.js
services/common/tests/unit/test_blocklist_certificates.js
services/common/tests/unit/test_blocklist_pinning.js
services/common/tests/unit/test_blocklist_signatures.js
services/common/tests/unit/test_blocklist_signatures.js
services/common/tests/unit/test_blocklist_updater.js

Comment 6

[Mathieu Leplatre [:leplatrem]]

Comment on attachment 8842001 [details]
Bug 1343216 - Add blocklist tags in xpcshell tests

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116032/diff/2-3/

Comment 7

[Kris Maglione [:kmag]]

Comment on attachment 8842001 [details]
Bug 1343216 - Add blocklist tags in xpcshell tests

https://reviewboard.mozilla.org/r/116032/#review119334

Thanks!

Comment 8

[Mathieu Leplatre [:leplatrem]]

Comment on attachment 8842001 [details]
Bug 1343216 - Add blocklist tags in xpcshell tests

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116032/diff/3-4/

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dd28321fb849
Add blocklist tags in xpcshell tests r=kmag

Comment 10

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/dd28321fb849