1369525 - Figure out how to make reftests run in automation when legacy extensions are disabled

Status: RESOLVED FIXED

(Testing :: Reftest, defect)

Description

[Andrew Swan [:aswan]]

Reftests use some browser extensions that are not webextensions and hence will not load in release and beta beginning with version 57.  These extensions currently circumvent the signing requirements by using temporary installation but that won't work for legacy extensions.

Comment 1

[Andrew Swan [:aswan]]

We will still allow legacy extensions to be loaded if Cu.isInAutomation is true and the extensions.legacy.enabled preference is set to true.  If we can arrange for that to happen for reftests, I think that would be the simplest solution.

The test for Cu.isInAutomation is here:
http://searchfox.org/mozilla-central/rev/1a0d9545b9805f50a70de703a3c04fc0d22e3839/js/xpconnect/src/xpcpublic.h#612-619

It looks like reftests already set the preference referenced there:
http://searchfox.org/mozilla-central/rev/972b7a5149bbb2eaab48aafa87678c33d5f2f045/layout/tools/reftest/runreftest.py#330-331

But apparently MOZ_DISABLE_NONLOCAL_CONNECTIONS is not set in reftests, if we can set it that would be great.
The needinfo here is kind of a shot in the dark, please redirect as necessary...

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

What info do you need?

Comment 3

[Andrew Swan [:aswan]]

Sorry, from comment 1, is it safe to set MOZ_DISABLE_NONLOCAL_CONNECTIONS in reftests?  If so, where is the right place to do it?

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to Andrew Swan [:aswan] from comment #3)
> Sorry, from comment 1, is it safe to set MOZ_DISABLE_NONLOCAL_CONNECTIONS in
> reftests?  If so, where is the right place to do it?

We should probably do so, though there might be some failures you'll have to sort through.

Probably the best place is in somewhere in runreftest.py and possibly also remotereftest.py.  jmaher probably knows the best place(s) that will catch as many reftest variants as possible.

Comment 5

[Andrew Swan [:aswan]]

Attachment: Bug 1369525: Make sure reftests still run when legacy extensions are disabled

This won't be an issue until the 57 release but at that time, the
reftest extensions will only load if the extensions.legacy.enabled
preference is set to true and Cu.isInAutomation, which depends on
another preference, is true.  Set the appropriate prefs here for reftests.

Review commit: https://reviewboard.mozilla.org/r/145102/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/145102/

Comment 6

[Andrew Swan [:aswan]]

Well, I had it backward in comment 1.  Turns out the NONLOCAL_CONNECTIONS environment variable is set during reftests, but the security.turn_of_all_security... preference is generally not.  The attached patch sets that preference unconditionally, and also sets extensions.legacy.enabled.  This try run shows reftests passing on linux64 opt with the attached patch and the build-time flag for legacy extensions disabled:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=3656aa79d0a389d240515bfa318a99062f789679&selectedJob=103960158

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

I would be concerned about autophone, could you push to try with:
try: -b do -p android-api-15 -u plain-reftest-1,jsreftest-1,crashtest-1,autophone-crashtest-dom-media,autophone-reftest-ogg-video -t none


notice there is jsreftest and crashtest to sort out as well, I would like to see that for linux (I have it in the above syntax)

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8873707 [details]
Bug 1369525: Make sure reftests still run when legacy extensions are disabled

https://reviewboard.mozilla.org/r/145102/#review149240

assuming things work on autophone and sanity checked on crashtest and jsreftest.

Comment 9

[Andrew Swan [:aswan]]

Since reviewboard only shows the most recent try run, here is the first one that covers reftests across all platforms:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=cd5724d7a7e4
And here is the new one requested in comment 7:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=fd69e3108b22

Comment 10

[Andrew Swan [:aswan]]

There's on failure in the first try run from comment 9 but it matches a known intermittent.  A few jobs in the second try run aren't finished but they're all tier 2.  Everything else is green, so I'm going to go ahead and land this.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fb9cc965d50b
Make sure reftests still run when legacy extensions are disabled r=jmaher

Comment 12

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/fb9cc965d50b