Figure out how to make reftests run in automation when legacy extensions are disabled

RESOLVED FIXED in Firefox 55

Status

Testing
Reftest
RESOLVED FIXED
8 months ago
8 months ago

People

(Reporter: aswan, Assigned: aswan)

Tracking

Version 3
mozilla55
Points:
---

Firefox Tracking Flags

(firefox55 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

Reftests use some browser extensions that are not webextensions and hence will not load in release and beta beginning with version 57.  These extensions currently circumvent the signing requirements by using temporary installation but that won't work for legacy extensions.
We will still allow legacy extensions to be loaded if Cu.isInAutomation is true and the extensions.legacy.enabled preference is set to true.  If we can arrange for that to happen for reftests, I think that would be the simplest solution.

The test for Cu.isInAutomation is here:
http://searchfox.org/mozilla-central/rev/1a0d9545b9805f50a70de703a3c04fc0d22e3839/js/xpconnect/src/xpcpublic.h#612-619

It looks like reftests already set the preference referenced there:
http://searchfox.org/mozilla-central/rev/972b7a5149bbb2eaab48aafa87678c33d5f2f045/layout/tools/reftest/runreftest.py#330-331

But apparently MOZ_DISABLE_NONLOCAL_CONNECTIONS is not set in reftests, if we can set it that would be great.
The needinfo here is kind of a shot in the dark, please redirect as necessary...
Flags: needinfo?(dbaron)
What info do you need?
Flags: needinfo?(dbaron) → needinfo?(aswan)
Sorry, from comment 1, is it safe to set MOZ_DISABLE_NONLOCAL_CONNECTIONS in reftests?  If so, where is the right place to do it?
Flags: needinfo?(aswan) → needinfo?(dbaron)
(In reply to Andrew Swan [:aswan] from comment #3)
> Sorry, from comment 1, is it safe to set MOZ_DISABLE_NONLOCAL_CONNECTIONS in
> reftests?  If so, where is the right place to do it?

We should probably do so, though there might be some failures you'll have to sort through.

Probably the best place is in somewhere in runreftest.py and possibly also remotereftest.py.  jmaher probably knows the best place(s) that will catch as many reftest variants as possible.
Flags: needinfo?(dbaron) → needinfo?(jmaher)
Comment hidden (mozreview-request)
Well, I had it backward in comment 1.  Turns out the NONLOCAL_CONNECTIONS environment variable is set during reftests, but the security.turn_of_all_security... preference is generally not.  The attached patch sets that preference unconditionally, and also sets extensions.legacy.enabled.  This try run shows reftests passing on linux64 opt with the attached patch and the build-time flag for legacy extensions disabled:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=3656aa79d0a389d240515bfa318a99062f789679&selectedJob=103960158
(Assignee)

Updated

8 months ago
Attachment #8873707 - Flags: review?(jmaher)
I would be concerned about autophone, could you push to try with:
try: -b do -p android-api-15 -u plain-reftest-1,jsreftest-1,crashtest-1,autophone-crashtest-dom-media,autophone-reftest-ogg-video -t none


notice there is jsreftest and crashtest to sort out as well, I would like to see that for linux (I have it in the above syntax)
Flags: needinfo?(jmaher)

Comment 8

8 months ago
mozreview-review
Comment on attachment 8873707 [details]
Bug 1369525: Make sure reftests still run when legacy extensions are disabled

https://reviewboard.mozilla.org/r/145102/#review149240

assuming things work on autophone and sanity checked on crashtest and jsreftest.
Attachment #8873707 - Flags: review?(jmaher) → review+
Since reviewboard only shows the most recent try run, here is the first one that covers reftests across all platforms:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=cd5724d7a7e4
And here is the new one requested in comment 7:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=fd69e3108b22
There's on failure in the first try run from comment 9 but it matches a known intermittent.  A few jobs in the second try run aren't finished but they're all tier 2.  Everything else is green, so I'm going to go ahead and land this.
(Assignee)

Updated

8 months ago
Assignee: nobody → aswan

Comment 11

8 months ago
Pushed by aswan@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/fb9cc965d50b
Make sure reftests still run when legacy extensions are disabled r=jmaher

Comment 12

8 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/fb9cc965d50b
Status: NEW → RESOLVED
Last Resolved: 8 months ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
You need to log in before you can comment on or make changes to this bug.