Closed Bug 1030045 Opened 8 years ago Closed 8 years ago

Marionette needs to define MOZ_DISABLE_NONLOCAL_CONNECTIONS

Categories

(Testing :: Marionette, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: emorley, Assigned: daleharvey)

References

Details

(Whiteboard: [systemsfe])

Attachments

(1 file)

Marionette doesn't currently define MOZ_DISABLE_NONLOCAL_CONNECTIONS (added by bug 995417) during automation runs, so we aren't protecting ourselves against non-loopback connections (eg external resource outside of the build network).

We need to:
a) Fix any failures caused by pre-existing bad behaviour (much of this will just be ensuring the automation prefs for disabling things like safebrowsing etc are set; see bug 1023483 comment 0 for what other suites use).
b) Start defining it during our automation test runs once we're green from (a).
This try run shows what will fail when we define MOZ_DISABLE_NONLOCAL_CONNECTIONS:
https://tbpl.mozilla.org/?tree=Try&rev=b6d1d8e8b633&jobname=marionette

Note that the jobs aren't turning orange (yet, since I disabled the crash, so we could catch multiple instances per test run, rather than aborting early), so you have to look at the logs.

In every desktop marionette run, we're making external connections, to most/all of the following:

{
TEST-UNEXPECTED-FAIL | Non-local network connections are disabled and a connection attempt to aus4.mozilla.org (63.245.217.219) was made.
TEST-UNEXPECTED-FAIL | Non-local network connections are disabled and a connection attempt to clients1.google.com (74.125.239.101) was made.
TEST-UNEXPECTED-FAIL | Non-local network connections are disabled and a connection attempt to ocsp.digicert.com (72.21.91.29) was made.
TEST-UNEXPECTED-FAIL | Non-local network connections are disabled and a connection attempt to ocsp.verisign.com (23.5.251.27) was made.
TEST-UNEXPECTED-FAIL | Non-local network connections are disabled and a connection attempt to safebrowsing-cache.google.com (74.125.239.100) was made.
TEST-UNEXPECTED-FAIL | Non-local network connections are disabled and a connection attempt to safebrowsing.google.com (74.125.239.100) was made.
TEST-UNEXPECTED-FAIL | Non-local network connections are disabled and a connection attempt to services.addons.mozilla.org (63.245.216.134) was made.
TEST-UNEXPECTED-FAIL | Non-local network connections are disabled and a connection attempt to versioncheck-bg.addons.mozilla.org (63.245.217.162) was made.
}

Example log:
Ubuntu VM 12.04 x64 try debug test marionette on 2014-06-25 02:43:10 PDT for push b6d1d8e8b633
https://tbpl.mozilla.org/php/getParsedLog.php?id=42430450&tree=Try

I'll file dependant bug(s) for fixing these.
Assignee: nobody → dale
Blocks: 960072
Depends on: 1095097
I think we need to allow these connections, but just make them fail the test. There's no way to re-write the OS to not make them. Basically they should just run in the background and return 404s.
(In reply to Kevin Grandon :kgrandon from comment #2)
> I think we need to allow these connections, but just make them fail the
> test. There's no way to re-write the OS to not make them. Basically they
> should just run in the background and return 404s.

Just stub them out with dummy localhost URLs, like Firefox, Toolkit and Core does :-)

eg:
http://mxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js#43
Gij1 is passing with non local connections disabled, with https://github.com/mozilla-b2g/marionette-b2gdesktop-host/pull/16/files#r19976584 merged I should be able to get a better idea of exactly how many tests will be effected by this, but from my first quick look over the logs I think e.me is the vast majority of them
(In reply to Dale Harvey (:daleharvey) from comment #4)
> be effected by this, but from my first quick look over the logs I think e.me
> is the vast majority of them

I still don't think that stubbing out *every* non-local URL for an operating system is the right approach. If we don't have any other options, the better thing to do would probably be to make E.me default to a local URL, and wait for the setting on a device.
location.services.mozilla.com
marketplace.firefox.com
fxos.cdn.mozilla.net

Are the 3 urls that cause the fail after 'fixing' e.me, fxos.cdn.mozilla.net seems like it may be an issue since we are actually using those assets in the test if I remember, the other 2 looks they should be disabled fairly easily
This might also break Gip too but there is an outstanding bug and patch (https://bugzilla.mozilla.org/show_bug.cgi?id=1084288) about skipping tests that use external resources on Treeherder which will unblock it.
Looking at e.me, and hopefully we can do this everywhere, I think we can define these urls via settings, remove any default references to them in the code, and use a test-settings.json in the Gij tests, this shouldnt effect how any of the other tests run
Depends on: 1095466
Depends on: 1095468
Depends on: 1095469
Whiteboard: [systemsfe]
Gij 1 is passing

Gij 3 has:
apps/system/test/marionette/browser_persist_title_in_app_test.js
apps/system/test/marionette/rocketbar_activity_layout_test.js
apps/system/test/marionette/software_home_permission_prompt_test.js

Gij 4 has:
apps/verticalhome/test/marionette/search_string_test.js
apps/verticalhome/test/marionette/search_test.js

Gij2 still running, all requests are too mozilla.org, expect we are browsing the web, taking a look now
ok, for some reason places.js is trying to make this request, I expect its for the icon although really it shouldnt be looking at the code, looking over the failures it seems to be be the vast majority of them, so close
ok so this issue is, we have 2 default preloaded sites, when you make a search request with those sites preloaded that matches either of them, we display them in the search results, when we do that we try to fetch their favicon

I think the only real fix for this is to be able to run the tests with optional build customisations, but afaik thats a fairly hard thing to fix, I can put some setting in there and not fetch data based on that, but that seems messy, any ideas?
Flags: needinfo?(kgrandon)
Flags: needinfo?(jlal)
Flags: needinfo?(gaye)
Green with MOZ_DISABLE_NONLOCAL_CONNECTIONS defined and this bug fixed (I tested by just disabling the getIcon)

https://treeherder.mozilla.org/ui/#/jobs?repo=gaia-try&revision=9cda9e6897ce
so this is actually a default part of the build and not any customisation, being able to define build customisations wouldnt help

  if (options.PROFILE_FOLDER !== 'profile-test') {
    this.initTopsitesJSON();
  }

in https://github.com/mozilla-b2g/gaia/blob/master/apps/search/build/build.js#L50

fixes this for now, not ultra clean but cleaner than a setting, anyone think of a nicer way?
The build approach seems like a fine thing to do to me. Probably the easiest fix to make.
Flags: needinfo?(kgrandon)
Depends on: 1097726
Clearning NI ... On IRC we discussed custom profiles as the cleanest way to do this long term (we should do this)
Flags: needinfo?(jlal)
Clearing needinfo, fixed
Flags: needinfo?(gaye)
One unrelated intermittent, but yay all green on restart
Comment on attachment 8522870 [details] [review]
https://github.com/mozilla-b2g/gaia/pull/26140

r+ (but!) Please announce this change on dev-gaia right before landing.
Attachment #8522870 - Flags: review?(jlal) → review+
Made an announcement and landed

https://github.com/mozilla-b2g/gaia/commit/7fda06da51e7a201ad57bd8e9fa9a48941064c9a
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.