1023450 - Experiments are running during mochitests

Status: RESOLVED WORKSFORME

(Firefox Health Report Graveyard :: Client: Desktop, defect)

Description

[Ed Morley [:emorley]]

In bug 1023423, the following is interspersed is the test's log output:
{
08:21:04     INFO -  1402413664376	Browser.Experiments.Experiments	TRACE	Experiments #0::enabled=true, true
08:21:04     INFO -  1402413664377	Browser.Experiments.Experiments	TRACE	Experiments #0::Registering instance with Addon Manager.
08:21:04     INFO -  1402413664377	Browser.Experiments.Experiments	TRACE	Experiments #0::Registering previous experiment add-on provider.
08:21:04     INFO -  1402413664378	Browser.Experiments.Experiments	TRACE	PreviousExperimentProvider #0::startup()
08:21:04     INFO -  1402413664378	Browser.Experiments.Experiments	TRACE	Experiments #0::_loadFromCache
08:21:04     INFO -  1402413664383	Browser.Experiments.Experiments	TRACE	Experiments #0::_loadTask finished ok
08:21:04     INFO -  1402413664383	Browser.Experiments.Experiments	TRACE	Experiments #0::_run
08:21:04     INFO -  1402413664384	Browser.Experiments.Experiments	TRACE	Experiments #0::_main iteration
08:21:04     INFO -  1402413664384	Browser.Experiments.Experiments	TRACE	Experiments #0::_evaluateExperiments
08:21:04     INFO -  1402413664386	Browser.Experiments.Experiments	TRACE	Experiments #0::_main finished, scheduling next run
}

We should set whatever prefs we need to in automation to ensure this doesn't happen outside of when we are testing experiments code (similar to what we do for updates, telemetry etc). We'll need to make sure prefs are set for all platforms and all test suites (there are multiple places for all of these prefs sadly).

Georg, would you mind taking a look?

Comment 1

[Ed Morley [:emorley]]

Another one in bug 1023809.

Comment 2

[Benjamin Smedberg]

We do not want to completely disable the experiment system during tests. We do set the manifest URL to a dummy URI at http://mxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js#61 so that no experiments will actually be activated except in the experiment tests.

I think this is WFM.

Comment 3

[Georg Fritzsche [:gfritzsche]]

Indeed, this should be fine and not break other tests.
The experiments code is highly async and shouldn't block other operations noticably.

Comment 4

[Ed Morley [:emorley]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #2)
> We do not want to completely disable the experiment system during tests. We
> do set the manifest URL to a dummy URI at
> http://mxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.
> js#61 so that no experiments will actually be activated except in the
> experiment tests.
> 
> I think this is WFM.

Note that that file only applies to mochitests and a few others - for everything else prefs have to be set in additional places (sigh; bug 1023483 comment 0) - and so experiments are in fact being pulled from telemetry-experiment.cdn.mozilla.net during talos runs :-( (see bug 1030093)