Experiment active in automated integration tests for Firefox Telemetry
Categories
(Firefox :: Normandy Client, defect)
Tracking
()
People
(Reporter: raphael, Unassigned)
References
Details
Hi folks,
I help manage an integration test suite for Firefox Telemetry with symbol tt(c). While investigating intermittent test failures on CI in bug 1579840, we discovered that the Firefox under test is enrolled in an Experiment.
The failing test verifies that the Telemetry component in Firefox manages Telemetry sessions as designed using automation: We start Firefox, open a new tab and then restart the browser to trigger a main ping with reason shutdown to be submitted to our local test server. We then verify that this ping payload does not contain a previousSubsessionId (see test_subsession_management).
Looking at the logs provided we discovered that prior to receiving this main ping with reason shutdown, we receive another main ping with reason environment-change. This happens at least once because the Firefox under test is enrolled in an Experiment, which changes the Telemetry Environment.
Relevant log entries:
Do you expect Experiments to target Firefox clients running in automation? If so, we can probably disable studies and experiments in this suite. However we want to make sure that this behavior is intentional.
|
||
Comment 1•5 years ago
|
||
Do you expect Experiments to target Firefox clients running in automation?
As it stands now, clients running in automation would be eligible for Normandy recipes, including Experiments. Recipes almost always include targeting to only apply to Release, Beta, Dev Edition, or Nightly. I expected most automation runs to have a different update channel set, but I see now that's not true. Just to confirm though, it would be good to know what the experiment that the client was enrolled in, to check that it was behaving as expected. The value passed to setExperimentActive would be enough to determine that.
I'm not sure if a blanket disable of Normandy when Firefox is being automated is the right choice, since we may also want to target things like automation clusters running web tests with some recipes. (For example a hotfix to unbreak TLS) I think that disabling Normandy during Mozilla's automated test suites would sense though. I'm not quite sure how to do that, but if you have any ideas I would like to explore how Normandy can be a better citizen here.
The quick fix for you here would be to set the preference app.normandy.enabled to false. in your test If that's the case, Normandy will not run automatically.
|
Reporter | |
Comment 2•5 years ago
|
||
Thank you for clarifying that clients running in automation would be eligible for Normandy recipes. Chutten recently landed a patch in bug 1636130 that added the experiment ID to the Telemetry logs when an experiment is set active. The patch was uplifted to beta and I was able to find at least one intermittent failure since then where Telemetry logged the experiment:
The logs for this job show that an experiment with ID bug-1635870-message-privacy-protections-report-onboarding-n4s-beta-76-77 and branch n4s_protections_report_position1 was active for that client.
|
|
Reporter | |
Comment 3•5 years ago
|
||
Based on your recommendation in Comment 1, we have disabled Normandy for the telemetry-tests-client suite. Thanks again!
|
||
Comment 4•5 years ago
|
||
The severity field is not set for this bug.
:mythmon, could you have a look please?
For more information, please visit auto_nag documentation.
|
|
||
Comment 5•5 years ago
|
||
It sounds like from comment 3 that this issue is resolved.
Description
•