1444869 - [QA] [Normandy] Update timer validation

Status: VERIFIED FIXED

(Firefox :: Normandy Client, defect)

Description

[Adrian Florinescu [:aflorinescu]]

[Description:]
Create two staging pref. experiment that modifies update timer: one for app.normandy.run_interval_seconds and one for extensions.shield-recipe-client.run_interval_seconds
The scope is to test both the update timer and the pref experiment targeting the two preferences.

[Prerequisites:]

You need access to New Admin interface (https://normandy-admin.stage.mozaws.net/control-new)
1. Set the app.normandy.logging.level preference to 0 to enable more logging.
2. Set the security.content.signature.root_hash preference to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90.
3. Set the preference value for app.normandy.api_url set to https://normandy.stage.mozaws.net/api/v1

[Staging:]

I. Create, approve and publish the pref. experiment for app.normandy.run_interval_seconds
1. Open Control Center: https://normandy-admin.stage.mozaws.net/
2. View all.
3. New recipe:
    type= preference-experiment
    filter: 'app.normandy.run_interval_seconds'|preferenceExists   
    preference name: app.normandy.run_interval_seconds
    preference type: integer
    preference branch type: Default
    branches: 
              - 1h / pref. value 3200 / ratio 1 
              - 2h / pref. value 7200 / ratio 1
4. Approve and Publish.

II. Create, approve and publish the pref. experiment for extensions.shield-recipe-client.run_interval_seconds
1. Open Control Center: https://normandy-admin.stage.mozaws.net/
2. View all.
3. New recipe:
    type= preference-experiment
    filter: 'extensions.shield-recipe-client.run_interval_seconds'|preferenceExists   
    preference name: extensions.shield-recipe-client.run_interval_seconds
    preference type: integer
    preference branch type: Default
    branches: 
              - 1h / pref. value 3200 / ratio 1 
              - 2h / pref. value 7200 / ratio 1
4. Approve and Publish.             


[Steps:]
1. Run Nightly FF59 or Nightly FF60 before 2018-03-03 with new profile, disable autoupdate
2. Set pre-requisites, restart.
3. Open about:support, profile location, open folder and view contents for shield-preference-experiments.json, to figure out in which branch you are.
8. Open Browser console and wait ~1h/2h depending on the branch.
5. Run Nightly FF60 with new profile - latest version.
6. Set pre-requisites, restart.
7. Open about:support, profile location, open folder and view contents for shield-preference-experiments.json, to figure out in which branch you are.
8. Open Browser console and wait ~1h/2h depending on the branch.

[Expected Results:]
4. && 8. The Shield/Normandy recipe runner is triggered and fetching/executing recipes after the amount of time set by the pref-experiment

Comment 1

[Adrian Florinescu [:aflorinescu]]

Addition to the above scenarios:
a. From profile manager, just create the profile folder (create the profile but not start FF with it)
b. Open the profile location and create an empty profiles.json, in which add the following lines, save afterwards:

user_pref("security.content.signature.root_hash", "DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90");
user_pref("extensions.shield-recipe-client.api_url", "https://normandy.stage.mozaws.net/api/v1");
user_pref("app.normandy.logging.level", 0);

Using this makes setting the prerequisites in the Step 2 and 6 unnecessary.

Comment 2

[Michael Cooper [:mythmon]]

I see two potential problems with this:

1. A default pref study won't work for extensions.shield-recipe-client preferences, because of the way the default values are set. Even if the preference changes successfully (and I'm not convinced it does), it won't be used for the first timer run, due to the order of events.


2. Fx60 won't read the values for extensions.shield-recipe-client.run_interval_servings, since it is in the old name space, and the value won't be migrated, since it is a default value, not user set.

I doubt we will be running this study on Fx59 and below, given our timing. I recommend we only target Fx60 and above, and use the pref app.normandy.run_interval_seconds.

Comment 3

[Adrian Florinescu [:aflorinescu]]

Related to concern 1., I am planning to run an experiment for the Nightlies 59 and 60 which still have Shield as system add-on. IMO, the order of events should be: set default, run the experiment, modify the preference.

Related to concern 2. Using Comment 1 complementary to comment 0, with my test for app.normandy.run_interval_seconds, the pref. experiment (https://normandy-admin.stage.mozaws.net/recipe/121/) is run at start-up due to the fact it's first run. After restart, the value is one from one of the branches set -> 7200 in my case: now waiting for the 2 hrs to pass to validate it runs again.

Comment 4

[Adrian Florinescu [:aflorinescu]]

I've verified the timer change using STR. as described in comment 0 and comment 1. I've listed a few results here for app.normandy.run_interval_seconds on Nightly 61: https://docs.google.com/spreadsheets/d/16X29mFhjYbQEgwEWUpkOEMGXSCxWR5e_YfoJcA3oe1k/edit#gid=0.
The only thing I've noticed is that on restart, I get the fetch again, see line 4, after-which the timer is consistent with the change. AFAIK, Normandy should fetch that early only at first run or when dev_mode is on. Mike, might it be because the pre-exp. changed the timer and then Normandy considers that it needs to run again?

Comment 5

[Michael Cooper [:mythmon]]

You're right that only dev_mode or first run should cause a fetch immediately on startup. One thing I can think of is that the dev_mode and first run runs don't set the preference for the last time the timer ran. That's because we don't manage that preference directly, and instead rely on the update timer manager. I think that this can cause Normandy to run twice the first time on a new profile: once right away, and the a few moments later when the update timer kicks in. Could that explain what you saw Adrian?

Comment 6

[Adrian Florinescu [:aflorinescu]]

(In reply to Mike Cooper [:mythmon] from comment #5)
> You're right that only dev_mode or first run should cause a fetch
> immediately on startup. One thing I can think of is that the dev_mode and
> first run runs don't set the preference for the last time the timer ran.
> That's because we don't manage that preference directly, and instead rely on
> the update timer manager. I think that this can cause Normandy to run twice
> the first time on a new profile: once right away, and the a few moments
> later when the update timer kicks in. Could that explain what you saw Adrian?

I've logged bug 1446421 for the two Normandy runs for new profile and also for dev_mode. Related to the case in which I get the two Normandy runs after restart, if I give the first run enough time, then I get desired results for the first restart as well: no fetch until the timer tics.

Comment 7

[Michael Cooper [:mythmon]]

Adrian, is there anything else blocking this timer update? Can we run this study on Beta 60?

Comment 8

[Adrian Florinescu [:aflorinescu]]

I was actually waiting to hear from you after you reviewed the data after the study on Nightly, in case there's a certain area that might be concerns around. 

If there are no concerns from the data on the Nightly study, then from the manual testing point of view, I would say we are ready for the timer update. Is it enough if I mark the bug as verified in order to run the beta study? Or do you need me to reply to the PI request?

I originally wanted to keep this open until beta study is done, but I realise now it's a bit confusing from the testing status point of view. 

Summing up, verifying this for Fx60/Fx61. Removing the flag for 59 since this change doesn't affect FX59

Comment 9

[Michael Cooper [:mythmon]]

> I was actually waiting to hear from you after you reviewed the data after the study on Nightly, in case there's a certain area that might be concerns around. 

The Nightly study didn't show much, as expected. The servers barely noticed the load, and enrollment/unenrollment have gone as expected

> If there are no concerns from the data on the Nightly study, then from the manual testing point of view, I would say we are ready for the timer update. Is it enough if I mark the bug as verified in order to run the beta study? Or do you need me to reply to the PI request?

I think it would good to reply to intent to ship email. I'll make sure you're CCed on that.