1604159 - The survey is not triggered if the tip is ignored

Status: RESOLVED WORKSFORME

(Firefox :: Address Bar, defect)

Description

[Cosmin Muntean [:cmuntean], Ecosystem QA]

Attachment: survey not triggered if tip is ignored.gif

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Affected Versions]:

• Firefox 72 Unbranded build
• Nightly 73.0a1

[Prerequisites]:

• Have 72 Unbranded build downloaded or installed.
• Have access to the stage delivery console.
• Have an active branched-addon recipe for the Search Interventions experiment. You can use my created recipe.
• Have a user.js file that contains all the prefs needed to be targeted by the Normandy recipe. You can use my created user.js.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. In the profile directory copy the user.js file.
3. Restart the browser.
4. After enrolling in the experiment, type "Clear" in the URL bar.
5. Dismiss the suggestions panel.
6. Observe the behavior.

[Expected result]:

• The suggestions panel is dismissed and the survey is triggered in a new tab.

[Actual result]:

• The suggestions panel is dismissed but the survey is not triggered.

[Notes]:

• The issue is also reproducible if a normal result from the panel is clicked.
• The issue is reproducible on both branches (control and treatment).
• Attached a screen recording with the issue.

Comment 1

[Drew Willcoxon :adw]

Did you run the browser.storage.local.set({ forceSurvey: 1 }) snippet that the testing doc mentions? Please see the end of the surveys bullet point, right above the Telemetry section ("Normally, the survey page is always opened when the user picks a tip..."). You need to inspect the add-on and run that snippet in the inspector window.

Comment 2

[Cosmin Muntean [:cmuntean], Ecosystem QA]

I have tried to run the snippet, but unfortunately, the Urlbar Interventions add-on is not displayed in "about:debugging#/runtime/this-firefox" page. Are there any other prefs that I need to add for the add-on to be displayed in "about:debugging"?
Also, Is this the only way to trigger the survey for this case, even if we are testing on new Firefox profiles where it was never triggered?

My understanding is that the survey is triggered once per session if the script is not run. Considering this, I have tested this issue on multiple new firefox profiles where the survey was never triggered and I can still reproduce this.
The survey is not triggered if the panel was dismissed or a normal result was clicked, but it is triggered if I click the button from the displayed interventions.

Comment 3

[Drew Willcoxon :adw]

OK, I now see that the add-on isn't listed in about:debugging when the signed add-on is installed. You need to run some code similar to the snippet in the testing doc in order to see surveys when interventions are ignored. That's because there's only a very small chance (2%) the add-on opens a survey in that case.

Fortunately there's another way to force the surveys. Please run this snippet in the normal browser console. You should see "OK" followed by a JS object that looks like { forceSurvey: 1 }. There may be other properties in this object, and that’s OK. The forceSurvey property is the important one in this case. I'll update the testing doc too.

(async function() {
  const { WebExtensionPolicy } = Cu.getGlobalForObject(
    ChromeUtils.import("resource://gre/modules/Services.jsm", {})
  );
  const { ExtensionStorageIDB } = ChromeUtils.import(
    "resource://gre/modules/ExtensionStorageIDB.jsm",
    {}
  );
  let addonID = "[e-mail]";
  let policy = WebExtensionPolicy.getByID(addonID);
  let storagePrincipal = ExtensionStorageIDB.getStoragePrincipal(
    policy.extension
  );
  let conn = await ExtensionStorageIDB.open(storagePrincipal);
  await conn.set({ forceSurvey: 1 });
  console.log(`OK`, await conn.get());
})();

Comment 4

[Drew Willcoxon :adw]

Actually you don't need the example.com part. (but it doesn't hurt either). I'll update the snippet.

Comment 5

[Drew Willcoxon :adw]

OK, I updated the snippet above and in the testing doc.

Comment 6

[Cosmin Muntean [:cmuntean], Ecosystem QA]

Thanks Drew for clarifying this!

I have retested using the provided snippet and the survey is correctly triggered when the tips are ignored or a normal result is clicked.
Considering this I will close this bug as RESOLVED - WORKSFORME.