1761052 - Implement remotely configurable frequency capping for non-sponsored and sponsored Suggest suggestions

Status: VERIFIED FIXED

(Firefox :: Address Bar, task, P1)

Description

[Drew Willcoxon :adw]

We want to be able to cap the number of Suggest suggestions an individual user sees per time period. We should have two caps, one for non-sponsored and one for sponsored. The cap config will be stored in remote settings similar to the best match config.

Comment 1

[Drew Willcoxon :adw]

Attachment: Bug 1761052 - Implement remotely configurable frequency capping for Suggest suggestions.

This implements Suggest impression frequency capping as described in the spec.
Capping is performed separately for sponsored and non-sponsored suggestions.
Capping is not per-suggestion; it's per type of suggestion, either sponsored or
non-sponsored.

Each time a suggestion is shown, this increments a counter for the suggestion's
type (sponsored or non-sponsored). Once the counter reaches a max count, we stop
adding that type of suggestion. After a certain interval has elapsed, that type
of suggestion can be shown again, and the counter is reset the next time that
type of suggestion is fetched. The time intervals are non-overlapping.

The config for the max count and time interval will be stored in the remote
settings config object. If the config isn't present (which it isn't right
now), then suggestions can always be added.

The user's local counters are stored in a pref.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/62fd3eab50b4
Implement remotely configurable frequency capping for Suggest suggestions. r=daisuke,nanj

Comment 3

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/62fd3eab50b4

Comment 4

[Drew Willcoxon :adw]

STR for QA

Overview

This bug adds frequency caps for Suggest suggestions. The caps are reset periodically so verifying this will require triggering suggestions, waiting some amount of time, and trying to trigger them again.

• Each time a suggestion impression occurs, a counter is incremented.
• Counters are stored per type of suggestion, either sponsored or non-sponsored. They aren't stored per suggestion. So if you have impressions for 5 different sponsored suggestions, then sponsored suggestions will have a count of 5.
• Counters are incremented only on "impressions", not when suggestions are triggered. An impression is when a result is picked while a Suggest suggestion is visble. Any result can be picked, not only the Suggest suggestion. So to increment a counter, you need to trigger a suggestion and then pick any result.
• When a counter reaches a cap for a suggestion type, that type cannot be triggered again until the cap's interval period elapses.
• The implementation supports multiple overlapping caps, for example: per hour, per day, per week, and lifetime caps.
• By default no caps exist.

STR prerequisites

Enable logging to make verificaton easier:

browser.urlbar.loglevel = "Info"

Keep the browser console open and filter on UrlbarProviderQuickSuggest to show only messages related to quick suggest.

The following test cases are cumulative, so please do them one after the other using the same profile.

Test case 1: Caps disabled

1. Trigger a sponsored suggestion (e.g. "amazon")
2. Click any result
3. Verify the following is logged (should be at the bottom, lots of other things will be logged too):

   urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Impression caps disabled, skipping update 
   

4. Repeat with a non-sponsored suggestion (e.g. "wikipedia")

Test case 2: Caps undefined

1. Set the following prefs to enable caps. They'll be enabled but not defined, so there shouldn't be any capping.

   browser.urlbar.quicksuggest.impressionCaps.nonSponsoredEnabled = true
   browser.urlbar.quicksuggest.impressionCaps.sponsoredEnabled = true
   

2. Trigger a sponsored suggestion (e.g. "amazon")
3. Click any result
4. Verify the following is logged (lots of other things will be logged too):

   urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Impression caps undefined, skipping update 
   

5. Repeat with a non-sponsored suggestion (e.g. "wikipedia")

Test case 3: Caps enabled and defined

Run the following JS in the browser console to define some caps:

(function() {
Cu.import("resource:///modules/UrlbarQuickSuggest.jsm");
UrlbarQuickSuggest._setConfig({
  impression_caps: {
    sponsored: {
      lifetime: 4,
      custom: [
        { interval_s: 30, max_count: 2 },
        { interval_s: 60, max_count: 3 },
      ],
    },
    nonsponsored: {
      lifetime: 2,
      custom: [
        { interval_s: 30, max_count: 1 },
      ],
    },
  },
});
})();

The following should be logged immediately:

urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Validating impression stats 

FYI the snippet sets up the following caps:

• Sponsored:
  • 2 per 30 seconds
  • 3 per 60 seconds
  • 4 lifetime total
• Non-sponsored:
  • 1 per 30 seconds
  • 2 lifetime total

As I mentioned, verification is timing sensitive. The STR below are organized on a timeline by the number of seconds after starting. i.e., in the first 30s, complete steps 1-9. Then wait until 30s total have elapsed since the start and complete steps 10-15 within seconds 30-60, etc. You can start at any time after running the snippet above, but once you start, you'll need to follow the timeline. It may be helpful to use a stopwatch.

If it's hard to do these STR within the time limits, let me know. You can also try changing the interval_s values in the JS snippet so larger values. interval_s is the number of seconds in the interval period.

If it's hard to verify the logging, you can just skip it because the important thing is whether or not suggestions appear. If you do verify the logging, note that many other things besides what's mentioned in the STR will be logged too. Generally the logs in the STR will appear at/near the bottom of the console.

If necessary, you can reset the counters/cap state at any time by clearing browser.urlbar.quicksuggest.impressionCaps.stats. You should restart the STR if you do.

0-30s

1. Trigger a sponsored suggestion (e.g. "amazon").
2. Click any result.
3. Verify the following is logged:

   urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Starting impression stats update
   urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Finished impression stats update 
   

4. Trigger a sponsored suggestion again and verify it appears.
5. Click any result. This will hit the 2 per 30s sponsored cap.
6. Verify the following is logged:

   urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Starting impression stats update
   urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: 'sponsored' impression cap hit
   urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Finished impression stats update 
   

7. Try to trigger a sponsored suggestion. Verify it does not appear.
8. Verify the following is logged:

   urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Impression cap(s) hit, not adding suggestion 
   

9. Trigger a non-sponsored suggestion. Verify that it does appear but don't click it.

30-60s

The 2 per 30s sponsored cap should reset now.

10. Trigger a sponsored suggestion and verify it appears.
11. Click any result. The 3 per 60s sponsored cap should be hit.
12. Verify the following is logged:

    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Starting impression stats update
    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: 'sponsored' impression cap hit
    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Finished impression stats update 
    

13. Try to trigger a sponsored suggestion. Verify it does not appear.
14. Verify the following is logged:

    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Impression cap(s) hit, not adding suggestion 
    

15. Trigger a non-sponsored suggestion. Verify that it does appear but don't click it.

60-90s

The 2 per 30s and 3 per 60s sponsored caps should both reset now.

16. Trigger a sponsored suggestion and verify it appears.
17. Click any result. The 4 per lifetime sponsored cap should be hit.
18. Verify the following is logged:

    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Starting impression stats update
    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: 'sponsored' impression cap hit
    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Finished impression stats update 
    

19. Try to trigger a sponsored suggestion. Verify it does not appear.
20. Verify the following is logged:

    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Impression cap(s) hit, not adding suggestion 
    

21. Trigger a non-sponsored suggestion. Verify that it does appear.
22. Click it. The 1 per 30s non-sponsored cap should be hit.
23. Verify the following is logged:

    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Starting impression stats update
    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: 'nonsponsored' impression cap hit
    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Finished impression stats update 
    

24. Try to trigger a non-sponsored suggestion. Verify it does not appear.
25. Verify the following is logged:

    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Impression cap(s) hit, not adding suggestion 
    

90-120s

26. Trigger a non-sponsored suggestion and verify it appears.
27. Click it. The 2 per lifetime non-sponsored cap should be hit.
28. Verify the following is logged:

    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Starting impression stats update
    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: 'nonsponsored' impression cap hit
    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Finished impression stats update 
    

29. Try to trigger a non-sponsored suggestion. Verify it does not appear.
30. Verify the following is logged:

    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Impression cap(s) hit, not adding suggestion 
    

31. Try to trigger a sponsored suggestion. Verify it does not appear.
32. Verify the following is logged:

    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Impression cap(s) hit, not adding suggestion 
    

120s

33. Try to trigger a non-sponsored suggestion. Verify it does not appear.
34. Verify the following is logged:

    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Impression cap(s) hit, not adding suggestion 
    

35. Try to trigger a sponsored suggestion. Verify it does not appear.
36. Verify the following is logged:

    urlbar	INFO	Provider.UrlbarProviderQuickSuggest :: Impression cap(s) hit, not adding suggestion 
    

Comment 5

[Drew Willcoxon :adw]

Hi Cosmin, I finally wrote some STR for this bug. Could you have a look at comment 4 please? This may be a hard bug to verify due to the timing sensitivity, so please let me know if you run into problems.

Comment 6

[Cosmin Muntean [:cmuntean], Ecosystem QA]

Hi Drew! Thank you for the detailed STR, they are very helpful. It was very interesting to verify this bug, after we learned the steps, we managed to verify exactly how the STR are described.
We have verified this on the latest Nightly 101.0a1 build (Build ID: 20220419214607) on Windows 10 x64, macOS 10.15.7 and Ubuntu 20.04 x64.

• In order to verify we have followed the exact steps described in comment 4.
• We haven't found any issues during testing.

I know that this implementation is only for non-sponsored and sponsored Suggest suggestions, but we have also wanted to see what happens if we enable Best Match. After running the provided script, the Best Match results become Suggest suggestions. I assume that the script forces this and this will not be a problem in the future with Best Match, is this correct?

Comment 7

[Drew Willcoxon :adw]

Thanks for verifying this! Yes, best match is expected to stop working when you use the config in the snippet. That's because the snippet does not specify a best match config. I don't think you need to test with best matches, but if you'd like, then you can use the following snippet. Note that this config is different from the usual one because it will allow every suggestion to be a best match, including the main Amazon and Wikipedia suggestions, as long as your search string is >= 4 characters.

(function() {
Cu.import("resource:///modules/UrlbarQuickSuggest.jsm");
UrlbarQuickSuggest._setConfig({
  best_match: {
    blocked_suggestion_ids: [],
    min_search_string_length: 4,
  },
  impression_caps: {
    sponsored: {
      lifetime: 4,
      custom: [
        { interval_s: 30, max_count: 2 },
        { interval_s: 60, max_count: 3 },
      ],
    },
    nonsponsored: {
      lifetime: 2,
      custom: [
        { interval_s: 30, max_count: 1 },
      ],
    },
  },
});
})();