1761058 - Implement telemetry for Suggest suggestion frequency capping

Status: VERIFIED FIXED

(Firefox :: Address Bar, task, P1)

Description

[Drew Willcoxon :adw]

We need to implement telemetry for Suggest suggestion frequency capping.

Comment 1

[Drew Willcoxon :adw]

Attachment: Bug 1761058 - Implement telemetry for Suggest suggestion frequency capping.

This adds a new "impression_cap" telemetry event with several things recorded in
the extra object. Please see the updated documentation and Events.yaml for
details.

This is another big patch but like D142152 most of it is the test.

Depends on D142152

Comment 2

[Drew Willcoxon :adw]

Attachment: data-request.md

Data review request for a new telemetry event called contextservices.quicksuggest.impression_cap

Comment 3

[Chris H-C :chutten]

Comment on attachment 9270941 [details]
data-request.md

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Drew Wilcoxon is responsible.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does the data collection use a third-party collection tool?

No.

---

Result: datareview+

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f616a1fad14b
Implement telemetry for Suggest suggestion frequency capping. r=daisuke

Comment 5

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/f616a1fad14b

Comment 6

[Drew Willcoxon :adw]

STR for QA

This bug builds on frequency caps in bug 1761052 so the STR will be similar. Only telemetry events need to be verified.

STR prerequisites

Set these prefs to enable caps:

browser.urlbar.quicksuggest.impressionCaps.nonSponsoredEnabled = true
browser.urlbar.quicksuggest.impressionCaps.sponsoredEnabled = true

Use the following snippet to set up some caps. For this bug, instead of running the snippet and then starting the STR whenever later, please run the snippet as part of step 1 in the STR, in the 0-10s range on the timeline.

(function() {
Cu.import("resource:///modules/UrlbarQuickSuggest.jsm");
UrlbarQuickSuggest._setConfig({
  impression_caps: {
    sponsored: {
      custom: [
        { interval_s: 10, max_count: 1 },
        { interval_s: 20, max_count: 2 },
      ],
    },
    nonsponsored: {
      custom: [
        { interval_s: 60, max_count: 1 },
      ],
    },
  },
});
})();

FYI the snippet sets up the following caps:

• Sponsored:
  • 1 per 10 seconds
  • 2 per 20 seconds
• Non-sponsored:
  • 1 per 60 seconds

STR

0-10s

1. Run the snippet above to define some caps.

10-20s

2. Trigger a sponsored suggestion (e.g. "amazon").
3. Click any result.

20-30s

3. Trigger a sponsored suggestion again.
4. Click any result.

30-40s

5. Trigger a sponsored suggestion again.
6. Click any result.

40-50s

7. Trigger a nonsponsored suggestion (e.g. "wikipedia").
8. Click any result.

50-60s

9. Don't do anything :-)

60s

10. Type anything in the urlbar but don't pick anything. (This is necessary to trigger the counter reset logic)

Expected results

Open about:telemetry#events-tab_search=impression_cap to view telemetry events filtered on "impression_cap". You should see many events in the contextservices.quicksuggest category with impression_cap for the method.

For the object, reset means a cap's counter was reset, and hit means a cap was hit (i.e., the counter became equal to the cap max count).

The value field is unused.

The extra is some JSON with various values. For verification purposes, the following values are important: type, maxCount, intervalSeconds, count

The following events should appear in the order below. There may be extra reset events at the end if you happened to wait more than 10s to open about:telemetry.

1. • object: reset
   • extra:
     • type: "sponsored"
     • maxCount: "1"
     • intervalSeconds: "10"
     • count: "0"
2. • object: hit
   • extra:
     • type: "sponsored"
     • maxCount: "1"
     • intervalSeconds: "10"
     • count: "1"
3. • object: reset
   • extra:
     • type: "sponsored"
     • maxCount: "1"
     • intervalSeconds: "10"
     • count: "1"
4. • object: reset
   • extra:
     • type: "sponsored"
     • maxCount: "2"
     • intervalSeconds: "20"
     • count": "1"
5. • object: hit
   • extra:
     • type: "sponsored"
     • maxCount: "1"
     • intervalSeconds: "10"
     • count": "1"
6. • object: reset
   • extra:
     • type: "sponsored"
     • maxCount: "1"
     • intervalSeconds: "10"
     • count: "1"
7. • object: hit
   • extra:
     • type: "sponsored"
     • maxCount: "1"
     • intervalSeconds: "10"
     • count: "1"
8. • object: hit
   • extra:
     • type: "sponsored"
     • maxCount: "2"
     • intervalSeconds: "20"
     • count: "2"
9. • object: reset
   • extra:
     • type: "sponsored"
     • maxCount: "1"
     • intervalSeconds: "10"
     • count: "1"
10. • object: reset
    • extra:
      • type: "sponsored"
      • maxCount: "2"
      • intervalSeconds: "20"
      • count: "2"
11. • object: hit
    • extra:
      • type: "nonsponsored"
      • maxCount: "1"
      • intervalSeconds: "60"
      • count: "1"
12. • object: reset
    • extra:
      • type: "sponsored"
      • maxCount: "1"
      • intervalSeconds: "10"
      • count: "0"
13. • object: reset
    • extra:
      • type: "sponsored"
      • maxCount: "1"
      • intervalSeconds: "10"
      • count: "0"
14. • object: reset
    • extra:
      • type: "sponsored"
      • maxCount: "2"
      • intervalSeconds: "20"
      • count: "0"
15. • object: reset
    • extra:
      • type: "nonsponsored"
      • maxCount: "1"
      • intervalSeconds: "60"
      • count: "1"

Comment 7

[Drew Willcoxon :adw]

Hi Cosmin, it would be good to verify this bug if it's not too hard. The STR are in comment 6 and are similar to bug 1761052 but not quite the same.

Comment 8

[Cosmin Muntean [:cmuntean], Ecosystem QA]

Hi Drew! We have managed to verify this bug on the latest Nightly 101.0a1 build (Build ID: 20220420215300) on Windows 10 x64, macOS 10.15.7 and Ubuntu 20.04 x64.

We have followed the described steps in comment 6 and we get the correct telemetry events.

However, if we are following the same steps, but if we are opening a new tab at each step (10s timeline) and trigger the sponsored/non-sponsored result in the newly opened tab, we get different telemetry events. We have lots of "reset" events. Shouldn't we have the same telemetry register?
Here is a screen recording of this scenario: Link.

Comment 9

[Drew Willcoxon :adw]

It looks like the video begins after you start up Firefox, right? My guess is that although you're testing immediately after startup, you're using a profile that you previously tested with, in other app session(s). The impression counters and stats are stored in the profile's prefs and they are not cleared on startup or shutdown, and the way the logic works is to record reset events for each interval of time that has passed, even periods where you weren't using Firefox or where you weren't performing searches in the address bar.

If you start Firefox, clear browser.urlbar.quicksuggest.impressionCaps.stats, and then run the STR (starting from running the snippet in step 1), then you should see the expected number of reset events.

Thanks for raising this point because I didn't really think about it. We probably don't want to record a million reset events if a million time intervals happened to pass between the last time you started Firefox and now. In practice intervals will be long -- we're discussing 9 days -- so this may not be a big problem, but we should have some mitigation in place. I filed bug 1765881.