1751746 - Change search-telemetry to use strict partner code matching

Status: VERIFIED FIXED

(Firefox :: Search, defect)

Description

[Mark Banner (:standard8)]

We want to change to strict matching for partner codes rather than prefix for the in-content search telemetry.

Comment 1

[Mark Banner (:standard8)]

Attachment: Bug 1751746 - Change search-telemetry to use strict partner code matching. r?daleharvey!,mkaply!

Comment 2

[Adrian Florinescu [:aflorinescu]]

As the test plan for this change needs to cover Remote Settings changes as well as code changes, the QA team has tested this on a try-build using both the try-build dump and the staging dump-disabled configuration - (configuration changes approved meanwhile.)

1. (done)Reproduce the issue on affected builds.
2. (done)Ensure that the used telemetry RS bucket is v2 not v1.
3. (done)Ensure that the issue doesn't reproduce and the telemetry logged is (google telementry used as an example):

• "google:organic:other" when the code is not maching the new strict list
• "google:organic:none" when the code is not present
• "google:tagged-follow-on:firefox-b-d" for follow-on

4. Ensure all telemetry recording engines are behaving taking in account point 3.

• google (done)
• ddg (done)
• bing (done)
• baidu (done)

5. (not done) Wide regression testing on search general telemetry
6. (not done) Multiple OS coverage.
7. (not done) Regression testing covering the removal of v1 RS telemetry bucket and effects on old versions esr 79 / esr 91 + 1-2 non-train builds
8. (not done/not ready) Verification(98) and Uplift verification (96 dot / 97b)
9. (not done) Telemetry Test coverage updates to account for the spill.
10. (not in QA scope) Data sanitize

All in all, with the current testing, with the fix and the RS changes accounted, the results look good. I will update the above list when complete and follow-up with the verifications.

Comment 3

[Mark Banner (:standard8)]

Comment on attachment 9260438 [details]
Bug 1751746 - Change search-telemetry to use strict partner code matching. r?daleharvey!,mkaply!

Beta/Release Uplift Approval Request

• User impact if declined: Prevents unexpected data being submitted as part of search telemetry.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: QE have been testing already and have steps.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Code changes contained within a self-contained code module.
• String changes made/needed: None

Comment 4

[Mark Banner (:standard8)]

Comment on attachment 9260438 [details]
Bug 1751746 - Change search-telemetry to use strict partner code matching. r?daleharvey!,mkaply!

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: See beta/release approval.
• User impact if declined:
• Fix Landed on Version:
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky):

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9260438 [details]
Bug 1751746 - Change search-telemetry to use strict partner code matching. r?daleharvey!,mkaply!

Dot release driver, approved for 97.0b8 so we can start validating ASAP.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/63bf391388c3

Comment 7

[Mark Banner (:standard8)]

Note: esr does not have the file browser/components/urlbar/tests/browser/browser_urlbar_telemetry_handoff.js but the rest of the patch applies fine.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/efc90ca80209

Comment 9

[Dianna Smith [:diannaS] - on Leave, back on August 9th]

Comment on attachment 9260438 [details]
Bug 1751746 - Change search-telemetry to use strict partner code matching. r?daleharvey!,mkaply!

Approved for 96.0.3

Comment 10

[Dianna Smith [:diannaS] - on Leave, back on August 9th]

https://hg.mozilla.org/releases/mozilla-release/rev/08a730393ae6

Comment 11

[Dianna Smith [:diannaS] - on Leave, back on August 9th]

Comment on attachment 9260438 [details]
Bug 1751746 - Change search-telemetry to use strict partner code matching. r?daleharvey!,mkaply!

Approved for 91.5.1esr

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr91/rev/9c78a8c65b7b (default - 91.6esr+)
https://hg.mozilla.org/releases/mozilla-esr91/rev/f560cd3cd43b (FIREFOX_ESR_91_5_X_RELBRANCH - 91.5.1esr)

Comment 13

[Adrian Florinescu [:aflorinescu]]

(In reply to Adrian Florinescu [:aflorinescu] from comment #2)

As the test plan for this change needs to cover Remote Settings changes as well as code changes, the QA team has tested this using both the staging dump-disabled configuration -

1. (done)Reproduce the issue on affected builds.
2. (done)Ensure that the used telemetry RS bucket is v2 not v1.
3. (done)Ensure that the issue doesn't reproduce and the telemetry logged is (google telementry used as an example):

• "google:organic:other" when the code is not maching the new strict list
• "google:organic:none" when the code is not present
• "google:tagged-follow-on:firefox-b-d" for follow-on

4. (done) Ensure all telemetry recording engines are behaving taking in account point 3.

• google (done)
• ddg (done)
• bing (done)
• baidu (done)

5. (done) Wide regression testing on search general telemetry

• [test run] (https://testrail.stage.mozaws.net/index.php?/plans/view/52957)

6. (done) Multiple OS coverage.
   -windows 10
   -mac 11
   -ubuntu 20

   with:
   - 91.5.1esr
   - 96.0.3
   - 97.0b8
   - 98.0a1

7. (done) Regression testing covering the removal of v1 RS telemetry bucket and effects on old versions esr 79 / esr 91 + 1-2 non-train builds (main-preview / RS staging)

• 85.0.2 (Windows 10 + Mac 11)
• 92.0a1 (20210713214232)
• 92.0.1 (Windows 10 + Mac 11

8. (done) Verification(98) and Uplift verification (96 dot / 97b)
9. (not done) Telemetry Test coverage updates to account for the spill.
10. (not in QA scope) Data sanitize