Bugzilla

(In reply to Adrian Florinescu [:aflorinescu] from comment #2)

As the test plan for this change needs to cover Remote Settings changes as well as code changes, the QA team has tested this on a [try-build](https://treeherder.mozilla.org/jobs?repo=try&revision=d420f54654efa13c0076618e142bb3c58caaf336) using both the try-build dump and the [staging dump-disabled configuration](https://testrail.stage.mozaws.net/index.php?/cases/view/838881&group_by=cases:section_id&group_order=asc&group_id=130516) - (configuration changes approved meanwhile.)

1. (done)Reproduce the issue on affected builds.
2. (done)Ensure that the used telemetry RS bucket is v2 not v1.
3. (done)Ensure that the issue doesn't reproduce and the telemetry logged is (google telementry used as an example):
-   "google:organic:other" when the code is not maching the new strict list
-    "google:organic:none" when the code is not present
-   "google:tagged-follow-on:firefox-b-d" for follow-on
4. (done) Ensure all telemetry recording engines are behaving taking in account point 3.
- google (done)
- ddg (done)
- bing (done)
- baidu (done)

5. (done)  Wide regression testing on search general telemetry
- [test run] (https://testrail.stage.mozaws.net/index.php?/plans/view/52957)
6. (done) Multiple OS coverage.
-windows 10
-mac 11
-ubuntu 20

with

91.5.1esr
96.0.3


7. (done) Regression testing covering the removal of v1 RS telemetry bucket and effects on **old** versions esr 79 / esr 91 + 1-2 non-train builds (main-preview / RS staging)
- 85.0.2 (Windows 10 + Mac 11)
- 92.0a1 (20210713214232)
- 92.0.1 (Windows 10 + Mac 11

8. (done) Verification(98) and Uplift verification (96 dot / 97b)
9. (not done) Telemetry Test coverage updates to account for the spill.
10. (not in QA scope)  Data sanitize

(In reply to Adrian Florinescu [:aflorinescu] from comment #2)

As the test plan for this change needs to cover Remote Settings changes as well as code changes, the QA team has tested this on a [try-build](https://treeherder.mozilla.org/jobs?repo=try&revision=d420f54654efa13c0076618e142bb3c58caaf336) using both the try-build dump and the [staging dump-disabled configuration](https://testrail.stage.mozaws.net/index.php?/cases/view/838881&group_by=cases:section_id&group_order=asc&group_id=130516) - (configuration changes approved meanwhile.)

1. (done)Reproduce the issue on affected builds.
2. (done)Ensure that the used telemetry RS bucket is v2 not v1.
3. (done)Ensure that the issue doesn't reproduce and the telemetry logged is (google telementry used as an example):
-   "google:organic:other" when the code is not maching the new strict list
-    "google:organic:none" when the code is not present
-   "google:tagged-follow-on:firefox-b-d" for follow-on
4. (done) Ensure all telemetry recording engines are behaving taking in account point 3.
- google (done)
- ddg (done)
- bing (done)
- baidu (done)

5. (done)  Wide regression testing on search general telemetry
- [test run] (https://testrail.stage.mozaws.net/index.php?/plans/view/52957)
6. (done) Multiple OS coverage.
-windows 10
-mac 11
-ubuntu 20

with

91.5.1esr
96.0.3
97.0b8
98.0a1

7. (done) Regression testing covering the removal of v1 RS telemetry bucket and effects on **old** versions esr 79 / esr 91 + 1-2 non-train builds (main-preview / RS staging)
- 85.0.2 (Windows 10 + Mac 11)
- 92.0a1 (20210713214232)
- 92.0.1 (Windows 10 + Mac 11

8. (done) Verification(98) and Uplift verification (96 dot / 97b)
9. (not done) Telemetry Test coverage updates to account for the spill.
10. (not in QA scope)  Data sanitize

(In reply to Adrian Florinescu [:aflorinescu] from comment #2)

As the test plan for this change needs to cover Remote Settings changes as well as code changes, the QA team has tested this  using both the [staging dump-disabled configuration](https://testrail.stage.mozaws.net/index.php?/cases/view/838881&group_by=cases:section_id&group_order=asc&group_id=130516) - 

1. (done)Reproduce the issue on affected builds.
2. (done)Ensure that the used telemetry RS bucket is v2 not v1.
3. (done)Ensure that the issue doesn't reproduce and the telemetry logged is (google telementry used as an example):
-   "google:organic:other" when the code is not maching the new strict list
-    "google:organic:none" when the code is not present
-   "google:tagged-follow-on:firefox-b-d" for follow-on
4. (done) Ensure all telemetry recording engines are behaving taking in account point 3.
- google (done)
- ddg (done)
- bing (done)
- baidu (done)

5. (done)  Wide regression testing on search general telemetry
- [test run] (https://testrail.stage.mozaws.net/index.php?/plans/view/52957)
6. (done) Multiple OS coverage.
-windows 10
-mac 11
-ubuntu 20

with

91.5.1esr
96.0.3
97.0b8
98.0a1

7. (done) Regression testing covering the removal of v1 RS telemetry bucket and effects on **old** versions esr 79 / esr 91 + 1-2 non-train builds (main-preview / RS staging)
- 85.0.2 (Windows 10 + Mac 11)
- 92.0a1 (20210713214232)
- 92.0.1 (Windows 10 + Mac 11

8. (done) Verification(98) and Uplift verification (96 dot / 97b)
9. (not done) Telemetry Test coverage updates to account for the spill.
10. (not in QA scope)  Data sanitize

(In reply to Adrian Florinescu [:aflorinescu] from comment #2)

As the test plan for this change needs to cover Remote Settings changes as well as code changes, the QA team has tested this  using both the [staging dump-disabled configuration](https://testrail.stage.mozaws.net/index.php?/cases/view/838881&group_by=cases:section_id&group_order=asc&group_id=130516) - 

1. (done)Reproduce the issue on affected builds.
2. (done)Ensure that the used telemetry RS bucket is v2 not v1.
3. (done)Ensure that the issue doesn't reproduce and the telemetry logged is (google telementry used as an example):
-   "google:organic:other" when the code is not maching the new strict list
-    "google:organic:none" when the code is not present
-   "google:tagged-follow-on:firefox-b-d" for follow-on
4. (done) Ensure all telemetry recording engines are behaving taking in account point 3.
- google (done)
- ddg (done)
- bing (done)
- baidu (done)

5. (done)  Wide regression testing on search general telemetry
- [test run] (https://testrail.stage.mozaws.net/index.php?/plans/view/52957)
6. (done) Multiple OS coverage.
-windows 10
-mac 11
-ubuntu 20

     with:
         - 91.5.1esr
         - 96.0.3
         - 97.0b8
         - 98.0a1

7. (done) Regression testing covering the removal of v1 RS telemetry bucket and effects on **old** versions esr 79 / esr 91 + 1-2 non-train builds (main-preview / RS staging)
- 85.0.2 (Windows 10 + Mac 11)
- 92.0a1 (20210713214232)
- 92.0.1 (Windows 10 + Mac 11

8. (done) Verification(98) and Uplift verification (96 dot / 97b)
9. (not done) Telemetry Test coverage updates to account for the spill.
10. (not in QA scope)  Data sanitize