As the test plan for this change needs to cover Remote Settings changes as well as code changes, the QA team has tested this on a [try-build](https://treeherder.mozilla.org/jobs?repo=try&revision=d420f54654efa13c0076618e142bb3c58caaf336) using both the try-build dump and the [staging dump-disabled configuration](https://testrail.stage.mozaws.net/index.php?/cases/view/838881&group_by=cases:section_id&group_order=asc&group_id=130516) - (configuration changes approved meanwhile.) 1. (done)Reproduce the issue on affected builds. 2. (done)Ensure that the used telemetry RS bucket is v2 not v1. 3. (done)Ensure that the issue doesn't reproduce and the telemetry logged is (google telementry used as an example): - "google:organic:other" when the code is not maching the new strict list - "google:organic:none" when the code is not present - "google:tagged-follow-on:firefox-b-d" for follow-on 4. Ensure all telemetry recording engines are behaving taking in account point 3. - google (done) - ddg (done) - bing (done) - baidu (not done - need ln build) 5. (not done) Wide regression testing on search general telemetry 6. (not done) Regression testing covering the removal of v1 RS telemetry bucket and effects on **old** versions esr 79 / esr 91 + 1-2 non-train builds 7. (not done/not ready) Verification(98) and Uplift verification (96 dot / 97b) 8. (not done) Telemetry Test coverage updates to account for the spill. 9. (not in QA scope) Data sanitize All in all, with the current testing, with the fix and the RS changes accounted, the results look good. I will update the above list when complete and follow-up with the verifications.
Bug 1751746 Comment 2 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
As the test plan for this change needs to cover Remote Settings changes as well as code changes, the QA team has tested this on a [try-build](https://treeherder.mozilla.org/jobs?repo=try&revision=d420f54654efa13c0076618e142bb3c58caaf336) using both the try-build dump and the [staging dump-disabled configuration](https://testrail.stage.mozaws.net/index.php?/cases/view/838881&group_by=cases:section_id&group_order=asc&group_id=130516) - (configuration changes approved meanwhile.) 1. (done)Reproduce the issue on affected builds. 2. (done)Ensure that the used telemetry RS bucket is v2 not v1. 3. (done)Ensure that the issue doesn't reproduce and the telemetry logged is (google telementry used as an example): - "google:organic:other" when the code is not maching the new strict list - "google:organic:none" when the code is not present - "google:tagged-follow-on:firefox-b-d" for follow-on 4. Ensure all telemetry recording engines are behaving taking in account point 3. - google (done) - ddg (done) - bing (done) - baidu (not done - need ln build) 5. (not done) Wide regression testing on search general telemetry 6. (not done) Multiple OS coverage. 7. (not done) Regression testing covering the removal of v1 RS telemetry bucket and effects on **old** versions esr 79 / esr 91 + 1-2 non-train builds 8. (not done/not ready) Verification(98) and Uplift verification (96 dot / 97b) 9. (not done) Telemetry Test coverage updates to account for the spill. 10. (not in QA scope) Data sanitize All in all, with the current testing, with the fix and the RS changes accounted, the results look good. I will update the above list when complete and follow-up with the verifications.
As the test plan for this change needs to cover Remote Settings changes as well as code changes, the QA team has tested this on a [try-build](https://treeherder.mozilla.org/jobs?repo=try&revision=d420f54654efa13c0076618e142bb3c58caaf336) using both the try-build dump and the [staging dump-disabled configuration](https://testrail.stage.mozaws.net/index.php?/cases/view/838881&group_by=cases:section_id&group_order=asc&group_id=130516) - (configuration changes approved meanwhile.) 1. (done)Reproduce the issue on affected builds. 2. (done)Ensure that the used telemetry RS bucket is v2 not v1. 3. (done)Ensure that the issue doesn't reproduce and the telemetry logged is (google telementry used as an example): - "google:organic:other" when the code is not maching the new strict list - "google:organic:none" when the code is not present - "google:tagged-follow-on:firefox-b-d" for follow-on 4. Ensure all telemetry recording engines are behaving taking in account point 3. - google (done) - ddg (done) - bing (done) - baidu (done) 5. (not done) Wide regression testing on search general telemetry 6. (not done) Multiple OS coverage. 7. (not done) Regression testing covering the removal of v1 RS telemetry bucket and effects on **old** versions esr 79 / esr 91 + 1-2 non-train builds 8. (not done/not ready) Verification(98) and Uplift verification (96 dot / 97b) 9. (not done) Telemetry Test coverage updates to account for the spill. 10. (not in QA scope) Data sanitize All in all, with the current testing, with the fix and the RS changes accounted, the results look good. I will update the above list when complete and follow-up with the verifications.