(In reply to Ryan VanderMeulen [:RyanVM] from comment #4) > We need to find a way to add tests for this too. It's pretty concerning that nothing in CI went orange with this bug. This is an issue that we've faced in IndexedDB and I think the general proposal would be that to detect profile-related regressions we potentially not only need point-in-time profiles (ex: a profile from the last release, a profile from the last upgrade-path-release), but also for the test infrastructure to have tests run against rolling profiles where the tests run against the preceding night's nightly profile at the end of the run which is based on the previous night's nightly run and exposed to the next night's nightly as long as it passed, etc. In particular, in IDB we regularly create tests that involve point-in-time snapshots, but a regression we encountered was due to newly introduced code that affected nightlies going forward. Manually created snapshots can't cover that unless a team manually creates snapshots every day, etc. (And in the specific case in question someone changed something without review by the people who would know that a new snapshot was required if the change in question hadn't been unsound, etc. I will note herald rules do potentially help address that underlying review and awareness-related issue.)
Bug 1750188 Comment 7 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
(In reply to Ryan VanderMeulen [:RyanVM] from comment #4) > We need to find a way to add tests for this too. It's pretty concerning that nothing in CI went orange with this bug. This is an issue that we've faced in IndexedDB and I think the general proposal would be that to detect profile-related regressions we potentially not only need point-in-time profiles (ex: a profile from the last release, a profile from the last upgrade-path-release), but also for the test infrastructure to have tests run against rolling profiles where the tests run against the preceding night's nightly profile at the end of the run which is based on the previous night's nightly run and exposed to the next night's nightly as long as it passed, etc. edit: Also clearly a fresh profile needs to be in the mix too. In particular, in IDB we regularly create tests that involve point-in-time snapshots, but a regression we encountered was due to newly introduced code that affected nightlies going forward. Manually created snapshots can't cover that unless a team manually creates snapshots every day, etc. (And in the specific case in question someone changed something without review by the people who would know that a new snapshot was required if the change in question hadn't been unsound, etc. I will note herald rules do potentially help address that underlying review and awareness-related issue.)