Closed Bug 1778009 Opened 3 months ago Closed 3 months ago

latest update of browser compatibility data breaks tests

Categories

(DevTools :: Inspector: Compatibility, defect)

defect

Tracking

(firefox-esr102 fixed, firefox102 fixed, firefox103 fixed, firefox104 fixed)

RESOLVED FIXED
104 Branch
Tracking Status
firefox-esr102 --- fixed
firefox102 --- fixed
firefox103 --- fixed
firefox104 --- fixed

People

(Reporter: aryx, Assigned: jdescottes)

Details

Attachments

(1 file)

See this failure - browser compatibility data tests fail after the latest data update which had to be rolled back because of this.

Daisuke, could you investigate this or redirect the request, please?

Flags: needinfo?(daisuke)

Thank you very much for the report, Sebastian!

Julian, as we talked on slack, could you take a look at this, please?

Flags: needinfo?(daisuke) → needinfo?(jdescottes)

The data dump removed IE, which fails a lot of expectations in our test suite.
For instance in browser_compatibility_css-property_issue.js, we expect 2 issues for the selected element, but one of them is for IE and since it's not a listed browser anymore, the issue is not displayed anymore.
Others are more straightforward such as browser_compatibility_settings.js where the list of browsers simply no longer matches.

Nicolas: do we have any process to address this scenario, when a data update breaks our existing tests? Or should we rewrite our tests to use a mock data set?

In the meantime, I can provide a patch to fix our tests with this data dump, but they will have to land together.

Flags: needinfo?(jdescottes) → needinfo?(nchevobbe)

I should actually have a version that works both with central and with the data dump, taking the bug.

Assignee: nobody → jdescottes
Status: NEW → ASSIGNED

The compatibility panel tests should now work both with the current browser data and with the update from D150961
We will still need to find a reliable way to avoid breakage when a data dump occurs.

Pushed by dakatsuka.birchill@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/01aaf17a51fe
[devtools] Stop expecting IE warnings in compatibility panel tests r=daisuke
Regressions: 1778034
No longer regressions: 1778034

(In reply to Julian Descottes [:jdescottes] from comment #3)

The data dump removed IE, which fails a lot of expectations in our test suite.
For instance in browser_compatibility_css-property_issue.js, we expect 2 issues for the selected element, but one of them is for IE and since it's not a listed browser anymore, the issue is not displayed anymore.
Others are more straightforward such as browser_compatibility_settings.js where the list of browsers simply no longer matches.

Nicolas: do we have any process to address this scenario, when a data update breaks our existing tests? Or should we rewrite our tests to use a mock data set?

Not really. I had an issue at some point because a property we were using in a test got supported by a browser and I fixed the test by using an other property. I can see that you used a similar approach in your patch, so we should probably be good

Flags: needinfo?(nchevobbe)

That feels a bit flaky :) Ideally we should either run our tests with mocked data (or a fixed version of the data, to be updated periodically as a release task), or whatever process triggers the data dumps should also run our tests and involve us in case fixes are needed.

Status: ASSIGNED → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → 104 Branch

FWIW, the search team has a similar issue when occasionally they need to update search configs and some tests in sync. When that happens they coordinate updating the config in remote-settings and the test changes with relman so nobody is surprised.

Oops didn't mean to set all those flags.

You need to log in before you can comment on or make changes to this bug.