1778009 - latest update of browser compatibility data breaks tests

Status: RESOLVED FIXED

(DevTools :: Inspector: Compatibility, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

See this failure - browser compatibility data tests fail after the latest data update which had to be rolled back because of this.

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Daisuke, could you investigate this or redirect the request, please?

Comment 2

[Daisuke Akatsuka (:daisuke)]

Thank you very much for the report, Sebastian!

Julian, as we talked on slack, could you take a look at this, please?

Comment 3

[Julian Descottes [:jdescottes]]

The data dump removed IE, which fails a lot of expectations in our test suite.
For instance in browser_compatibility_css-property_issue.js, we expect 2 issues for the selected element, but one of them is for IE and since it's not a listed browser anymore, the issue is not displayed anymore.
Others are more straightforward such as browser_compatibility_settings.js where the list of browsers simply no longer matches.

Nicolas: do we have any process to address this scenario, when a data update breaks our existing tests? Or should we rewrite our tests to use a mock data set?

In the meantime, I can provide a patch to fix our tests with this data dump, but they will have to land together.

Comment 4

[Julian Descottes [:jdescottes]]

I should actually have a version that works both with central and with the data dump, taking the bug.

Comment 5

[Julian Descottes [:jdescottes]]

Attachment: Bug 1778009 - [devtools] Stop expecting IE warnings in compatibility panel tests

The compatibility panel tests should now work both with the current browser data and with the update from D150961
We will still need to find a reliable way to avoid breakage when a data dump occurs.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/01aaf17a51fe
[devtools] Stop expecting IE warnings in compatibility panel tests r=daisuke

Comment 7

[Nicolas Chevobbe [:nchevobbe]]

(In reply to Julian Descottes [:jdescottes] from comment #3)

The data dump removed IE, which fails a lot of expectations in our test suite.
For instance in browser_compatibility_css-property_issue.js, we expect 2 issues for the selected element, but one of them is for IE and since it's not a listed browser anymore, the issue is not displayed anymore.
Others are more straightforward such as browser_compatibility_settings.js where the list of browsers simply no longer matches.

Nicolas: do we have any process to address this scenario, when a data update breaks our existing tests? Or should we rewrite our tests to use a mock data set?

Not really. I had an issue at some point because a property we were using in a test got supported by a browser and I fixed the test by using an other property. I can see that you used a similar approach in your patch, so we should probably be good

Comment 8

[Julian Descottes [:jdescottes]]

That feels a bit flaky :) Ideally we should either run our tests with mocked data (or a fixed version of the data, to be updated periodically as a release task), or whatever process triggers the data dumps should also run our tests and involve us in case fixes are needed.

Comment 9

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/01aaf17a51fe

Comment 10

[Julien Cristau [:jcristau]]

FWIW, the search team has a similar issue when occasionally they need to update search configs and some tests in sync. When that happens they coordinate updating the config in remote-settings and the test changes with relman so nobody is surprised.

Comment 11

[Julien Cristau [:jcristau]]

Oops didn't mean to set all those flags.

Comment 12

[Donal Meehan [:dmeehan]]

https://hg.mozilla.org/releases/mozilla-beta/rev/dcfe56ed0822

Comment 13

[Donal Meehan [:dmeehan]]

https://hg.mozilla.org/releases/mozilla-esr102/rev/3226ab8b82ee

Comment 14

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/7551f0f20faa