1641400 - Thunderbird test failures in toolkit/mozapps/extensions/test/xpcshellrs-blocklist

Status: RESOLVED FIXED

(Toolkit :: Blocklist Implementation, defect)

Description

[Geoff Lankow (:darktrojan)]

Bug 1633466 introduced some new tests with telemetry, which fail when run on a product that doesn't have the appropriate telemetry probes set up (e.g. Thunderbird). This is not an uncommon problem so I'm proposing the usual fix, which is for the tests to pretend the probes aren't missing.

Comment 1

[Geoff Lankow (:darktrojan)]

Attachment: Bug 1641400 - Add blocklist v3 telemetry scalars for Thunderbird. r?robwu

Comment 2

[Rob Wu [:robwu]]

Some questions before we go in the direction of setting the toolkit.telemetry.testing.overrideProductsCheck pref:

• Is telemetry used by Thunderbird?
• Does Thunderbird intend to switch to the v3 blocklist implementation (bug 1620621) at some point? The v2 blocklist implementation will probably be removed at some point.

There are already some blocklist-related telemetry probes for Thunderbird, e.g. at https://searchfox.org/mozilla-central/rev/bc3600def806859c31b2c7ac06e3d69271052a89/toolkit/components/telemetry/Scalars.yaml#4127

If Thunderbird intends to support the v3 blocklist at some point, then I would suggest to add thunderbird to the products for the scalars defined in bug 1633466.

If Thunderbird is not likely going to use the v3 blocklist, then I suggest to disable the test with a skip-if

Comment 3

[Geoff Lankow (:darktrojan)]

I don't know the answer to that, but I think it's highly likely we will switch to v3 if v2 is going away. We might as well add the scalars.

Comment 4

[Magnus Melin [:mkmelin]]

(In reply to Rob Wu [:robwu] from comment #2)

If Thunderbird is not likely going to use the v3 blocklist, then I suggest to disable the test with a skip-if

I agree.
Thunderbird is (soon) using Telemetry. Unclear with the blocklist v3, but either way, this data seems irrelevant for Thunderbird even if we'd use that.

Comment 5

[Magnus Melin [:mkmelin]]

Attachment: Bug 1641400 - skip Thunderbird for blocklist telemetry. r=robwu

Comment 6

[Rob Wu [:robwu]]

(In reply to Magnus Melin [:mkmelin] from comment #4)

(In reply to Rob Wu [:robwu] from comment #2)

If Thunderbird is not likely going to use the v3 blocklist, then I suggest to disable the test with a skip-if

I agree.
Thunderbird is (soon) using Telemetry. Unclear with the blocklist v3, but either way, this data seems irrelevant for Thunderbird even if we'd use that.

This data is used to monitor the health of the addons blocklist. If you're going to use the v3 blocklist, then this data looks relevant to you.

If this data is not of any use to you, then the other two blocklist-related probes at are probably also obsolete: https://searchfox.org/mozilla-central/rev/bc3600def806859c31b2c7ac06e3d69271052a89/toolkit/components/telemetry/Scalars.yaml#4127,4168

With this explanation, are the probes of interest after all?
If not, do you want to update the patch to remove Thunderbird from the two scalars that I listed?

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1633466

Comment 8

[Magnus Melin [:mkmelin]]

Re blocklist v3 we're probably going to have to use it, to avoid problems. The times Thunderbird has had to block add-ons are however, very very few. So in this view, it's very unlikely we'd have any desire to put resources into analysing the data of how a block affected us. At least in the foreseeable future.

The other probes you mention are also not super interesting, but at least they would let us verify blocklists are working.

I'd prefer to just go with the current patch to fix the failures.

Comment 9

[Rob Wu [:robwu]]

(In reply to Magnus Melin [:mkmelin] from comment #8)

The other probes you mention are also not super interesting, but at least they would let us verify blocklists are working.

It only covers the v2 blocklist for addons and the plugin blocklist (does Thunderbird have plugins at all...?).
Once the v3 blocklist is in use, these probes become obsolete and you won't have any coverage any more. But since you acknowledged that and think that it's fine, I have no problem with accepting that.

I'd prefer to just go with the current patch to fix the failures.

Fine by me.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7d6333b49ac8
skip Thunderbird for blocklist telemetry. r=robwu

Comment 11

[Magnus Melin [:mkmelin]]

Thanks! Thunderbird has disabled plug-ins completely.

Comment 12

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/7d6333b49ac8

Comment 13

[Intermittent Failures Robot]

69 failures in 1048 pushes (0.066 failures/push) were associated with this bug yesterday.

Repository breakdown:

• comm-central: 69

Platform breakdown:

• linux64-shippable: 4
• windows10-64-shippable: 6
• macosx1014-64-shippable: 6
• linux32-shippable: 3
• windows7-32: 11
• linux64: 8
• macosx1014-64: 9
• windows7-32-shippable: 6
• linux32: 6
• windows10-64: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1641400&startday=2020-05-28&endday=2020-05-28&tree=all

Comment 14

[Intermittent Failures Robot]

69 failures in 5458 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

This is the #13 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• comm-central: 69

Platform breakdown:

• linux64-shippable: 4
• windows10-64-shippable: 6
• macosx1014-64-shippable: 6
• linux32-shippable: 3
• windows7-32: 11
• linux64: 8
• macosx1014-64: 9
• windows7-32-shippable: 6
• linux32: 6
• windows10-64: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1641400&startday=2020-05-25&endday=2020-05-31&tree=all