1668124 - [beta tree] Intermittent toolkit/components/telemetry/tests/unit/test_TelemetryReportingPolicy.js | test_prefs - [test_prefs : 141] We must be able to upload after the policy is accepted. - false == true

Status: RESOLVED FIXED

(Thunderbird :: Testing Infrastructure, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: thunderbird [at] calypsoblue.org
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=317062374&repo=comm-beta
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/WI2Pl0GBSVGfOr7wb6ueCQ/runs/0/artifacts/public/logs/live_backing.log

---

Seeing this on beta releases on all platforms. I don't see it on esr78 however.```

Comment 1

[Intermittent Failures Robot]

1 failures in 4195 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-beta: 1

Platform breakdown:

• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1668124&startday=2020-09-28&endday=2020-10-04&tree=all

Comment 2

[Intermittent Failures Robot]

3 failures in 4230 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-beta: 3

Platform breakdown:

• windows7-32-shippable: 1
• windows10-64-shippable: 1
• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1668124&startday=2020-10-19&endday=2020-10-25&tree=all

Comment 3

[Khushil Mistry [:khushil324]]

I am not able to reproduce this failure on the local machine with Beta.

Comment 4

[Rob Lemley [:rjl]]

This still happens on every beta on every platform. It would be nice if it didn't.

Comment 5

[Wayne Mery (:wsmwk)]

hasn't happened in over 5 months -> WFM

Comment 6

[Rob Lemley [:rjl]]

I don't know what everyone else is looking at, but we get this test failure on every Thunderbird beta build. On every platform. Every build for 3 years.

Comment 7

[Rob Lemley [:rjl]]

Attachment: Bug 1668124 - Update privacy policy version. r=sancus

The Thunderbird Privacy Policy was updated on March 16, 2021. That should
have triggered a policy version bump to display the new version. This is
the long awaited policy version bump.
This also fixes a perma-fail telemetry test that only occurs on comm-beta
builds every time for the last 3 years.

Comment 8

[Rob Lemley [:rjl]]

Explanation of the submitted patch:

The failing function call in test_TelemetryReportingPolicy.js is TelemetryReportingPolicy.canUpload(). The reason why that call returns false is unfortunately wiped out and never displayed anywhere in the output or console log though.

Eventually, canUpload() will call isUserNotifiedOfCurrentPolicy(). That has this code:

// The accepted policy version should not be less than the minimum policy version.
if (this.dataSubmissionPolicyAcceptedVersion < this.minimumPolicyVersion) {
  return false;
}

At this point in the test, the "user" has just accepted the "currentPolicyVersion". For Thunderbird, that is 1. The minimumPolicyVersion is not explicitly set in Thunderbird's prefs, but it is set in all.js.

#ifdef MOZ_DATA_REPORTING
  pref("datareporting.policy.dataSubmissionEnabled", true);
  pref("datareporting.policy.dataSubmissionPolicyNotifiedTime", "0");
  pref("datareporting.policy.dataSubmissionPolicyAcceptedVersion", 0);
  pref("datareporting.policy.dataSubmissionPolicyBypassNotification", false);
  pref("datareporting.policy.currentPolicyVersion", 2);
  pref("datareporting.policy.minimumPolicyVersion", 1);
  pref("datareporting.policy.minimumPolicyVersion.channel-beta", 2);
  pref("datareporting.policy.firstRunURL", "https://www.mozilla.org/privacy/firefox/");
#endif

But, as the astute reader will no doubt notice, (1 < 1) is false, so the test fails and the function returns true... right?

Not exactly... minimumPolicyVersion is not so simple.

  get minimumPolicyVersion() {
    const minPolicyVersion = Services.prefs.getIntPref(
      TelemetryUtils.Preferences.MinimumPolicyVersion,
      1
    );

    // First check if the current channel has a specific minimum policy version. If not,
    // use the general minimum policy version.
    let channel = "";
    try {
      channel = TelemetryUtils.getUpdateChannel();
    } catch (e) {
      this._log.error(
        "minimumPolicyVersion - Unable to retrieve the current channel."
      );
      return minPolicyVersion;
    }
    const channelPref =
      TelemetryUtils.Preferences.MinimumPolicyVersion + ".channel-" + channel;
    return Services.prefs.getIntPref(channelPref, minPolicyVersion);
  },

The real minimumPolicyVersion for beta build (Firefox and Thunderbird) is in the pref datareporting.policy.minimumPolicyVersion.channel-beta. So, it's 2, and 1 < 2.

You'll notice that currentPolicyVersion as set in all.js is 2 and Thunderbird's all-thunderbird.js overrides it and sets it to 1. (That code is from 2019 and predates this bug, so timeline-wise this all makes sense.)

So the fix is to make the versions align with test expectations. As noted in the commit message, Thunderbird's Privacy Policy was updated in March 2021, and the currentVersion pref was never bumped.

Alternatively, all-thunderbird.js could explicitly set currentPolicyVersion, minimumPolicyVersion, minimumPolicyVersion.channel-beta to completely different values. (Could even have a minimumPolicyVersion.channel-release and minimumPolicyVersion.channel-esr if desired.) Arguably, perhaps that should be done anyway since it's coincidence that the version numbers align with the Firefox ones right now. I'll leave that up to the data stewards. I just want the test permafail to go away.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/29e74e8a589e
Update privacy policy version. r=sancus

Comment 10

[Intermittent Failures Robot]

5 failures in 3885 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-beta: 5

Platform and build breakdown:

• windows11-32-2009-shippable-qr: 1
  • opt: 1
• linux1804-64-shippable-qr: 1
  • opt: 1
• windows11-64-2009-shippable-qr: 2
  • opt: 2
• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1668124&startday=2023-06-05&endday=2023-06-11&tree=all

Comment 11

[Rob Lemley [:rjl]]

Comment on attachment 9337931 [details]
Bug 1668124 - Update privacy policy version. r=sancus

[Approval Request Comment]
Regression caused by (bug #): N/A
User impact if declined: none
Testing completed (on c-c, etc.):
Risk to taking this patch (and alternatives if risky):
This is to fix a perma fail orange test only seen on comm-beta for the past 3 years or something insane. The effect will be that users will get prompted to view the privacy policy again. Note that the policy was updated in 2021 for the Mailfence partnership and Thunderbird did not bump the version number in order to force the prompt.

Comment 12

[Wayne Mery (:wsmwk)]

Comment on attachment 9337931 [details]
Bug 1668124 - Update privacy policy version. r=sancus

[Triage Comment]
Approved for beta

Comment 13

[Rob Lemley [:rjl]]

Thunderbird 115.0b2:
https://hg.mozilla.org/releases/comm-beta/rev/858e68cb718d

Comment 14

[Magnus Melin [:mkmelin]]

\o/
Thanks Rob, for getting this cleared up!