1605297 - Intermittent browser/extensions/doh-rollout/test/browser/browser_policyOverride.js | Uncaught exception - undefined - timed out after 50 tries.

Status: RESOLVED FIXED

(Firefox :: Security, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: rmaries [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=282033576&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/GaSxCXqkRomtlyXokwhzLA/runs/0/artifacts/public/logs/live_backing.log

---

[task 2019-12-20T03:18:09.482Z] 03:18:09 INFO - TEST-START | browser/extensions/doh-rollout/test/browser/browser_policyOverride.js
[task 2019-12-20T03:18:17.610Z] 03:18:17 INFO - TEST-INFO | started process screencapture
[task 2019-12-20T03:18:17.792Z] 03:18:17 INFO - TEST-INFO | screencapture: exit 0
[task 2019-12-20T03:18:17.792Z] 03:18:17 INFO - Buffered messages logged at 03:18:09
[task 2019-12-20T03:18:17.792Z] 03:18:17 INFO - Entering test bound testPolicyOverride
[task 2019-12-20T03:18:17.792Z] 03:18:17 INFO - TEST-PASS | browser/extensions/doh-rollout/test/browser/browser_policyOverride.js | Sanity check the temporary file doesn't exist. - true == true -
[task 2019-12-20T03:18:17.792Z] 03:18:17 INFO - TEST-PASS | browser/extensions/doh-rollout/test/browser/browser_policyOverride.js | Policy engine is active. -
[task 2019-12-20T03:18:17.792Z] 03:18:17 INFO - Buffered messages finished
[task 2019-12-20T03:18:17.793Z] 03:18:17 INFO - TEST-UNEXPECTED-FAIL | browser/extensions/doh-rollout/test/browser/browser_policyOverride.js | Uncaught exception - undefined - timed out after 50 tries.
[task 2019-12-20T03:18:17.793Z] 03:18:17 INFO - Leaving test bound testPolicyOverride
[task 2019-12-20T03:18:17.793Z] 03:18:17 INFO - GECKO(2044) | MEMORY STAT | vsize 7594MB | residentFast 308MB | heapAllocated 93MB
[task 2019-12-20T03:18:17.793Z] 03:18:17 INFO - TEST-OK | browser/extensions/doh-rollout/test/browser/browser_policyOverride.js | took 8142ms
[task 2019-12-20T03:18:17.793Z] 03:18:17 INFO - checking window state

Comment 1

[Razvan Maries]

I think https://hg.mozilla.org/integration/autoland/rev/f92d4db5d7e4 is the culprit.

Nihanth, could you take a look?

Comment 2

[Intermittent Failures Robot]

39 failures in 767 pushes (0.051 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 30
• mozilla-central: 8
• try: 1

Platform breakdown:

• windows7-32-shippable: 2
• linux64-shippable: 3
• windows10-64-shippable: 2
• macosx1014-64-shippable: 21
• windows10-64: 4
• linux64: 5
• windows7-32: 1
• windows10-64-shippable-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2019-12-20&endday=2019-12-20&tree=all

Comment 3

[Cristina Coroiu [:ccoroiu]]

Retriggers: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=282297553&searchStr=OS%2CX%2C10.14%2CShippable%2Copt%2CMochitests%2Ctest-macosx1014-64-shippable%2Fopt-mochitest-browser-chrome-e10s-3%2CM%28bc3%29&tochange=da4d394f0f2c5bf38cf1d883595cd92f19cc630f&fromchange=9cb9c105a8036a2eea1003e7773cf725de89865f

Comment 4

[Intermittent Failures Robot]

60 failures in 4582 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

This is the #20 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 41
• mozilla-central: 17
• try: 2

Platform breakdown:

• macosx1014-64-shippable: 33
• windows7-32-shippable: 2
• linux64-shippable: 3
• windows10-64-shippable: 2
• windows10-64: 7
• linux64: 9
• windows7-32: 1
• windows10-64-shippable-qr: 2
• macosx1014-64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2019-12-16&endday=2019-12-22&tree=all

Comment 5

[Andreea Pavel [:apavel]]

Retrigger results confirm Razvan's comment above

Comment 6

[(inactive) Nihanth Subramanya [:nhnt11]]

I'll dig into this tomorrow. Took a quick look today and the problem was not obvious.

Comment 7

[Intermittent Failures Robot]

15 failures in 375 pushes (0.04 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• try: 2
• mozilla-beta: 1

Platform breakdown:

• macosx1014-64-shippable: 11
• linux64: 3
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2019-12-23&endday=2019-12-23&tree=all

Comment 8

[Intermittent Failures Robot]

17 failures in 235 pushes (0.072 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 10
• try: 4
• mozilla-central: 3

Platform breakdown:

• windows10-64: 4
• macosx1014-64-shippable: 6
• macosx1014-64-devedition: 3
• linux64: 2
• windows10-64-shippable: 1
• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2019-12-24&endday=2019-12-24&tree=all

Comment 9

[Intermittent Failures Robot]

16 failures in 221 pushes (0.072 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 7
• mozilla-central: 3
• try: 3
• mozilla-beta: 3

Platform breakdown:

• macosx1014-64-shippable: 9
• linux64: 2
• windows10-64: 2
• macosx1014-64-devedition: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2019-12-26&endday=2019-12-26&tree=all

Comment 10

[Intermittent Failures Robot]

15 failures in 304 pushes (0.049 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 9
• mozilla-beta: 1
• mozilla-central: 3
• try: 2

Platform breakdown:

• macosx1014-64-shippable: 10
• macosx1014-64-devedition: 2
• linux64: 1
• windows10-64-qr: 1
• windows10-64-shippable-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2019-12-27&endday=2019-12-27&tree=all

Comment 11

[Intermittent Failures Robot]

17 failures in 89 pushes (0.191 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 6
• autoland: 3
• try: 8

Platform breakdown:

• windows10-64-shippable-qr: 1
• macosx1014-64-shippable: 8
• linux64: 2
• windows7-32-shippable: 1
• macosx1014-64-devedition: 4
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2019-12-28&endday=2019-12-28&tree=all

Comment 12

[Daniel Varga [:dvarga]]

Attachment: Bug 1605297 - disable browser_policyOverride.js for frequent failures on windows, linux and macosx. r=#intermittent-reviewers

Comment 13

[Daniel Varga [:dvarga]]

Nihanth, this made it to the disable list, with 122 failures in the last 21 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-12-08&endday=2019-12-29&tree=trunk&bug=1605297
Please let us know if you want us to disable this test or if you can fix it.

Comment 14

[Intermittent Failures Robot]

91 failures in 1365 pushes (0.067 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 45
• try: 22
• mozilla-beta: 5
• mozilla-central: 19

Platform breakdown:

• macosx1014-64-shippable: 50
• linux64: 11
• windows10-64: 10
• macosx1014-64-devedition: 13
• windows10-64-shippable: 1
• windows10-64-qr: 2
• windows10-64-shippable-qr: 3
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2019-12-23&endday=2019-12-29&tree=all

Comment 15

[Daniel Varga [:dvarga]]

Attachment: Bug 1605297 - disable browser_policyOverride.js for frequent failures. r=#intermittent-reviewers

Comment 16

[(inactive) Nihanth Subramanya [:nhnt11]]

Working on it today, but please go ahead and disable it if this is crossing our failure-rate thresholds.

Comment 17

[(inactive) Nihanth Subramanya [:nhnt11]]

Attachment: Bug 1605297 - Wait for balrog migration at test start to prevent intermittent failures in DoH tests. r=JuniorHsu!

There was a race between the add-on adding its pref listener and us flipping the enabled pref. By waiting for
the balrog migration, we give the add-on time to start listening for the pref before setting it.

Comment 18

[Intermittent Failures Robot]

24 failures in 309 pushes (0.078 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• try: 2
• mozilla-central: 1

Platform breakdown:

• linux64: 5
• windows10-64: 5
• linux64-shippable: 2
• macosx1014-64-shippable: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2019-12-31&endday=2019-12-31&tree=all

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b2027d7f6480
Wait for balrog migration at test start to prevent intermittent failures in DoH tests. r=JuniorHsu

Comment 20

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/eee9db89e4a7
Backed out changeset b2027d7f6480 for failures in browser_cleanFlow.js CLOSED TREE

Comment 21

[Noemi Erli[:noemi_erli]]

Backed out changeset b2027d7f6480 (Bug 1605297) for failures in browser_cleanFlow.js CLOSED TREE

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=283230101&resultStatus=testfailed%2Cbusted%2Cexception&revision=b2027d7f6480ca2b3275c0e97a859749cc137307

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=283230101&repo=autoland&lineNumber=2388

Backout: https://hg.mozilla.org/integration/autoland/rev/eee9db89e4a7ac42ed3fcf479de2541a2abeab1e

Comment 22

[Intermittent Failures Robot]

27 failures in 403 pushes (0.067 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 9
• try: 3
• autoland: 15

Platform breakdown:

• windows10-64-qr: 2
• linux64-shippable: 4
• macosx1014-64-devedition: 1
• windows10-64: 3
• windows10-64-shippable: 3
• windows10-64-shippable-qr: 1
• macosx1014-64-shippable: 9
• windows7-32-shippable: 2
• linux64: 1
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2020-01-02&endday=2020-01-02&tree=all

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bbf630f15308
disable browser_policyOverride.js for frequent failures. r=jmaher

Comment 24

[Intermittent Failures Robot]

18 failures in 477 pushes (0.038 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 9
• mozilla-central: 2
• try: 6
• mozilla-release: 1

Platform breakdown:

• macosx1014-64-shippable: 8
• windows10-64: 3
• linux64: 4
• macosx1014-64-devedition: 2
• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2020-01-03&endday=2020-01-03&tree=all

Comment 25

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/bbf630f15308

Comment 26

[Intermittent Failures Robot]

88 failures in 1862 pushes (0.047 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 52
• mozilla-beta: 1
• mozilla-central: 19
• try: 13
• mozilla-release: 3

Platform breakdown:

• macosx1014-64-shippable: 37
• macosx1014-64-devedition: 4
• linux64: 15
• windows10-64: 15
• linux64-shippable: 8
• windows10-64-qr: 2
• windows10-64-shippable: 3
• windows10-64-shippable-qr: 1
• windows7-32-shippable: 2
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2019-12-30&endday=2020-01-05&tree=all

Comment 27

[Intermittent Failures Robot]

6 failures in 4472 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 2
• try: 4

Platform breakdown:

• macosx1014-64-shippable: 4
• windows10-64-qr: 1
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2020-01-06&endday=2020-01-12&tree=all

Comment 28

[Intermittent Failures Robot]

8 failures in 3773 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3
• autoland: 2
• mozilla-release: 1
• mozilla-central: 2

Platform breakdown:

• linux64: 1
• windows7-32: 3
• windows7-32-shippable: 3
• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2020-01-13&endday=2020-01-19&tree=all

Comment 29

[Intermittent Failures Robot]

21 failures in 732 pushes (0.029 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 10
• mozilla-central: 9
• try: 2

Platform breakdown:

• windows7-32-shippable: 14
• windows7-32: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2020-01-21&endday=2020-01-21&tree=all

Comment 30

[Intermittent Failures Robot]

30 failures in 827 pushes (0.036 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 17
• mozilla-central: 6
• try: 5
• mozilla-beta: 2

Platform breakdown:

• windows7-32-shippable: 22
• windows7-32: 5
• windows7-32-devedition: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2020-01-22&endday=2020-01-22&tree=all

Comment 31

[(inactive) Nihanth Subramanya [:nhnt11]]

Attachment: Bug 1605297 - Don't expect telemetry after restarting add-on in policy override test. r=johannh!

Comment 32

[Intermittent Failures Robot]

23 failures in 838 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-beta: 20
• mozilla-central: 2
• try: 1

Platform breakdown:

• windows7-32-shippable: 11
• windows7-32-devedition: 10
• windows7-32: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2020-01-23&endday=2020-01-23&tree=all

Comment 33

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2a0d5d66eb16
Don't expect telemetry after restarting add-on in policy override test. r=johannh

Comment 34

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/2a0d5d66eb16

Comment 35

[Bogdan Tara[:bogdan_tara | bogdant]]

This is perma failing on Beta:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&searchStr=windows%2C7%2Cdevedition%2Copt%2Cmochitests%2Ctest-windows7-32-devedition%2Fopt-mochitest-browser-chrome-e10s-6%2Cm%28bc6%29&fromchange=6a9e651e08318cd1a5da7f1bc2b489e47d4acf3f&tochange=38fc7c5ed55f4bdbdc837e11bb67eb6bfde947c9&selectedJob=286481745

nhnt, could you, please, take a look?

Comment 36

[Cosmin Sabou [:CosminS]]

This applies cleanly on beta rev https://hg.mozilla.org/releases/mozilla-beta/rev/d3ab563a434bfafa13aaad2dcde764104367336a.
Nihanth, could you please submit an uplift request for rev in comment 33 to get rid of the permafails in browser_policyOverride on beta? Thank you.

Comment 37

[Intermittent Failures Robot]

115 failures in 4004 pushes (0.029 failures/push) were associated with this bug in the last 7 days.

This is the #4 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• try: 29
• autoland: 28
• mozilla-central: 18
• mozilla-beta: 40

Platform breakdown:

• linux64: 1
• windows7-32-shippable: 67
• windows7-32: 26
• windows7-32-devedition: 21

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2020-01-20&endday=2020-01-26&tree=all

Comment 38

[(inactive) Nihanth Subramanya [:nhnt11]]

Comment on attachment 9122571 [details]
Bug 1605297 - Don't expect telemetry after restarting add-on in policy override test. r=johannh!

Beta/Release Uplift Approval Request

• User impact if declined: None
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This fixes a permafailing test. The fix has been verified. It's a one-line change in a test script. No user-facing code touched.
• String changes made/needed:

Comment 39

[Julien Cristau [:jcristau]]

Comment on attachment 9122571 [details]
Bug 1605297 - Don't expect telemetry after restarting add-on in policy override test. r=johannh!

test-only change, doesn't need approval.

Comment 40

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/releases/mozilla-beta/rev/c24dfbda267e

Comment 41

[Intermittent Failures Robot]

2 failures in 2190 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• windows7-32-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2020-01-27&endday=2020-02-02&tree=all

Comment 42

[Intermittent Failures Robot]

2 failures in 3364 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• windows7-32-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1605297&startday=2020-02-03&endday=2020-02-09&tree=all

Comment 43

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 44

[(inactive) Nihanth Subramanya [:nhnt11]]

The DoH heuristics code was ported into a JS component, and several mechanics reworked. The tests live in a new directory and have all been re-enabled, and I'd like to track any new intermittents in new bugs. Closing all open intermittent test failure bugs associated with the old extension.