1516340 - Perma beta browser_privacypane_2.js - deleteOnClose should not be disabled - Got true, expected false

Status: VERIFIED FIXED

(Firefox :: Site Identity, defect, P1)

Description

[Cosmin Sabou [:CosminS]]

[Tracking Requested - why for this release]:

Central as late Beta simulation: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception&revision=a30fb730b7143aecb52e1a2e4e237a04190cebbe&selectedJob=218761682&searchStr=browser%2Cchrome

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=218761682&repo=try&lineNumber=10430

[task 2018-12-25T12:25:03.875Z] 12:25:03     INFO - TEST-PASS | browser/components/preferences/in-content/tests/browser_privacypane_2.js | rememberForms should not be disabled - 
[task 2018-12-25T12:25:03.876Z] 12:25:03     INFO - Buffered messages finished
[task 2018-12-25T12:25:03.877Z] 12:25:03     INFO - TEST-UNEXPECTED-FAIL | browser/components/preferences/in-content/tests/browser_privacypane_2.js | deleteOnClose should not be disabled - Got true, expected false
[task 2018-12-25T12:25:03.878Z] 12:25:03     INFO - Stack trace:
[task 2018-12-25T12:25:03.879Z] 12:25:03     INFO - chrome://mochikit/content/browser-test.js:test_is:1316
[task 2018-12-25T12:25:03.880Z] 12:25:03     INFO - chrome://mochitests/content/browser/browser/components/preferences/in-content/tests/privacypane_tests_perwindow.js:expect_disabled/<:76
[task 2018-12-25T12:25:03.881Z] 12:25:03     INFO - chrome://mochitests/content/browser/browser/components/preferences/in-content/tests/privacypane_tests_perwindow.js:expect_disabled:75
[task 2018-12-25T12:25:03.882Z] 12:25:03     INFO - chrome://mochitests/content/browser/browser/components/preferences/in-content/tests/privacypane_tests_perwindow.js:test_dependent_elements:103
[task 2018-12-25T12:25:03.883Z] 12:25:03     INFO - chrome://mochitests/content/browser/browser/components/preferences/in-content/tests/privacypane_tests_perwindow.js:runTestOnPrivacyPrefPane:11

This happens only on late beta simulation and looks to be a regression from Bug 1466817. Paolo could you please take a look over this? Thank you.

Comment 1

[:Paolo Amadini]

The only test that was changed in bug 1466817 is browser_spotlight.js, and in the log it is executed _after_ the test that fails here. The production code change seems totally unrelated to what browser_privacypane_2.js is testing. There is a new attribute added in the privacy pane implementation file, but after a quick look it seems strange that this would affect the test.

Cosmin, can you verify that the regression bug you identified is correct?

Comment 2

[Raul Gurzau (:RaulG)]

Backed out bug 1466817 in beta sim and the failure is not present anymore.

Central as beta simulation:
https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&revision=bd4531e9dd487fc05d38aa62b04e0b1011a0bf67

Comment 3

[:Paolo Amadini]

Thanks! I'll check this out in the next few days.

Comment 4

[Intermittent Failures Robot]

15 failures in 219 pushes (0.068 failures/push) were associated with this bug yesterday.

Repository breakdown:
* try: 15

Platform breakdown:
* linux64: 2
* linux64-devedition: 1
* windows10-64: 1
* windows7-32-nightly: 1
* windows10-64-devedition: 1
* osx-10-10: 1
* windows7-32-devedition: 1
* linux32-nightly: 1
* windows10-64-nightly: 1
* windows7-32: 1
* linux32-devedition: 1
* linux32: 1
* macosx64-nightly: 1
* macosx64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1516340&startday=2018-12-28&endday=2018-12-28&tree=all

Comment 5

[Intermittent Failures Robot]

19 failures in 1008 pushes (0.019 failures/push) were associated with this bug in the last 7 days.

This is the #40 most frequent failure this week.

Repository breakdown:
* try: 19

Platform breakdown:
* linux64: 3
* windows10-64: 1
* windows7-32-nightly: 1
* windows10-64-devedition: 1
* windows7-32: 1
* linux32-devedition: 2
* linux64-nightly: 1
* windows10-64-nightly: 1
* linux32-nightly: 2
* osx-10-10: 1
* linux32: 1
* macosx64-nightly: 1
* windows7-32-devedition: 1
* macosx64-devedition: 1
* linux64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1516340&startday=2018-12-24&endday=2018-12-30&tree=all

Comment 6

[:Paolo Amadini]

Bug 1466817 isn't the correct regression range, as comment 1 originally suggested.

Comment 2 is irrelevant because the backout was done on an early beta simulation (66.0b1) instead of the original late beta (66.0b12), so this test would likely have succeeded even without the backout. I figured this out after spending quite some time for a full build of this changeset.

I've then built revision a30fb730b714 mentioned in comment 0 locally, and verified that browser_privacypane_2.js fails when run together with its entire test folder, but not individually:

  ./mach test browser/components/preferences/in-content/tests/

However, backing out bug 1466817 from that revision and running the tests in the folder again does not fix the failure.

You need to find the correct regression range here, and I suggest double checking the regressing bug with a cherry-picked backout and posting both the changeset that fails and the one that succeeds.

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Sorry for the time spent on this, Paolo.

Late beta simulation with a backout of https://hg.mozilla.org/mozilla-central/rev/724652219657 / bug 1514853 points to that bug (none of the oranges have a failure message for this test): https://treeherder.mozilla.org/#/jobs?repo=try&revision=a45702316af9a507640676f60b2847323afe2d6a

Ehsan, please take a look. This permafails on beta.

Comment 8

[Intermittent Failures Robot]

32 failures in 312 pushes (0.103 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 32

Platform breakdown:
* macosx64-nightly: 2
* osx-10-10: 2
* linux32-nightly: 2
* linux64-devedition: 2
* linux32-devedition: 2
* windows7-32-nightly: 2
* windows7-32: 2
* linux32: 2
* windows10-64-nightly: 2
* windows10-64: 2
* macosx64-devedition: 2
* windows7-32-devedition: 2
* linux64-nightly: 2
* linux64: 4
* windows10-64-devedition: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1516340&startday=2019-01-02&endday=2019-01-02&tree=all

Comment 9

[(no longer active)]

Johann/Erica, can you please take this?  I don't have the bandwidth right now... Thanks!

Comment 10

[Intermittent Failures Robot]

38 failures in 425 pushes (0.089 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 35
* try: 3

Platform breakdown:
* linux32: 6
* macosx64-nightly: 2
* windows7-32-devedition: 2
* windows7-32: 1
* osx-10-10: 6
* linux32-nightly: 3
* windows10-64: 1
* linux32-devedition: 3
* linux64-nightly: 2
* windows10-64-nightly: 3
* windows10-64-devedition: 1
* linux64-devedition: 2
* linux64: 3
* windows7-32-nightly: 2
* macosx64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1516340&startday=2019-01-03&endday=2019-01-03&tree=all

Comment 11

[Intermittent Failures Robot]

16 failures in 421 pushes (0.038 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 16

Platform breakdown:
* linux64-nightly: 1
* linux32: 1
* windows7-32-nightly: 1
* windows10-64: 1
* linux32-devedition: 1
* windows10-64-devedition: 1
* macosx64-nightly: 1
* linux64-devedition: 1
* windows7-32: 1
* windows10-64-nightly: 1
* macosx64-devedition: 1
* linux32-nightly: 1
* linux64: 2
* osx-10-10: 1
* windows7-32-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1516340&startday=2019-01-04&endday=2019-01-04&tree=all

Comment 12

[Intermittent Failures Robot]

97 failures in 1668 pushes (0.058 failures/push) were associated with this bug in the last 7 days.

This is the #6 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. ** 

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. ** 

Repository breakdown:
* try: 14
* mozilla-beta: 83

Platform breakdown:
* linux32: 10
* windows7-32: 6
* linux64-nightly: 5
* linux64: 11
* windows10-64: 6
* windows10-64-devedition: 4
* windows7-32-nightly: 5
* linux32-nightly: 7
* osx-10-10: 10
* windows7-32-devedition: 5
* macosx64-nightly: 6
* macosx64-devedition: 5
* linux64-devedition: 5
* linux32-devedition: 6
* windows10-64-nightly: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1516340&startday=2018-12-31&endday=2019-01-06&tree=all

Comment 13

[Intermittent Failures Robot]

54 failures in 441 pushes (0.122 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-beta: 48
• try: 6

Platform breakdown:

• windows10-64-devedition: 3
• macosx64-nightly: 3
• windows7-32-nightly: 3
• osx-10-10: 4
• macosx64-devedition: 3
• windows10-64: 3
• linux32-devedition: 3
• linux32: 4
• linux64-devedition: 4
• windows7-32-devedition: 3
• windows7-32: 3
• linux64: 8
• windows10-64-nightly: 3
• linux64-nightly: 4
• linux32-nightly: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1516340&startday=2019-01-07&endday=2019-01-07&tree=all

Comment 14

[Johann Hofmann [:johannh]]

Attachment: Bug 1516340 - Properly clean up prefs in browser_contentblocking.js. r=ewright

Comment 15

[Intermittent Failures Robot]

76 failures in 468 pushes (0.162 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 9
• mozilla-beta: 67

Platform breakdown:

• linux64-devedition: 6
• linux64: 9
• windows10-64-devedition: 5
• windows7-32-nightly: 5
• windows7-32: 5
• osx-10-10: 4
• linux32-nightly: 4
• linux64-nightly: 4
• windows7-32-devedition: 5
• windows10-64: 5
• linux32-devedition: 4
• linux32: 5
• windows10-64-nightly: 5
• macosx64-devedition: 5
• macosx64-nightly: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1516340&startday=2019-01-09&endday=2019-01-09&tree=all

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/993c073ab27a
Properly clean up prefs in browser_contentblocking.js. r=ewright

Comment 17

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/45c1f862dccf

Comment 18

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/993c073ab27a

Comment 19

[Cristina Coroiu [:ccoroiu]]

Verified as fixed with this beta simulation:
https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=c32e089d8ccac4d097bef756163872830c6e32bb

Comment 20

[Intermittent Failures Robot]

140 failures in 2627 pushes (0.053 failures/push) were associated with this bug in the last 7 days.

This is the #4 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• try: 25
• mozilla-beta: 115

Platform breakdown:

• linux32: 9
• linux32-devedition: 8
• windows10-64: 8
• macosx64-devedition: 9
• osx-10-10: 8
• macosx64-nightly: 9
• windows10-64-devedition: 8
• linux32-nightly: 9
• windows10-64-nightly: 8
• linux64: 17
• windows7-32-nightly: 9
• windows7-32: 9
• windows7-32-devedition: 9
• linux64-devedition: 11
• linux64-nightly: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1516340&startday=2019-01-07&endday=2019-01-13&tree=all