1386226 - Perma failure when 56 merges to beta in browser/base/content/test/newtab/browser_newtab_focus.js | Validate focus count in the new tab page. on OSX and Windows with opt or pgo with e10s

Status: VERIFIED FIXED

(Firefox :: New Tab Page, defect, P1)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Merge day is tomorrow (August 2nd):

Test run with failures: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b47de49ad58a9f315a1a6ddcbe3c8fbd5db1a38d&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=running&filter-resultStatus=pending&filter-resultStatus=runnable&filter-searchStr=browser-chrome

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=119721671&repo=try

01:55:30     INFO - TEST-START | browser/base/content/test/newtab/browser_newtab_focus.js
01:55:31     INFO - TEST-INFO | started process screencapture
01:55:31     INFO - TEST-INFO | screencapture: exit 0
01:55:31     INFO - Buffered messages logged at 01:55:30
01:55:31     INFO - innerHeight,,884
01:55:31     INFO - innerWidth,,1030
01:55:31     INFO - Entering test bound setup
01:55:31     INFO - Leaving test bound setup
01:55:31     INFO - Entering test bound 
01:55:31     INFO - Buffered messages finished
01:55:31     INFO - TEST-UNEXPECTED-FAIL | browser/base/content/test/newtab/browser_newtab_focus.js | Validate focus count in the new tab page. - 
01:55:31     INFO - Stack trace:
01:55:31     INFO - chrome://mochitests/content/browser/browser/base/content/test/newtab/browser_newtab_focus.js:countFocus:58
01:55:31     INFO - chrome://mochitests/content/browser/browser/base/content/test/newtab/browser_newtab_focus.js:null:34
01:55:31     INFO - TEST-PASS | browser/base/content/test/newtab/browser_newtab_focus.js | Validate focus count in the new tab page. - 
01:55:31     INFO - Leaving test bound

Comment 1

[Ed Lee :Mardak]

Can someone help take a look if not busy with a-s Fx56 issues? This shouldn't be activity stream as it's preffed off in beta and newtab tests should be running with activity stream off anyway.

Comment 2

[Ed Lee :Mardak]

Fischer, I see that there were a few recent changes to this test for onboarding. Do you know if onboarding could be affecting this test is a strange way when it goes to beta?

Comment 3

[Fischer [:Fischer]]

(In reply to Ed Lee :Mardak from comment #2)
> Fischer, I see that there were a few recent changes to this test for
> onboarding. Do you know if onboarding could be affecting this test is a
> strange way when it goes to beta?
The failure reason is in fact simple.
The bug 1375793 turned on the onboarding on all channel.
However, the browser_newtab_focus.js still bear the AppConstants.NIGHTLY_BUILD condition [1].
So it thought the onboarding is disabled on Beta but actually enabled.
I will update the test right away.

[1] http://searchfox.org/mozilla-central/rev/bbc1c59e460a27b20929b56489e2e55438de81fa/browser/base/content/test/newtab/browser_newtab_focus.js#12

Comment 4

[Fischer [:Fischer]]

Attachment: Bug 1386226 - Perma failure when 56 merges to beta in browser/base/content/test/newtab/browser_newtab_focus.js,

This commit

- Fixes the focus count failure. The bug 1375793 turned on the onboarding on all channels but this test still checks `AppConstants.NIGHTLY_BUILD`. Hence although the onboarding is enabled on Beta, this test thought it is disabled and missed counting the onboarding elements. The solution is to remove `AppConstants.NIGHTLY_BUILD` condition.

- Updates the `promiseTourNotificationOpened` function the same as the bug 1383070 btw

Review commit: https://reviewboard.mozilla.org/r/163750/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/163750/

Comment 5

[Dave Townsend [:mossop]]

Comment on attachment 8892774 [details]
Bug 1386226 - Perma failure when 56 merges to beta in browser/base/content/test/newtab/browser_newtab_focus.js,

https://reviewboard.mozilla.org/r/163750/#review169324

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/04e37b6c85d5
Perma failure when 56 merges to beta in browser/base/content/test/newtab/browser_newtab_focus.js, r=mossop

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/04e37b6c85d5

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/cc969bb39edc

Comment 9

[Intermittent Failures Robot]

20 failures in 888 pushes (0.023 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* try: 20

Platform breakdown:
* windows7-32: 8
* windows8-64: 7
* osx-10-10: 2
* macosx64-nightly: 2
* macosx64-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1386226&startday=2017-07-31&endday=2017-08-06&tree=all

Comment 10

[Fischer [:Fischer]]

(In reply to OrangeFactor Robot from comment #9)
> 20 failures in 888 pushes (0.023 failures/push) were associated with this
> bug in the last 7 days.   
> 
> Repository breakdown:
> * try: 20
> 
> Platform breakdown:
> * windows7-32: 8
> * windows8-64: 7
> * osx-10-10: 2
> * macosx64-nightly: 2
> * macosx64-devedition: 1
> 
> For more details, see:
> https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1386226&startday=2017-07-31&endday=2017-08-06&tree=all
Should be fine because the graph shows after the patch landed no more failure was there.

Comment 11

[Justin [:JW_SoftvisionQA]]

Verifying per comment 10.