1554724 - Perma Talos FATAL ERROR: Non-local network connections are disabled and a connection attempt to push.services.mozilla.com

Status: RESOLVED FIXED

(Core :: DOM: Push Subscriptions, defect)

Description

[Narcis Beleuzu [:NarcisB]]

Failures started with:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&revision=944c410b7e9185a0cb90a4fbc0970299f1ff3e2b&selectedJob=248561110

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=248561110&repo=mozilla-central&lineNumber=717

Comment 1

[Narcis Beleuzu [:NarcisB]]

Log snippet:

15:17:52 INFO - TEST-INFO | started process 3723 (/home/cltbld/tasks/task_1558969282/build/application/firefox/firefox -profile /tmp/tmpFZFdId/profile)
15:17:53 INFO - PID 3723 | ATTENTION: default value of option force_s3tc_enable overridden by environment.
15:17:53 INFO - PID 3723 | ATTENTION: default value of option force_s3tc_enable overridden by environment.
15:17:55 INFO - PID 3723 | FATAL ERROR: Non-local network connections are disabled and a connection attempt to push.services.mozilla.com (35.163.237.31) was made.
15:17:55 INFO - PID 3723 | You should only access hostnames available via the test networking proxy (if running mochitests) or from a test-specific httpd.js server (if running xpcshell tests). Browser services should be disabled or redirected to a local server.
15:17:55 INFO - PID 3723 | ExceptionHandler::GenerateDump cloned child 3926
15:17:55 INFO - PID 3723 | ExceptionHandler::SendContinueSignalToChild sent continue signal to child
15:17:55 INFO - PID 3723 | ExceptionHandler::WaitForContinueSignal waiting for continue signal...
15:17:55 INFO - PID 3723 | Exiting due to channel error.
15:17:55 INFO - PID 3723 | [Child 3821, Chrome_ChildThread] WARNING: pipe error (3): Connection reset by peer: file /builds/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 358
15:17:55 INFO - PID 3723 | [Child 3791, Chrome_ChildThread] WARNING: pipe error (3): Connection reset by peer: file /builds/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 358
15:17:55 INFO - PID 3723 | Exiting due to channel error.
15:17:55 INFO - PID 3723 | Exiting due to channel error.
15:17:55 INFO - Terminating psutil.Process(pid=3723, name='firefox-bin', started='15:17:52')
15:17:55 INFO - TEST-INFO | 3723: exit 11
15:17:55 INFO - TEST-UNEXPECTED-ERROR | tp5o_scroll | Could not find report in browser output: [('tsformat', ('__start_report', '__end_report')), ('tpformat', ('__start_tp_report', '__end_tp_report'))]
15:17:55 ERROR - Traceback (most recent call last):
15:17:55 INFO - File "/home/cltbld/tasks/task_1558969282/build/tests/talos/talos/run_tests.py", line 300, in run_tests
15:17:55 INFO - talos_results.add(mytest.runTest(browser_config, test))
15:17:55 INFO - File "/home/cltbld/tasks/task_1558969282/build/tests/talos/talos/ttest.py", line 64, in runTest
15:17:55 INFO - return self._runTest(browser_config, test_config, setup)
15:17:55 INFO - File "/home/cltbld/tasks/task_1558969282/build/tests/talos/talos/ttest.py", line 278, in _runTest
15:17:55 INFO - else None)
15:17:55 INFO - File "/home/cltbld/tasks/task_1558969282/build/tests/talos/talos/results.py", line 95, in add
15:17:55 INFO - global_counters=self.global_counters
15:17:55 INFO - File "/home/cltbld/tasks/task_1558969282/build/tests/talos/talos/results.py", line 326, in init
15:17:55 INFO - self.parse()
15:17:55 INFO - File "/home/cltbld/tasks/task_1558969282/build/tests/talos/talos/results.py", line 353, in parse
15:17:55 INFO - % self.report_tokens)
15:17:55 INFO - File "/home/cltbld/tasks/task_1558969282/build/tests/talos/talos/results.py", line 337, in error
15:17:55 INFO - raise utils.TalosError(message)

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This got uncovered after bug 1542244 got backed out and push notifications reenabled after having been disabled for 6 days.

Comment 3

[Edouard Oger [:eoger]]

Vidjay can you have a look at this? I believe this is caused by the new FxA toolbar menu.

Comment 4

[Lina Butler [:lina]]

We might need to add an entry for dom.push.connection.enabled to this file.

Comment 5

[Intermittent Failures Robot]

982 failures in 632 pushes (1.554 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 329
• mozilla-inbound: 271
• autoland: 382

Platform breakdown:

• linux64: 30
• linux64-qr: 34
• windows10-64-qr: 28
• windows10-64: 30
• windows7-32: 17
• macosx1014-64-shippable: 17
• windows10-aarch64: 24
• windows7-32-shippable: 169
• windows10-64-shippable: 170
• windows10-64-ux: 3
• windows10-64-shippable-qr: 159
• linux64-shippable: 156
• linux64-shippable-qr: 145

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1554724&startday=2019-05-27&endday=2019-05-27&tree=all

Comment 6

[Ed Lee :Mardak]

I did a try with

+++ b/testing/profiles/common/user.js
+user_pref("dom.push.connection.enabled", false);

and talos seemed to run fine https://treeherder.mozilla.org/#/jobs?repo=try&revision=5071a859e83134fe95ac7843f604dbfc919efb30

Comment 7

[Edouard Oger [:eoger]]

Also currently running https://treeherder.mozilla.org/#/jobs?repo=try&revision=60c120a7a7daaeb8fae78a075dd6a6f9c72570f4
I'll post a patch if it's all green.

Comment 8

[Edouard Oger [:eoger]]

Attachment: Bug 1554724 - Disable push connection for perf tests.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ad2f980e2363
Disable push connection for perf tests. r=lina

Comment 10

[Intermittent Failures Robot]

2611 failures in 841 pushes (3.105 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 1219
• mozilla-central: 1002
• mozilla-inbound: 368
• try: 22

Platform breakdown:

• windows7-32-shippable: 411
• linux64-shippable: 401
• linux64-shippable-qr: 380
• windows10-64-shippable: 437
• windows10-64-shippable-qr: 413
• linux64: 80
• linux64-qr: 72
• windows7-32: 90
• windows10-aarch64: 72
• macosx1014-64-shippable: 76
• windows10-64: 90
• windows10-64-qr: 83
• windows10-64-ux: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1554724&startday=2019-05-28&endday=2019-05-28&tree=all

Comment 11

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/ad2f980e2363

Comment 12

[Daniel Varga [:dvarga]]

This is still happening: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=249078135&repo=autoland&lineNumber=27190

00:53:41     INFO -  PID 1504 | JavaScript note: http://127.0.0.1:49776/tests/tp5n/sohu.com/images.sohu.com/cs/sohuim/xiaozt/version/2.0/js/loader.sohu.js, line 1: { opened at line 1, column 14866
00:53:41     INFO -  PID 1504 | [Child 4860, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1559165341/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
00:53:41     INFO -  PID 1504 | [Child 4860, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1559165341/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
00:53:41     INFO -  PID 1504 | [Child 4860, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1559165341/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
00:53:41     INFO -  PID 1504 | [Child 4860, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1559165341/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
00:53:41     INFO -  PID 1504 | [Child 4860, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1559165341/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
00:53:41     INFO -  PID 1504 | JavaScript note: http://127.0.0.1:49776/tests/tp5n/sohu.com/www.sohu.com/sohuflash_1.js, line 1: { opened at line 1, column 1295
00:53:41     INFO -  PID 1504 | FATAL ERROR: Non-local network connections are disabled and a connection attempt to ping.ie.sogou.com (211.159.235.146) was made.
00:53:41     INFO -  PID 1504 | You should only access hostnames available via the test networking proxy (if running mochitests) or from a test-specific httpd.js server (if running xpcshell tests). Browser services should be disabled or redirected to a local server.
00:53:41     INFO -  PID 1504 | Exiting due to channel error.
00:53:41     INFO -  PID 1504 | Exiting due to channel error.
00:53:41     INFO -  PID 1504 | Exiting due to channel error.
00:53:41     INFO -  ERROR:root:[Errno 10054] An existing connection was forcibly closed by the remote host
00:53:41     INFO -  ERROR:root:[Errno 10054] An existing connection was forcibly closed by the remote host
00:53:41     INFO -  ERROR:root:[Errno 10054] An existing connection was forcibly closed by the remote host
00:53:41     INFO -  ERROR:root:[Errno 10054] An existing connection was forcibly closed by the remote host
00:53:41     INFO -  ERROR:root:[Errno 10054] An existing connection was forcibly closed by the remote host
00:53:41     INFO -  ERROR:root:[Errno 10054] An existing connection was forcibly closed by the remote host
00:53:41     INFO -  TEST-INFO | 1504: exit 1
00:53:42     INFO -  TEST-UNEXPECTED-ERROR | tp5o_webext | Could not find report in browser output: [('tsformat', ('__start_report', '__end_report')), ('tpformat', ('__start_tp_report', '__end_tp_report'))]
00:53:42    ERROR -  Traceback (most recent call last):
00:53:42     INFO -    File "C:\Users\task_1559174416\build\tests\talos\talos\run_tests.py", line 300, in run_tests
00:53:42     INFO -      talos_results.add(mytest.runTest(browser_config, test))
00:53:42     INFO -    File "C:\Users\task_1559174416\build\tests\talos\talos\ttest.py", line 64, in runTest
00:53:42     INFO -      return self._runTest(browser_config, test_config, setup)
00:53:42     INFO -    File "C:\Users\task_1559174416\build\tests\talos\talos\ttest.py", line 278, in _runTest
00:53:42     INFO -      else None)
00:53:42     INFO -    File "C:\Users\task_1559174416\build\tests\talos\talos\results.py", line 95, in add
00:53:42     INFO -      global_counters=self.global_counters
00:53:42     INFO -    File "C:\Users\task_1559174416\build\tests\talos\talos\results.py", line 326, in __init__
00:53:42     INFO -      self.parse()
00:53:42     INFO -    File "C:\Users\task_1559174416\build\tests\talos\talos\results.py", line 353, in parse
00:53:42     INFO -      % self.report_tokens)
00:53:42     INFO -    File "C:\Users\task_1559174416\build\tests\talos\talos\results.py", line 337, in error
00:53:42     INFO -      raise utils.TalosError(message)
00:53:42     INFO -  TalosError: Could not find report in browser output: [('tsformat', ('__start_report', '__end_report')), ('tpformat', ('__start_tp_report', '__end_tp_report'))]
00:53:42     INFO -  TEST-INFO took 491252ms

Comment 13

[Lina Butler [:lina]]

That looks like a different hostname, not related to push?

FATAL ERROR: Non-local network connections are disabled and a connection attempt to ping.ie.sogou.com (211.159.235.146) was made.

Comment 14

[Daniel Varga [:dvarga]]

True, found bug 1554570 for that hostname.

Comment 15

[Intermittent Failures Robot]

330 failures in 917 pushes (0.36 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 210
• mozilla-inbound: 4
• mozilla-central: 116

Platform breakdown:

• windows7-32-shippable: 59
• windows10-64-shippable: 62
• windows10-64-shippable-qr: 55
• linux64-shippable: 59
• linux64-shippable-qr: 56
• linux64: 1
• windows10-64: 5
• windows10-64-qr: 3
• windows10-aarch64: 13
• macosx1014-64-shippable: 9
• windows7-32: 7
• windows10-64-ux: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1554724&startday=2019-05-29&endday=2019-05-29&tree=all

Comment 16

[Intermittent Failures Robot]

3944 failures in 4337 pushes (0.909 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 1453
• mozilla-inbound: 648
• autoland: 1821
• try: 22

Platform breakdown:

• linux64: 111
• linux64-qr: 107
• windows10-64-qr: 115
• windows10-64: 125
• windows7-32: 115
• macosx1014-64-shippable: 102
• windows10-aarch64: 111
• windows7-32-shippable: 643
• windows10-64-shippable: 670
• windows10-64-ux: 10
• windows10-64-shippable-qr: 629
• linux64-shippable: 623
• linux64-shippable-qr: 583

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1554724&startday=2019-05-27&endday=2019-06-02&tree=all