1563794 - Intermittent FATAL ERROR: Non-local network connections are disabled and a connection attempt to services.addons.mozilla.org (35.166.104.37) was made with pid: None | application crashed [@ nsresult mozilla::net::nsSocketTransport::InitiateSocket()]

Status: RESOLVED FIXED

(Toolkit :: Add-ons Manager, defect, P2)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=254869221&repo=mozilla-central
Full log: https://queue.taskcluster.net/v1/task/ScRFNRgARhyhr89CZarW3Q/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/ScRFNRgARhyhr89CZarW3Q/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

09:20:20 INFO - REFTEST INFO | Removed pending crash reports at 'C:\Users\testdroid\AppData\Roaming\Mozilla\Firefox\Crash Reports'
09:20:20 INFO - REFTEST INFO | Running with e10s: True
09:20:20 INFO - REFTEST INFO | Application command: C:\tasks\task_1562317170\build\application\firefox\firefox.exe -marionette --wait-for-browser -profile c:\users\testdr1\appdata\local\temp\tmpaztg3r.mozrunner
09:20:21 INFO - 1562318421663 [e-mail] WARN Loading extension '[e-mail]': Reading manifest: Invalid extension permission: mozillaAddons
09:20:21 INFO - 1562318421663 [e-mail] WARN Loading extension '[e-mail]': Reading manifest: Invalid extension permission: telemetry
09:20:21 INFO - 1562318421663 [e-mail] WARN Loading extension '[e-mail]': Reading manifest: Invalid extension permission: resource://pdf.js/
09:20:21 INFO - 1562318421664 [e-mail] WARN Loading extension '[e-mail]': Reading manifest: Invalid extension permission: about:reader*
09:20:21 INFO - 1562318421825 Marionette TRACE Received observer notification profile-after-change
09:20:21 INFO - 1562318421888 Marionette TRACE Received observer notification toplevel-window-ready
09:20:21 INFO - 1562318421931 Marionette TRACE Received observer notification command-line-startup
09:20:21 INFO - 1562318421932 Marionette TRACE Received observer notification nsPref:changed
09:20:21 INFO - 1562318421932 Marionette DEBUG Init aborted (running=false, enabled=true, finalUIStartup=false)
09:20:22 INFO - 1562318422097 Marionette TRACE Received observer notification toplevel-window-ready
09:20:23 INFO - FATAL ERROR: Non-local network connections are disabled and a connection attempt to services.addons.mozilla.org (35.166.104.37) was made.
09:20:23 INFO - You should only access hostnames available via the test networking proxy (if running mochitests) or from a test-specific httpd.js server (if running xpcshell tests). Browser services should be disabled or redirected to a local server.
09:20:24 INFO - Exiting due to channel error.
09:20:24 INFO - Exiting due to channel error.
09:23:21 ERROR - TEST-UNEXPECTED-FAIL | None | application terminated with exit code 1
09:23:21 INFO - REFTEST INFO | Downloading symbols from: https://queue.taskcluster.net/v1/task/EWTIadtOTLSAv2SE4-ejnA/artifacts/public/build/target.crashreporter-symbols.zip
09:23:25 INFO - REFTEST INFO | Copy/paste: C:\tasks\task_1562317170\build\win32-minidump_stackwalk.exe c:\users\testdr1\appdata\local\temp\tmpaztg3r.mozrunner\minidumps\e7c87d49-5225-45ed-921e-895e7534d4b7.dmp c:\users\testdr1\appdata\local\temp\tmpb5befu
09:23:34 INFO - REFTEST INFO | Saved minidump as C:\tasks\task_1562317170\build\blobber_upload_dir\e7c87d49-5225-45ed-921e-895e7534d4b7.dmp
09:23:34 INFO - REFTEST INFO | Saved app info as C:\tasks\task_1562317170\build\blobber_upload_dir\e7c87d49-5225-45ed-921e-895e7534d4b7.extra
09:23:34 INFO - REFTEST PROCESS-CRASH | pid: None | application crashed [@ nsresult mozilla::net::nsSocketTransport::InitiateSocket()]
09:23:34 INFO - Crash dump filename: c:\users\testdr1\appdata\local\temp\tmpaztg3r.mozrunner\minidumps\e7c87d49-5225-45ed-921e-895e7534d4b7.dmp
09:23:34 INFO - Operating system: Windows NT
09:23:34 INFO - 10.0.17134
09:23:34 INFO - CPU: 0x000c
09:23:34 INFO - 8 CPUs
09:23:34 INFO -
09:23:34 INFO - GPU: UNKNOWN
09:23:34 INFO -
09:23:34 INFO - Crash reason: EXCEPTION_ILLEGAL_INSTRUCTION
09:23:34 INFO - Crash address: 0x7ffaa2781518
09:23:34 INFO - Process uptime: 4 seconds
09:23:34 INFO -
09:23:34 INFO - Thread 17 (crashed)
09:23:34 INFO - 0 xul.dll!nsresult mozilla::net::nsSocketTransport::InitiateSocket() [nsSocketTransport2.cpp:ac50a7bd09b0a9262eed73f9dee2ffbd0e1a33b2 : 1285 + 0x0]
09:23:34 INFO -
09:23:34 INFO - Found by: given as instruction pointer in context
09:23:34 INFO - 1 xul.dll!nsresult mozilla::net::nsSocketTransport::InitiateSocket() [nsSocketTransport2.cpp:ac50a7bd09b0a9262eed73f9dee2ffbd0e1a33b2 : 1274 + 0x1c]
09:23:34 INFO -
09:23:34 INFO - Found by: previous frame's frame pointer
09:23:34 INFO - 2 xul.dll!nsresult mozilla::net::nsSocketEvent::Run() [nsSocketTransport2.cpp:ac50a7bd09b0a9262eed73f9dee2ffbd0e1a33b2 : 91 + 0x328]
09:23:34 INFO -
09:23:34 INFO - Found by: previous frame's frame pointer
09:23:34 INFO - 3 xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:ac50a7bd09b0a9262eed73f9dee2ffbd0e1a33b2 : 1225 + 0x4]
09:23:34 INFO -
09:23:34 INFO - Found by: previous frame's frame pointer
09:23:34 INFO - 4 xul.dll!NS_ProcessNextEvent(nsIThread *,bool) [nsThreadUtils.cpp:ac50a7bd09b0a9262eed73f9dee2ffbd0e1a33b2 : 486 + 0x10]
09:23:34 INFO -
09:23:34 INFO - Found by: previous frame's frame pointer
09:23:34 INFO - 5 xul.dll!nsresult mozilla::net::nsSocketTransportService::Run() [nsSocketTransportService2.cpp:ac50a7bd09b0a9262eed73f9dee2ffbd0e1a33b2 : 1011 + 0x0]
09:23:34 INFO -
09:23:34 INFO - Found by: previous frame's frame pointer
09:23:34 INFO - 6 xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:ac50a7bd09b0a9262eed73f9dee2ffbd0e1a33b2 : 1225 + 0x4]
09:23:34 INFO -
09:23:34 INFO - Found by: previous frame's frame pointer
09:23:34 INFO - 7 xul.dll!NS_ProcessNextEvent(nsIThread *,bool) [nsThreadUtils.cpp:ac50a7bd09b0a9262eed73f9dee2ffbd0e1a33b2 : 486 + 0x10]
09:23:34 INFO -
09:23:34 INFO - Found by: previous frame's frame pointer
09:23:34 INFO - 8 xul.dll!mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate *) [MessagePump.cpp:ac50a7bd09b0a9262eed73f9dee2ffbd0e1a33b2 : 333 + 0x8]
09:23:34 INFO -
09:23:34 INFO - Found by: previous frame's frame pointer
09:23:34 INFO - 9 xul.dll!MessageLoop::RunHandler() [message_loop.cc:ac50a7bd09b0a9262eed73f9dee2ffbd0e1a33b2 : 308 + 0x10]
09:23:34 INFO -
09:23:34 INFO - Found by: previous frame's frame pointer
09:23:34 INFO - 10 xul.dll!static void nsThread::ThreadFunc(void *) [nsThread.cpp:ac50a7bd09b0a9262eed73f9dee2ffbd0e1a33b2 : 459 + 0x34]
09:23:34 INFO -
09:23:34 INFO - Found by: previous frame's frame pointer
09:23:34 INFO - 11 nss3.dll!static void _PR_NativeRunThread(void *) [pruthr.c:ac50a7bd09b0a9262eed73f9dee2ffbd0e1a33b2 : 397 + 0x4]
09:23:34 INFO -
09:23:34 INFO - Found by: previous frame's frame pointer
09:23:34 INFO - 12 nss3.dll!unsigned int pr_root(void *) [w95thred.c:ac50a7bd09b0a9262eed73f9dee2ffbd0e1a33b2 : 137 + 0x4]
09:23:34 INFO -
09:23:34 INFO - Found by: previous frame's frame pointer
09:23:34 INFO - 13 ucrtbase.dll!SortGetSortKey + 0x7a4
09:23:34 INFO -
09:23:34 INFO - Found by: previous frame's frame pointer
09:23:34 INFO - 14 kernel32.dll!BaseThreadInitThunk + 0x30
09:23:34 INFO -
09:23:34 INFO - Found by: previous frame's frame pointer
09:23:34 INFO - 15 mozglue.dll!static void patched_BaseThreadInitThunk(int, void *, void *) [WindowsDllBlocklist.cpp:ac50a7bd09b0a9262eed73f9dee2ffbd0e1a33b2 : 617 + 0x8]
09:23:34 INFO -
09:23:34 INFO - Found by: previous frame's frame pointer
09:23:34 INFO - 16 ntdll.dll!AslHashFree + 0x5468
09:23:34 INFO -
09:23:34 INFO - Found by: previous frame's frame pointer

Comment 1

[Intermittent Failures Robot]

12 failures in 3501 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 12

Platform breakdown:

• windows10-aarch64: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1563794&startday=2019-07-01&endday=2019-07-07&tree=all

Comment 2

[Intermittent Failures Robot]

22 failures in 4710 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 22

Platform breakdown:

• windows10-aarch64: 22

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1563794&startday=2019-07-08&endday=2019-07-14&tree=all

Comment 3

[Natalia Csoregi [:nataliaCs]]

These failures seem to be very high frequency- R/B up to https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&selectedJob=246047384&searchStr=windows%2C10%2Caarch64%2Copt%2Creftests%2Ctest-windows10-aarch64%2Fopt-reftest-e10s-1%2Cr%28r1%29&revision=4b3945b758896f5153e61a1b84a2b3614a98021a

Comment 4

[Alexandru Michis [:malexandru]]

Continued with the r/b: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&searchStr=windows%2C10%2Caarch64%2Copt%2Creftests%2Ctest-windows10-aarch64%2Fopt-reftest-e10s-1%2Cr%28r1%29&tochange=df3eadfa74a8061f4c88496404d51baf47e21070&fromchange=7995904ca18e2d85e768bc4317c56c1cddff44aa

Comment 5

[Intermittent Failures Robot]

39 failures in 4298 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

This is the #46 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 39

Platform breakdown:

• windows10-aarch64: 39

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1563794&startday=2019-07-15&endday=2019-07-21&tree=all

Comment 6

[Andrei Ciure[:aciure]]

There are 32 failures associated to this bug in the last 7 days. These are occurring on windows10-aarch64 opt builds.

:TomGrab can you take a look?

Comment 7

[Intermittent Failures Robot]

34 failures in 4608 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

This is the #38 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 34

Platform breakdown:

• windows10-aarch64: 34

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1563794&startday=2019-07-22&endday=2019-07-28&tree=all

Comment 9

[Junior [inactive]]

Hello :aswan,
Please see https://searchfox.org/mozilla-central/diff/aeee1208082b4ec27927d1ac8910a7778c5e6e5c/browser/app/profile/firefox.js#47
Could you replace the test config from extensions.getAddons.getWithPerformance.url to extensions.getAddons.compatOverides.url especially here?

Comment 10

[Intermittent Failures Robot]

36 failures in 4439 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

This is the #48 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 36

Platform breakdown:

• windows10-aarch64: 36

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1563794&startday=2019-07-29&endday=2019-08-04&tree=all

Comment 11

[Andrew Swan [:aswan]]

Attachment: Bug 1563794 Don't make external requests for addon compat overrides in automation r=zombie

Also, remove an old obsolete preference (getWithPerformance) from test profiles.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4be642df3822
Don't make external requests for addon compat overrides in automation r=zombie

Comment 13

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/4be642df3822

Comment 14

[Andrei Ciure[:aciure]]

This is still failing on tier2:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&selectedJob=260185285&searchStr=windows%2C10%2Caarch64%2Copt%2Creftests%2Ctest-windows10-aarch64%2Fopt-crashtest-e10s%2Cr%28c%29

Filed a new bug for this: https://bugzilla.mozilla.org/show_bug.cgi?id=1571885

Comment 15

[Intermittent Failures Robot]

6 failures in 4666 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6

Platform breakdown:

• windows10-aarch64: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1563794&startday=2019-08-05&endday=2019-08-11&tree=all

Comment 16

[Alex Cornestean]

Hello,

Will this fix require manual validation? If yes, please provide some steps to reproduce in order to correctly test it and also, please set the "qe-verify+" flag. Otherwise, could the "qe-verify-" flag be added? Thanks!