1498831 - intermittent ccov [tier 2] application crashed [@ mozilla::plugins::FunctionHookArray::~FunctionHookArray()] after xpcshell return code: 1

Status: RESOLVED FIXED

(Core Graveyard :: Plug-ins, defect)

Description

[Tiberius Oros[:tiberius_oros]]

https://treeherder.mozilla.org/logviewer.html#?job_id=205253784&repo=mozilla-central&lineNumber=4419

11:15:41     INFO -  TEST-START | browser/components/places/tests/unit/test_clearHistory_shutdown.js
11:15:46  WARNING -  TEST-UNEXPECTED-FAIL | browser/components/places/tests/unit/test_clearHistory_shutdown.js | xpcshell return code: 1
11:15:46     INFO -  TEST-INFO took 5817ms
11:15:46     INFO -  >>>>>>>
11:15:46     INFO -  PID 608 | [608, Main Thread] WARNING: Couldn't get the user appdata directory. Crash events may not be produced.: file z:/build/build/src/toolkit/crashreporter/nsExceptionHandler.cpp, line 2751
11:15:46     INFO -  (xpcshell/head.js) | test MAIN run_test pending (1)
11:15:46     INFO -  (xpcshell/head.js) | test run_next_test 0 pending (2)
11:15:46     INFO -  (xpcshell/head.js) | test MAIN run_test finished (2)
11:15:46     INFO -  running event loop
11:15:46     INFO -  browser/components/places/tests/unit/test_clearHistory_shutdown.js | Starting test_execute
11:15:46     INFO -  (xpcshell/head.js) | test test_execute pending (2)
11:15:46     INFO -  "Initialize browserglue before Places"
11:15:46     INFO -  "Add visits."
11:15:46     INFO -  (xpcshell/head.js) | test run_next_test 0 finished (2)
11:15:46     INFO -  "Add cache."
11:15:46     INFO -  TEST-PASS | browser/components/places/tests/unit/test_clearHistory_shutdown.js | test_execute - [test_execute : 137] 0 == 0
11:15:46     INFO -  "Add form history."
11:15:46     INFO -  TEST-PASS | browser/components/places/tests/unit/test_clearHistory_shutdown.js | test_execute - [test_execute : 76] Added form history - 1 == 1
11:15:46     INFO -  "Simulate and wait shutdown."
11:15:46     INFO -  "shutdownPlaces: starting"
11:15:46     INFO -  "shutdownPlaces: sent profile-change-teardown"
11:15:46     INFO -  PID 608 | [608, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80520012: file z:/build/build/src/extensions/cookie/nsPermissionManager.cpp, line 2932
11:15:46     INFO -  PID 608 | [608, Main Thread] WARNING: Couldn't get the user appdata directory, crash dumps will go in an unusual location: file z:/build/build/src/toolkit/crashreporter/nsExceptionHandler.cpp, line 2828
11:15:46     INFO -  PID 608 | [NPAPI 11472, Main Thread] WARNING: NS_ENSURE_TRUE(InitStaticMembers()) failed: file z:/build/build/src/modules/libpref/Preferences.cpp, line 5009
11:15:46     INFO -  PID 608 | [NPAPI 11472, Main Thread] WARNING: NS_ENSURE_TRUE(InitStaticMembers()) failed: file z:/build/build/src/modules/libpref/Preferences.cpp, line 5050
11:15:46     INFO -  PID 608 | [NPAPI 11472, Main Thread] WARNING: NS_ENSURE_TRUE(InitStaticMembers()) failed: file z:/build/build/src/modules/libpref/Preferences.cpp, line 5009
11:15:46     INFO -  PID 608 | [608, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file z:/build/build/src/netwerk/cache/nsCacheService.cpp, line 362
11:15:46     INFO -  PID 608 | JavaScript strict warning: resource://gre/modules/PushService.jsm, line 18: ReferenceError: reference to undefined property "getCryptoParams"
11:15:46     INFO -  PID 608 | [608, Main Thread] WARNING: This method is lossy. Use GetCanonicalPath !: file z:/build/build/src/xpcom/io/nsLocalFileWin.cpp, line 3462
11:15:46     INFO -  PID 608 | [608, Main Thread] WARNING: This method is lossy. Use GetCanonicalPath !: file z:/build/build/src/xpcom/io/nsLocalFileWin.cpp, line 3462
11:15:46     INFO -  PID 608 | [608, Main Thread] WARNING: This method is lossy. Use GetCanonicalPath !: file z:/build/build/src/xpcom/io/nsLocalFileWin.cpp, line 3462
11:15:46     INFO -  "CONSOLE_MESSAGE: (warn) [JavaScript Warning: "ReferenceError: reference to undefined property "getCryptoParams"" {file: "resource://gre/modules/PushService.jsm" line: 18}]"
11:15:46     INFO -  "CONSOLE_MESSAGE: (info) "
11:15:46     INFO -  "shutdownPlaces: sent test-simulate-places-shutdown"
11:15:46     INFO -  "shutdownPlaces: complete"
11:15:46     INFO -  TEST-PASS | browser/components/places/tests/unit/test_clearHistory_shutdown.js | test_execute - [test_execute : 81] Form history cleared - 0 == 0
11:15:46     INFO -  TEST-PASS | browser/components/places/tests/unit/test_clearHistory_shutdown.js | test_execute - [test_execute : 90] true == true
11:15:46     INFO -  TEST-PASS | browser/components/places/tests/unit/test_clearHistory_shutdown.js | test_execute - [test_execute : 90] true == true
11:15:46     INFO -  TEST-PASS | browser/components/places/tests/unit/test_clearHistory_shutdown.js | test_execute - [test_execute : 90] true == true
11:15:46     INFO -  TEST-PASS | browser/components/places/tests/unit/test_clearHistory_shutdown.js | test_execute - [test_execute : 90] true == true
11:15:46     INFO -  "Check cache"
11:15:46     INFO -  TEST-PASS | browser/components/places/tests/unit/test_clearHistory_shutdown.js | test_execute - [test_execute : 168] 2152398909 == 2152398909
11:15:46     INFO -  (xpcshell/head.js) | test run_next_test 1 pending (2)
11:15:46     INFO -  (xpcshell/head.js) | test test_execute finished (2)
11:15:46     INFO -  (xpcshell/head.js) | test run_next_test 1 finished (1)
11:15:46     INFO -  exiting test
11:15:46     INFO -  PID 608 | [608, Main Thread] WARNING: NS_ENSURE_TRUE(mDB) failed: file z:/build/build/src/netwerk/cache/nsDiskCacheDeviceSQL.cpp, line 1424
11:15:46     INFO -  PID 608 | [NPAPI 11472, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
11:15:46     INFO -  PID 608 | [NPAPI 11472, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
11:15:46     INFO -  PID 608 | [608, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
11:15:46     INFO -  PID 608 | [608, Main Thread] WARNING: '!aObserver', file z:/build/build/src/xpcom/ds/nsObserverService.cpp, line 243
11:15:46     INFO -  PID 608 | nsStringStats
11:15:46     INFO -  PID 608 |  => mAllocCount:             99
11:15:46     INFO -  PID 608 |  => mReallocCount:            0
11:15:46     INFO -  PID 608 |  => mFreeCount:              99
11:15:46     INFO -  PID 608 |  => mShareCount:            227
11:15:46     INFO -  PID 608 |  => mAdoptCount:              0
11:15:46     INFO -  PID 608 |  => mAdoptFreeCount:          0
11:15:46     INFO -  PID 608 |  => Process ID: 11472, Thread ID: 9092
11:15:46     INFO -  PID 608 | [608, Main Thread] WARNING: NS_ENSURE_TRUE(maybeContext) failed: file z:/build/build/src/xpcom/threads/nsThread.cpp, line 948
11:15:46     INFO -  <<<<<<<
11:15:46     INFO -  mozcrash Copy/paste: Z:\task_1539428255\build\win32-minidump_stackwalk.exe c:\users\task_1539428255\appdata\local\temp\xpc-other-0xoewy\7b96c86b-124c-4faa-9b04-ce7718cab168.dmp Z:\task_1539428255\build\symbols
11:15:56     INFO -  mozcrash Saved minidump as Z:\task_1539428255\build\blobber_upload_dir\7b96c86b-124c-4faa-9b04-ce7718cab168.dmp
11:15:56     INFO -  mozcrash Saved app info as Z:\task_1539428255\build\blobber_upload_dir\7b96c86b-124c-4faa-9b04-ce7718cab168.extra
11:15:56  WARNING -  PROCESS-CRASH | browser/components/places/tests/unit/test_clearHistory_shutdown.js | application crashed [@ mozilla::plugins::FunctionHookArray::~FunctionHookArray()]
11:15:56     INFO -  Crash dump filename: c:\users\task_1539428255\appdata\local\temp\xpc-other-0xoewy\7b96c86b-124c-4faa-9b04-ce7718cab168.dmp
11:15:56     INFO -  Operating system: Windows NT
11:15:56     INFO -                    10.0.15063
11:15:56     INFO -  CPU: amd64
11:15:56     INFO -       family 6 model 63 stepping 2
11:15:56     INFO -       8 CPUs
11:15:56     INFO -  GPU: UNKNOWN
11:15:56     INFO -  Crash reason:  EXCEPTION_ACCESS_VIOLATION_READ
11:15:56     INFO -  Crash address: 0xffffffffffffffff
11:15:56     INFO -  Process uptime: 3 seconds
11:15:56     INFO -  Thread 0 (crashed)
11:15:56     INFO -   0  xul.dll!mozilla::plugins::FunctionHookArray::~FunctionHookArray() [FunctionHook.h:0dfb3afc73572ea842e1fbf6ee08644e5570d79d : 89 + 0x14]
11:15:56     INFO -      rax = 0x0000000000000001   rdx = 0x0000000000000001
11:15:56     INFO -      rcx = 0xe4e4e4e4e4e4e4e4   rbx = 0x000000000000001a
11:15:56     INFO -      rsi = 0x000001b70898ddf0   rdi = 0x0000000000000019
11:15:56     INFO -      rbp = 0x0000000000000000   rsp = 0x00000012179ff4c0
11:15:56     INFO -       r8 = 0x00007fff1de049b0    r9 = 0x0000000000000000
11:15:56     INFO -      r10 = 0x000001b7089fd800   r11 = 0x000001b7089fef60
11:15:56     INFO -      r12 = 0x00007ffef6cb2700   r13 = 0x00007ffef6cb2260
11:15:56     INFO -      r14 = 0x00007ffef8d68f70   r15 = 0x00007ffef8d2ced0
11:15:56     INFO -      rip = 0x00007ffee91fbf84
11:15:56     INFO -      Found by: given as instruction pointer in context
11:15:56     INFO -   1  xul.dll!mozilla::ClearOnShutdown_Internal::PointerClearer<mozilla::StaticAutoPtr<mozilla::plugins::FunctionHookArray> >::Shutdown() [ClearOnShutdown.h:0dfb3afc73572ea842e1fbf6ee08644e5570d79d : 81 + 0x6c]
11:15:56     INFO -      rbx = 0x000000000000001a   rbp = 0x0000000000000000
11:15:56     INFO -      rsp = 0x00000012179ff530   r12 = 0x00007ffef6cb2700
11:15:56     INFO -      r13 = 0x00007ffef6cb2260   r14 = 0x00007ffef8d68f70
11:15:56     INFO -      r15 = 0x00007ffef8d2ced0   rip = 0x00007ffee91fc321
11:15:56     INFO -      Found by: call frame info

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Triggered by bug 1488439, it seems.

Comment 2

[Intermittent Failures Robot]

19 failures in 147 pushes (0.129 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 19

Platform breakdown:
* windows10-64-ccov: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1498831&startday=2018-10-13&endday=2018-10-13&tree=all

Comment 3

[Intermittent Failures Robot]

45 failures in 2740 pushes (0.016 failures/push) were associated with this bug in the last 7 days.

This is the #35 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-central: 45

Platform breakdown:
* windows10-64-ccov: 45

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1498831&startday=2018-10-08&endday=2018-10-14&tree=all

Comment 4

[David Parks [:handyman]]

Attachment: Bug 1498831: Exclude ID_GetFileAttributesW enum from Windows builds without MOZ_SANDBOX (r?jmathies)

All of the enum entries must have a FunctionHook object but GetFileAttributesW only gets one if MOZ_SANDBOX is defined.  This aligns the #define behavior of the enum with the #define behavior of its usage in FunctionHook.

Comment 5

[David Parks [:handyman]]

Jimm, I'm not sure if this is the cause of the intermittent failure but it is a bug in the use of this enum from patch 2 in bug 1488439.

Comment 6

[Intermittent Failures Robot]

45 failures in 475 pushes (0.095 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 45

Platform breakdown:
* windows10-64-ccov: 45

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1498831&startday=2018-10-15&endday=2018-10-15&tree=all

Comment 8

[David Parks [:handyman]]

I'm now pretty certain that this does fix the bug.  I'm hanging hopes on this push:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=205912630&revision=bb06de7a97d21a1b6032ad8dc1e3962d0af7d804

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/943c165146e6
Exclude ID_GetFileAttributesW enum from Windows builds without MOZ_SANDBOX (r=jmathies)

Comment 14

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/943c165146e6

Comment 15

[Intermittent Failures Robot]

26 failures in 515 pushes (0.05 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 26

Platform breakdown:
* windows10-64-ccov: 26

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1498831&startday=2018-10-16&endday=2018-10-16&tree=all

Comment 16

[Intermittent Failures Robot]

71 failures in 2759 pushes (0.026 failures/push) were associated with this bug in the last 7 days.

This is the #28 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-central: 71

Platform breakdown:
* windows10-64-ccov: 71

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1498831&startday=2018-10-15&endday=2018-10-21&tree=all

Comment 20

[David Parks [:handyman]]

Comment on attachment 9017222 [details]
Bug 1498831: Exclude ID_GetFileAttributesW enum from Windows builds without MOZ_SANDBOX (r?jmathies)

[Beta/Release Uplift Approval Request]

Feature/Bug causing the regression: N/A

User impact if declined: (Takenfrom beta uplift request in bug 1488439)

This failure, which breaks Comcast streaming video, is due to the restricting SIDs work that is currently active only in trunk.  This bug is a touch-up for a patch in bug 1488439 -- together they fix the issue.  This is being uplifted so that we can (later this week) uplift that work from bug 1426733.  We want to uplift since there is a low likelihood of stressing these patches in nightly.

Is this code covered by automated tests?: No

Has the fix been verified in Nightly?: Yes

Needs manual test from QE?: No

If yes, steps to reproduce: 

List of other uplifts needed: Bug 1488439

Risk to taking this patch: Medium

Why is the change risky/not risky? (and alternatives if risky): See other uplift request in bug 1488439

String changes made/needed: N/A

Comment 21

[[Back August 16] Ryan VanderMeulen]

Comment on attachment 9017222 [details]
Bug 1498831: Exclude ID_GetFileAttributesW enum from Windows builds without MOZ_SANDBOX (r?jmathies)

This is already on Beta.