916757 - WinXP browser-chrome failures on multiple trees due to "Assertion failure: rv, at e:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/nsprpub/pr/src/md/windows/w95thred.c:220"

Reporter

Description

•

11 years ago

These failures have started occurring on multiple trees, and on pushes that were previously green. Clobbering hasn't helped.

Guessing one of:
1) Slave/infra change.
2) Test that relies on external resource (eg like the now fixed zombocom issue).
3) ???

All trunk trees closed.

Logs to follow.

Ed Morley [:emorley]

Reporter

Comment 1

•

11 years ago

eg:
https://tbpl.mozilla.org/php/getParsedLog.php?id=27917258&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=27917286&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=27917301&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=27913944&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=27922841&tree=Mozilla-Inbound

{
21:13:54     INFO -  ************************************************************
21:13:54     INFO -  * Call to xpconnect wrapped JSObject produced this error:  *
21:13:54     INFO -  [Exception... "'[JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 483}]' when calling method: [nsIController::isCommandEnabled]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: chrome://global/content/globalOverlay.js :: goUpdateCommand :: line 78"  data: yes]
21:13:54     INFO -  ************************************************************
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "nodes[i] is null" {file: "chrome://browser/content/places/controller.js" line: 137}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "An error occurred updating the cmd_cut command: [Exception... "'[JavaScript Error: "nodes[i] is null" {file: "chrome://browser/content/places/controller.js" line: 137}]' when calling method: [nsIController::isCommandEnabled]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: chrome://global/content/globalOverlay.js :: goUpdateCommand :: line 78"  data: yes]" {file: "chrome://global/content/globalOverlay.js" line: 84}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 483}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "An error occurred updating the cmd_paste command: [Exception... "'[JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 483}]' when calling method: [nsIController::isCommandEnabled]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: chrome://global/content/globalOverlay.js :: goUpdateCommand :: line 78"  data: yes]" {file: "chrome://global/content/globalOverlay.js" line: 84}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 162}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "An error occurred updating the cmd_delete command: [Exception... "'[JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 162}]' when calling method: [nsIController::isCommandEnabled]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: chrome://global/content/globalOverlay.js :: goUpdateCommand :: line 78"  data: yes]" {file: "chrome://global/content/globalOverlay.js" line: 84}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 483}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 162}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "An error occurred updating the cmd_delete command: [Exception... "'[JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 162}]' when calling method: [nsIController::isCommandEnabled]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: chrome://global/content/globalOverlay.js :: goUpdateCommand :: line 78"  data: yes]" {file: "chrome://global/content/globalOverlay.js" line: 84}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 483}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "An error occurred updating the cmd_paste command: [Exception... "'[JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 483}]' when calling method: [nsIController::isCommandEnabled]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: chrome://global/content/globalOverlay.js :: goUpdateCommand :: line 78"  data: yes]" {file: "chrome://global/content/globalOverlay.js" line: 84}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "nodes[i] is null" {file: "chrome://browser/content/places/controller.js" line: 137}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "An error occurred updating the cmd_cut command: [Exception... "'[JavaScript Error: "nodes[i] is null" {file: "chrome://browser/content/places/controller.js" line: 137}]' when calling method: [nsIController::isCommandEnabled]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: chrome://global/content/globalOverlay.js :: goUpdateCommand :: line 78"  data: yes]" {file: "chrome://global/content/globalOverlay.js" line: 84}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 483}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "An error occurred updating the cmd_paste command: [Exception... "'[JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 483}]' when calling method: [nsIController::isCommandEnabled]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: chrome://global/content/globalOverlay.js :: goUpdateCommand :: line 78"  data: yes]" {file: "chrome://global/content/globalOverlay.js" line: 84}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 162}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "An error occurred updating the cmd_delete command: [Exception... "'[JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 162}]' when calling method: [nsIController::isCommandEnabled]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: chrome://global/content/globalOverlay.js :: goUpdateCommand :: line 78"  data: yes]" {file: "chrome://global/content/globalOverlay.js" line: 84}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 483}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 162}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "An error occurred updating the cmd_delete command: [Exception... "'[JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 162}]' when calling method: [nsIController::isCommandEnabled]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: chrome://global/content/globalOverlay.js :: goUpdateCommand :: line 78"  data: yes]" {file: "chrome://global/content/globalOverlay.js" line: 84}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 483}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "An error occurred updating the cmd_paste command: [Exception... "'[JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 483}]' when calling method: [nsIController::isCommandEnabled]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: chrome://global/content/globalOverlay.js :: goUpdateCommand :: line 78"  data: yes]" {file: "chrome://global/content/globalOverlay.js" line: 84}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "nodes[i] is null" {file: "chrome://browser/content/places/controller.js" line: 137}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "An error occurred updating the cmd_cut command: [Exception... "'[JavaScript Error: "nodes[i] is null" {file: "chrome://browser/content/places/controller.js" line: 137}]' when calling method: [nsIController::isCommandEnabled]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: chrome://global/content/globalOverlay.js :: goUpdateCommand :: line 78"  data: yes]" {file: "chrome://global/content/globalOverlay.js" line: 84}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 483}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "An error occurred updating the cmd_paste command: [Exception... "'[JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 483}]' when calling method: [nsIController::isCommandEnabled]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: chrome://global/content/globalOverlay.js :: goUpdateCommand :: line 78"  data: yes]" {file: "chrome://global/content/globalOverlay.js" line: 84}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 162}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "An error occurred updating the cmd_delete command: [Exception... "'[JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 162}]' when calling method: [nsIController::isCommandEnabled]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: chrome://global/content/globalOverlay.js :: goUpdateCommand :: line 78"  data: yes]" {file: "chrome://global/content/globalOverlay.js" line: 84}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 483}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 162}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "An error occurred updating the cmd_delete command: [Exception... "'[JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 162}]' when calling method: [nsIController::isCommandEnabled]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: chrome://global/content/globalOverlay.js :: goUpdateCommand :: line 78"  data: yes]" {file: "chrome://global/content/globalOverlay.js" line: 84}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 483}]
21:13:54     INFO -  TEST-INFO | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Console message: [JavaScript Error: "An error occurred updating the cmd_paste command: [Exception... "'[JavaScript Error: "aNode is null" {file: "resource://gre/modules/PlacesUtils.jsm" line: 483}]' when calling method: [nsIController::isCommandEnabled]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: chrome://global/content/globalOverlay.js :: goUpdateCommand :: line 78"  data: yes]" {file: "chrome://global/content/globalOverlay.js" line: 84}]
21:13:54     INFO -  [Parent 2108] WARNING: NS_ENSURE_TRUE(aSelection->GetRangeCount()) failed: file e:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/editor/libeditor/base/nsEditor.cpp, line 3806
21:13:54     INFO -  [Parent 2108] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file e:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/editor/libeditor/base/nsEditor.cpp, line 3785
21:13:54     INFO -  [Parent 2108] WARNING: NS_ENSURE_SUCCESS(res, res) failed with result 0x80004005: file e:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/editor/libeditor/text/nsTextEditRules.cpp, line 417
21:13:54     INFO -  [Parent 2108] WARNING: NS_ENSURE_TRUE(aSelection->GetRangeCount()) failed: file e:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/editor/libeditor/base/nsEditor.cpp, line 3806
21:13:54     INFO -  [Parent 2108] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file e:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/editor/libeditor/base/nsEditor.cpp, line 3785
21:13:54     INFO -  [Parent 2108] WARNING: NS_ENSURE_SUCCESS(res, res) failed with result 0x80004005: file e:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/editor/libeditor/text/nsTextEditRules.cpp, line 417
21:13:54     INFO -  ++DOCSHELL 1F9EEF58 == 76 [id = 1167]
21:13:54     INFO -  ++DOMWINDOW == 254 (1F9A23B8) [serial = 3212] [outer = 00000000]
21:13:54     INFO -  ++DOMWINDOW == 255 (143959C8) [serial = 3213] [outer = 1F9A23B8]
21:13:54     INFO -  [Parent 2108] WARNING: NS_ENSURE_TRUE(mMutable) failed: file e:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/netwerk/base/src/nsSimpleURI.cpp, line 266
21:13:54     INFO -  TEST-PASS | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Tab has been opened in current browser window
21:13:54     INFO -  ++DOCSHELL 1ED6D788 == 77 [id = 1168]
21:13:54     INFO -  ++DOMWINDOW == 256 (0CE70988) [serial = 3214] [outer = 00000000]
21:13:54     INFO -  ++DOMWINDOW == 257 (153DF9E8) [serial = 3215] [outer = 0CE70988]
21:13:54     INFO -  [Parent 2108] WARNING: NS_ENSURE_TRUE(mMutable) failed: file e:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/netwerk/base/src/nsSimpleURI.cpp, line 266
21:13:54     INFO -  TEST-PASS | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | We have opened 2 new tab(s)
21:13:54     INFO -  TEST-PASS | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Tab has been opened in current browser window
21:13:54     INFO -  [Parent 2108] WARNING: NS_ENSURE_TRUE(mMutable) failed: file e:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/netwerk/base/src/nsSimpleURI.cpp, line 266
21:13:54     INFO -  0[38a5f8]: Assertion failure: rv != 0, at e:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/nsprpub/pr/src/md/windows/w95cv.c:153
21:13:54     INFO -  Assertion failure: rv != 0, at e:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/nsprpub/pr/src/md/windows/w95cv.c:153
21:13:54     INFO -  nsStringStats
21:13:54     INFO -   => mAllocCount:        1652829
21:13:54     INFO -   => mReallocCount:       104177
21:13:54     INFO -   => mFreeCount:         1602399  --  LEAKED 50430 !!!
21:13:54     INFO -   => mShareCount:        3186231
21:13:54     INFO -   => mAdoptCount:         134579
21:13:54     INFO -   => mAdoptFreeCount:     134545  --  LEAKED 34 !!!
21:13:54     INFO -  Assertion failure: isEmpty(), at e:\builds\moz2_slave\m-in-w32-d-0000000000000000000\build\obj-firefox\dist\include\mozilla/LinkedList.h:304
21:13:55  WARNING -  TEST-UNEXPECTED-FAIL | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | Exited with code -2147483645 during test run
21:13:55     INFO -  INFO | automation.py | Application ran for: 0:08:00.891000
21:13:55     INFO -  INFO | zombiecheck | Reading PID log: c:\docume~1\cltbld~1.t-x\locals~1\temp\tmp9ypcqspidlog
21:13:55     INFO -  ==> process 2108 launched child process 2936 ("C:\slave\test\build\application\firefox\plugin-container.exe" --channel=2108.d01eb68.332495401 "c:\docume~1\cltbld~1.t-x\locals~1\temp\tmptm2rbt\plugins\nptest.dll" -greomni "C:\slave\test\build\application\firefox\omni.ja" -appomni "C:\slave\test\build\application\firefox\browser\omni.ja" -appdir "C:\slave\test\build\application\firefox\browser" - 2108 "\\.\pipe\gecko-crash-server-pipe.2108" plugin)
21:13:55     INFO -  ==> process 2108 launched child process 3064 ("C:\slave\test\build\application\firefox\plugin-container.exe" --channel=2108.1556e6b0.1421611392 "c:\docume~1\cltbld~1.t-x\locals~1\temp\tmptm2rbt\plugins\npsecondtest.dll" -greomni "C:\slave\test\build\application\firefox\omni.ja" -appomni "C:\slave\test\build\application\firefox\browser\omni.ja" -appdir "C:\slave\test\build\application\firefox\browser" - 2108 "\\.\pipe\gecko-crash-server-pipe.2108" plugin)
21:13:55     INFO -  ==> process 2108 launched child process 3116 ("C:\slave\test\build\application\firefox\plugin-container.exe" --channel=2108.c764848.2076014064 "c:\docume~1\cltbld~1.t-x\locals~1\temp\tmptm2rbt\plugins\nptest.dll" -greomni "C:\slave\test\build\application\firefox\omni.ja" -appomni "C:\slave\test\build\application\firefox\browser\omni.ja" -appdir "C:\slave\test\build\application\firefox\browser" - 2108 "\\.\pipe\gecko-crash-server-pipe.2108" plugin)
21:13:55     INFO -  ==> process 2108 launched child process 3176 ("C:\slave\test\build\application\firefox\plugin-container.exe" --channel=2108.14b17c58.1390524998 "c:\docume~1\cltbld~1.t-x\locals~1\temp\tmptm2rbt\plugins\nptest.dll" -greomni "C:\slave\test\build\application\firefox\omni.ja" -appomni "C:\slave\test\build\application\firefox\browser\omni.ja" -appdir "C:\slave\test\build\application\firefox\browser" - 2108 "\\.\pipe\gecko-crash-server-pipe.2108" plugin)
21:13:55     INFO -  ==> process 2108 launched child process 3252 ("C:\slave\test\build\application\firefox\plugin-container.exe" --channel=2108.143efd28.1659807659 "c:\docume~1\cltbld~1.t-x\locals~1\temp\tmptm2rbt\plugins\nptest.dll" -greomni "C:\slave\test\build\application\firefox\omni.ja" -appomni "C:\slave\test\build\application\firefox\browser\omni.ja" -appdir "C:\slave\test\build\application\firefox\browser" - 2108 "\\.\pipe\gecko-crash-server-pipe.2108" plugin)
21:13:55     INFO -  ==> process 2108 launched child process 3448 ("C:\slave\test\build\application\firefox\plugin-container.exe" --channel=2108.139b59f0.758692337 -greomni "C:\slave\test\build\application\firefox\omni.ja" -appomni "C:\slave\test\build\application\firefox\browser\omni.ja" -appdir "C:\slave\test\build\application\firefox\browser" - 2108 "\\.\pipe\gecko-crash-server-pipe.2108" tab)
21:13:55     INFO -  ==> process 2108 launched child process 3596 ("C:\slave\test\build\application\firefox\plugin-container.exe" --channel=2108.1337d1e8.312671343 -greomni "C:\slave\test\build\application\firefox\omni.ja" -appomni "C:\slave\test\build\application\firefox\browser\omni.ja" -appdir "C:\slave\test\build\application\firefox\browser" - 2108 "\\.\pipe\gecko-crash-server-pipe.2108" tab)
21:13:55     INFO -  ==> process 2108 launched child process 3464 ("C:\slave\test\build\application\firefox\plugin-container.exe" --channel=2108.155025b0.335575857 -greomni "C:\slave\test\build\application\firefox\omni.ja" -appomni "C:\slave\test\build\application\firefox\browser\omni.ja" -appdir "C:\slave\test\build\application\firefox\browser" - 2108 "\\.\pipe\gecko-crash-server-pipe.2108" tab)
21:13:55     INFO -  ==> process 2108 launched child process 3836 ("C:\slave\test\build\application\firefox\plugin-container.exe" --channel=2108.c54ad80.150726913 -greomni "C:\slave\test\build\application\firefox\omni.ja" -appomni "C:\slave\test\build\application\firefox\browser\omni.ja" -appdir "C:\slave\test\build\application\firefox\browser" - 2108 "\\.\pipe\gecko-crash-server-pipe.2108" tab)
21:13:55     INFO -  ==> process 2108 launched child process 3804 ("C:\slave\test\build\application\firefox\plugin-container.exe" --channel=2108.147dfba0.1165321350 -greomni "C:\slave\test\build\application\firefox\omni.ja" -appomni "C:\slave\test\build\application\firefox\browser\omni.ja" -appdir "C:\slave\test\build\application\firefox\browser" - 2108 "\\.\pipe\gecko-crash-server-pipe.2108" tab)
21:14:02  WARNING -  PROCESS-CRASH | chrome://mochitests/content/browser/browser/components/places/tests/browser/browser_library_middleclick.js | application crashed [@ ntdll.dll + 0x120e]
21:14:02     INFO -  Crash dump filename: c:\docume~1\cltbld~1.t-x\locals~1\temp\tmptm2rbt\minidumps\03a2adb2-ec36-43cd-9c00-e19e869ee4cb.dmp
21:14:02     INFO -  Operating system: Windows NT
21:14:02     INFO -                    5.1.2600 Service Pack 3
21:14:02     INFO -  CPU: x86
21:14:02     INFO -       GenuineIntel family 6 model 30 stepping 5
21:14:02     INFO -       8 CPUs
21:14:02     INFO -  Crash reason:  EXCEPTION_BREAKPOINT
21:14:02     INFO -  Crash address: 0x7c90120e
21:14:02     INFO -  Thread 0 (crashed)
21:14:02     INFO -   0  ntdll.dll + 0x120e
21:14:02     INFO -      eip = 0x7c90120e   esp = 0x0012f2d4   ebp = 0x0012f2e4   ebx = 0x00a8f3b8
21:14:02     INFO -      esi = 0x10261440   edi = 0x00a94cb8   eax = 0x00000000   ecx = 0xf4fa5e34
21:14:02     INFO -      edx = 0x10361f48   efl = 0x00000206
21:14:02     INFO -      Found by: given as instruction pointer in context
21:14:02     INFO -   1  nss3.dll!md_UnlockAndPostNotifies [w95cv.c:2980ed8667f3 : 153 + 0x17]
21:14:02     INFO -      eip = 0x009cf08a   esp = 0x0012f2ec   ebp = 0x0012f36c
21:14:02     INFO -      Found by: previous frame's frame pointer
21:14:02     INFO -   2  nss3.dll!_PR_MD_UNLOCK [w95cv.c:2980ed8667f3 : 363 + 0xb]
21:14:02     INFO -      eip = 0x009cf506   esp = 0x0012f374   ebp = 0x0012f378
21:14:02     INFO -      Found by: call frame info
21:14:02     INFO -   3  nss3.dll!PR_Unlock [prulock.c:2980ed8667f3 : 315 + 0x8]
21:14:02     INFO -      eip = 0x009cbef4   esp = 0x0012f380   ebp = 0x0012f38c
21:14:02     INFO -      Found by: call frame info
21:14:02     INFO -   4  nss3.dll!PR_ExitMonitor [prmon.c:2980ed8667f3 : 100 + 0x7]
21:14:02     INFO -      eip = 0x009beb1b   esp = 0x0012f394   ebp = 0x0012f3a0
21:14:02     INFO -      Found by: call frame info
21:14:02     INFO -   5  xul.dll!mozilla::ReentrantMonitor::Exit() [BlockingResourceBase.cpp:2980ed8667f3 : 290 + 0x8]
21:14:02     INFO -      eip = 0x0364705f   esp = 0x0012f3a8   ebp = 0x0012f3c0
21:14:02     INFO -      Found by: call frame info
}

Carsten Book [:Tomcat]

Comment 2

•

11 years ago

kaie, wan-the, : do you have a idea what could cause this error ?

Ed Morley [:emorley]

Reporter

Comment 3

•

11 years ago

Some rough ranges... (though seeming not gecko checkin related)

Inbound:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=xp.*debug.*chrome&fromchange=0d4a80f8e1df&tochange=2a4109a72ff7

Fx-team:
https://tbpl.mozilla.org/?tree=Fx-Team&jobname=xp.*debug.*chrome&fromchange=4d44bc8da75c&tochange=7b5b8819ac56

mozilla-central:
https://tbpl.mozilla.org/?jobname=xp.*debug.*chrome&fromchange=2b4aa0f8515e&tochange=dc909122bcf5

(no longer active)

Comment 4

•

11 years ago

The assertion failure is caused by PR_ASSERTing the return value of a CloseHandle call.  One reason why that might happen is that we may pass an invalid handle to CloseHandle.

Kai Engert (:KaiE:)

Comment 5

•

11 years ago

This crash occurred in low level thread synchronization code, and md_UnlockAndPostNotifies is Windows/OS2 only code. I doubt that code has changed recently, unless you can show that you upgraded NSPR in the change window.

Memory corruption?

(no longer active)

Comment 6

•

11 years ago

Landed a debugging patch to print out the value of the handle and the last error when CloseHandle fails:

https://hg.mozilla.org/integration/mozilla-inbound/rev/861e1c86d759

Whiteboard: [leave open]

Ed Morley [:emorley]

Reporter

Comment 7

•

11 years ago

Few extra points:
1) This has happened on many different machines.
2) The frequency of occurrence has varied wildly, making it very hard to tell whether it's gecko-checkin caused (and just fairly low frequency on the whole, except a few flukes near tip) or else puppet-slave-deployment (or similar - Windows update? though they should be disabled afaik), or else "prior bustage but only visible once that build slave has had a clobber" or something equally annoying to find the regression range of.

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #6)
> Landed a debugging patch to print out the value of the handle and the last
> error when CloseHandle fails:
> 
> https://hg.mozilla.org/integration/mozilla-inbound/rev/861e1c86d759

That's great - thank you :-)

Benjamin Smedberg

Comment 8

•

11 years ago

These are all over the place, in terms of cause:

https://tbpl.mozilla.org/php/getParsedLog.php?id=27917258&tree=Mozilla-Inbound
In _PR_MD_CLEAN_THREAD we're closing the thread->md.handle and asserting success

https://tbpl.mozilla.org/php/getParsedLog.php?id=27917286&tree=Mozilla-Inbound
We're failing a WaitForSingleObject on the monitor which protects a non-main-thread event queue.

https://tbpl.mozilla.org/php/getParsedLog.php?id=27917301&tree=Mozilla-Inbound
Error from ReleaseSemaphore while shutting down a non-main thread using nsThread::Shutdown

Similar things for the last few.

If this were a code change, I think it's highly likely that somebody is calling nsIThread->Shutdown multiple times for the same thread. What's the earliest build on which we saw this?

Ed Morley [:emorley]

Reporter

Comment 9

•

11 years ago

The earliest I can see is either:
https://tbpl.mozilla.org/php/getParsedLog.php?id=27918062&tree=Mozilla-Central

or:
https://tbpl.mozilla.org/php/getParsedLog.php?id=27918327&tree=Mozilla-Inbound

Ed Morley [:emorley]

Reporter

Comment 10

•

11 years ago

Revised ranges for each tree (seems to be centered around merges):

m-c:
https://tbpl.mozilla.org/?jobname=xp.*debug.*chrome&fromchange=2b4aa0f8515e&tochange=64e01f43e027

inbound:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=xp.*debug.*chrome&fromchange=0d4a80f8e1df&tochange=1f7fddaf46fd

fx-team:
https://tbpl.mozilla.org/?tree=Fx-Team&jobname=xp.*debug.*chrome&fromchange=4d44bc8da75c&tochange=7cb2b73c7d25

(note the fromchange for each is much earlier than likely, but not taking anything for granted, awaiting more retriggers)

Kyle Huey (Exited; not receiving bugmail, old account, do not use)

Comment 11

•

11 years ago

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #8)
> If this were a code change, I think it's highly likely that somebody is
> calling nsIThread->Shutdown multiple times for the same thread. What's the
> earliest build on which we saw this?

nsThread::Shutdown prevents multiple calls by setting mShutdownRequired, no?

Benjamin Smedberg

Comment 12

•

11 years ago

I believe khuey is correct. The only other way I can think of this happening is if we were actually leaking a ref to nsThread objects and then using them. Looking through both the commit logs and the diffstat/patch of the large regression range from comment 10, there's really nothing terribly suspicious: some Firefox debugger patches which reference threads, but nothing that immediately jumps out.

I'm currently stumped.

Ed Morley [:emorley]

Reporter

Comment 13

•

11 years ago

Assertion post Ehsan's patch:
https://tbpl.mozilla.org/php/getParsedLog.php?id=27930724&tree=Mozilla-Inbound

Ed Morley [:emorley]

Reporter

Comment 14

•

11 years ago

Err though the printf output isn't there?!

Matt Brubeck (:mbrubeck)

Comment 15

•

11 years ago

(In reply to Ed Morley [:edmorley UTC+1] from comment #13)
> Assertion post Ehsan's patch:
> https://tbpl.mozilla.org/php/getParsedLog.php?id=27930724&tree=Mozilla-Inbound

This failed in a different location than comment 1: nsprpub/pr/src/md/windows/w95cv.c:153

Matt Brubeck (:mbrubeck)

Comment 16

•

11 years ago

(In reply to Matt Brubeck (:mbrubeck) from comment #15)
> This failed in a different location than comment 1:
> nsprpub/pr/src/md/windows/w95cv.c:153

Actually comment 1 had several examples of failures at w95cv.c:153, as well as a third location, w95cv.c:253.

Matt Brubeck (:mbrubeck)

Comment 17

•

11 years ago

Attached patch more logging — Details — Splinter Review

Matt Brubeck (:mbrubeck)

Comment 18

•

11 years ago

Comment on attachment 805406 [details] [diff] [review]
more logging

https://hg.mozilla.org/integration/mozilla-inbound/rev/1713de744990

Ed Morley [:emorley]

Reporter

Comment 19

•

11 years ago

More useful logs :-)

https://tbpl.mozilla.org/php/getParsedLog.php?id=27933731&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=27933746&tree=Mozilla-Inbound

Alex Keybl [:akeybl]

Updated

•

11 years ago

tracking-firefox26: --- → ?

(no longer active)

Comment 20

•

11 years ago

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #12)
> I believe khuey is correct. The only other way I can think of this happening
> is if we were actually leaking a ref to nsThread objects and then using
> them. Looking through both the commit logs and the diffstat/patch of the
> large regression range from comment 10, there's really nothing terribly
> suspicious: some Firefox debugger patches which reference threads, but
> nothing that immediately jumps out.
> 
> I'm currently stumped.

Perhaps we can run the tests through vtune's memory checker and see if it screams about something?  Ben, do you know if we can do that relatively easily?

Flags: needinfo?(bent.mozilla)

Kyle Huey (Exited; not receiving bugmail, old account, do not use)

Updated

•

11 years ago

Depends on: 917075

Phil Ringnalda (:philor)

Comment 21

•

11 years ago

Because nobody can stand to see WinXP get all the attention, https://tbpl.mozilla.org/php/getParsedLog.php?id=27942096&tree=Mozilla-Inbound is on Win8.

Kyle Huey (Exited; not receiving bugmail, old account, do not use)

Comment 22

•

11 years ago

I got this in a debugger.

First-chance exception at 0x7c90e4df in firefox.exe: 0xC0000235: NtClose was called on a handle that was protected from close via NtSetInformationObject.
Unhandled exception at 0x7c90e4df in firefox.exe: 0xC0000235: NtClose was called on a handle that was protected from close via NtSetInformationObject.

Kyle Huey (Exited; not receiving bugmail, old account, do not use)

Comment 23

•

11 years ago

Unfortunately we don't save the PDB files for builds so I can't actually debug it ...

It looks like we finally found the right patch to back out though.

Phil Ringnalda (:philor)

Comment 24

•

11 years ago

https://hg.mozilla.org/mozilla-central/rev/861e1c86d759
https://hg.mozilla.org/mozilla-central/rev/1713de744990

(no longer active)

Comment 25

•

11 years ago

(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #22)
> I got this in a debugger.
> 
> First-chance exception at 0x7c90e4df in firefox.exe: 0xC0000235: NtClose was
> called on a handle that was protected from close via NtSetInformationObject.
> Unhandled exception at 0x7c90e4df in firefox.exe: 0xC0000235: NtClose was
> called on a handle that was protected from close via NtSetInformationObject.

Hmm, see <http://www.nynaeve.net/?p=203>.  It would be interesting to see where that handle is coming from, and who prevents it from being closed...

(no longer active)

Comment 26

•

11 years ago

Backed out the debugging patches:

https://hg.mozilla.org/integration/mozilla-inbound/rev/a0fcc3eb892f

Ed Morley [:emorley]

Reporter

Comment 27

•

11 years ago

The following backouts were performed, which has resolved the problem:
https://hg.mozilla.org/mozilla-central/rev/ee7366ad4698
https://hg.mozilla.org/mozilla-central/rev/1fbc1a15e4bb
https://hg.mozilla.org/mozilla-central/rev/ef0f3d62cbb2

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #26)
> Backed out the debugging patches:
> 
> https://hg.mozilla.org/integration/mozilla-inbound/rev/a0fcc3eb892f

Double landed on m-c, so it makes the uplift:
remote:   https://hg.mozilla.org/mozilla-central/rev/2520866d5874

We're now good to proceed with the uplift; leaving bug open to continue debugging so that the backed out changesets can reland - but the priority is lower now.

Blocks: 906620, 914435, 913110

Severity: blocker → normal

Ben Turner (not reading bugmail, use the needinfo flag!)

Comment 28

•

11 years ago

It sounds like vtune won't help here really... We're not touching dead things, just protected things?

Flags: needinfo?(bent.mozilla)

Lukas Blakk [:lsblakk] use ?needinfo

Updated

•

11 years ago

status-firefox26: --- → affected

status-firefox27: --- → fixed

tracking-firefox26: ? → +

tracking-firefox27: --- → +

Shane Caraveo (:mixedpuppy)

Comment 29

•

11 years ago

I've narrowed this down to bug 914435 and am trying some changes in the test file to see if I can avoid the crash.  Assuming that works, I believe it is e10s related since the test is using a remote iframe (which I am removing from that test).

Kyle Huey (Exited; not receiving bugmail, old account, do not use)

Comment 30

•

11 years ago

There's a platform bug here that still needs to be found and fixed though.

Shane Caraveo (:mixedpuppy)

Comment 31

•

11 years ago

(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #30)
> There's a platform bug here that still needs to be found and fixed though.

Sorry, didn't mean to imply it isn't a platform bug, but I think the test changes I've done may indicate that it has to do with e10s.  The test was using a remote frameworker, I removed that from the test and am rerunning a bunch of the tests to see if the crashes still exist.

https://tbpl.mozilla.org/?tree=Try&rev=04f8af703949

Ben Turner (not reading bugmail, use the needinfo flag!)

Comment 32

•

11 years ago

bsmedberg suggests that bug 920397 may be the underlying platform bug here.

Kyle Huey (Exited; not receiving bugmail, old account, do not use)

Comment 33

•

11 years ago

That's plausible.  I ran out of steam trying to investigate this in a debugger :-/

Mark Hammond [:markh] [:mhammond]

Comment 34

•

11 years ago

(In reply to ben turner [:bent] (needinfo? encouraged) from comment #32)
> bsmedberg suggests that bug 920397 may be the underlying platform bug here.

I commented in that bug, but I don't think it is identical (although may be related).  However... thinking a little more...

Windows may reuse handle values.  In that bug we have 2 bits of code closing the same handle.Let's say:

1) Handle A closed first time.
2) Other code runs that gets a *new* handle - it just happens to get the same handle value (as A is now closed), but is handle B.
3) Handle A closed second time.

Given handle A and handle B have the same value, bad things happen...

Mark Hammond [:markh] [:mhammond]

Comment 35

•

11 years ago

(In reply to Mark Hammond (:markh) from comment #34)
> related).  However... thinking a little more...

printf debugging tells me this is indeed what is happening in at least one case.  ipc/glue/SyncChannel.cpp has an mHandle that occasionally throws the InvalidHandle exception as it it closed.  The printf debugging shows that the exact same handle value was previously used as a handle to a process and was closed twice as per bug 920397.

While I'm yet to catch it, it doesn't take much imagination to picture the exact same thing happening in w95thred.c

Mark Hammond [:markh] [:mhammond]

Comment 36

•

11 years ago

I pushed the fix in bug 920397, plus the patch from bug 914435 which initially triggered this bug, to try - https://tbpl.mozilla.org/?tree=Try&rev=ed2cc81209ec - and could not reproduce this problem.

The patch from bug 914435 didn't apply cleanly - Shane, can you please check https://hg.mozilla.org/try/rev/f4fa9a9ccfeb and ensure it includes the necessary parts which caused the issue (I think it does - it has a provider with a worker, and remote-frameworkers has been enabled).  Assuming that is the case, I think we can close this as a dupe of bug 920397.

Flags: needinfo?(mixedpuppy)

Mark Hammond [:markh] [:mhammond]

Comment 37

•

11 years ago

I spoke to Shane and he believes that try run would also have crashed if the bug wasn't fixed - so marking this as a dupe of bug 920397 where the fix landed.

Status: NEW → RESOLVED

Closed: 11 years ago

Flags: needinfo?(mixedpuppy)

Resolution: --- → DUPLICATE

Lukas Blakk [:lsblakk] use ?needinfo

Updated

•

11 years ago

tracking-firefox26: + → ---

tracking-firefox27: + → ---