1373572 - Intermittent firefox-ui-update PROCESS-CRASH | marionette.py | application crashed [None]

Status: RESOLVED INCOMPLETE

(Testing :: Firefox UI Tests, defect)

Description

[Treeherder Bug Filer]

Filed by: hskupin [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=107314371&repo=mozilla-central

https://firefox-ui-tests.s3.amazonaws.com/66b100c1-3a5d-48a3-8ecf-6b00a3011939/log_info.log

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Sadly the minidump cannot be analyzed by the stackwalk binary:

06:05:15     INFO -  *** UTM:SVC TimerManager:registerTimer - id: telemetry_modules_ping
06:05:16     INFO -  ###!!! [Child][MessageChannel::SendAndWait] Error: Channel error: cannot send/recv
06:05:16     INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/TiJctOmpSPKLVfCpUYmmZA/artifacts/public%2Fbuild%2Ffirefox-56.0a1.en-US.win32.crashreporter-symbols.zip
06:05:24     INFO - mozcrash Copy/paste: c:\jenkins\workspace\mozilla-central_update\build\win32-minidump_stackwalk.exe c:\jenkins\workspace\mozilla-central_update\build\tmpca1fin.mozrunner-1497531891\minidumps\41193d7d-6f1e-429f-bc1e-10e94c3d7478-browser.dmp c:\users\mozauto\appdata\local\temp\tmpqjljf9
06:06:08     INFO - mozcrash Saved minidump as c:\jenkins\workspace\mozilla-central_update\build\blobber_upload_dir\41193d7d-6f1e-429f-bc1e-10e94c3d7478-browser.dmp
06:06:08     INFO - PROCESS-CRASH | marionette.py | application crashed [None]
06:06:08     INFO - Crash dump filename: c:\jenkins\workspace\mozilla-central_update\build\tmpca1fin.mozrunner-1497531891\minidumps\41193d7d-6f1e-429f-bc1e-10e94c3d7478-browser.dmp
06:06:08     INFO - stderr from minidump_stackwalk:
06:06:08     INFO - 2017-06-15 06:05:24: minidump.cc:4359: INFO: Minidump opened minidump c:\jenkins\workspace\mozilla-central_update\build\tmpca1fin.mozrunner-1497531891\minidumps\41193d7d-6f1e-429f-bc1e-10e94c3d7478-browser.dmp
06:06:08     INFO - 2017-06-15 06:05:24: minidump.cc:4479: INFO: Minidump not byte-swapping minidump
06:06:08     INFO - 2017-06-15 06:05:24: minidump.cc:3833: INFO: MinidumpBreakpadInfo has no dump thread
06:06:08     INFO - 2017-06-15 06:05:24: minidump.cc:3856: INFO: MinidumpBreakpadInfo has no requesting thread
06:06:08     INFO - 2017-06-15 06:05:24: minidump.cc:4956: INFO: GetStream: type 1197932546 not present
06:06:08     INFO - 2017-06-15 06:05:24: minidump_processor.cc:146: INFO: Found 232 memory regions.
06:06:08     INFO - 2017-06-15 06:05:24: minidump_processor.cc:156: INFO: Minidump c:\jenkins\workspace\mozilla-central_update\build\tmpca1fin.mozrunner-1497531891\minidumps\41193d7d-6f1e-429f-bc1e-10e94c3d7478-browser.dmp has CPU info, OS info, Breakpad info, exception, module list, thread list, no dump thread, requesting thread, and process create time
06:06:08     INFO - 2017-06-15 06:05:24: minidump_processor.cc:195: INFO: Looking at thread c:\jenkins\workspace\mozilla-central_update\build\tmpca1fin.mozrunner-1497531891\minidumps\41193d7d-6f1e-429f-bc1e-10e94c3d7478-browser.dmp:0/55 id 0xcb0
06:06:08     INFO - 2017-06-15 06:05:24: simple_symbol_supplier.cc:196: INFO: No symbol file at c:\users\mozauto\appdata\local\temp\tmpqjljf9/xul.pdb/4DA2D0249B484F5ABEF5868EBBCDF56C2/xul.sym
06:06:08     INFO - 2017-06-15 06:05:24: http_symbol_supplier.cc:553: INFO: HTTPSymbolSupplier: querying http://s3-us-west-2.amazonaws.com/org.mozilla.crash-stats.symbols-public/v1/xul.pdb/4DA2D0249B484F5ABEF5868EBBCDF56C2/xul.sym
06:06:08     INFO - 2017-06-15 06:05:24: http_symbol_supplier.cc:342: INFO: HTTPSymbolSupplier: need to manually un-gzip
06:06:08     INFO - 2017-06-15 06:05:24: http_symbol_supplier.cc:351: INFO: HTTPSymbolSupplier: symbol exists, saving to C:\Users\mozauto\AppData\Local\Temp\sym1.tmp
06:06:08     INFO - 2017-06-15 06:05:28: source_line_resolver_base.cc:236: INFO: Loading symbols for module c:\jenkins\workspace\mozilla-central_update\build\application.copy\xul.dll from memory buffer
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f108
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f12c
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f1d4
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f1e0
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f2c4
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f2fc
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x3f560100
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f3fc
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f404
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f430
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f468
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f488
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f498
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f4a8
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f4bc
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f6b0
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f6e4
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f840
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80f84c
06:06:08     INFO - 2017-06-15 06:05:32: basic_code_modules.cc:110: INFO: No module at 0x80fcc8
06:06:08     INFO - 2017-06-15 06:05:32: simple_symbol_supplier.cc:196: INFO: No symbol file at c:\users\mozauto\appdata\local\temp\tmpqjljf9/firefox.pdb/FFBFC215CC1F4C2A98DCE16A1F3CAB0B2/firefox.sym
06:06:08     INFO - 2017-06-15 06:05:32: http_symbol_supplier.cc:553: INFO: HTTPSymbolSupplier: querying http://s3-us-west-2.amazonaws.com/org.mozilla.crash-stats.symbols-public/v1/firefox.pdb/FFBFC215CC1F4C2A98DCE16A1F3CAB0B2/firefox.sym
06:06:08     INFO - 2017-06-15 06:05:32: http_symbol_supplier.cc:342: INFO: HTTPSymbolSupplier: need to manually un-gzip
06:06:08     INFO - 2017-06-15 06:05:32: http_symbol_supplier.cc:351: INFO: HTTPSymbolSupplier: symbol exists, saving to C:\Users\mozauto\AppData\Local\Temp\sym1.tmp
06:06:08     INFO - 2017-06-15 06:05:35: source_line_resolver_base.cc:236: INFO: Loading symbols for module c:\jenkins\workspace\mozilla-central_update\build\application.copy\firefox.exe from memory buffer
06:06:08     INFO - 2017-06-15 06:05:39: simple_symbol_supplier.cc:196: INFO: No symbol file at c:\users\mozauto\appdata\local\temp\tmpqjljf9/wkernel32.pdb/DFB9F6A1661C482C87B4D0D454332CB81/wkernel32.sym
06:06:08     INFO - 2017-06-15 06:05:39: http_symbol_supplier.cc:553: INFO: HTTPSymbolSupplier: querying http://s3-us-west-2.amazonaws.com/org.mozilla.crash-stats.symbols-public/v1/wkernel32.pdb/DFB9F6A1661C482C87B4D0D454332CB81/wkernel32.sym
06:06:08     INFO - 2017-06-15 06:05:39: http_symbol_supplier.cc:351: INFO: HTTPSymbolSupplier: symbol exists, saving to C:\Users\mozauto\AppData\Local\Temp\sym1.tmp
06:06:08     INFO - 2017-06-15 06:05:39: source_line_resolver_base.cc:236: INFO: Loading symbols for module C:\Windows\System32\kernel32.dll from memory buffer
06:06:08     INFO - 2017-06-15 06:05:39: basic_code_modules.cc:110: INFO: No module at 0x808e39b3
06:06:08     INFO - 2017-06-15 06:05:39: basic_code_modules.cc:110: INFO: No module at 0x80fd6c
06:06:08     INFO - 2017-06-15 06:05:39: simple_symbol_supplier.cc:196: INFO: No symbol file at c:\users\mozauto\appdata\local\temp\tmpqjljf9/wntdll.pdb/238AACDDAEFE41C98EAA92CEB7374AAD2/wntdll.sym
06:06:08     INFO - 2017-06-15 06:05:39: http_symbol_supplier.cc:553: INFO: HTTPSymbolSupplier: querying http://s3-us-west-2.amazonaws.com/org.mozilla.crash-stats.symbols-public/v1/wntdll.pdb/238AACDDAEFE41C98EAA92CEB7374AAD2/wntdll.sym
06:06:08     INFO - 2017-06-15 06:05:40: http_symbol_supplier.cc:342: INFO: HTTPSymbolSupplier: need to manually un-gzip
06:06:08     INFO - 2017-06-15 06:05:40: http_symbol_supplier.cc:351: INFO: HTTPSymbolSupplier: symbol exists, saving to C:\Users\mozauto\AppData\Local\Temp\sym1.tmp
06:06:08     INFO - 2017-06-15 06:05:42: source_line_resolver_base.cc:236: INFO: Loading symbols for module C:\Windows\System32\ntdll.dll from memory buffer
06:06:08     INFO - 2017-06-15 06:05:47: minidump_processor.cc:195: INFO: Looking at thread c:\jenkins\workspace\mozilla-central_update\build\tmpca1fin.mozrunner-1497531891\minidumps\41193d7d-6f1e-429f-bc1e-10e94c3d7478-browser.dmp:1/55 id 0xa5c
06:06:08     INFO - 2017-06-15 06:05:47: basic_code_modules.cc:110: INFO: No module at 0x82533ec7
06:06:08     INFO - 2017-06-15 06:05:47: basic_code_modules.cc:110: INFO: No module at 0x25dfa64
06:06:08     INFO - 2017-06-15 06:05:47: simple_symbol_supplier.cc:196: INFO: No symbol file at c:\users\mozauto\appdata\local\temp\tmpqjljf9/mozglue.pdb/3DBED4EEC627498282E16048E393F6C32/mozglue.sym
06:06:08     INFO - 2017-06-15 06:05:47: http_symbol_supplier.cc:553: INFO: HTTPSymbolSupplier: querying http://s3-us-west-2.amazonaws.com/org.mozilla.crash-stats.symbols-public/v1/mozglue.pdb/3DBED4EEC627498282E16048E393F6C32/mozglue.sym
06:06:08     INFO - 2017-06-15 06:05:47: http_symbol_supplier.cc:342: INFO: HTTPSymbolSupplier: need to manually un-gzip
06:06:08     INFO - 2017-06-15 06:05:47: http_symbol_supplier.cc:351: INFO: HTTPSymbolSupplier: symbol exists, saving to C:\Users\mozauto\AppData\Local\Temp\sym1.tmp
06:06:08     INFO - 2017-06-15 06:05:49: source_line_resolver_base.cc:236: INFO: Loading symbols for module c:\jenkins\workspace\mozilla-central_update\build\application.copy\mozglue.dll from memory buffer
06:06:08     INFO - 2017-06-15 06:05:53: minidump_processor.cc:195: INFO: Looking at thread c:\jenkins\workspace\mozilla-central_update\build\tmpca1fin.mozrunner-1497531891\minidumps\41193d7d-6f1e-429f-bc1e-10e94c3d7478-browser.dmp:2/55 id 0x994
06:06:08     INFO - 2017-06-15 06:05:53: simple_symbol_supplier.cc:196: INFO: No symbol file at c:\users\mozauto\appdata\local\temp\tmpqjljf9/wkernelbase.pdb/B2F9E12F3A084995A241C9912D94E9CD2/wkernelbase.sym
06:06:08     INFO - 2017-06-15 06:05:53: http_symbol_supplier.cc:553: INFO: HTTPSymbolSupplier: querying http://s3-us-west-2.amazonaws.com/org.mozilla.crash-stats.symbols-public/v1/wkernelbase.pdb/B2F9E12F3A084995A241C9912D94E9CD2/wkernelbase.sym
06:06:08     INFO - 2017-06-15 06:05:53: http_symbol_supplier.cc:342: INFO: HTTPSymbolSupplier: need to manually un-gzip
06:06:08     INFO - 2017-06-15 06:05:53: http_symbol_supplier.cc:351: INFO: HTTPSymbolSupplier: symbol exists, saving to C:\Users\mozauto\AppData\Local\Temp\sym1.tmp
06:06:08     INFO - 2017-06-15 06:05:56: source_line_resolver_base.cc:236: INFO: Loading symbols for module C:\Windows\System32\KERNELBASE.dll from memory buffer
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0xffffffff
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0x68df97c
06:06:08     INFO - 2017-06-15 06:06:00: minidump_processor.cc:195: INFO: Looking at thread c:\jenkins\workspace\mozilla-central_update\build\tmpca1fin.mozrunner-1497531891\minidumps\41193d7d-6f1e-429f-bc1e-10e94c3d7478-browser.dmp:3/55 id 0xdf0
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0x6cbfcd4
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0xffffffff
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0x0
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0xb0db80
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0x6cbfd08
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0x6cbfd28
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0x6cbfd38
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0x6cbfd70
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0x6cbfd90
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0x6cbfe50
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0x6cbfe58
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0x6cbfe6c
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0x6cbfe80
06:06:08     INFO - 2017-06-15 06:06:00: minidump_processor.cc:195: INFO: Looking at thread c:\jenkins\workspace\mozilla-central_update\build\tmpca1fin.mozrunner-1497531891\minidumps\41193d7d-6f1e-429f-bc1e-10e94c3d7478-browser.dmp:4/55 id 0xaf8
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0x1
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0xb0b2d4
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0x0
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0x0
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0xb5b5e0
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0xa1ff758
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0xa1ff778
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0xa1ff7b0
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0xa1ff7d0
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0xa1ff7f4
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0xb153e0
06:06:08     INFO - 2017-06-15 06:06:00: basic_code_modules.cc:110: INFO: No module at 0xa1ff814
06:06:08     INFO - 2017-06-15 06:06:00: simple_symbol_supplier.cc:196: INFO: No symbol file at c:\users\mozauto\appdata\local\temp\tmpqjljf9/nss3.pdb/E33AAC8B85864622ABB427168427F6572/nss3.sym
06:06:08     INFO - 2017-06-15 06:06:00: http_symbol_supplier.cc:553: INFO: HTTPSymbolSupplier: querying http://s3-us-west-2.amazonaws.com/org.mozilla.crash-stats.symbols-public/v1/nss3.pdb/E33AAC8B85864622ABB427168427F6572/nss3.sym
06:06:08     INFO - 2017-06-15 06:06:00: http_symbol_supplier.cc:342: INFO: HTTPSymbolSupplier: need to manually un-gzip
06:06:08     INFO - 2017-06-15 06:06:00: http_symbol_supplier.cc:351: INFO: HTTPSymbolSupplier: symbol exists, saving to C:\Users\mozauto\AppData\Local\Temp\sym1.tmp
06:06:08     INFO - 2017-06-15 06:06:02: source_line_resolver_base.cc:236: INFO: Loading symbols for module c:\jenkins\workspace\mozilla-central_update\build\application.copy\nss3.dll from memory buffer
06:06:08     INFO - 
06:06:08     INFO - This application has requested the Runtime to terminate it in an unusual way.
06:06:08     INFO - Please contact the application's support team for more information.
06:06:08     INFO - terminate called after throwing an instance of 'std::bad_alloc'
06:06:08     INFO -   what():  std::bad_alloc

Ted, is there anything you can see here which cause that? If you need a minidump I might be able to retrieve one when it happens again.

Comment 2

[(not currently active) Ted Mielczarek]

It looks like the stackwalker is crashing trying to read a symbol file, which is certainly odd. If you can get me a minidump file I can take a look at it. It looks like this job does have MINIDUMP_SAVE_PATH set, but perhaps nothing is actually uploading those files anywhere?
06:06:38     INFO - mozcrash Saved minidump as c:\jenkins\workspace\mozilla-central_update\build\blobber_upload_dir\41193d7d-6f1e-429f-bc1e-10e94c3d7478.dmp

Comment 3

[Intermittent Failures Robot]

1 failures in 814 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373572&startday=2017-06-12&endday=2017-06-18&tree=all

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The underlying crash might actually be related to bug 1375407.

Ted, I will try to figure out how to get one of the stacks out of the box. Looks like uploading doesn't work right now.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: cae897e9-0f86-4635-88d9-e784ea02720d.zip

Ted, attached you can find such a broken minidump file. The failure actually seem to appear in the one with the -browser suffix.

Comment 6

[Intermittent Failures Robot]

2 failures in 718 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 2

Platform breakdown:
* windows8-64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373572&startday=2017-06-26&endday=2017-07-02&tree=all

Comment 7

[Intermittent Failures Robot]

1 failures in 656 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373572&startday=2017-07-03&endday=2017-07-09&tree=all

Comment 8

[Intermittent Failures Robot]

1 failures in 939 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* windows10-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373572&startday=2017-08-28&endday=2017-09-03&tree=all

Comment 9

[(not currently active) Ted Mielczarek]

Oh, so I was able to reproduce this on Windows. I think the stackwalker is OOMing because it's a 32-bit binary. All of our Windows automation is 64-bit nowadays, we could swap it out for a 64-bit binary. Fixing bug 1391408 would make that easier.

Comment 10

[Firefox Bug Husbandry Bot]

https://wiki.mozilla.org/Bugmasters#Intermittent_Test_Failure_Cleanup