Closed
Bug 1585365
Opened 6 years ago
Closed 6 years ago
--geckoProfile broken locally
Categories
(Testing :: Talos, defect, P3)
Tracking
(Not tracked)
RESOLVED
INCOMPLETE
People
(Reporter: davidwalsh, Unassigned)
Details
I've run the following two commands, both of which break at the very end of the run:
./mach talos-test --activeTests damp --cycles 1 --tppagecycles 1 --geckoProfile --geckoProfileEntries 100000000 --subtests custom.jsdebugger
•./mach talos-test --activeTests damp --geckoProfile
....
14:34:38 INFO - Tue Oct 1 09:34:32 2019 MainThread Parsing SYM file at /var/folders/94/n0b8629s1vb79l84c9knt2jr0000gn/T/tmpz72gSa/libsystem_secinit.dylib/EF1EA47B7B2235E8BD9BF7003DCB96AE0/libsystem_secinit.dylib.sym
14:34:38 INFO - Tue Oct 1 09:34:32 2019 MainThread Parsing SYM file at /var/folders/94/n0b8629s1vb79l84c9knt2jr0000gn/T/tmpz72gSa/libsystem_symptoms.dylib/03F1C2DD0F5A3D9D88F6B26C0F94EB520/libsystem_symptoms.dylib.sym
14:34:38 INFO - Tue Oct 1 09:34:32 2019 MainThread Parsing SYM file at /var/folders/94/n0b8629s1vb79l84c9knt2jr0000gn/T/tmpz72gSa/libunwind.dylib/24A97A67F0173CFCB0D06BD0224B13360/libunwind.dylib.sym
14:34:38 INFO - Adding profile profile_damp/page_0_pagecycle_1/cycle_0.profile to archive /Users/davidwalsh/Projects/mozilla-central/testing/mozharness/build/blobber_upload_dir/profile_damp.zip
14:34:39 INFO - mozcrash Saved minidump as /Users/davidwalsh/Projects/mozilla-central/testing/mozharness/build/blobber_upload_dir/A5464998-F5B1-45C9-9592-3018FCED895B-browser.dmp
14:34:39 INFO - PROCESS-CRASH | damp | application crashed [unknown top frame]
14:34:39 INFO - Crash dump filename: /var/folders/94/n0b8629s1vb79l84c9knt2jr0000gn/T/tmpzF5fNo/profile/minidumps/A5464998-F5B1-45C9-9592-3018FCED895B-browser.dmp
14:34:39 INFO - MINIDUMP_STACKWALK not set, can't process dump.
14:34:39 INFO - mozcrash Saved minidump as /Users/davidwalsh/Projects/mozilla-central/testing/mozharness/build/blobber_upload_dir/10C7E49D-D86F-4490-9B82-29E569CD74A4.dmp
14:34:39 INFO - mozcrash Saved app info as /Users/davidwalsh/Projects/mozilla-central/testing/mozharness/build/blobber_upload_dir/10C7E49D-D86F-4490-9B82-29E569CD74A4.extra
14:34:39 INFO - PROCESS-CRASH | damp | application crashed [unknown top frame]
14:34:39 INFO - Crash dump filename: /var/folders/94/n0b8629s1vb79l84c9knt2jr0000gn/T/tmpzF5fNo/profile/minidumps/10C7E49D-D86F-4490-9B82-29E569CD74A4.dmp
14:34:39 INFO - MINIDUMP_STACKWALK not set, can't process dump.
14:34:39 INFO - mozcrash Saved minidump as /Users/davidwalsh/Projects/mozilla-central/testing/mozharness/build/blobber_upload_dir/10C7E49D-D86F-4490-9B82-29E569CD74A4-browser.dmp
14:34:39 INFO - PROCESS-CRASH | damp | application crashed [unknown top frame]
14:34:39 INFO - Crash dump filename: /var/folders/94/n0b8629s1vb79l84c9knt2jr0000gn/T/tmpzF5fNo/profile/minidumps/10C7E49D-D86F-4490-9B82-29E569CD74A4-browser.dmp
14:34:39 INFO - MINIDUMP_STACKWALK not set, can't process dump.
14:34:39 INFO - mozcrash Saved minidump as /Users/davidwalsh/Projects/mozilla-central/testing/mozharness/build/blobber_upload_dir/A5464998-F5B1-45C9-9592-3018FCED895B.dmp
14:34:39 INFO - mozcrash Saved app info as /Users/davidwalsh/Projects/mozilla-central/testing/mozharness/build/blobber_upload_dir/A5464998-F5B1-45C9-9592-3018FCED895B.extra
14:34:39 INFO - PROCESS-CRASH | damp | application crashed [unknown top frame]
14:34:39 INFO - Crash dump filename: /var/folders/94/n0b8629s1vb79l84c9knt2jr0000gn/T/tmpzF5fNo/profile/minidumps/A5464998-F5B1-45C9-9592-3018FCED895B.dmp
14:34:39 INFO - MINIDUMP_STACKWALK not set, can't process dump.
14:34:39 INFO - TEST-UNEXPECTED-ERROR | damp | Found crashes after test run, terminating test
14:34:39 ERROR - Traceback (most recent call last):
14:34:39 INFO - File "/Users/davidwalsh/Projects/mozilla-central/testing/talos/talos/run_tests.py", line 276, in run_tests
14:34:39 INFO - talos_results.add(mytest.runTest(browser_config, test))
14:34:39 INFO - File "/Users/davidwalsh/Projects/mozilla-central/testing/talos/talos/ttest.py", line 64, in runTest
14:34:39 INFO - return self._runTest(browser_config, test_config, setup)
14:34:39 INFO - File "/Users/davidwalsh/Projects/mozilla-central/testing/talos/talos/ttest.py", line 286, in _runTest
14:34:39 INFO - test_config['name'])
14:34:39 INFO - File "/Users/davidwalsh/Projects/mozilla-central/testing/talos/talos/ttest.py", line 48, in check_for_crashes
14:34:39 INFO - raise TalosCrash('Found crashes after test run, terminating test')
14:34:39 INFO - TalosCrash: Found crashes after test run, terminating test
14:34:39 INFO - TEST-INFO took 413393ms
14:34:39 INFO - SUITE-END | took 413s
14:34:40 ERROR - Return code: 2
14:34:40 WARNING - setting return code to 2
14:34:40 ERROR - # TBPL FAILURE #
14:34:40 INFO - Running post-action listener: _package_coverage_data
14:34:40 INFO - Running post-action listener: _resource_record_post_action
14:34:40 INFO - Running post-action listener: process_java_coverage_data
14:34:40 INFO - [mozharness: 2019-10-01 14:34:40.129916Z] Finished run-tests step (success)
14:34:40 INFO - Running post-run listener: _resource_record_post_run
14:34:40 INFO - Total resource usage - Wall time: 413s; CPU: 40.0%; Read bytes: 383397888; Write bytes: 414244864; Read time: 13220; Write time: 4333
14:34:40 INFO - TinderboxPrint: CPU usage<br/>39.7%
14:34:40 INFO - TinderboxPrint: I/O read bytes / time<br/>383,397,888 / 13,220
14:34:40 INFO - TinderboxPrint: I/O write bytes / time<br/>414,244,864 / 4,333
14:34:40 INFO - TinderboxPrint: CPU idle<br/>997.7 (60.3%)
14:34:40 INFO - TinderboxPrint: CPU system<br/>113.3 (6.9%)
14:34:40 INFO - TinderboxPrint: CPU user<br/>542.4 (32.8%)
14:34:40 INFO - TinderboxPrint: Swap in / out<br/>307,220,480 / 303,104
14:34:40 INFO - run-tests - Wall time: 414s; CPU: 40.0%; Read bytes: 383397888; Write bytes: 414244864; Read time: 13220; Write time: 4333
|
||
Comment 1•6 years ago
|
||
:bebe, is there someone on the SV team that can try to reproduce this please? I'd start with the 2nd command ./mach talos-test --activeTests damp --geckoProfile. Also maybe push to try to talos with --geckoProfile for damp too. Thanks!
Flags: needinfo?(fstrugariu)
Priority: -- → P2
|
|
||
Updated•6 years ago
|
Assignee: nobody → marian.raiciof
Status: NEW → ASSIGNED
|
||
Updated•6 years ago
|
Flags: needinfo?(fstrugariu)
|
||
Comment 2•6 years ago
|
||
David, what do you mean with break at the very end? I can see a lot of Firefox crashes, which cannot be analyzed because the minidump stackwalk binary is not available at your machine. So maybe you can clarify? Thanks.
Assignee: marian.raiciof → nobody
Status: ASSIGNED → NEW
Flags: needinfo?(dwalsh)
Priority: P2 → P3
|
|
Reporter | |
Comment 3•6 years ago
|
||
I ran ./mach talos-test --activeTests damp --geckoProfile and received the following feedback:
15:11:15 INFO - PID 19483 | [#70] server.protocoljs.DAMP Cycles:2 Average:925.53 Median:953.34 stddev:39.34 (4.1%)
15:11:15 INFO - PID 19483 | Values: 897.7 953.3
15:11:15 INFO - PID 19483 | -------- Summary: end --------
15:11:15 INFO - PID 19483 |
15:11:16 INFO - TEST-INFO | 19483: exit 0
15:11:46 INFO - Adding profile profile_damp/page_0_pagecycle_2/cycle_1.profile to archive /Users/davidwalsh/Projects/mozilla-central/testing/mozharness/build/blobber_upload_dir/profile_damp.zip
15:12:25 INFO - Adding profile profile_damp/page_0_pagecycle_1/cycle_1.profile to archive /Users/davidwalsh/Projects/mozilla-central/testing/mozharness/build/blobber_upload_dir/profile_damp.zip
15:12:27 INFO - mozcrash Saved minidump as /Users/davidwalsh/Projects/mozilla-central/testing/mozharness/build/blobber_upload_dir/0D48583F-23D8-463B-B133-C73A92E5E4A2.dmp
15:12:27 INFO - mozcrash Saved app info as /Users/davidwalsh/Projects/mozilla-central/testing/mozharness/build/blobber_upload_dir/0D48583F-23D8-463B-B133-C73A92E5E4A2.extra
15:12:27 INFO - PROCESS-CRASH | damp | application crashed [unknown top frame]
15:12:27 INFO - Crash dump filename: /var/folders/x_/zt4lhsl551q6dxkct7n2fqdh0000gn/T/tmpuC0SEZ/profile/minidumps/0D48583F-23D8-463B-B133-C73A92E5E4A2.dmp
15:12:27 INFO - MINIDUMP_STACKWALK not set, can't process dump.
15:12:27 INFO - mozcrash Saved minidump as /Users/davidwalsh/Projects/mozilla-central/testing/mozharness/build/blobber_upload_dir/0D48583F-23D8-463B-B133-C73A92E5E4A2-browser.dmp
15:12:27 INFO - PROCESS-CRASH | damp | application crashed [unknown top frame]
15:12:27 INFO - Crash dump filename: /var/folders/x_/zt4lhsl551q6dxkct7n2fqdh0000gn/T/tmpuC0SEZ/profile/minidumps/0D48583F-23D8-463B-B133-C73A92E5E4A2-browser.dmp
15:12:27 INFO - MINIDUMP_STACKWALK not set, can't process dump.
15:12:27 INFO - TEST-UNEXPECTED-ERROR | damp | Found crashes after test run, terminating test
15:12:27 ERROR - Traceback (most recent call last):
15:12:27 INFO - File "/Users/davidwalsh/Projects/mozilla-central/testing/talos/talos/run_tests.py", line 306, in run_tests
15:12:27 INFO - talos_results.add(mytest.runTest(browser_config, test))
15:12:27 INFO - File "/Users/davidwalsh/Projects/mozilla-central/testing/talos/talos/ttest.py", line 64, in runTest
15:12:27 INFO - return self._runTest(browser_config, test_config, setup)
15:12:27 INFO - File "/Users/davidwalsh/Projects/mozilla-central/testing/talos/talos/ttest.py", line 286, in _runTest
15:12:27 INFO - test_config['name'])
15:12:27 INFO - File "/Users/davidwalsh/Projects/mozilla-central/testing/talos/talos/ttest.py", line 48, in check_for_crashes
15:12:27 INFO - raise TalosCrash('Found crashes after test run, terminating test')
15:12:27 INFO - TalosCrash: Found crashes after test run, terminating test
15:12:27 INFO - TEST-INFO took 852874ms
15:12:27 INFO - SUITE-END | took 852s
15:12:28 ERROR - Return code: 2
15:12:28 WARNING - setting return code to 2
15:12:28 ERROR - # TBPL FAILURE #
15:12:28 INFO - Running post-action listener: _package_coverage_data
15:12:28 INFO - Running post-action listener: _resource_record_post_action
15:12:28 INFO - Running post-action listener: process_java_coverage_data
15:12:28 INFO - [mozharness: 2020-02-17 15:12:28.164531Z] Finished run-tests step (success)
15:12:28 INFO - Running post-run listener: _resource_record_post_run
15:12:28 INFO - Total resource usage - Wall time: 853s; CPU: 9.0%; Read bytes: 432537600; Write bytes: 1902424064; Read time: 2714; Write time: 23228
15:12:28 INFO - TinderboxPrint: CPU usage<br/>8.8%
15:12:28 INFO - TinderboxPrint: I/O read bytes / time<br/>432,537,600 / 2,714
15:12:28 INFO - TinderboxPrint: I/O write bytes / time<br/>1,902,424,064 / 23,228
15:12:28 INFO - TinderboxPrint: CPU idle<br/>12,449.4 (91.2%)
15:12:28 INFO - TinderboxPrint: CPU system<br/>188.4 (1.4%)
15:12:28 INFO - TinderboxPrint: CPU user<br/>1,006.3 (7.4%)
15:12:28 INFO - TinderboxPrint: Swap in / out<br/>378,593,280 / 0
15:12:28 INFO - run-tests - Wall time: 854s; CPU: 9.0%; Read bytes: 432537600; Write bytes: 1902424064; Read time: 2714; Write time: 23228
Since I see timings listed at the end, and could see the tests run on my desktop, I know that the tests ran correctly. I don't know what other feedback I can provide outside of what's above.
Flags: needinfo?(dwalsh)
|
|
||
Comment 4•6 years ago
|
||
Could you please download the minidump stackwalk binary from tooltool, and export the MINIDUMP_STACKWALK environment variable with its path on your local disk? That would given us an idea which code caused those crashes, and if that is related to content or not.
Flags: needinfo?(dwalsh)
|
|
Reporter | |
Comment 5•6 years ago
|
||
Hello :whimboo, I've not used ToolTool before -- is there a good guide for doing this?
Flags: needinfo?(dwalsh)
|
|
||
Comment 6•6 years ago
|
||
I actually haven't used tooltool for a while I don't know who to download a recent version of the mindump stackwalk binary. Gabriele, do you know?
Flags: needinfo?(gsvelto)
|
||
Comment 7•6 years ago
|
||
tooltool is not required anymore, we've got a toolchain build now which you can find here. I'm assuming you need the macOS version, the artifact is here.
Flags: needinfo?(gsvelto)
|
|
||
Comment 8•6 years ago
|
||
Oh great!
David, here is the direct link to the download:
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LIvLqbsJQcehagJnkH-1cA/runs/0/artifacts/public/build/minidump_stackwalk.tar.xz
Flags: needinfo?(dwalsh)
|
|
Reporter | |
Comment 9•6 years ago
|
||
Thank you for passing this on :whimboo. I'm still unclear on how to use this though. Is there a guide?
Flags: needinfo?(dwalsh)
|
|
||
Comment 10•6 years ago
|
||
Just download and extract the minidump stackwalk binary on your local disk. Then set the environment variable MINIDUMP_STACKWALK which has to point to the absolute executable path like MINIDUMP_STACKWALK=/Users/henrik/bin/macosx64_minidump_stackwalk.
Then run the above test again, and a full process crash should be reported.
Flags: needinfo?(dwalsh)
|
|
||
Comment 11•6 years ago
|
||
Marking as incomplete due to lack of response. Feel free to reopen if you can still reproduce this.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → INCOMPLETE
|
|
Reporter | |
Updated•6 years ago
|
Flags: needinfo?(dwalsh)
You need to log in
before you can comment on or make changes to this bug.
Description
•