1571216 - Permafailing xpcshell tools/profiler/tests/xpcshell/test_feature_stackwalking.js | xpcshell return code: 0 | Expected native stacks to be interleaved between some frame labels. when Gecko 70 merges to Beta on 2019-08-26

Status: VERIFIED FIXED

(Core :: Gecko Profiler, defect, P1)

Description

[Natalia Csoregi [:nataliaCs]]

[Tracking Requested - why for this release]:
Central as Beta: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&selectedJob=259780822&revision=2116e7173752c044711f7e81e6d574afde735382&searchStr=xpcshell-e10s-2%2Cx%28x2%29

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=259780822&repo=try&lineNumber=4725

[task 2019-08-03T15:36:55.120Z] 15:36:55 INFO - TEST-START | tools/profiler/tests/xpcshell/test_feature_stackwalking.js
[task 2019-08-03T15:36:55.320Z] 15:36:55 WARNING - TEST-UNEXPECTED-FAIL | tools/profiler/tests/xpcshell/test_feature_stackwalking.js | xpcshell return code: 0
[task 2019-08-03T15:36:55.321Z] 15:36:55 INFO - TEST-INFO took 200ms
[task 2019-08-03T15:36:55.321Z] 15:36:55 INFO - >>>>>>>
[task 2019-08-03T15:36:55.321Z] 15:36:55 INFO - (xpcshell/head.js) | test MAIN run_test pending (1)
[task 2019-08-03T15:36:55.321Z] 15:36:55 INFO - (xpcshell/head.js) | test run_next_test 0 pending (2)
[task 2019-08-03T15:36:55.321Z] 15:36:55 INFO - (xpcshell/head.js) | test MAIN run_test finished (2)
[task 2019-08-03T15:36:55.321Z] 15:36:55 INFO - running event loop
[task 2019-08-03T15:36:55.321Z] 15:36:55 INFO - tools/profiler/tests/xpcshell/test_feature_stackwalking.js | Starting
[task 2019-08-03T15:36:55.321Z] 15:36:55 INFO - (xpcshell/head.js) | test pending (2)
[task 2019-08-03T15:36:55.321Z] 15:36:55 INFO - (xpcshell/head.js) | test run_next_test 0 finished (2)
[task 2019-08-03T15:36:55.322Z] 15:36:55 INFO - "CONSOLE_MESSAGE: (info) No chrome package registered for chrome://branding/locale/brand.properties"
[task 2019-08-03T15:36:55.322Z] 15:36:55 INFO - PID 5193 | console.log: "Actual stack: " ["(root)", "xpcshell argument processing", "js::RunScript", "js::RunScript", "XPCWrappedJS method call", "js::RunScript"]
[task 2019-08-03T15:36:55.322Z] 15:36:55 INFO - PID 5193 | console.log: "Expected to contain: " ["(root)", "/^0x[0-9a-f]+$/", "/^0x[0-9a-f]+$/", "js::RunScript", "/^0x[0-9a-f]+$/", "/^0x[0-9a-f]+$/"]
[task 2019-08-03T15:36:55.322Z] 15:36:55 INFO - "Could not find a match for: "/^0x[0-9a-f]+$/""
[task 2019-08-03T15:36:55.322Z] 15:36:55 WARNING - TEST-UNEXPECTED-FAIL | tools/profiler/tests/xpcshell/test_feature_stackwalking.js | - Expected native stacks to be interleaved between some frame labels. There shouldbe more than one native stack if stack walking is working correctly. There is no attempt here to determine if the memory addresses point to the correct symbols - false == true
[task 2019-08-03T15:36:55.322Z] 15:36:55 INFO - /Users/cltbld/tasks/task_1564843075/build/tests/xpcshell/tests/tools/profiler/tests/xpcshell/test_feature_stackwalking.js:null:33
[task 2019-08-03T15:36:55.322Z] 15:36:55 INFO - /Users/cltbld/tasks/task_1564843075/build/tests/xpcshell/head.js:_do_main:245
[task 2019-08-03T15:36:55.322Z] 15:36:55 INFO - /Users/cltbld/tasks/task_1564843075/build/tests/xpcshell/head.js:_execute_test:572
[task 2019-08-03T15:36:55.323Z] 15:36:55 INFO - -e:null:1
[task 2019-08-03T15:36:55.323Z] 15:36:55 INFO - exiting test
[task 2019-08-03T15:36:55.323Z] 15:36:55 INFO - Unexpected exception NS_ERROR_ABORT:
[task 2019-08-03T15:36:55.323Z] 15:36:55 INFO - _abort_failed_test@/Users/cltbld/tasks/task_1564843075/build/tests/xpcshell/head.js:788:20
[task 2019-08-03T15:36:55.323Z] 15:36:55 INFO - do_report_result@/Users/cltbld/tasks/task_1564843075/build/tests/xpcshell/head.js:889:23
[task 2019-08-03T15:36:55.323Z] 15:36:55 INFO - Assert<@/Users/cltbld/tasks/task_1564843075/build/tests/xpcshell/head.js:66:21
[task 2019-08-03T15:36:55.323Z] 15:36:55 INFO - proto.report@resource://testing-common/Assert.jsm:233:10
[task 2019-08-03T15:36:55.323Z] 15:36:55 INFO - proto.ok@resource://testing-common/Assert.jsm:259:10
[task 2019-08-03T15:36:55.323Z] 15:36:55 INFO - expectStackToContain@/Users/cltbld/tasks/task_1564843075/build/tests/xpcshell/tests/tools/profiler/tests/xpcshell/head_profiler.js:143:16
[task 2019-08-03T15:36:55.323Z] 15:36:55 INFO - @/Users/cltbld/tasks/task_1564843075/build/tests/xpcshell/tests/tools/profiler/tests/xpcshell/test_feature_stackwalking.js:33:23
[task 2019-08-03T15:36:55.323Z] 15:36:55 INFO - _do_main@/Users/cltbld/tasks/task_1564843075/build/tests/xpcshell/head.js:245:6
[task 2019-08-03T15:36:55.324Z] 15:36:55 INFO - _execute_test@/Users/cltbld/tasks/task_1564843075/build/tests/xpcshell/head.js:572:13
[task 2019-08-03T15:36:55.324Z] 15:36:55 INFO - @-e:1:14
[task 2019-08-03T15:36:55.324Z] 15:36:55 INFO - exiting test
[task 2019-08-03T15:36:55.324Z] 15:36:55 INFO - PID 5193 | JavaScript error: resource://testing-common/PromiseTestUtils.jsm, line 112: uncaught exception: Object
[task 2019-08-03T15:36:55.324Z] 15:36:55 INFO - <<<<<<<

:gregtatum looks like it is a regression from Bug 1567390. Can you take a look please?
Thank you.

Comment 1

[Natalia Csoregi [:nataliaCs]]

Also fails on: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=259781081&repo=try&lineNumber=4767

[task 2019-08-03T15:42:54.829Z] 15:42:54 WARNING - TEST-UNEXPECTED-FAIL | tools/profiler/tests/xpcshell/test_merged_stacks.js | xpcshell return code: 0

Comment 2

[Greg Tatum [:gregtatum]]

I'm guessing the permafailure is because we don't have frame pointers turned on with OS X on beta. The solution here is to disable the test for that configuration.

Comment 3

[Greg Tatum [:gregtatum]]

Attachment: Bug 1571216 - Disable stack walking test on macOS beta and release; r?canaltinova

Comment 4

[Greg Tatum [:gregtatum]]

Is there a preferred workflow for verifying that my patch fixes the problem? I'm not familiar with the inner workings of cutting a release. I could pull from the beta repo, but then it won't have the dependent patches in nightly. Or are you comfortable with me just landing the disabling without doing my own try run.

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Should test_merged_stacks.js (comment 1) also be disabled? I don't think you need a Try push here. If you want to do it (with mozilla-central):

1. Run the ./mach try release command from https://wiki.mozilla.org/Sheriffing/How_To/Beta_simulations#TRUNK_AS_EARLY_BETA and append --no-push
2. Commit the changes.
3. Pick macOS and xpcshell from ./mach try chooser

Comment 6

[Greg Tatum [:gregtatum]]

Thanks for the help everyone, I also included the test_merged_stacks.js which I had originally missed. I did not push to try, since this is only disabling tests.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/eaae932bdaef
Disable stack walking test on macOS beta and release; r=canaltinova

Comment 8

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/eaae932bdaef

Comment 9

[Alexandru Michis [:malexandru]]

After the disable, the failures have moved from OS X to Linux64 and have changed chunks from X2 to X5: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&selectedJob=260930996&revision=a13fcfac8049db22f62e60c2d7363632df9677df&searchStr=%28X5

Comment 10

[Greg Tatum [:gregtatum]]

I think we're permanently failing to stackwalk through JavaScript on Linux 64. Here is Dev Edition. I'm not as familiar with what's a frame label on Linux. I'll work on confirming.

https://perfht.ml/2Tpp3Oe

Comment 11

[Greg Tatum [:gregtatum]]

Looking further into the profile, I found some successful stackwalking through baseline and interpreter: https://perfht.ml/2Tp1zJh

Focus on node shows native stacks: https://perfht.ml/2TqYlEQ
JS only view shows no ion code: https://perfht.ml/2Tp1zJh

Comment 12

[Greg Tatum [:gregtatum]]

Also, this failure appears to be Bug 1434402, where we can't stackwalk through ion.

Comment 13

[Greg Tatum [:gregtatum]]

Attachment: Bug 1571216 - Disable stackwalking tests on optimized linux builds; r?canaltinova

Comment 14

[Greg Tatum [:gregtatum]]

After talking this over with Gerald. I think we are fine disabling this on any platform that is permafailing. I think we have the known issues on file. I want to keep it on where we can to keep regressions, but we know that our native stackwalking is not great on every platform. I did more investigations yesterday, and most of the remaining issues seem to be Bug 1434402.

Comment 15

[Greg Tatum [:gregtatum]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7d41fdb35e090b36c3954c1bfbd36b58f504a247

Comment 16

[Intermittent Failures Robot]

1 failures in 4839 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• linux64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1571216&startday=2019-08-12&endday=2019-08-18&tree=all

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bdf0e0445b4f
Disable stackwalking tests on optimized linux builds; r=canaltinova

Comment 18

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/bdf0e0445b4f

Comment 19

[Alexandru Michis [:malexandru]]

Verified in today's central as beta simulation -
Log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=262461074&repo=try&lineNumber=2694

Push: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&revision=716be811e619f562523e2df550f33f4063610f69&searchStr=%28X&group_state=expanded