1563997 - Perma remoteautomation.py | application timed out after 370 seconds with no output when Gecko when Gecko 69 merges to Beta on 2019-07-01

Status: VERIFIED FIXED

(Firefox for Android Graveyard :: Testing, defect, P1)

Description

[Andreea Pavel [:apavel]]

Central as Beta simulation

Treeherder link: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=ca012cf6a330dfd906dcae6078c2b45e5dfbcbec&selectedJob=255192559

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=255192559&repo=try

[task 2019-07-08T02:10:48.237Z] 02:10:48 INFO - TEST-START | dom/imptests/html/webgl/test_uniformMatrixNfv.html
[task 2019-07-08T02:10:48.237Z] 02:10:48 INFO - TEST-SKIP | dom/imptests/html/webgl/test_uniformMatrixNfv.html | took 0ms
[task 2019-07-08T02:10:48.237Z] 02:10:48 INFO - Running manifest: dom/bindings/test/mochitest.ini
[task 2019-07-08T02:10:48.587Z] 02:10:48 INFO - pk12util: PKCS12 IMPORT SUCCESSFUL
[task 2019-07-08T02:10:48.640Z] 02:10:48 INFO - MochitestServer : launching [u'/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64/xpcshell', '-g', '/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64', '-f', '/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpTcv9cq.mozrunner'; const _SERVER_PORT = '8854'; const _SERVER_ADDR = '10.0.2.2'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', '/builds/worker/workspace/build/tests/mochitest/server.js']
[task 2019-07-08T02:10:48.640Z] 02:10:48 INFO - runtests.py | Server pid: 1195
[task 2019-07-08T02:10:48.642Z] 02:10:48 INFO - runtests.py | Websocket server pid: 1198
[task 2019-07-08T02:10:48.646Z] 02:10:48 INFO - runtests.py | SSL tunnel pid: 1201
[task 2019-07-08T02:10:48.959Z] 02:10:48 INFO - runtests.py | Running with scheme: http
[task 2019-07-08T02:10:48.960Z] 02:10:48 INFO - runtests.py | Running with e10s: True
[task 2019-07-08T02:10:48.960Z] 02:10:48 INFO - runtests.py | Running with serviceworker_e10s: False
[task 2019-07-08T02:10:48.960Z] 02:10:48 INFO - runtests.py | Running with socketprocess_e10s: False
[task 2019-07-08T02:10:48.960Z] 02:10:48 INFO - runtests.py | Running tests: start.
[task 2019-07-08T02:10:49.268Z] 02:10:49 INFO - adb Granting important runtime permissions to org.mozilla.geckoview.test
[task 2019-07-08T02:10:50.483Z] 02:10:50 INFO - adb launch_application: am start -W -n org.mozilla.geckoview.test/org.mozilla.geckoview.test.TestRunnerActivity -a android.intent.action.MAIN --es env9 MOZ_UPLOAD_DIR=/sdcard/tests/mozlog --es env8 R_LOG_DESTINATION=stderr --es args "-no-remote -profile /sdcard/tests/profile//" --es env3 R_LOG_VERBOSE=1 --es env2 XPCOM_DEBUG_BREAK=stack --es env1 MOZ_WEBRENDER=0 --es env0 MOZ_CRASHREPORTER=1 --es env7 MOZ_CRASHREPORTER_SHUTDOWN=1 --es env6 MOZ_IN_AUTOMATION=1 --es env5 MOZ_DISABLE_NONLOCAL_CONNECTIONS=1 --es env4 DISABLE_UNSAFE_CPOW_WARNINGS=1 --ez use_multiprocess True --es env12 MOZ_HIDE_RESULTS_TABLE=1 --es env11 R_LOG_LEVEL=6 --es env10 MOZ_CRASHREPORTER_NO_REPORT=1 -d "http://mochi.test:8888/tests?autorun=1&closeWhenDone=1&logFile=%2Fsdcard%2Ftests%2Flogs%2Fmochitest.log&fileLevel=INFO&consoleLevel=INFO&hideResultsTable=1&manifestFile=tests.json&dumpOutputDirectory=%2Fsdcard%2Ftests"
[task 2019-07-08T02:10:51.492Z] 02:10:51 INFO - remoteautomation.py | Application pid: 2780
[task 2019-07-08T02:17:19.608Z] 02:17:19 INFO - wait for org.mozilla.geckoview.test complete; top activity=org.mozilla.geckoview.test
[task 2019-07-08T02:17:19.711Z] 02:17:19 INFO - org.mozilla.geckoview.test unexpectedly found running. Killing...
[task 2019-07-08T02:17:19.711Z] 02:17:19 INFO - TEST-INFO | started process screentopng
[task 2019-07-08T02:17:20.009Z] 02:17:20 INFO - TEST-INFO | screentopng: exit 0
[task 2019-07-08T02:17:34.348Z] 02:17:34 WARNING - TEST-UNEXPECTED-FAIL | remoteautomation.py | application timed out after 370 seconds with no output
[task 2019-07-08T02:17:34.348Z] 02:17:34 INFO - remoteautomation.py | Application ran for: 0:06:45.285699
[task 2019-07-08T02:17:34.761Z] 02:17:34 INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/XKyaRAQJRDSOSe-HPWMFpg/artifacts/public/build/target.crashreporter-symbols.zip
[task 2019-07-08T02:17:37.695Z] 02:17:37 INFO - mozcrash Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmpIP8WHM/65423ca3-1438-4e41-6758-ac2c0934f301.dmp /tmp/tmpR3CbKq
[task 2019-07-08T02:17:41.702Z] 02:17:41 INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/65423ca3-1438-4e41-6758-ac2c0934f301.dmp
[task 2019-07-08T02:17:41.702Z] 02:17:41 INFO - mozcrash Saved app info as /builds/worker/workspace/build/blobber_upload_dir/65423ca3-1438-4e41-6758-ac2c0934f301.extra
[task 2019-07-08T02:17:41.705Z] 02:17:41 WARNING - PROCESS-CRASH | remoteautomation.py | application crashed [@ libc.so + 0x8c66a]

This is failing mochitest Android 8.0 Pixel2 debug and pgo; Android 7.0 x86-64 debug and opt

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Push log: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=6a2bd09a6bf9aada14581e923408d7308479b76d&tochange=1dd4e16a1a9633c661abd2da224d1046259513e0

Kats, could this be from bug 1563264?

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Sorry, meant bug 1525314.

Comment 3

[Kartikaya Gupta (email:[e-mail])]

Unlikely to be bug 1525314, I think. The android-qr jobs are only enabled in central and the patches shouldn't affect the non-qr jobs.

Comment 4

[Kartikaya Gupta (email:[e-mail])]

The logcat has this which seems related:

GeckoDump: TEST-UNEXPECTED-FAIL: manifestLibrary.js | error parsing http://mochi.test:8888/tests.json (TypeError: RunSet is undefined)

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Thanks, let's see if it gets fixed by https://hg.mozilla.org/mozilla-central/rev/7e6657f88b7694ecd841088963ff71d767e4ec22

Comment 6

[Ryan VanderMeulen [:RyanVM]]

This blocks our ability to ship anything mobile off Beta69.

Comment 7

[Andreea Pavel [:apavel]]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #5)

Thanks, let's see if it gets fixed by https://hg.mozilla.org/mozilla-central/rev/7e6657f88b7694ecd841088963ff71d767e4ec22

Does not look like that worked: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&revision=3ad352c33f8a556ac2679445a1985e2f41e5efa3&selectedJob=255472570

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to Andreea Pavel [:apavel] from comment #7)

Does not look like that worked: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&revision=3ad352c33f8a556ac2679445a1985e2f41e5efa3&selectedJob=255472570
That got reverted.

A beta simulation with the SpecialPowers changes backed out has no issues during the execution of the Android 7.0 gpu job: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8683d4eaf7665661730be0aae1bc8f9f7ddeb767

While the same revision without the backout still has this issue: https://treeherder.mozilla.org/#/jobs?repo=try&revision=811b38e227002d0aac15f9f3662be3530837326b

Comment 9

[Andreea Pavel [:apavel]]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #8)

(In reply to Andreea Pavel [:apavel] from comment #7)

Does not look like that worked: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&revision=3ad352c33f8a556ac2679445a1985e2f41e5efa3&selectedJob=255472570
That got reverted.

A beta simulation with the SpecialPowers changes backed out has no issues during the execution of the Android 7.0 gpu job: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8683d4eaf7665661730be0aae1bc8f9f7ddeb767

While the same revision without the backout still has this issue: https://treeherder.mozilla.org/#/jobs?repo=try&revision=811b38e227002d0aac15f9f3662be3530837326b

Saw it now, i just noticed it's a central link so I assumed the changes already are included in the beta sims.

Comment 10

[Chris Peterson [:cpeterson]]

Adding [geckoview:fenix:m7] whiteboard tag because RyanVM says we should fix this bug before Fenix updates from GV 69.

Comment 11

[Nick Alexander :nalexander [he/him]]

I do not know much about the real issue, but this looks very similar to https://bugzilla.mozilla.org/show_bug.cgi?id=1451484.

Comment 12

[James Willcox (:snorp) ([e-mail]) (he/him)]

Geoff, can you help out here? Somehow mochitest is going wrong on Beta. It looks SpecialPowers related? comment #4 has the relevant error.

Comment 13

[Kris Maglione [:kmag]]

My best guess is that the page is somehow being unloaded or reloaded before the manifest callback is called, since that's only obvious way I can think of for the RunTest global to wind up null here.

I'm still looking into it, but I'm probably going to need to get my new test device rooted so I can try to reproduce locally before I make much progress. The turnaround time on try is way too long.

I also have no idea why this would only happen in beta builds and not in central.

Comment 14

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Kris Maglione [:kmag] from comment #13)

My best guess is that the page is somehow being unloaded or reloaded before the manifest callback is called, since that's only obvious way I can think of for the RunTest global to wind up null here.

I'm still looking into it, but I'm probably going to need to get my new test device rooted so I can try to reproduce locally before I make much progress. The turnaround time on try is way too long.

You shouldn't need to root any device. It should be possible to reproduce using the same emulator that automation is using with mach android-emulator --version=x86-7.0

Comment 15

[Kris Maglione [:kmag]]

(In reply to James Willcox (:snorp) ([e-mail]) (he/him) from comment #14)

You shouldn't need to root any device. It should be possible to reproduce using the same emulator that automation is using with mach android-emulator --version=x86-7.0

I've never been able to get tests to work on an emulator.

Comment 16

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Kris Maglione [:kmag] from comment #15)

(In reply to James Willcox (:snorp) ([e-mail]) (he/him) from comment #14)

You shouldn't need to root any device. It should be possible to reproduce using the same emulator that automation is using with mach android-emulator --version=x86-7.0

I've never been able to get tests to work on an emulator.

People can and do regularly run mochitest against a local emulator. Please find us on Slack/IRC if it's not working for you.

Comment 17

[Kris Maglione [:kmag]]

OK, so there are apparently two problems here:

1. We disable sending console logging to logcat on beta and release, which means the only error that shows up in this log is the one that is explicitly caught and dumped in manifestLibrary.js.

2. We enable the fake, web-exposed Components object on beta and release, which breaks the fallback to SpecialPowers that we need when running on Android.

For #1, we really should set the "consoleservice.logcat" at least when running tests, or people are going to wind up (again) wasting a lot of time tracking down simple errors.

For #2, I'll just adjust the check to handle the fake Components object.

Comment 18

[Kris Maglione [:kmag]]

Attachment: Bug 1563997: Handle webcompat Components stub in MozillaFileLogger.js. r=nika

Comment 19

[Bogdan Tara[:bogdan_tara | bogdant]]

The patch seems to work:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7c891a7092619e9f2bf3abb68b4542062420e0d4&searchStr=android%2C7.0%2Cx86-64%2Copt%2Cmochitests%2Ctest-android-em-7.0-x86_64%2Fopt-mochitest-gpu-e10s%2Cm%28gpu%29

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5999f810cba92a3a56cf034c9007a73804801e5a&searchStr=android%2C7.0%2Cx86-64%2Copt%2Cmochitests%2Ctest-android-em-7.0-x86_64%2Fopt-mochitest-gpu-e10s%2Cm%28gpu%29

Comment 20

[Kris Maglione [:kmag]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/9eec8911e1825433064c122838401e473e298f52
Bug 1563997: Handle webcompat Components stub in MozillaFileLogger.js. r=nika

Comment 21

[Kris Maglione [:kmag]]

https://hg.mozilla.org/releases/mozilla-beta/rev/e0a81e0687285f20853b5a5494d0c34ae5acdf1b

Comment 22

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/9eec8911e182

Comment 23

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

This bug has been identified as part of a pilot on determining root causes of blocking and dot release drivers.

It needs a root-cause set for it. Please see the list at https://docs.google.com/document/d/1FFEGsmoU8T0N8R9kk-MXWptOPtXXXRRIe4vQo3_HgMw/.

Add the root cause as a whiteboard tag in the form [rca - <cause> ] and remove the rca-needed keyword.

If you have questions, please contact :tmaity.

Comment 24

[Geoff Brown [:gbrown]]

:kmag -- Can you help with comment 23?

Comment 25

[Sorina Florean [:sflorean]]

From a manual testing point of view, we consider that [rca-testing error] whiteboard should stand for out of scope testing, due to the fact that the issue is not applicable for manual testing.