Closed Bug 1563997 Opened 5 months ago Closed 5 months ago

Perma remoteautomation.py | application timed out after 370 seconds with no output when Gecko when Gecko 69 merges to Beta on 2019-07-01

Categories

(Firefox for Android :: Testing, defect, P1, critical)

defect

Tracking

()

VERIFIED FIXED
Firefox 70
Tracking Status
firefox-esr60 --- unaffected
firefox-esr68 --- unaffected
firefox67 --- unaffected
firefox67.0.1 --- unaffected
firefox68 --- unaffected
firefox69 blocking verified
firefox70 --- verified

People

(Reporter: apavel, Assigned: kmag)

References

(Regression)

Details

(Keywords: regression, Whiteboard: [geckoview:fenix:m7])

Attachments

(1 file)

Central as Beta simulation

Treeherder link: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=ca012cf6a330dfd906dcae6078c2b45e5dfbcbec&selectedJob=255192559

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=255192559&repo=try

[task 2019-07-08T02:10:48.237Z] 02:10:48 INFO - TEST-START | dom/imptests/html/webgl/test_uniformMatrixNfv.html
[task 2019-07-08T02:10:48.237Z] 02:10:48 INFO - TEST-SKIP | dom/imptests/html/webgl/test_uniformMatrixNfv.html | took 0ms
[task 2019-07-08T02:10:48.237Z] 02:10:48 INFO - Running manifest: dom/bindings/test/mochitest.ini
[task 2019-07-08T02:10:48.587Z] 02:10:48 INFO - pk12util: PKCS12 IMPORT SUCCESSFUL
[task 2019-07-08T02:10:48.640Z] 02:10:48 INFO - MochitestServer : launching [u'/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64/xpcshell', '-g', '/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64', '-f', '/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpTcv9cq.mozrunner'; const _SERVER_PORT = '8854'; const _SERVER_ADDR = '10.0.2.2'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', '/builds/worker/workspace/build/tests/mochitest/server.js']
[task 2019-07-08T02:10:48.640Z] 02:10:48 INFO - runtests.py | Server pid: 1195
[task 2019-07-08T02:10:48.642Z] 02:10:48 INFO - runtests.py | Websocket server pid: 1198
[task 2019-07-08T02:10:48.646Z] 02:10:48 INFO - runtests.py | SSL tunnel pid: 1201
[task 2019-07-08T02:10:48.959Z] 02:10:48 INFO - runtests.py | Running with scheme: http
[task 2019-07-08T02:10:48.960Z] 02:10:48 INFO - runtests.py | Running with e10s: True
[task 2019-07-08T02:10:48.960Z] 02:10:48 INFO - runtests.py | Running with serviceworker_e10s: False
[task 2019-07-08T02:10:48.960Z] 02:10:48 INFO - runtests.py | Running with socketprocess_e10s: False
[task 2019-07-08T02:10:48.960Z] 02:10:48 INFO - runtests.py | Running tests: start.
[task 2019-07-08T02:10:49.268Z] 02:10:49 INFO - adb Granting important runtime permissions to org.mozilla.geckoview.test
[task 2019-07-08T02:10:50.483Z] 02:10:50 INFO - adb launch_application: am start -W -n org.mozilla.geckoview.test/org.mozilla.geckoview.test.TestRunnerActivity -a android.intent.action.MAIN --es env9 MOZ_UPLOAD_DIR=/sdcard/tests/mozlog --es env8 R_LOG_DESTINATION=stderr --es args "-no-remote -profile /sdcard/tests/profile//" --es env3 R_LOG_VERBOSE=1 --es env2 XPCOM_DEBUG_BREAK=stack --es env1 MOZ_WEBRENDER=0 --es env0 MOZ_CRASHREPORTER=1 --es env7 MOZ_CRASHREPORTER_SHUTDOWN=1 --es env6 MOZ_IN_AUTOMATION=1 --es env5 MOZ_DISABLE_NONLOCAL_CONNECTIONS=1 --es env4 DISABLE_UNSAFE_CPOW_WARNINGS=1 --ez use_multiprocess True --es env12 MOZ_HIDE_RESULTS_TABLE=1 --es env11 R_LOG_LEVEL=6 --es env10 MOZ_CRASHREPORTER_NO_REPORT=1 -d "http://mochi.test:8888/tests?autorun=1&closeWhenDone=1&logFile=%2Fsdcard%2Ftests%2Flogs%2Fmochitest.log&fileLevel=INFO&consoleLevel=INFO&hideResultsTable=1&manifestFile=tests.json&dumpOutputDirectory=%2Fsdcard%2Ftests"
[task 2019-07-08T02:10:51.492Z] 02:10:51 INFO - remoteautomation.py | Application pid: 2780
[task 2019-07-08T02:17:19.608Z] 02:17:19 INFO - wait for org.mozilla.geckoview.test complete; top activity=org.mozilla.geckoview.test
[task 2019-07-08T02:17:19.711Z] 02:17:19 INFO - org.mozilla.geckoview.test unexpectedly found running. Killing...
[task 2019-07-08T02:17:19.711Z] 02:17:19 INFO - TEST-INFO | started process screentopng
[task 2019-07-08T02:17:20.009Z] 02:17:20 INFO - TEST-INFO | screentopng: exit 0
[task 2019-07-08T02:17:34.348Z] 02:17:34 WARNING - TEST-UNEXPECTED-FAIL | remoteautomation.py | application timed out after 370 seconds with no output
[task 2019-07-08T02:17:34.348Z] 02:17:34 INFO - remoteautomation.py | Application ran for: 0:06:45.285699
[task 2019-07-08T02:17:34.761Z] 02:17:34 INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/XKyaRAQJRDSOSe-HPWMFpg/artifacts/public/build/target.crashreporter-symbols.zip
[task 2019-07-08T02:17:37.695Z] 02:17:37 INFO - mozcrash Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmpIP8WHM/65423ca3-1438-4e41-6758-ac2c0934f301.dmp /tmp/tmpR3CbKq
[task 2019-07-08T02:17:41.702Z] 02:17:41 INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/65423ca3-1438-4e41-6758-ac2c0934f301.dmp
[task 2019-07-08T02:17:41.702Z] 02:17:41 INFO - mozcrash Saved app info as /builds/worker/workspace/build/blobber_upload_dir/65423ca3-1438-4e41-6758-ac2c0934f301.extra
[task 2019-07-08T02:17:41.705Z] 02:17:41 WARNING - PROCESS-CRASH | remoteautomation.py | application crashed [@ libc.so + 0x8c66a]

This is failing mochitest Android 8.0 Pixel2 debug and pgo; Android 7.0 x86-64 debug and opt

Unlikely to be bug 1525314, I think. The android-qr jobs are only enabled in central and the patches shouldn't affect the non-qr jobs.

Flags: needinfo?(kats)

The logcat has this which seems related:

GeckoDump: TEST-UNEXPECTED-FAIL: manifestLibrary.js | error parsing http://mochi.test:8888/tests.json (TypeError: RunSet is undefined)

This blocks our ability to ship anything mobile off Beta69.

Severity: normal → critical
Priority: -- → P1
Flags: needinfo?(aryx.bugmail)

(In reply to Andreea Pavel [:apavel] from comment #7)

Does not look like that worked: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&revision=3ad352c33f8a556ac2679445a1985e2f41e5efa3&selectedJob=255472570
That got reverted.

A beta simulation with the SpecialPowers changes backed out has no issues during the execution of the Android 7.0 gpu job: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8683d4eaf7665661730be0aae1bc8f9f7ddeb767

While the same revision without the backout still has this issue: https://treeherder.mozilla.org/#/jobs?repo=try&revision=811b38e227002d0aac15f9f3662be3530837326b

Flags: needinfo?(aryx.bugmail)

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #8)

(In reply to Andreea Pavel [:apavel] from comment #7)

Does not look like that worked: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&revision=3ad352c33f8a556ac2679445a1985e2f41e5efa3&selectedJob=255472570
That got reverted.

A beta simulation with the SpecialPowers changes backed out has no issues during the execution of the Android 7.0 gpu job: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8683d4eaf7665661730be0aae1bc8f9f7ddeb767

While the same revision without the backout still has this issue: https://treeherder.mozilla.org/#/jobs?repo=try&revision=811b38e227002d0aac15f9f3662be3530837326b

Saw it now, i just noticed it's a central link so I assumed the changes already are included in the beta sims.

Blocks: 1564450

Adding [geckoview:fenix:m7] whiteboard tag because RyanVM says we should fix this bug before Fenix updates from GV 69.

Whiteboard: [geckoview:fenix:m7]

I do not know much about the real issue, but this looks very similar to https://bugzilla.mozilla.org/show_bug.cgi?id=1451484.

Geoff, can you help out here? Somehow mochitest is going wrong on Beta. It looks SpecialPowers related? comment #4 has the relevant error.

Flags: needinfo?(gbrown)

My best guess is that the page is somehow being unloaded or reloaded before the manifest callback is called, since that's only obvious way I can think of for the RunTest global to wind up null here.

I'm still looking into it, but I'm probably going to need to get my new test device rooted so I can try to reproduce locally before I make much progress. The turnaround time on try is way too long.

I also have no idea why this would only happen in beta builds and not in central.

Flags: needinfo?(kmaglione+bmo)

(In reply to Kris Maglione [:kmag] from comment #13)

My best guess is that the page is somehow being unloaded or reloaded before the manifest callback is called, since that's only obvious way I can think of for the RunTest global to wind up null here.

I'm still looking into it, but I'm probably going to need to get my new test device rooted so I can try to reproduce locally before I make much progress. The turnaround time on try is way too long.

You shouldn't need to root any device. It should be possible to reproduce using the same emulator that automation is using with mach android-emulator --version=x86-7.0

(In reply to James Willcox (:snorp) (jwillcox@mozilla.com) (he/him) from comment #14)

You shouldn't need to root any device. It should be possible to reproduce using the same emulator that automation is using with mach android-emulator --version=x86-7.0

I've never been able to get tests to work on an emulator.

(In reply to Kris Maglione [:kmag] from comment #15)

(In reply to James Willcox (:snorp) (jwillcox@mozilla.com) (he/him) from comment #14)

You shouldn't need to root any device. It should be possible to reproduce using the same emulator that automation is using with mach android-emulator --version=x86-7.0

I've never been able to get tests to work on an emulator.

People can and do regularly run mochitest against a local emulator. Please find us on Slack/IRC if it's not working for you.

OK, so there are apparently two problems here:

  1. We disable sending console logging to logcat on beta and release, which means the only error that shows up in this log is the one that is explicitly caught and dumped in manifestLibrary.js.

  2. We enable the fake, web-exposed Components object on beta and release, which breaks the fallback to SpecialPowers that we need when running on Android.

For #1, we really should set the "consoleservice.logcat" at least when running tests, or people are going to wind up (again) wasting a lot of time tracking down simple errors.

For #2, I'll just adjust the check to handle the fake Components object.

Assignee: nobody → kmaglione+bmo
Flags: needinfo?(gbrown)
Depends on: 1565699
https://hg.mozilla.org/integration/mozilla-inbound/rev/9eec8911e1825433064c122838401e473e298f52
Bug 1563997: Handle webcompat Components stub in MozillaFileLogger.js. r=nika
Status: NEW → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 70
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.