Closed
Bug 1623209
Opened 4 years ago
Closed 3 years ago
Intermittent Android Reftest IndexError: list index out of range
Categories
(Testing :: Mozbase, defect, P5)
Tracking
(Not tracked)
RESOLVED
WORKSFORME
People
(Reporter: intermittent-bug-filer, Assigned: gbrown)
Details
(Keywords: intermittent-failure)
Attachments
(1 file)
Filed by: btara [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=293515667&repo=try
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RD6MNUW1SzSe91M8a0L2pw/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RD6MNUW1SzSe91M8a0L2pw/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1
[task 2020-03-17T15:34:24.236Z] 15:34:24 INFO - [mozharness: 2020-03-17 15:34:24.236017Z] Running run-tests step.
[task 2020-03-17T15:34:24.236Z] 15:34:24 INFO - Running pre-action listener: _resource_record_pre_action
[task 2020-03-17T15:34:24.236Z] 15:34:24 INFO - Running pre-action listener: _set_gcov_prefix
[task 2020-03-17T15:34:24.236Z] 15:34:24 INFO - Running pre-action listener: timed_screenshots
[task 2020-03-17T15:34:24.236Z] 15:34:24 INFO - Running main action method: run_tests
[task 2020-03-17T15:34:24.239Z] 15:34:24 INFO - Running the command /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/reftest/remotereftest.py --app=org.mozilla.geckoview.test --ignore-window-size --remote-webserver=10.0.2.2 --xre-path=/builds/worker/workspace/build/hostutils/host-utils-71.0a1.en-US.linux-x86_64 --utility-path=/builds/worker/workspace/build/hostutils/host-utils-71.0a1.en-US.linux-x86_64 --http-port=8854 --ssl-port=4454 --httpd-path /builds/worker/workspace/build/tests/modules --symbols-path=/builds/worker/workspace/build/symbols --extra-profile-file=fonts --extra-profile-file=hyphenation --suite=reftest --log-raw=/builds/worker/workspace/build/blobber_upload_dir/reftest_raw.log --log-raw-level=info --log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/reftest_errorsummary.log --log-tbpl-level=info --deviceSerial=emulator-5554 --topsrcdir=tests --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --setpref=layers.d3d11.enable-blacklist=false --this-chunk 2 --total-chunks 2 --enable-webrender -- tests/layout/reftests/reftest.list
[task 2020-03-17T15:34:24.239Z] 15:34:24 INFO - ##### reftest log begins
[task 2020-03-17T15:34:24.239Z] 15:34:24 INFO - Running command: ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/reftest/remotereftest.py', '--app=org.mozilla.geckoview.test', '--ignore-window-size', '--remote-webserver=10.0.2.2', '--xre-path=/builds/worker/workspace/build/hostutils/host-utils-71.0a1.en-US.linux-x86_64', '--utility-path=/builds/worker/workspace/build/hostutils/host-utils-71.0a1.en-US.linux-x86_64', '--http-port=8854', '--ssl-port=4454', '--httpd-path', '/builds/worker/workspace/build/tests/modules', '--symbols-path=/builds/worker/workspace/build/symbols', '--extra-profile-file=fonts', '--extra-profile-file=hyphenation', '--suite=reftest', '--log-raw=/builds/worker/workspace/build/blobber_upload_dir/reftest_raw.log', '--log-raw-level=info', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/reftest_errorsummary.log', '--log-tbpl-level=info', '--deviceSerial=emulator-5554', '--topsrcdir=tests', '--setpref=media.peerconnection.mtransport_process=false', '--setpref=network.process.enabled=false', '--setpref=layers.d3d11.enable-blacklist=false', '--this-chunk', '2', '--total-chunks', '2', '--enable-webrender', '--', 'tests/layout/reftests/reftest.list'] in /builds/worker/workspace/build/tests/reftest
[task 2020-03-17T15:34:24.239Z] 15:34:24 INFO - Copy/paste: /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/reftest/remotereftest.py --app=org.mozilla.geckoview.test --ignore-window-size --remote-webserver=10.0.2.2 --xre-path=/builds/worker/workspace/build/hostutils/host-utils-71.0a1.en-US.linux-x86_64 --utility-path=/builds/worker/workspace/build/hostutils/host-utils-71.0a1.en-US.linux-x86_64 --http-port=8854 --ssl-port=4454 --httpd-path /builds/worker/workspace/build/tests/modules --symbols-path=/builds/worker/workspace/build/symbols --extra-profile-file=fonts --extra-profile-file=hyphenation --suite=reftest --log-raw=/builds/worker/workspace/build/blobber_upload_dir/reftest_raw.log --log-raw-level=info --log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/reftest_errorsummary.log --log-tbpl-level=info --deviceSerial=emulator-5554 --topsrcdir=tests --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --setpref=layers.d3d11.enable-blacklist=false --this-chunk 2 --total-chunks 2 --enable-webrender -- tests/layout/reftests/reftest.list
[task 2020-03-17T15:34:24.239Z] 15:34:24 INFO - Using env: (same as previous command)
[task 2020-03-17T15:34:24.564Z] 15:34:24 INFO - Using adb 1.0.41
[task 2020-03-17T15:34:24.871Z] 15:34:24 INFO - adbd running as root
[task 2020-03-17T15:34:24.973Z] 15:34:24 INFO - su -c setenforce 0 exitcode 0, stdout: None
[task 2020-03-17T15:34:25.280Z] 15:34:25 INFO - su 0 supported
[task 2020-03-17T15:34:25.488Z] 15:34:25 INFO - /system/bin/ls -1A supported
[task 2020-03-17T15:34:25.595Z] 15:34:25 INFO - Native cp support: True
[task 2020-03-17T15:34:25.698Z] 15:34:25 INFO - Native chmod -R support: True
[task 2020-03-17T15:34:25.801Z] 15:34:25 INFO - Native chown -R support: True
[task 2020-03-17T15:34:25.903Z] 15:34:25 INFO - Setting SELinux Permissive
[task 2020-03-17T15:34:26.110Z] 15:34:26 INFO - Native pidof support: True
[task 2020-03-17T15:34:28.172Z] 15:34:28 INFO - REFTEST INFO | Checking for ssltunnel processes...
[task 2020-03-17T15:34:28.180Z] 15:34:28 INFO - REFTEST INFO | Checking for xpcshell processes...
[task 2020-03-17T15:34:28.189Z] 15:34:28 INFO - INFO | remotereftests.py | Server pid: 1175
[task 2020-03-17T15:34:29.190Z] 15:34:29 INFO - Device info:
[task 2020-03-17T15:34:29.926Z] 15:34:29 INFO - uptime: 15:34:29 up 1 min, 0 users, load average: 0.13, 0.06, 0.02
[task 2020-03-17T15:34:29.926Z] 15:34:29 INFO - battery: 100.0
[task 2020-03-17T15:34:29.926Z] 15:34:29 INFO - process:
[task 2020-03-17T15:34:29.926Z] 15:34:29 INFO - USER PID PPID VSIZE RSS WCHAN PC NAME
[task 2020-03-17T15:34:29.926Z] 15:34:29 INFO - root 1 0 10312 1676 ep_poll 00004fa1ea S /init
[task 2020-03-17T15:34:29.926Z] 15:34:29 INFO - root 2 0 0 0 kthreadd 0000000000 S kthreadd
[task 2020-03-17T15:34:29.926Z] 15:34:29 INFO - root 3 2 0 0 smpboot_th 0000000000 S ksoftirqd/0
[task 2020-03-17T15:34:29.926Z] 15:34:29 INFO - root 4 2 0 0 worker_thr 0000000000 S kworker/0:0
[task 2020-03-17T15:34:29.926Z] 15:34:29 INFO - root 5 2 0 0 worker_thr 0000000000 S kworker/0:0H
[task 2020-03-17T15:34:29.926Z] 15:34:29 INFO - root 6 2 0 0 worker_thr 0000000000 S kworker/u8:0
[task 2020-03-17T15:34:29.926Z] 15:34:29 INFO - root 7 2 0 0 smpboot_th 0000000000 S migration/0
[task 2020-03-17T15:34:29.926Z] 15:34:29 INFO - root 8 2 0 0 rcu_gp_kth 0000000000 S rcu_preempt
[task 2020-03-17T15:34:29.926Z] 15:34:29 INFO - root 9 2 0 0 rcu_gp_kth 0000000000 S rcu_bh
[task 2020-03-17T15:34:29.926Z] 15:34:29 INFO - root 10 2 0 0 rcu_gp_kth 0000000000 S rcu_sched
[task 2020-03-17T15:34:29.927Z] 15:34:29 INFO - root 11 2 0 0 smpboot_th 0000000000 S migration/1
[task 2020-03-17T15:34:29.927Z] 15:34:29 INFO - root 12 2 0 0 smpboot_th 0000000000 S ksoftirqd/1
[task 2020-03-17T15:34:29.927Z] 15:34:29 INFO - root 13 2 0 0 worker_thr 0000000000 S kworker/1:0
[task 2020-03-17T15:34:29.928Z] 15:34:29 INFO - root 14 2 0 0 worker_thr 0000000000 S kworker/1:0H
[task 2020-03-17T15:34:52.406Z] 15:34:52 INFO - Granting important runtime permissions to org.mozilla.geckoview.test
[task 2020-03-17T15:34:53.925Z] 15:34:53 INFO - launch_application: am start -W -n org.mozilla.geckoview.test/org.mozilla.geckoview.test.TestRunnerActivity -a android.intent.action.MAIN --es env9 MOZ_CRASHREPORTER_NO_REPORT=1 --es env8 MOZ_ACCELERATED=1 --es args "-no-remote -profile /sdcard/tests/reftest/profile/" --es env3 R_LOG_VERBOSE=1 --es env2 XPCOM_DEBUG_BREAK=stack --es env1 MOZ_WEBRENDER=1 --es env0 MOZ_CRASHREPORTER=1 --es env7 R_LOG_DESTINATION=stderr --es env6 MOZ_CRASHREPORTER_SHUTDOWN=1 --es env5 MOZ_IN_AUTOMATION=1 --es env4 MOZ_DISABLE_NONLOCAL_CONNECTIONS=1 --ez use_multiprocess True --es env11 R_LOG_LEVEL=6 --es env10 MOZ_DEVELOPER_REPO_DIR=tests
[task 2020-03-17T15:34:54.947Z] 15:34:54 INFO - Automation Error: Exception caught while running tests
[task 2020-03-17T15:34:54.947Z] 15:34:54 INFO - Traceback (most recent call last):
[task 2020-03-17T15:34:54.948Z] 15:34:54 INFO - File "/builds/worker/workspace/build/tests/reftest/remotereftest.py", line 424, in run_test_harness
[task 2020-03-17T15:34:54.949Z] 15:34:54 INFO - retVal = reftest.runTests(options.tests, options)
[task 2020-03-17T15:34:54.950Z] 15:34:54 INFO - File "/builds/worker/workspace/build/tests/reftest/runreftest.py", line 613, in runTests
[task 2020-03-17T15:34:54.950Z] 15:34:54 INFO - return self.runSerialTests(manifests, options, cmdargs)
[task 2020-03-17T15:34:54.951Z] 15:34:54 INFO - File "/builds/worker/workspace/build/tests/reftest/runreftest.py", line 953, in runSerialTests
[task 2020-03-17T15:34:54.952Z] 15:34:54 INFO - return run()
[task 2020-03-17T15:34:54.952Z] 15:34:54 INFO - File "/builds/worker/workspace/build/tests/reftest/runreftest.py", line 944, in run
[task 2020-03-17T15:34:54.953Z] 15:34:54 INFO - **kwargs)
[task 2020-03-17T15:34:54.954Z] 15:34:54 INFO - File "/builds/worker/workspace/build/tests/reftest/remotereftest.py", line 383, in runApp
[task 2020-03-17T15:34:54.954Z] 15:34:54 INFO - e10s=options.e10s)
[task 2020-03-17T15:34:54.954Z] 15:34:54 INFO - File "/builds/worker/workspace/build/tests/reftest/remoteautomation.py", line 67, in runApp
[task 2020-03-17T15:34:54.954Z] 15:34:54 INFO - self.log.info("remoteautomation.py | Application pid: %d", self.pid)
[task 2020-03-17T15:34:54.954Z] 15:34:54 INFO - File "/builds/worker/workspace/build/tests/reftest/remoteautomation.py", line 244, in pid
[task 2020-03-17T15:34:54.954Z] 15:34:54 INFO - procs = self.device.get_process_list()
[task 2020-03-17T15:34:54.955Z] 15:34:54 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 2565, in get_process_list
[task 2020-03-17T15:34:54.998Z] 15:34:54 INFO - ret.append([int(els[pid_i]), els[-1], els[user_i]])
[task 2020-03-17T15:34:54.998Z] 15:34:54 INFO - IndexError: list index out of range
[task 2020-03-17T15:34:55.118Z] 15:34:55 INFO - --------- beginning of main
[task 2020-03-17T15:34:55.118Z] 15:34:55 INFO - 03-17 15:34:27.111 I/log ( 2634): logcat cleared
...
|
||
Comment 2•4 years ago
|
||
Root of the python error is actually in mozdevice. The get_process_list() code is presumably making some assumptions that are invalid
Component: Reftest → Mozbase
Priority: P5 → --
|
||
Comment 3•4 years ago
|
||
gbrown: We don't log any information unless we have a ValueError. I think we should do so for any exception for debugging purposes. I'll work up a patch that we can use to diagnose this.
|
|
||
Comment 4•4 years ago
|
||
|
||
Updated•4 years ago
|
Assignee: nobody → bob
Status: NEW → ASSIGNED
|
|
||
Updated•4 years ago
|
Keywords: leave-open
Pushed by bclary@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/033bea932f79 [mozdevice] Catch and report IndexError when parsing ps output in get_process_list, r=gbrown.
|
||
Comment 6•4 years ago
|
||
| bugherder | ||
| Comment hidden (Intermittent Failures Robot) |
|
|
||
Comment 8•4 years ago
|
||
Nothing to do until we can reproduce and get data.
Assignee: bob → nobody
Status: ASSIGNED → NEW
Priority: -- → P5
|
Assignee | |
Updated•3 years ago
|
Assignee: nobody → whole.grains
Keywords: leave-open
|
|
Assignee | |
Updated•3 years ago
|
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → WORKSFORME
You need to log in
before you can comment on or make changes to this bug.
Description
•