Closed Bug 1568092 Opened 7 years ago Closed 6 years ago

Intermittent Android tp6 IndexError: list index out of range

Categories

(Testing :: Raptor, defect, P2)

Product:
Testing
Component:
Raptor
Version:
Version 3
Type:
defect

Tracking

(firefox-esr68 unaffected, firefox72 wontfix, firefox73 wontfix)

Status:
RESOLVED INCOMPLETE
Milestone:
mozilla74
Tracking Flags:
Tracking Status
firefox-esr68 --- unaffected
firefox72 --- wontfix
firefox73 --- wontfix

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: intermittent-failure, regression)

Attachments

(1 file, 1 obsolete file)

Bug 1568092 - don't ignore tooltool_download() exit code and add a retry
47 bytes, text/x-phabricator-request
Details | Review

Filed by: rmaries [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=257778332&repo=mozilla-central
Full log: https://queue.taskcluster.net/v1/task/GW4_Q1vcS829dwfn_GUPBg/runs/0/artifacts/public/logs/live_backing.log

23:41:16 INFO - raptor-control-server Info: shutting down control server
23:41:16 INFO - raptor-main Info: finished
23:41:16 ERROR - Traceback (most recent call last):
23:41:16 INFO - File "/builds/task_1563838790/workspace/build/tests/raptor/raptor/raptor.py", line 1385, in <module>
23:41:16 INFO - main()
23:41:16 INFO - File "/builds/task_1563838790/workspace/build/tests/raptor/raptor/raptor.py", line 1353, in main
23:41:16 INFO - success = raptor.run_tests(raptor_test_list, raptor_test_names)
23:41:16 INFO - File "/builds/task_1563838790/workspace/build/tests/raptor/raptor/raptor.py", line 1088, in run_tests
23:41:16 INFO - return super(RaptorAndroid, self).run_tests(tests, test_names)
23:41:16 INFO - File "/builds/task_1563838790/workspace/build/tests/raptor/raptor/raptor.py", line 189, in run_tests
23:41:16 INFO - self.run_test(test, timeout=int(test.get('page_timeout')))
23:41:16 INFO - File "/builds/task_1563838790/workspace/build/tests/raptor/raptor/raptor.py", line 1118, in run_test
23:41:16 INFO - self.__run_test_cold(test, timeout)
23:41:16 INFO - File "/builds/task_1563838790/workspace/build/tests/raptor/raptor/raptor.py", line 1169, in __run_test_cold
23:41:16 INFO - self.start_playback(test)
23:41:16 INFO - File "/builds/task_1563838790/workspace/build/tests/raptor/raptor/raptor.py", line 395, in start_playback
23:41:16 INFO - self.playback.start()
23:41:16 INFO - File "/builds/task_1563838790/workspace/build/venv/lib/python2.7/site-packages/mozproxy/backends/mitm.py", line 125, in start
23:41:16 INFO - self.setup()
23:41:16 INFO - File "/builds/task_1563838790/workspace/build/venv/lib/python2.7/site-packages/mozproxy/backends/mitm.py", line 384, in setup
23:41:16 INFO - self.install_mitmproxy_cert(self.browser_path)
23:41:16 INFO - File "/builds/task_1563838790/workspace/build/venv/lib/python2.7/site-packages/mozproxy/backends/mitm.py", line 457, in install_mitmproxy_cert
23:41:16 INFO - )[0]
23:41:16 INFO - IndexError: list index out of range
23:41:16 ERROR - Return code: 1

Can we investigate this error and return the proper error.

Flags: needinfo?(marian.raiciof)
Flags: needinfo?(fstrugariu)
Flags: needinfo?(alexandru.ionescu)
Depends on: 1558047
Flags: needinfo?(fstrugariu)
Assignee: nobody → alexandru.ionescu
Flags: needinfo?(alexandru.ionescu)

:rwood, the issue appears to be from mozproxy's utils.py
if could try doing:

try:
   tooltool_download(...)
except Exception:
   raise Exception("failed to download the hostutils from tooltool")

but it's not a really good way of catching an exception. If I do:

if _dest:
   tooltool_download(...)
else:
   raise Exception("failed to download the hostutils from tooltool")

it won't really affect, because the parameters are good, utils.py is failing.

Flags: needinfo?(rwood)

Thanks Alex. Maybe mozproxy's tooltool_download needs to be modified to return any exception to Raptor?

Flags: needinfo?(rwood)
Flags: needinfo?(alexandru.ionescu)

I agree. I asked you because wasn't sure that I should modify it, but looking at annotations, bebe did some updates to the code too. I will modify it to a better form. Thannks!

Flags: needinfo?(alexandru.ionescu)
Flags: needinfo?(marian.raiciof)

The latest failures here that are not on Android are misclassifications and that is a regression from Bug 1583700 https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=Linux%2Cx64%2Casan%2CWeb%2Cplatform%2Ctests%2Ctest-linux64-asan%2Fopt-web-platform-tests-e10s-15%2CW%28wpt15%29&tochange=25b533bff4051e6a8177bbc83eed49ac460e109e&fromchange=ab3acf0f24a4c59de4b25d489c4a0aeaf9720448&selectedJob=272462114
Bug 1590604 tracks those failures.

Summary: Intermittent tp6 IndexError: list index out of range → Intermittent Android tp6 IndexError: list index out of range
Assignee: aionescu → nobody
Assignee: nobody → aionescu
Priority: P5 → P2

This patch will check the exit code of the ProcessHandler() that
calls the tooltool script, and will also retry when it fails.

Attachment #9121529 - Attachment is obsolete: true

one failure is related to the removal of mitm4-motog5-gve-cnn.manifest in https://phabricator.services.mozilla.com/rMOZILLACENTRALf3b31e49295529387aec1b9de7f36fec488b6edb

Pushed by tziade@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/4f49e933eb83 don't ignore tooltool_download() exit code and add a retry r=AlexandruIonescu,Bebe

https://hg.mozilla.org/mozilla-central/rev/4f49e933eb83

Status: NEW → RESOLVED
Closed: 6 years ago
status-firefox74: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla74

Reopening it as the intermittent rule is to close the bug if there were no intermittents for 21 days.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

== Change summary for alert #24736 (as of Wed, 22 Jan 2020 05:03:37 GMT) ==

Improvements:

56% raptor-tp6m-cnn-geckoview-cold fcp android-hw-p2-8-0-android-aarch64 pgo 3,199.83 -> 1,403.83
55% raptor-tp6m-cnn-geckoview-cold fcp android-hw-g5-7-0-arm7-api-16 pgo 6,822.21 -> 3,048.83
40% raptor-tp6m-cnn-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 pgo 7,199.21 -> 4,305.17
38% raptor-tp6m-cnn-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 pgo 7,216.58 -> 4,468.25
32% raptor-tp6m-cnn-geckoview-cold android-hw-p2-8-0-android-aarch64 pgo 2,025.68 -> 1,383.26
31% raptor-tp6m-cnn-geckoview-cold android-hw-g5-7-0-arm7-api-16 pgo 3,709.36 -> 2,547.48
30% raptor-tp6m-cnn-geckoview-cold loadtime android-hw-p2-8-0-android-aarch64 pgo 4,206.96 -> 2,944.75

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24736

Tarek, is it just me or the tooltool_download didn't retry?

Flags: needinfo?(tarek)

It's normal.This new occurrence of intermittent happened on mozilla-beta. The patch was not backported there.

Flags: needinfo?(tarek)

I think bug 1612279 should cover all cases whether on Linux or Android, so personally I think this bug should not be used for the fallout from bug 1611339.

Flags: needinfo?(egao)
Assignee: aionescu → nobody

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup

Status: REOPENED → RESOLVED
Closed: 6 years ago → 6 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: