Intermittent webgl talos TypeError: list indices must be integers, not float followed by task timeout
Categories
(Testing :: Talos, defect, P5)
Tracking
(Not tracked)
People
(Reporter: intermittent-bug-filer, Assigned: gbrown)
References
(Regression)
Details
(Keywords: intermittent-failure)
Filed by: btara [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=325654832&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Sy6mum_XQS6T5PcP-lkXKA/runs/0/artifacts/public/logs/live_backing.log
[task 2021-01-02T03:01:54.233Z] 03:01:54 INFO - TEST-INFO | started process 4764 (C:\Users\task_1609544445\build\application\firefox\firefox -wait-for-browser -no-deelevate -profile c:\users\task_1609544445\appdata\local\temp\tmpuefvtq\profile)
[task 2021-01-02T03:01:59.261Z] 03:01:59 INFO - PID 4764 | 0.WebGL-terrain-alpha-no-AA-no= Average: 1.86 stddev: 1.3 (71.5%)
[task 2021-01-02T03:01:59.261Z] 03:01:59 INFO - PID 4764 | Intervals: 1.4 [_6.2_] 1.5 1.2 1.5 2.4 0.9 1.1 1.4 1.8 [_5.2_] 1.5 1.2 1.8 0.9 1.9 1.1 0.9 2.2 0.9 [_4.5_] 1.0 1.6 1.2 1.5 1.0 1.5 1.7 [_6.4_] 1.1 1.6 1.1 2.0 2.0 1.1 1.4 1.4 [_5.2_] 1.0 1.5 2.4 0.9 1.5 1.4 1.0 1.7 [_5.7_] 1.0 2.0 0.9 1.6 1.0 1.7 1.3 1.6 [_4.8_] 1.6 1.2 1.8 1.2 1.2 2.6 0.8 1.5 [_5.4_] 0.9 1.4 2.2 1.0 1.5 1.6 1.7 1.0 [_4.4_] 1.0 1.3 1.2 1.5 1.5 2.0 0.9 1.7 [_5.8_] 1.5 1.1 1.4 1.7 1.6 1.1 1.7 1.0 [_6.1_] 1.0 1.5 1.1 1.6 1.0 2.4 1.6 1.2
[task 2021-01-02T03:01:59.261Z] 03:01:59 INFO - PID 4764 |
[task 2021-01-02T03:01:59.261Z] 03:01:59 INFO - PID 4764 | 1.WebGL-terrain-alpha-no-AA-yes= Average: 2.00 stddev: 1.3 (64.5%)
[task 2021-01-02T03:01:59.261Z] 03:01:59 INFO - PID 4764 | Intervals: 1.4 [_5.2_] 0.9 1.4 1.3 [_5.0_] 0.9 1.3 1.8 [_4.1_] 1.0 1.2 2.1 1.0 3.4 0.9 1.5 2.2 1.1 [_5.4_] 1.1 1.1 1.4 1.6 3.0 1.0 1.4 2.8 1.0 2.9 2.4 0.9 [_5.6_] 1.0 1.0 1.4 3.4 1.3 1.3 2.8 1.0 3.7 1.0 1.8 [_6.1_] 0.9 1.1 1.1 2.1 2.2 1.7 2.3 1.0 1.8 3.0 1.8 1.5 2.5 0.9 [_6.2_] 1.3 1.3 1.1 1.9 2.0 2.3 1.8 2.2 [_5.0_] 1.5 1.6 1.8 1.2 1.5 1.7 2.5 1.0 3.3 1.1 1.5 1.2 2.0 1.1 2.3 1.1 2.1 [_4.4_] 1.5 1.0 1.8 1.1 1.9 1.1 1.6 1.1 [_6.0_] 1.6 1.1 1.7 1.5
[task 2021-01-02T03:01:59.261Z] 03:01:59 INFO - PID 4764 |
[task 2021-01-02T03:01:59.261Z] 03:01:59 INFO - PID 4764 | 2.WebGL-terrain-alpha-yes-AA-no= Average: 1.87 stddev: 1.0 (53.7%)
[task 2021-01-02T03:01:59.262Z] 03:01:59 INFO - PID 4764 | Intervals: 1.4 1.7 1.2 2.2 1.1 [_4.4_] 1.5 2.1 1.2 1.8 1.2 2.2 1.3 1.9 3.7 1.5 1.3 1.6 1.7 1.3 1.9 1.4 1.9 [_4.3_] 1.6 1.2 2.1 1.3 2.1 1.7 1.2 2.1 3.7 1.5 1.8 2.0 1.6 1.3 1.7 1.0 1.7 3.1 1.9 1.1 1.6 1.6 2.0 1.3 1.6 1.2 [_5.0_] 1.3 1.6 1.1 1.5 1.4 2.0 1.2 2.0 3.5 1.7 1.2 1.7 1.1 1.5 1.3 1.5 1.7 [_5.3_] 1.1 1.6 1.2 1.6 1.1 1.6 1.0 1.6 [_6.1_] 1.8 1.1 1.5 1.4 1.7 1.1 2.1 1.2 [_5.1_] 1.4 1.6 1.2 2.0 1.2 2.2 1.1 1.5 [_4.6_] 1.7 2.2 1.2 2.2
[task 2021-01-02T03:01:59.262Z] 03:01:59 INFO - PID 4764 |
[task 2021-01-02T03:01:59.262Z] 03:01:59 INFO - PID 4764 | 3.WebGL-terrain-alpha-yes-AA-yes= Average: 1.98 stddev: 1.0 (51.5%)
[task 2021-01-02T03:01:59.262Z] 03:01:59 INFO - PID 4764 | Intervals: 2.0 2.6 2.2 1.3 1.9 1.5 1.8 1.3 1.9 [_5.0_] 2.1 1.2 1.9 1.8 1.9 1.9 2.3 2.0 1.9 2.1 2.5 1.2 2.0 1.6 1.8 1.4 3.4 1.2 2.5 1.6 1.8 1.7 1.7 2.3 [_5.2_] 0.9 1.6 1.6 1.1 1.7 1.4 3.5 1.0 [_4.1_] 1.1 1.2 1.5 3.2 1.5 1.3 [_5.3_] 0.8 1.7 1.6 1.2 1.7 1.1 1.5 [_4.4_] 1.0 1.7 1.7 1.2 1.6 1.3 1.7 2.1 1.9 [_5.6_] 2.3 1.0 1.8 1.3 2.0 1.3 3.6 0.9 3.9 1.3 1.7 3.6 0.9 1.6 1.6 1.9 2.6 0.9 1.5 1.6 2.1 1.5 3.7 1.1 [_4.1_] 1.6 1.5 2.6 1.2 1.4 1.6
[task 2021-01-02T03:01:59.262Z] 03:01:59 INFO - PID 4764 |
[task 2021-01-02T03:01:59.262Z] 03:01:59 INFO - PID 4764 | Cycle 1(1): loaded http://127.0.0.1:49897/tests/webgl/benchmarks/terrain/perftest.html (next: http://127.0.0.1:49897/tests/webgl/benchmarks/terrain/perftest.html)
[task 2021-01-02T03:02:03.547Z] 03:02:03 INFO - PID 4764 | 0.WebGL-terrain-alpha-no-AA-no= Average: 1.87 stddev: 0.9 (46.5%)
[task 2021-01-02T03:02:03.547Z] 03:02:03 INFO - PID 4764 | Intervals: 2.0 1.8 1.4 1.3 2.4 [_4.5_] 1.6 1.6 1.5 2.7 1.5 1.9 1.9 1.6 2.9 1.8 1.1 1.5 2.4 0.9 1.8 1.4 1.6 [_4.4_] 1.8 0.9 2.1 1.0 2.0 1.7 1.0 1.6 [_3.9_] 0.8 1.8 1.8 2.5 1.0 1.7 1.1 1.7 [_4.4_] 2.0 0.9 1.8 1.6 1.4 1.5 1.9 2.2 3.5 1.9 1.2 1.7 1.5 1.6 1.7 2.1 1.0 [_4.7_] 1.0 1.8 1.1 1.9 1.3 1.7 1.1 2.0 [_3.9_] 1.3 1.3 1.5 1.1 1.5 2.7 1.6 1.1 [_4.7_] 1.7 1.4 2.3 1.6 1.4 1.7 1.0 1.9 [_4.0_] 2.0 1.2 1.5 1.2 2.4 1.6 1.8 1.5 2.9 1.4 1.4 2.0 2.1
[task 2021-01-02T03:02:03.548Z] 03:02:03 INFO - PID 4764 |
[task 2021-01-02T03:02:03.548Z] 03:02:03 INFO - PID 4764 | 1.WebGL-terrain-alpha-no-AA-yes= Average: 2.14 stddev: 1.2 (55.2%)
[task 2021-01-02T03:02:03.548Z] 03:02:03 INFO - PID 4764 | Intervals: 1.5 [_5.1_] 1.0 1.7 [_4.5_] 1.1 1.4 2.0 [_4.6_] 1.3 1.7 1.9 2.2 2.1 1.5 [_6.0_] 0.9 1.9 2.3 1.6 1.6 2.7 1.5 4.2 1.4 1.7 [_4.9_] 1.3 2.2 1.6 2.5 1.2 2.2 1.0 3.6 1.6 1.6 2.2 2.1 [_6.9_] 1.2 1.3 2.1 2.0 1.7 2.1 2.3 2.0 2.5 2.9 [_4.7_] 1.1 1.8 2.3 1.3 2.4 2.4 2.0 2.0 2.0 2.6 3.8 1.3 3.2 1.6 2.5 1.6 1.3 1.5 1.5 2.3 1.0 3.9 1.0 1.5 2.6 1.1 2.0 1.3 1.4 1.7 1.4 1.7 1.9 1.2 1.5 1.8 2.0 1.6 1.0 [_6.1_] 1.1 1.6 3.2 0.9 2.3 1.3 1.1 2.0 3.3
[task 2021-01-02T03:02:03.548Z] 03:02:03 INFO - PID 4764 |
[task 2021-01-02T03:02:03.548Z] 03:02:03 INFO - PID 4764 | 2.WebGL-terrain-alpha-yes-AA-no= Average: 1.94 stddev: 0.8 (41.1%)
[task 2021-01-02T03:02:03.549Z] 03:02:03 INFO - PID 4764 | Intervals: 2.4 1.4 2.0 1.2 1.6 1.6 1.8 1.4 2.0 2.0 1.8 2.3 1.9 [_4.3_] 1.8 1.3 2.8 1.3 1.8 1.2 2.5 1.0 2.7 1.3 1.6 1.6 1.8 1.8 1.8 2.2 [_4.0_] 1.8 2.0 2.9 1.4 1.7 1.2 1.9 1.3 2.7 1.8 1.4 1.8 3.2 1.1 2.1 1.9 3.6 2.3 1.8 1.8 2.0 1.2 1.8 1.5 1.2 2.5 1.1 1.9 1.1 2.1 2.3 2.2 1.2 [_5.2_] 1.2 1.6 1.9 2.1 1.6 1.4 1.8 1.6 3.7 2.0 1.3 2.7 1.2 1.8 3.1 1.1 1.9 1.8 1.5 1.4 2.3 1.0 2.6 1.1 1.7 [_4.6_] 2.7 1.4 2.4 1.1 1.7 1.2 1.7 1.2 3.6
[task 2021-01-02T03:02:03.549Z] 03:02:03 INFO - PID 4764 |
[task 2021-01-02T03:02:03.549Z] 03:02:03 INFO - PID 4764 | 3.WebGL-terrain-alpha-yes-AA-yes= Average: 2.07 stddev: 0.9 (45.2%)
[task 2021-01-02T03:02:03.549Z] 03:02:03 INFO - PID 4764 | Intervals: 2.3 1.6 2.1 1.8 2.3 1.7 1.2 2.3 [_5.9_] 1.3 2.1 2.2 1.3 1.7 1.8 1.1 4.1 1.1 2.7 3.2 1.2 1.9 1.4 1.8 1.3 [_4.3_] 0.9 [_4.3_] 1.5 1.4 2.4 1.7 2.3 1.4 2.3 2.7 2.2 1.3 1.7 [_4.2_] 1.3 1.6 1.4 3.6 1.2 1.9 3.7 1.9 1.9 1.2 1.5 2.6 2.7 1.6 1.6 1.9 1.2 2.8 1.0 4.1 2.5 2.9 1.9 1.6 1.2 1.7 1.4 [_4.4_] 1.0 2.2 1.7 1.5 2.2 2.7 1.7 1.8 2.3 1.9 1.2 1.9 1.6 2.3 1.3 [_4.9_] 1.2 1.9 1.8 1.9 1.4 2.1 1.5 1.9 3.2 2.0 1.2 1.9 2.7 2.7 1.2 1.8
[task 2021-01-02T03:02:03.549Z] 03:02:03 INFO - PID 4764 |
[task 2021-01-02T03:02:03.550Z] 03:02:03 INFO - PID 4764 | Cycle 1(2): loaded http://127.0.0.1:49897/tests/webgl/benchmarks/terrain/perftest.html (next: http://127.0.0.1:49897/tests/webgl/benchmarks/terrain/perftest.html)
[task 2021-01-02T03:02:04.247Z] 03:02:04 INFO - PID 4764 | __start_tp_report
[task 2021-01-02T03:02:04.247Z] 03:02:04 INFO - PID 4764 | _x_x_mozilla_page_load
[task 2021-01-02T03:02:04.247Z] 03:02:04 INFO - PID 4764 | _x_x_mozilla_page_load_details
[task 2021-01-02T03:02:04.247Z] 03:02:04 INFO - PID 4764 | |i|pagename|runs|
[task 2021-01-02T03:02:04.247Z] 03:02:04 INFO - PID 4764 | |0;0.WebGL-terrain-alpha-no-AA-no;1.8577999999999997;1.8710000000000002
[task 2021-01-02T03:02:04.248Z] 03:02:04 INFO - PID 4764 | |1;1.WebGL-terrain-alpha-no-AA-yes;2.001400000000001;2.140600000000002
[task 2021-01-02T03:02:04.248Z] 03:02:04 INFO - PID 4764 | |2;2.WebGL-terrain-alpha-yes-AA-no;1.867199999999998;1.9376000000000022
[task 2021-01-02T03:02:04.248Z] 03:02:04 INFO - PID 4764 | |3;3.WebGL-terrain-alpha-yes-AA-yes;1.98;2.069200000000001
[task 2021-01-02T03:02:04.248Z] 03:02:04 INFO - PID 4764 | __end_tp_report
[task 2021-01-02T03:02:04.248Z] 03:02:04 INFO - PID 4764 | __start_cc_report
[task 2021-01-02T03:02:04.248Z] 03:02:04 INFO - PID 4764 | _x_x_mozilla_cycle_collect,64.54525253773409
[task 2021-01-02T03:02:04.248Z] 03:02:04 INFO - PID 4764 | __end_cc_report
[task 2021-01-02T03:02:04.248Z] 03:02:04 INFO - PID 4764 | __startTimestamp9021.162030951493__endTimestamp
[task 2021-01-02T03:02:04.249Z] 03:02:04 INFO - PID 4764 |
[task 2021-01-02T03:02:04.249Z] 03:02:04 INFO - PID 4764 | ------- Summary: start -------
[task 2021-01-02T03:02:04.249Z] 03:02:04 INFO - PID 4764 | Number of tests: 4
[task 2021-01-02T03:02:04.249Z] 03:02:04 INFO - PID 4764 |
[task 2021-01-02T03:02:04.249Z] 03:02:04 INFO - PID 4764 | [#0] 0.WebGL-terrain-alpha-no-AA-no Cycles:2 Average:1.86 Median:1.87 stddev:0.01 (0.5%)
[task 2021-01-02T03:02:04.249Z] 03:02:04 INFO - PID 4764 | Values: 1.9 1.9
[task 2021-01-02T03:02:04.249Z] 03:02:04 INFO - PID 4764 |
[task 2021-01-02T03:02:04.249Z] 03:02:04 INFO - PID 4764 | [#1] 1.WebGL-terrain-alpha-no-AA-yes Cycles:2 Average:2.07 Median:2.14 stddev:0.10 (4.6%)
[task 2021-01-02T03:02:04.249Z] 03:02:04 INFO - PID 4764 | Values: 2.0 2.1
[task 2021-01-02T03:02:04.250Z] 03:02:04 INFO - PID 4764 |
[task 2021-01-02T03:02:04.250Z] 03:02:04 INFO - PID 4764 | [#2] 2.WebGL-terrain-alpha-yes-AA-no Cycles:2 Average:1.90 Median:1.94 stddev:0.05 (2.6%)
[task 2021-01-02T03:02:04.250Z] 03:02:04 INFO - PID 4764 | Values: 1.9 1.9
[task 2021-01-02T03:02:04.250Z] 03:02:04 INFO - PID 4764 |
[task 2021-01-02T03:02:04.250Z] 03:02:04 INFO - PID 4764 | [#3] 3.WebGL-terrain-alpha-yes-AA-yes Cycles:2 Average:2.02 Median:2.07 stddev:0.06 (3.0%)
[task 2021-01-02T03:02:04.250Z] 03:02:04 INFO - PID 4764 | Values: 2.0 2.1
[task 2021-01-02T03:02:04.250Z] 03:02:04 INFO - PID 4764 | -------- Summary: end --------
[task 2021-01-02T03:02:04.250Z] 03:02:04 INFO - PID 4764 |
[task 2021-01-02T03:02:04.355Z] 03:02:04 INFO - PID 4764 | [Parent 4000, IPC I/O Parent] WARNING: file /builds/worker/checkouts/gecko/ipc/chromium/src/base/process_util_win.cc:167
[task 2021-01-02T03:02:04.557Z] 03:02:04 INFO - TEST-INFO | 4764: exit 0
[task 2021-01-02T03:02:04.559Z] 03:02:04 INFO - profiler Retrieving symbol zip from https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RSyy4rGHQuW-RzCRj9pxEg/artifacts/public/build/target.crashreporter-symbols.zip...
[task 2021-01-02T03:02:07.244Z] 03:02:07 INFO - Encountered an exception during profile symbolication c:\users\task_1609544445\appdata\local\temp\tmptot0o2\benchmarks_pagecycle_1.profile (cycle 0)
[task 2021-01-02T03:02:07.244Z] 03:02:07 ERROR - Traceback (most recent call last):
[task 2021-01-02T03:02:07.244Z] 03:02:07 INFO - File "C:\Users\task_1609544445\build\tests\talos\talos\gecko_profile.py", line 106, in _save_gecko_profile
[task 2021-01-02T03:02:07.244Z] 03:02:07 INFO - symbolicator.symbolicate_profile(profile)
[task 2021-01-02T03:02:07.244Z] 03:02:07 INFO - File "C:\Users\task_1609544445\build\venv\lib\site-packages\mozgeckoprofiler\symbolication.py", line 286, in symbolicate_profile
[task 2021-01-02T03:02:07.244Z] 03:02:07 INFO - addresses, shared_libraries
[task 2021-01-02T03:02:07.245Z] 03:02:07 INFO - File "C:\Users\task_1609544445\build\venv\lib\site-packages\mozgeckoprofiler\symbolication.py", line 328, in _assign_symbols_to_libraries
[task 2021-01-02T03:02:07.245Z] 03:02:07 INFO - lib = self._get_containing_library(int(address, 0), shared_libraries)
[task 2021-01-02T03:02:07.245Z] 03:02:07 INFO - File "C:\Users\task_1609544445\build\venv\lib\site-packages\mozgeckoprofiler\symbolication.py", line 317, in _get_containing_library
[task 2021-01-02T03:02:07.245Z] 03:02:07 INFO - if address >= libs[mid]["end"]:
[task 2021-01-02T03:02:07.245Z] 03:02:07 ERROR - TypeError: list indices must be integers, not float
[task 2021-01-02T03:02:07.255Z] 03:02:07 INFO - Adding profile profile_glterrain\benchmarks_pagecycle_1\cycle_0.profile to archive C:\Users\task_1609544445\build\blobber_upload_dir\profile_glterrain.zip
[task 2021-01-02T03:02:07.455Z] 03:02:07 INFO - Encountered an exception during profile symbolication c:\users\task_1609544445\appdata\local\temp\tmptot0o2\benchmarks_pagecycle_2.profile (cycle 0)
[task 2021-01-02T03:02:07.455Z] 03:02:07 ERROR - Traceback (most recent call last):
[task 2021-01-02T03:02:07.455Z] 03:02:07 INFO - File "C:\Users\task_1609544445\build\tests\talos\talos\gecko_profile.py", line 106, in _save_gecko_profile
[task 2021-01-02T03:02:07.455Z] 03:02:07 INFO - symbolicator.symbolicate_profile(profile)
[task 2021-01-02T03:02:07.455Z] 03:02:07 INFO - File "C:\Users\task_1609544445\build\venv\lib\site-packages\mozgeckoprofiler\symbolication.py", line 286, in symbolicate_profile
[task 2021-01-02T03:02:07.455Z] 03:02:07 INFO - addresses, shared_libraries
[task 2021-01-02T03:02:07.455Z] 03:02:07 INFO - File "C:\Users\task_1609544445\build\venv\lib\site-packages\mozgeckoprofiler\symbolication.py", line 328, in _assign_symbols_to_libraries
[task 2021-01-02T03:02:07.456Z] 03:02:07 INFO - lib = self._get_containing_library(int(address, 0), shared_libraries)
[task 2021-01-02T03:02:07.456Z] 03:02:07 INFO - File "C:\Users\task_1609544445\build\venv\lib\site-packages\mozgeckoprofiler\symbolication.py", line 317, in _get_containing_library
[task 2021-01-02T03:02:07.456Z] 03:02:07 INFO - if address >= libs[mid]["end"]:
[task 2021-01-02T03:02:07.456Z] 03:02:07 ERROR - TypeError: list indices must be integers, not float
[task 2021-01-02T03:02:07.461Z] 03:02:07 INFO - Adding profile profile_glterrain\benchmarks_pagecycle_2\cycle_0.profile to archive C:\Users\task_1609544445\build\blobber_upload_dir\profile_glterrain.zip
[task 2021-01-02T03:02:07.724Z] 03:02:07 INFO - mozcrash checking c:\users\task_1609544445\appdata\local\temp\tmpuefvtq\profile\minidumps for minidumps...
[task 2021-01-02T03:02:07.724Z] 03:02:07 INFO - TEST-OK | glterrain | took 17085ms```
Updated•3 years ago
|
Comment hidden (Intermittent Failures Robot) |
Updated•3 years ago
|
Comment hidden (Intermittent Failures Robot) |
Comment 3•3 years ago
|
||
This failure has started appearing with the backout of Bug 1649599:
https://treeherder.mozilla.org/jobs?repo=autoland&searchStr=windows%2C10%2Cx64%2Cwebrender%2Cshippable%2Copt%2Ctalos%2Cperformance%2Ctests%2Cwith%2Cwebgl%2Cover%2Cipc%2Cand%2Cgecko%2Cprofiling%2Ctest-windows10-64-shippable-qr%2Fopt-talos-webgl-profiling-gli-e10s%2Cwebgl&tochange=89a982fb950bbf31b1848d1ce20fa9c89b430349&fromchange=0e66f49f121db45362ee9197ef1d5cf1afa613ce&group_state=expanded&selectedTaskRun=Ke-HHU_9RtSzzClfP9bC5Q.0
The successful jobs below the backout also have this same failure line, but the job does not get reported as a failure and the run time is significantly lower (2min vs 30min).
sparky, could you please take a look at what might cause this behavior?
Comment 4•3 years ago
|
||
Michelle, would you be able to look into this?
Comment 5•3 years ago
|
||
Hmm, odd.
I worked on this patch together with emalysz, will ask her for her input.
Comment 6•3 years ago
•
|
||
We backed out the patch from Bug 1649599 and reverted it back to its original state, so since we haven't changed SessionFile yet, I'm not sure who should take a closer look at this. Mike, do you have any suggestions?
Updated•3 years ago
|
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Updated•3 years ago
|
Description
•