1921352 - structuredlog.py exception with "IndexError: list index out of range" in _handle_log

Status: RESOLVED FIXED

(Testing :: Mozbase, defect, P2)

Description

[Daniel Holbert [:dholbert]]

This Try log (posted on bug 1904552 comment 6)...
https://treeherder.mozilla.org/logviewer?job_id=475901417&repo=try&lineNumber=9515
...seems to have hit some sort of edge case and thrown an exception in _handle_log. I'm not sure precisely what happened, but maybe it can be inferred based on contextual lines in the log.

Here's the relevant snippet of the log (with leading [task...] stripped out for brevity):

13:33:16     INFO - REFTEST TEST-LOAD | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/pixel-rounding/check-image.html | 0 / 163 (0%)
13:33:17    ERROR - A content process crashed and MOZ_CRASHREPORTER_SHUTDOWN is set, shutting down
13:33:17     INFO - 1727357597956	Marionette	TRACE	Received observer notification quit-application
13:33:17     INFO - 1727357597957	Marionette	INFO	Stopped listening on port 2828
13:33:17     INFO - 1727357597957	Marionette	DEBUG	Marionette stopped listening
13:33:18     INFO - [Parent 9172, IPC I/O Parent] WARNING: process 9470 exited on signal 11: file /builds/worker/checkouts/gecko/ipc/chromium/src/base/process_util_posix.cc:335
13:33:18     INFO - !!! error running onStopped callback: TypeError: callback is not a function
13:33:19     INFO - REFTEST TEST-UNEXPECTED-FAIL | layout/reftests/pixel-rounding/check-image.html | application terminated with exit code 245
13:33:19     INFO -  mozlog.structuredlog: Failure calling log handler:
13:33:19     INFO -  Traceback (most recent call last):
13:33:19     INFO -    File "/builds/worker/workspace/build/venv/lib/python3.8/site-packages/mozlog/structuredlog.py", line 327, in _handle_log
13:33:19     INFO -      handler(data)
13:33:19     INFO -    File "/builds/worker/workspace/build/venv/lib/python3.8/site-packages/mozlog/handlers/base.py", line 74, in __call__
13:33:19     INFO -      formatted = self.formatter(data)
13:33:19     INFO -    File "/builds/worker/workspace/build/venv/lib/python3.8/site-packages/mozlog/handlers/base.py", line 42, in __call__
13:33:19     INFO -      return self.inner(item)
13:33:19     INFO -    File "/builds/worker/workspace/build/venv/lib/python3.8/site-packages/mozlog/formatters/errorsummary.py", line 31, in __call__
13:33:19     INFO -      rv = BaseFormatter.__call__(self, data)
13:33:19     INFO -    File "/builds/worker/workspace/build/venv/lib/python3.8/site-packages/mozlog/reader.py", line 72, in __call__
13:33:19     INFO -      return handler(data)
13:33:19     INFO -    File "/builds/worker/workspace/build/venv/lib/python3.8/site-packages/mozlog/formatters/errorsummary.py", line 161, in shutdown_failure
13:33:19     INFO -      data["group"] = [g for g in self.groups if item["group"].endswith(g)][0]
13:33:19     INFO -  IndexError: list index out of range

We might want to make structuredlog.py handle whatever-this-edge-case-was with a clearer error message rather than throwing an exception. I don't know right now if this is something we hit often vs. if it was just a one-off, but wanted to be sure it's tracked since it seems surprising.

Comment 1

[Daniel Holbert [:dholbert]]

Attachment: log

Here's the full log (from
https://firefoxci.taskcluster-artifacts.net/MmatVBz0Qu2m8QqgPAkB4Q/0/public/logs/live_backing.log )

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

this is something I am not familiar with, but we need to handle it. Thanks for filing it, I can at least make this a smoother exit.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1921352 - handle errorsummary edge case better. r=ahal

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jmaher, could you have a look please?

For more information, please visit BugBot documentation.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4c66a6806596
handle errorsummary edge case better. r=ahal

Comment 6

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/4c66a6806596