1019038 - TBPL shouldn't match against "Caught Exception: Remote Device Error: unable to connect to..."

Status: RESOLVED FIXED

(Tree Management Graveyard :: TBPL, defect)

Description

[Ed Morley [:emorley]]

Bug 976106 made this change:
https://hg.mozilla.org/build/tools/rev/cb82132dc905
    1.12 -            except:
    1.13 +            except BaseException as e:
    1.14 +                print "Caught Exception:", e

This means that instead of silently swallowing exceptions, we now print things like:
https://tbpl.mozilla.org/php/getParsedLog.php?id=40869231&tree=Mozilla-Inbound
07:51:08     INFO -  Caught Exception: Remote Device Error: unable to connect to panda-0501 after 5 attempts

However, the exception was still not intended to cause the run to fail, which prompted a change to errors.py in bug 1018531 to ensure parity with past behavior:
https://hg.mozilla.org/build/buildbotcustom/rev/199ec58139d8

However TBPL still matches against those lines, meaning the failure excerpt lists that line, even though it's not related to the failure.

We need to make a regexp change similar to that made to errors.py, to:
https://hg.mozilla.org/webtools/tbpl/file/0d9836ff8bf0/php/inc/GeneralErrorFilter.php#l53

Comment 1

[Pete Moore [:pmoore][:pete]]

This was fixed by bug 1018531.

Comment 2

[Pete Moore [:pmoore][:pete]]

Sorry, ignore my comment!

Comment 3

[Ed Morley [:emorley]]

I think the best fix for this is just to match against start of line, since at least for TBPL (where the timestamp + log level + leading whitespace is trimmed off), I can't see any valid instances of "" that would match against:


eg:
https://tbpl.mozilla.org/php/getParsedLog.php?id=41034984&tree=Mozilla-Inbound#error0
Remote Device Error: updateApp() call failed - exiting

https://tbpl.mozilla.org/php/getParsedLog.php?id=41037677&tree=Mozilla-Inbound#error5
09:03:18     INFO -  DMError: Remote Device Error: unable to connect to 10.12.128.145 after 5 attempts
-> This would match against /^[A-Za-z.]+Error: / anyway.

https://tbpl.mozilla.org/php/getParsedLog.php?id=41037494&tree=Mozilla-Inbound#error3
09:04:06     INFO -  Remote Device Error: Unhandled exception in cleanupDevice
09:04:06    ERROR - Return code: 1
-> The extra space is trimmed by:
https://hg.mozilla.org/webtools/tbpl/file/fdaf28ca2446/php/inc/GeneralErrorFilter.php#l31

And in fact, this is what we used to do, prior to bug 799564. The reason for changing it there was concerns over compatibility with mozharness, but since then TBPL trims the mozharness log level prefixes off, so that is no longer a problem.

(side note: roll on structured logging and not having to scratch around looking at old logs seeing what will and won't be matched *sigh*)

Comment 4

[Ed Morley [:emorley]]

Attachment: Prevent matching against "Caught Exception: Remote Device Error: ..."

Returns to TBPL only matching against "Remote Device Error" if at start of line, which is what we did prior to bug 799564 anyway (and that change was only made for mozharness compatibility, which has been made unnecessary since by bug 809436).

Comment 5

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8438570 [details] [diff] [review]
Prevent matching against "Caught Exception: Remote Device Error: ..."

Hi Ed,

Looks perfect. I enjoyed reading through the associated bugs!

Thanks,
Pete

Comment 6

[Ed Morley [:emorley]]

Thank you for the review :-)

https://hg.mozilla.org/webtools/tbpl/rev/78d6b2288d5c

Comment 7

[Ed Morley [:emorley]]

This was in production on 2014-06-12, forgot to update the bug then.