Ensure B2G failures are printed as ERRORS so TBPL can parse them

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
5 years ago
4 years ago

People

(Reporter: ahal, Assigned: ahal)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [mozharness][sheriff-want])

Attachments

(3 attachments, 1 obsolete attachment)

Comment hidden (empty)
(Assignee)

Comment 1

5 years ago
Created attachment 679253 [details] [diff] [review]
Patch 1.0 - Add error messages
Attachment #679253 - Flags: review?(jgriffin)
Attachment #679253 - Flags: review?(jgriffin) → review+
(Assignee)

Comment 2

5 years ago
https://hg.mozilla.org/build/mozharness/rev/e32ed3d9681f
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED

Comment 3

5 years ago
If I'm reading the attached patch correctly, it should catch errors of the form:
"03:22:54     INFO -  : Timeout waiting for marionette on port '40169'"

but we're unfortunately not, eg: 
https://tbpl.mozilla.org/php/getParsedLog.php?id=16986938&tree=Mozilla-Inbound
Blocks: 778688, 809436
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: [mozharness] → [mozharness][sheriff-want]

Comment 4

5 years ago
(Also, what's with the leading colon?)
(Assignee)

Comment 5

5 years ago
Weird, it's thrown here: http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/emulator.py#415

Mozharness must be munging the error message and the colon is probably why it isn't getting caught.

Comment 6

5 years ago
Is there a bug filed for making sure the last line of a traceback is always printed on an ERROR line?

Comment 7

5 years ago
(Seeing as it would stop us having to hardcode these)

Comment 8

5 years ago
Also:
23:16:13     INFO -  MarionetteException: Could not successfully complete transport of message to Gecko, socket closed?
https://tbpl.mozilla.org/php/getParsedLog.php?id=17057752&tree=Mozilla-Inbound#error0

Updated

5 years ago
Blocks: 812149
(Assignee)

Comment 9

5 years ago
So to recap, we have two exceptions (though we should hunt down the others while we are at it):
MarionetteException: Could not successfully complete transport of message to Gecko, socket closed?
Timeout waiting for marionette on port 'nnnnn'

Also the latter should read "TimeoutException: Timeout waiting for marionette on port 'nnnnn'" but for some reason mozharness is only dumping out  ": Timeout waiting for..."

:aki, would you be able to look into this? I'll do it otherwise.
Summary: Add two new error strings from bug 809437 to error list → Ensure B2G failures are printed as ERRORS so TBPL can parse them
And also:
23:33:46     INFO -  : Timeout waiting for the b2g process to start

But the v4 patch in bug 809437 removes some of the strings iirc?
(Assignee)

Comment 11

5 years ago
(In reply to Ed Morley [:edmorley UTC+0] from comment #10)
> But the v4 patch in bug 809437 removes some of the strings iirc?

Yep, it removes the "Timeout waiting for the B2G process to terminate" string. So we should remove that while we are at it. The other string is still possible and the "Could not successfully complete transport of message to Gecko" one is a generic error that can happen anywhere we use marionette when the server hasn't started up, so it should be added.

Comment 12

5 years ago
(In reply to Ed Morley [:edmorley UTC+0] from comment #6)
> Is there a bug filed for making sure the last line of a traceback is always
> printed on an ERROR line?

bug 688338 would allow us to easily.

Comment 13

5 years ago
Or less difficult, at least.

(In reply to Andrew Halberstadt [:ahal] from comment #5)
> Weird, it's thrown here:
> http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/
> marionette/emulator.py#415
> 
> Mozharness must be munging the error message and the colon is probably why
> it isn't getting caught.

substr shouldn't care where in the string it is...
I should be able to add TimeoutException and MarionetteException pretty easily.
(Assignee)

Comment 14

5 years ago
Theoretically any of these exceptions can be thrown: http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/errors.py

Though in practice I'm not sure how many of them are actually used.

Comment 15

5 years ago
Created attachment 682057 [details] [diff] [review]
all emulator exceptions in a regex
Attachment #682057 - Flags: review?(ahalberstadt)
(Assignee)

Comment 16

5 years ago
Comment on attachment 682057 [details] [diff] [review]
all emulator exceptions in a regex

Review of attachment 682057 [details] [diff] [review]:
-----------------------------------------------------------------

So this patch looks good, but I don't think it will work. The problem we are seeing is despite these messages being here they are still being printed at the INFO level. I have no idea why. (see https://tbpl.mozilla.org/php/getParsedLog.php?id=17071920&tree=Mozilla-Inbound). Instead all errors are just "Error: Return code 1"

Also could you remove the 'Timeout waiting for the b2g process to terminate' message? That is no longer being raised.
Attachment #682057 - Flags: review?(ahalberstadt) → review-

Comment 17

5 years ago
Created attachment 682087 [details] [diff] [review]
with a couple more tweaks

[11:03]	<aki>	ahal: i'm not sure why the line that starts with the : doesn't get caught, but it would catch this line above: 09:44:43 INFO - raise TimeoutException("Timeout waiting for marionette on port '%s'" % self.marionette_port)

My current guess is this error is eating the TimeoutException: "BaseException.message has been deprecated as of Python 2.6"

09:44:43     INFO -      raise TimeoutException("Timeout waiting for marionette on port '%s'" % self.marionette_port)
09:44:43     INFO -  marionette.errors.TimeoutException/home/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/marionette/errors.py:18: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
09:44:43     INFO -    return str(self.message)
Attachment #682057 - Attachment is obsolete: true
Attachment #682087 - Flags: review?(ahalberstadt)
(Assignee)

Comment 18

5 years ago
Comment on attachment 682087 [details] [diff] [review]
with a couple more tweaks

Review of attachment 682087 [details] [diff] [review]:
-----------------------------------------------------------------

Ok, let's land. I'll file a bug to get rid of the deprecation warning and we'll see what happens. Thanks! r+ with one nit.

::: scripts/marionette.py
@@ +79,5 @@
>  
>      error_list = [
>          {'substr': 'FAILED (errors=', 'level': ERROR},
>          {'substr': r'''Could not successfully complete transport of message to Gecko, socket closed''', 'level': ERROR},
>          {'substr': 'Timeout waiting for the b2g process to terminate', 'level': ERROR},

Forgot to remove this here
Attachment #682087 - Flags: review?(ahalberstadt) → review+

Comment 19

5 years ago
Comment on attachment 682087 [details] [diff] [review]
with a couple more tweaks

Also added a missing 'import re' that I found while running unit.sh.
http://hg.mozilla.org/build/mozharness/rev/817d3b48fefc

We should be able to see its effect on any job starting after that.
Attachment #682087 - Flags: checked-in+

Comment 20

5 years ago
And, as :ahal expected, not fixed:
http://buildbot-master24.build.scl1.mozilla.com:8201/builders/b2g_ics_armv7a_gecko_emulator%20mozilla-inbound%20opt%20test%20reftest-2/builds/83/steps/run_script/logs/stdio

Comment 21

5 years ago
Created attachment 682307 [details] [diff] [review]
duh
Attachment #682307 - Flags: review?(ahalberstadt)
(Assignee)

Comment 22

5 years ago
Comment on attachment 682307 [details] [diff] [review]
duh

Review of attachment 682307 [details] [diff] [review]:
-----------------------------------------------------------------

Heh
Attachment #682307 - Flags: review?(ahalberstadt) → review+

Comment 23

5 years ago
Comment on attachment 682307 [details] [diff] [review]
duh

http://hg.mozilla.org/build/mozharness/rev/d9fa46496849
Attachment #682307 - Flags: checked-in+

Updated

5 years ago
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.