797324 - Find specific "talosError:" cases that aren't easily caused on Try & can be set to RETRY

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Ed Morley [:emorley]]

Indicates a fatal talos run failure, so we didn't get any results and should RETRY.

Comment 1

[[e-mail] (:bhearsum)]

I have a vague recollection of someone (maybe philor?) not wanting to do this. In any case, it's easy to do by adding it to the list of global errors: https://github.com/mozilla/buildbotcustom/blob/master/status/errors.py#L5

Comment 2

[Ed Morley [:emorley]]

Attachment: Patch v1

Comment 3

[Ed Morley [:emorley]]

(In reply to Ben Hearsum [:bhearsum] from comment #1)
> I have a vague recollection of someone (maybe philor?) not wanting to do
> this. In any case, it's easy to do by adding it to the list of global
> errors:
> https://github.com/mozilla/buildbotcustom/blob/master/status/errors.py#L5

Sorry didn't get a mid-air on this.

Would you prefer the global list or the one I added in the patch?

philor is CCed and I'll make sure he agrees before landing.

Comment 4

[Ed Morley [:emorley]]

Attachment: Patch v2

Adding to the general errors list, since bhearsum mentioned that adding log_eval_func overrides the general list and we still want those on talos (and talosError won't come up anywhere else).

Comment 5

[Phil Ringnalda (:philor)]

Comment on attachment 667463 [details] [diff] [review]
Patch v2

With mobile and their... ways, the first thing you have to ask yourself while setting RETRY is "is there any way on earth that a patch can cause this condition (because if so a patch will cause it, and I'll have 200 jobs retrying all at once)?"

A patch that causes the browser to not start up presents as "talosError: 'failed to initialize browser'" and in fact the two logs in that bug from yesterday are how long it took me to realize I was starring a completely busted push, a completely busted push that stayed in the tree for probably 20 or 30 pushes.

There probably are individual talosErrors that can retry because they can't be caused by a patch, though given how the ateam tests talos (and the other harness bits that can be tested there because they are in m-c) with try pushes, that's risky too - I killed someone's push on try this weekend because it added a devicemanager.DMError to robocop, so it had retried 30 times by the time I saw it. Even if you figure this is okay because the retries will coalesce and we'll just kill the running jobs on the push before we backed out bustage, you still have to consider that this will cause infinite retries on unnoticed try pushes from someone breaking mobile and not understanding that their push will continue to run forever.

Comment 6

[Ed Morley [:emorley]]

Ah, good point; I should have thought of that :-/

I'll try looking into specific cases (like you say), but I'm guessing there won't be many of them (other than the already filed bug 716800).

Comment 7

[Ed Morley [:emorley]]

Given comment 5, the fact that this was just wallpaper, and that we can now star talosErrors so they are less of a burden; WONTFIX.