Make upload_symbols.py retry on 500 errors from the API

RESOLVED FIXED in Firefox 39

Status

()

defect
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: ted, Assigned: ted)

Tracking

Trunk
mozilla41
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox39 fixed, firefox40 fixed, firefox41 fixed, firefox-esr31 fixed, firefox-esr38 fixed, b2g-v2.0 fixed, b2g-v2.0M fixed, b2g-v2.1 fixed, b2g-v2.1S fixed, b2g-v2.2 fixed)

Details

Attachments

(1 attachment)

We got a 500 ISE from the symbol upload API today, but it was transient and retrying would have worked. We don't currently retry on 500.
Duplicate of this bug: 1171621
I have a patch, I tested locally with a python httpd that alternates returning 500 and 200:

 0:01.36 Uploading symbol file "dist/firefox-41.0a1.en-US.linux-x86_64.crashreporter-symbols-full.zip" to "http://0.0.0.0:48030/"...
 0:03.10 Error: got HTTP response 500: Internal Server Error
 0:03.11 Response body:
 0:03.11 ====================
 0:03.11 <head>
 0:03.11 <title>Error response</title>
 0:03.11 </head>
 0:03.11 <body>
 0:03.11 <h1>Error response</h1>
 0:03.11 <p>Error code 500.
 0:03.11 <p>Message: Internal Server Error.
 0:03.11 <p>Error code explanation: 500 = Server got itself in trouble.
 0:03.11 </body>
 0:03.11 
 0:03.11 ====================
 0:03.11 
 0:03.11 Retrying...
 0:15.85 Uploaded successfully!
Assignee: nobody → ted
bug 1171527 -  Make upload_symbols.py retry on 500 errors from the API. r?gps
Attachment #8615944 - Flags: review?(gps)
Comment on attachment 8615944 [details]
MozReview Request: bug 1171527 -  Make upload_symbols.py retry on 500 errors from the API. r?gps

https://reviewboard.mozilla.org/r/10383/#review9123

::: toolkit/crashreporter/tools/upload_symbols.py:72
(Diff revision 1)
> +            # 500 is likely to be a transient failure.
> +            # Break out for success or other error codes.
> +            if r.status_code  != 500:
> -            break
> +                break
> +            print_error(r)

While you are here, could you make sure we have a log event for every attempted HTTP request? As it stands, we don't have a good way of measuring 500 frequency from clients. If there's one thing I've learned from services it's that server logs only tell part of the story.
Attachment #8615944 - Flags: review?(gps) → review+
(In reply to Gregory Szorc [:gps] from comment #4)
> While you are here, could you make sure we have a log event for every
> attempted HTTP request? As it stands, we don't have a good way of measuring
> 500 frequency from clients. If there's one thing I've learned from services
> it's that server logs only tell part of the story.

You mean log something in the loop here for every request it makes?
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5)
> You mean log something in the loop here for every request it makes?

Yes.
https://hg.mozilla.org/mozilla-central/rev/68234efc22d7
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla41
Flags: needinfo?(ryanvm)
Flags: needinfo?(ryanvm)
You need to log in before you can comment on or make changes to this bug.