Closed Bug 1171527 Opened 10 years ago Closed 10 years ago

Make upload_symbols.py retry on 500 errors from the API

Categories

(Toolkit :: Crash Reporting, defect)

Product:
Toolkit
Component:
Crash Reporting
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla41
Tracking Flags:
Tracking Status
firefox39 --- fixed
firefox40 --- fixed
firefox41 --- fixed
firefox-esr31 --- fixed
firefox-esr38 --- fixed
b2g-v2.0 --- fixed
b2g-v2.0M --- fixed
b2g-v2.1 --- fixed
b2g-v2.1S --- fixed
b2g-v2.2 --- fixed

People

(Reporter: ted, Assigned: ted)

References

Details

Attachments

(1 file)

MozReview Request: bug 1171527 - Make upload_symbols.py retry on 500 errors from the API. r?gps
40 bytes, text/x-review-board-request
gps
: review+
Details
We got a 500 ISE from the symbol upload API today, but it was transient and retrying would have worked. We don't currently retry on 500.
I have a patch, I tested locally with a python httpd that alternates returning 500 and 200: 0:01.36 Uploading symbol file "dist/firefox-41.0a1.en-US.linux-x86_64.crashreporter-symbols-full.zip" to "http://0.0.0.0:48030/"... 0:03.10 Error: got HTTP response 500: Internal Server Error 0:03.11 Response body: 0:03.11 ==================== 0:03.11 <head> 0:03.11 <title>Error response</title> 0:03.11 </head> 0:03.11 <body> 0:03.11 <h1>Error response</h1> 0:03.11 <p>Error code 500. 0:03.11 <p>Message: Internal Server Error. 0:03.11 <p>Error code explanation: 500 = Server got itself in trouble. 0:03.11 </body> 0:03.11 0:03.11 ==================== 0:03.11 0:03.11 Retrying... 0:15.85 Uploaded successfully!
Assignee: nobody → ted
bug 1171527 - Make upload_symbols.py retry on 500 errors from the API. r?gps
Attachment #8615944 - Flags: review?(gps)
Comment on attachment 8615944 [details] MozReview Request: bug 1171527 - Make upload_symbols.py retry on 500 errors from the API. r?gps https://reviewboard.mozilla.org/r/10383/#review9123 ::: toolkit/crashreporter/tools/upload_symbols.py:72 (Diff revision 1) > + # 500 is likely to be a transient failure. > + # Break out for success or other error codes. > + if r.status_code != 500: > - break > + break > + print_error(r) While you are here, could you make sure we have a log event for every attempted HTTP request? As it stands, we don't have a good way of measuring 500 frequency from clients. If there's one thing I've learned from services it's that server logs only tell part of the story.
Attachment #8615944 - Flags: review?(gps) → review+
(In reply to Gregory Szorc [:gps] from comment #4) > While you are here, could you make sure we have a log event for every > attempted HTTP request? As it stands, we don't have a good way of measuring > 500 frequency from clients. If there's one thing I've learned from services > it's that server logs only tell part of the story. You mean log something in the loop here for every request it makes?
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5) > You mean log something in the loop here for every request it makes? Yes.
https://hg.mozilla.org/mozilla-central/rev/68234efc22d7
Status: NEW → RESOLVED
Closed: 10 years ago
status-firefox41: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla41
Flags: needinfo?(ryanvm)
Flags: needinfo?(ryanvm)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: