Closed Bug 1171527 Opened 8 years ago Closed 8 years ago

Make upload_symbols.py retry on 500 errors from the API

Categories

(Toolkit :: Crash Reporting, defect)

Product:
Toolkit
Component:
Crash Reporting
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla41
Tracking Flags:
Tracking Status
firefox39 --- fixed
firefox40 --- fixed
firefox41 --- fixed
firefox-esr31 --- fixed
firefox-esr38 --- fixed
b2g-v2.0 --- fixed
b2g-v2.0M --- fixed
b2g-v2.1 --- fixed
b2g-v2.1S --- fixed
b2g-v2.2 --- fixed

People

(Reporter: ted, Assigned: ted)

References

Details

Attachments

(1 file)

MozReview Request: bug 1171527 - Make upload_symbols.py retry on 500 errors from the API. r?gps
40 bytes, text/x-review-board-request
gps
: review+
Details 
We got a 500 ISE from the symbol upload API today, but it was transient and retrying would have worked. We don't currently retry on 500.
I have a patch, I tested locally with a python httpd that alternates returning 500 and 200:

 0:01.36 Uploading symbol file "dist/firefox-41.0a1.en-US.linux-x86_64.crashreporter-symbols-full.zip" to "http://0.0.0.0:48030/"...
 0:03.10 Error: got HTTP response 500: Internal Server Error
 0:03.11 Response body:
 0:03.11 ====================
 0:03.11 <head>
 0:03.11 <title>Error response</title>
 0:03.11 </head>
 0:03.11 <body>
 0:03.11 <h1>Error response</h1>
 0:03.11 <p>Error code 500.
 0:03.11 <p>Message: Internal Server Error.
 0:03.11 <p>Error code explanation: 500 = Server got itself in trouble.
 0:03.11 </body>
 0:03.11 
 0:03.11 ====================
 0:03.11 
 0:03.11 Retrying...
 0:15.85 Uploaded successfully!
Assignee: nobody → ted
bug 1171527 -  Make upload_symbols.py retry on 500 errors from the API. r?gps
Attachment #8615944 - Flags: review?(gps)
Comment on attachment 8615944 [details]
MozReview Request: bug 1171527 -  Make upload_symbols.py retry on 500 errors from the API. r?gps

https://reviewboard.mozilla.org/r/10383/#review9123

::: toolkit/crashreporter/tools/upload_symbols.py:72
(Diff revision 1)
> +            # 500 is likely to be a transient failure.
> +            # Break out for success or other error codes.
> +            if r.status_code  != 500:
> -            break
> +                break
> +            print_error(r)

While you are here, could you make sure we have a log event for every attempted HTTP request? As it stands, we don't have a good way of measuring 500 frequency from clients. If there's one thing I've learned from services it's that server logs only tell part of the story.
Attachment #8615944 - Flags: review?(gps) → review+
(In reply to Gregory Szorc [:gps] from comment #4)
> While you are here, could you make sure we have a log event for every
> attempted HTTP request? As it stands, we don't have a good way of measuring
> 500 frequency from clients. If there's one thing I've learned from services
> it's that server logs only tell part of the story.

You mean log something in the loop here for every request it makes?
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5)
> You mean log something in the loop here for every request it makes?

Yes.
https://hg.mozilla.org/mozilla-central/rev/68234efc22d7
Status: NEW → RESOLVED
Closed: 8 years ago
status-firefox41: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla41
Flags: needinfo?(ryanvm)
Flags: needinfo?(ryanvm)
You need to log in before you can comment on or make changes to this bug.