Closed
Bug 1165347
Opened 9 years ago
Closed 9 years ago
Use retries for symbolstore uploads
Categories
(Firefox Build System :: General, defect)
Firefox Build System
General
Tracking
(firefox39 fixed, firefox40 fixed, firefox41 fixed, firefox-esr31 fixed, firefox-esr38 fixed, b2g-v2.0 fixed, b2g-v2.0M fixed, b2g-v2.1 fixed, b2g-v2.1S fixed, b2g-v2.2 fixed, b2g-master fixed)
RESOLVED
FIXED
mozilla41
People
(Reporter: nthomas, Assigned: ted)
References
Details
Attachments
(1 file, 1 obsolete file)
Sheriffs have been reporting nightly builds failing when the symbolstore upload fails, eg http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/05/2015-05-15-03-02-02-mozilla-central/mozilla-central-macosx64-nightly-bm82-build1-build41.txt.gz http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/05/2015-05-15-00-40-04-mozilla-aurora/mozilla-aurora-macosx64-nightly-bm85-build1-build24.txt.gz Those both look to be fallout from the phx1 networking issues, but in general we should be using a retry to handle transient faults. The state of the whole build depends on the symbol upload working. It looks like we have some functions in tree for this already: http://hg.mozilla.org/mozilla-central/file/default/python/redo/redo/__init__.py
Assignee | ||
Updated•9 years ago
|
Assignee: nobody → ted
Assignee | ||
Comment 1•9 years ago
|
||
/r/8847 - bug 1165347 - use retries for symbol uploads. r?gps Pull down this commit: hg pull -r 2addddc35dca37e1236e7e157412f63c4363d04d https://reviewboard-hg.mozilla.org/gecko/
Attachment #8606385 -
Flags: review?(gps)
Assignee | ||
Comment 2•9 years ago
|
||
I didn't know we had redo in the tree, thanks for pointing that out!
Comment 3•9 years ago
|
||
Comment on attachment 8606385 [details] MozReview Request: bz://1165347/ted https://reviewboard.mozilla.org/r/8845/#review7803 Ship It!
Attachment #8606385 -
Flags: review?(gps) → review+
Assignee | ||
Comment 4•9 years ago
|
||
https://hg.mozilla.org/integration/mozilla-inbound/rev/7c82838a230c
Comment 5•9 years ago
|
||
https://hg.mozilla.org/mozilla-central/rev/7c82838a230c
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla41
Comment 6•9 years ago
|
||
04:47:44 INFO - /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/i386/_virtualenv/bin/python /builds/slave/m-cen-m64-ntly-000000000000000/build/src/toolkit/crashreporter/tools/upload_symbols.py 'dist/firefox-41.0a1.en-US.mac.crashreporter-symbols-full.zip' 04:52:23 INFO - Error: got HTTP response 400 04:52:23 INFO - Uploading symbol file "dist/firefox-41.0a1.en-US.mac.crashreporter-symbols-full.zip" to "https://crash-stats.mozilla.com/symbols/upload"... 04:52:23 INFO - Retrying... 04:52:23 INFO - make[1]: *** [uploadsymbols] Error 1 04:52:23 INFO - make: *** [automation/uploadsymbols] Error 2 Mighty quick retry there.
Assignee | ||
Comment 7•9 years ago
|
||
I assume this is just the output being buffered. We should probably run Python with unbuffered output to make this clearer.
Assignee | ||
Comment 8•9 years ago
|
||
Oh! Nevermind, I'm wrong here, it's not going to retry in this case, nothing raised an exception, it just got a bad HTTP status: https://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/upload_symbols.py#70 HTTP 400 is "bad request". I wonder if we log upload requests anywhere?
Comment 9•9 years ago
|
||
ni? me to uplift this across the branches once bug 1168535 is merged to m-c.
Flags: needinfo?(ryanvm)
Updated•9 years ago
|
Flags: needinfo?(ryanvm)
Comment 10•9 years ago
|
||
https://hg.mozilla.org/releases/mozilla-aurora/rev/0cd5f3f55ede
status-firefox40:
--- → fixed
Comment 11•9 years ago
|
||
https://hg.mozilla.org/releases/mozilla-beta/rev/103c7e2422c7
status-firefox39:
--- → fixed
Flags: in-testsuite-
Comment 12•9 years ago
|
||
https://hg.mozilla.org/releases/mozilla-esr38/rev/94cc6dc1058d
status-firefox-esr38:
--- → fixed
Comment 13•9 years ago
|
||
https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/15b2ed710814
status-b2g-v2.2:
--- → fixed
Comment 14•9 years ago
|
||
https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/5dcbf12ec339
status-b2g-v2.1:
--- → fixed
Comment 15•9 years ago
|
||
https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/c5677cf32671
status-b2g-v2.0:
--- → fixed
Comment 16•9 years ago
|
||
https://hg.mozilla.org/releases/mozilla-esr31/rev/b585afd5db93
status-firefox-esr31:
--- → fixed
Comment 18•9 years ago
|
||
https://hg.mozilla.org/releases/mozilla-b2g32_v2_0m/rev/c5677cf32671
status-b2g-v2.0M:
--- → fixed
Updated•9 years ago
|
status-b2g-v2.1S:
--- → fixed
status-b2g-master:
--- → fixed
Comment 19•9 years ago
|
||
For some reason, b2g34 and b2g32 aren't liking this: https://treeherder.mozilla.org/logviewer.html#?job_id=128045&repo=mozilla-b2g34_v2_1 I haven't a clue as to what's so special about them. They're on the same slaves and mozharness rev as b2g37, which has no issues.
Flags: needinfo?(ted)
B2G Nightlies are failing on all of the branches that got this except b2g37 (and on esr31, which doesn't do b2g nightlies). I think they're missing bug 1118774, which landed for firefox37.
Flags: needinfo?(ryanvm)
Comment 21•9 years ago
|
||
Or I could start with the obvious lack of bug 1118774.
Flags: needinfo?(ted)
Updated•9 years ago
|
Flags: needinfo?(ryanvm)
Assignee | ||
Comment 22•9 years ago
|
||
Attachment #8606385 -
Attachment is obsolete: true
Attachment #8620305 -
Flags: review+
Assignee | ||
Comment 23•9 years ago
|
||
Updated•6 years ago
|
Product: Core → Firefox Build System
You need to log in
before you can comment on or make changes to this bug.
Description
•