Closed Bug 1165347 Opened 5 years ago Closed 5 years ago
Use retries for symbolstore uploads
39 bytes, text/x-review-board-request
Sheriffs have been reporting nightly builds failing when the symbolstore upload fails, eg http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/05/2015-05-15-03-02-02-mozilla-central/mozilla-central-macosx64-nightly-bm82-build1-build41.txt.gz http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/05/2015-05-15-00-40-04-mozilla-aurora/mozilla-aurora-macosx64-nightly-bm85-build1-build24.txt.gz Those both look to be fallout from the phx1 networking issues, but in general we should be using a retry to handle transient faults. The state of the whole build depends on the symbol upload working. It looks like we have some functions in tree for this already: http://hg.mozilla.org/mozilla-central/file/default/python/redo/redo/__init__.py
/r/8847 - bug 1165347 - use retries for symbol uploads. r?gps Pull down this commit: hg pull -r 2addddc35dca37e1236e7e157412f63c4363d04d https://reviewboard-hg.mozilla.org/gecko/
Attachment #8606385 - Flags: review?(gps)
I didn't know we had redo in the tree, thanks for pointing that out!
Comment on attachment 8606385 [details] MozReview Request: bz://1165347/ted https://reviewboard.mozilla.org/r/8845/#review7803 Ship It!
Attachment #8606385 - Flags: review?(gps) → review+
04:47:44 INFO - /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/i386/_virtualenv/bin/python /builds/slave/m-cen-m64-ntly-000000000000000/build/src/toolkit/crashreporter/tools/upload_symbols.py 'dist/firefox-41.0a1.en-US.mac.crashreporter-symbols-full.zip' 04:52:23 INFO - Error: got HTTP response 400 04:52:23 INFO - Uploading symbol file "dist/firefox-41.0a1.en-US.mac.crashreporter-symbols-full.zip" to "https://crash-stats.mozilla.com/symbols/upload"... 04:52:23 INFO - Retrying... 04:52:23 INFO - make: *** [uploadsymbols] Error 1 04:52:23 INFO - make: *** [automation/uploadsymbols] Error 2 Mighty quick retry there.
I assume this is just the output being buffered. We should probably run Python with unbuffered output to make this clearer.
Oh! Nevermind, I'm wrong here, it's not going to retry in this case, nothing raised an exception, it just got a bad HTTP status: https://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/upload_symbols.py#70 HTTP 400 is "bad request". I wonder if we log upload requests anywhere?
ni? me to uplift this across the branches once bug 1168535 is merged to m-c.
For some reason, b2g34 and b2g32 aren't liking this: https://treeherder.mozilla.org/logviewer.html#?job_id=128045&repo=mozilla-b2g34_v2_1 I haven't a clue as to what's so special about them. They're on the same slaves and mozharness rev as b2g37, which has no issues.
B2G Nightlies are failing on all of the branches that got this except b2g37 (and on esr31, which doesn't do b2g nightlies). I think they're missing bug 1118774, which landed for firefox37.
Or I could start with the obvious lack of bug 1118774.
You need to log in before you can comment on or make changes to this bug.