Closed Bug 1165347 Opened 5 years ago Closed 5 years ago

Use retries for symbolstore uploads

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(firefox39 fixed, firefox40 fixed, firefox41 fixed, firefox-esr31 fixed, firefox-esr38 fixed, b2g-v2.0 fixed, b2g-v2.0M fixed, b2g-v2.1 fixed, b2g-v2.1S fixed, b2g-v2.2 fixed, b2g-master fixed)

RESOLVED FIXED
mozilla41
Tracking Status
firefox39 --- fixed
firefox40 --- fixed
firefox41 --- fixed
firefox-esr31 --- fixed
firefox-esr38 --- fixed
b2g-v2.0 --- fixed
b2g-v2.0M --- fixed
b2g-v2.1 --- fixed
b2g-v2.1S --- fixed
b2g-v2.2 --- fixed
b2g-master --- fixed

People

(Reporter: nthomas, Assigned: ted)

References

Details

Attachments

(1 file, 1 obsolete file)

Sheriffs have been reporting nightly builds failing when the symbolstore upload fails, eg
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/05/2015-05-15-03-02-02-mozilla-central/mozilla-central-macosx64-nightly-bm82-build1-build41.txt.gz
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/05/2015-05-15-00-40-04-mozilla-aurora/mozilla-aurora-macosx64-nightly-bm85-build1-build24.txt.gz

Those both look to be fallout from the phx1 networking issues, but in general we should be using a retry to handle transient faults. The state of the whole build depends on the symbol upload working.

It looks like we have some functions in tree for this already:
http://hg.mozilla.org/mozilla-central/file/default/python/redo/redo/__init__.py
Assignee: nobody → ted
Attached file MozReview Request: bz://1165347/ted (obsolete) —
/r/8847 - bug 1165347 - use retries for symbol uploads. r?gps

Pull down this commit:

hg pull -r 2addddc35dca37e1236e7e157412f63c4363d04d https://reviewboard-hg.mozilla.org/gecko/
Attachment #8606385 - Flags: review?(gps)
I didn't know we had redo in the tree, thanks for pointing that out!
Comment on attachment 8606385 [details]
MozReview Request: bz://1165347/ted

https://reviewboard.mozilla.org/r/8845/#review7803

Ship It!
Attachment #8606385 - Flags: review?(gps) → review+
https://hg.mozilla.org/mozilla-central/rev/7c82838a230c
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla41
04:47:44     INFO -  /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/i386/_virtualenv/bin/python /builds/slave/m-cen-m64-ntly-000000000000000/build/src/toolkit/crashreporter/tools/upload_symbols.py 'dist/firefox-41.0a1.en-US.mac.crashreporter-symbols-full.zip'
04:52:23     INFO -  Error: got HTTP response 400
04:52:23     INFO -  Uploading symbol file "dist/firefox-41.0a1.en-US.mac.crashreporter-symbols-full.zip" to "https://crash-stats.mozilla.com/symbols/upload"...
04:52:23     INFO -  Retrying...
04:52:23     INFO -  make[1]: *** [uploadsymbols] Error 1
04:52:23     INFO -  make: *** [automation/uploadsymbols] Error 2

Mighty quick retry there.
I assume this is just the output being buffered. We should probably run Python with unbuffered output to make this clearer.
Oh! Nevermind, I'm wrong here, it's not going to retry in this case, nothing raised an exception, it just got a bad HTTP status:
https://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/upload_symbols.py#70

HTTP 400 is "bad request". I wonder if we log upload requests anywhere?
Blocks: 1168368
Depends on: 1168535
ni? me to uplift this across the branches once bug 1168535 is merged to m-c.
Flags: needinfo?(ryanvm)
Flags: needinfo?(ryanvm)
For some reason, b2g34 and b2g32 aren't liking this:
https://treeherder.mozilla.org/logviewer.html#?job_id=128045&repo=mozilla-b2g34_v2_1

I haven't a clue as to what's so special about them. They're on the same slaves and mozharness rev as b2g37, which has no issues.
Flags: needinfo?(ted)
B2G Nightlies are failing on all of the branches that got this except b2g37 (and on esr31, which doesn't do b2g nightlies). I think they're missing bug 1118774, which landed for firefox37.
Flags: needinfo?(ryanvm)
Or I could start with the obvious lack of bug 1118774.
Flags: needinfo?(ted)
Flags: needinfo?(ryanvm)
Attachment #8606385 - Attachment is obsolete: true
Attachment #8620305 - Flags: review+
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.