Closed
Bug 1085557
Opened 10 years ago
Closed 9 years ago
Use Socorro symbol upload API for uploading Firefox desktop crash reporter symbols
Categories
(Firefox Build System :: General, defect)
Firefox Build System
General
Tracking
(firefox39 fixed, firefox40 fixed, firefox-esr31 fixed, firefox-esr38 fixed, b2g-v2.0 fixed, b2g-v2.0M fixed, b2g-v2.1 fixed, b2g-v2.1S fixed, b2g-v2.2 fixed, b2g-master fixed)
People
(Reporter: ted, Assigned: ted)
References
Details
Attachments
(1 file, 1 obsolete file)
I think this will be mostly build config work (as automation just calls "make buildsymbols") but we will likely need to at least configure an access token for builders to use.
Assignee | ||
Comment 1•9 years ago
|
||
/r/4423 - bug 1085557 - Add Socorro symbol upload token file to mozconfigs Pull down this commit: hg pull review -r 3c62d97274352feb330eaa8721c0c364e2dbfc8c
Attachment #8570472 -
Flags: review?(coop)
Assignee | ||
Comment 2•9 years ago
|
||
Obviously this patch needs to wait to land for bug 1119238 to go live.
Assignee: nobody → ted
Updated•9 years ago
|
Attachment #8570472 -
Flags: review?(coop) → review+
Comment 3•9 years ago
|
||
Comment on attachment 8570472 [details] MozReview Request: bz://1085557/ted https://reviewboard.mozilla.org/r/4421/#review3601 LGTM
Assignee | ||
Comment 4•9 years ago
|
||
Landed directly on central so we can retrigger nightlies and see if it works: https://hg.mozilla.org/mozilla-central/rev/b2e71f32548f
Comment 5•9 years ago
|
||
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #4) > Landed directly on central so we can retrigger nightlies and see if it works: > https://hg.mozilla.org/mozilla-central/rev/b2e71f32548f sorry had to back this out for nightly bustage like https://treeherder.mozilla.org/logviewer.html#?job_id=1197057&repo=mozilla-central
Status: RESOLVED → REOPENED
Flags: needinfo?(ted)
Resolution: FIXED → ---
Assignee | ||
Comment 6•9 years ago
|
||
Thanks, looks like the token file didn't actually get put on all the machines.
Flags: needinfo?(ted)
Assignee | ||
Comment 7•9 years ago
|
||
The fact that this broke on the Windows builders looks to be my fault: 10:45:53 INFO - c:/builds/moz2_slave/m-cen-w64-ntly-000000000000000/build/src/obj-firefox/_virtualenv/Scripts/python.exe c:/builds/moz2_slave/m-cen-w64-ntly-000000000000000/build/src/toolkit/crashreporter/tools/upload_symbols.py 'dist/firefox-39.0a1.en-US.win64.crashreporter-symbols-full.zip' 10:45:53 INFO - Error: SOCORRO_SYMBOL_UPLOAD_TOKEN_FILE "/c/builds/crash-stats-api.token" does not exist! The path needs to be in native Windows format, I forgot that we have Python code for loading the mozconfig nowadays so msys path translation isn't going to happen. The Linux bustage just looks like the file isn't where it's supposed to be.
Assignee | ||
Comment 8•9 years ago
|
||
On Mac we got a different error, maybe we need a network flow opened up? https://treeherder.mozilla.org/logviewer.html#?job_id=1197061&repo=mozilla-central 08:32:03 INFO - Uploading symbol file "dist/firefox-39.0a1.en-US.mac.crashreporter-symbols-full.zip" to "https://crash-stats.allizom.org/symbols/upload"... 08:32:03 INFO - Error: HTTPSConnectionPool(host='crash-stats.allizom.org', port=443): Read timed out. (read timeout=60) 08:32:03 INFO - make[1]: *** [uploadsymbols] Error 1
Assignee | ||
Comment 9•9 years ago
|
||
The Linux issue was taken care of in bug 1119238, it turns out the file needed to be added to the mock chroot environment. Callek looked into the network flow in bug 1119238 comment 17 and it appears to be open (and he added a test to that effect), so I might just try bumping the timeout up and see if that helps. I'm still kind of suspicious that anything takes more than 60 seconds, but I guess we will find out.
Assignee | ||
Comment 10•9 years ago
|
||
Trying again: https://hg.mozilla.org/mozilla-central/rev/e3d23172f0fe Also bumped the timeout to 120s to see if that fixes things: https://hg.mozilla.org/mozilla-central/rev/db0409de517a
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Component: General Automation → Build Config
Product: Release Engineering → Core
QA Contact: catlee
Resolution: --- → FIXED
Target Milestone: --- → mozilla39
Comment 11•9 years ago
|
||
Backed out the mozconfig changes for bustage again. https://hg.mozilla.org/mozilla-central/rev/5330c6f461a4
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee | ||
Comment 12•9 years ago
|
||
So on the plus side the Windows and OS X builds worked fine this time. The Linux builds are still failing not being able to find the token file: 10:27:31 INFO - /builds/slave/m-cen-lx-ntly-0000000000000000/build/src/obj-firefox/_virtualenv/bin/python /builds/slave/m-cen-lx-ntly-0000000000000000/build/src/toolkit/crashreporter/tools/upload_symbols.py 'dist/firefox-39.0a1.en-US.linux-i686.crashreporter-symbols-full.zip' 10:27:31 INFO - Error: SOCORRO_SYMBOL_UPLOAD_TOKEN_FILE "/builds/crash-stats-api.token" does not exist!
Assignee | ||
Comment 13•9 years ago
|
||
https://hg.mozilla.org/integration/mozilla-inbound/rev/b40fd507c175
Assignee | ||
Comment 14•9 years ago
|
||
I split the patch in two and re-landed just the Windows/Mac mozconfig changes. I'll wait on the Linux mozconfig until coop sorts out the Linux builder issue.
Whiteboard: [leave open]
Assignee | ||
Comment 17•9 years ago
|
||
Comment on attachment 8570472 [details] MozReview Request: bz://1085557/ted /r/4423 - bug 1085557 - Switch symbol upload to use Socorro production server Pull down this commit: hg pull -r 14fbb3eab6cc5217f078280241353e3771c9c76a https://reviewboard-hg.mozilla.org/gecko/
Attachment #8570472 -
Flags: review+ → review?(rhelmer)
Comment 18•9 years ago
|
||
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #17) > Comment on attachment 8570472 [details] > MozReview Request: bz://1085557/ted > > /r/4423 - bug 1085557 - Switch symbol upload to use Socorro production server > > Pull down this commit: > > hg pull -r 14fbb3eab6cc5217f078280241353e3771c9c76a > https://reviewboard-hg.mozilla.org/gecko/ We just shipped bug 1151435 to make our lives easier when this goes live, going to make sure everything worked fine tomorrow and will r+ if so.
Comment 19•9 years ago
|
||
Comment on attachment 8570472 [details] MozReview Request: bz://1085557/ted This lgtm, symbol upload is working fine on stage and content-type for .sym files is now "text/plain" for convenience: https://s3-us-west-2.amazonaws.com/org.mozilla.crash-stats.staging.symbols-public/v1/browsercomps.pdb/8427296CD2F446C0B8BE00027F8C00522/browsercomps.sym
Attachment #8570472 -
Flags: review?(rhelmer) → review+
Comment 20•9 years ago
|
||
Looks like b2g device builds aren't getting SOCORRO_SYMBOL_UPLOAD_TOKEN_FILE set, so no upload to S3 for them. There may be other types of builds also missing out too.
Assignee | ||
Comment 21•9 years ago
|
||
Nick, I'd love your help tracking down what we need to do to switch over all remaining builds that are uploading symbols currently. Getting the Firefox builds switched over was obviously highest-priority (and was enough of a project on its own), but the endgame here is that we want to turn off the SSH upload endpoint, so we'll need to clean up the stragglers.
Comment 22•9 years ago
|
||
Commenter's remorse. The bit of the b2g build harness you want is at http://hg.mozilla.org/build/mozharness/file/default/scripts/b2g_build.py#l599, so whatever that uploadsymbols target is in the b2g build system. We'll be calling that on any job that uses one these configs - http://mxr.mozilla.org/build/search?string=build-symbols&find=%2Fmozharness%2F&findi=&filter=^[^\0]*%24&hitlimit=&tree=build Looks like a lot of device and emulator builds from a quick skim, but there isn't a good mapping to buildbot or Treeherder job names. I can hack/grep something up if that'd help.
Assignee | ||
Comment 23•9 years ago
|
||
https://hg.mozilla.org/integration/mozilla-inbound/rev/4286f78f0933
Blocks: 1144865
Does this need to land for 2.2 and lower?
Flags: needinfo?(ted)
Flags: needinfo?(rhelmer)
Assignee | ||
Comment 26•9 years ago
|
||
We haven't sorted out any B2G-related symbol uploading with this yet. I'm not concerned about any branches for which Mozilla isn't running automated builds that need symbol uploads, FWIW. Anything outside of our infra our partners are likely to be using the upload API directly and not the build system integration for it. We will need to uplift this to aurora/beta/release before we can turn off SSH uploading.
Flags: needinfo?(ted)
Flags: needinfo?(rhelmer)
Comment 27•9 years ago
|
||
Hi, I am working on Bug 1148630, which adds symbol upload for Aries. Since Aries is built using Taskcluster infrastructure, it needs to use the new Socorro API, but as ssh uploads are still on [1], "./build.sh uploadsymbols" fails [2]. Is there a filed bug to turn ssh off? [1] http://hg.mozilla.org/integration/b2g-inbound/file/b592396987d7/Makefile.in#l259 [2] https://tools.taskcluster.net/task-inspector/#DFlZplm4R6Kf6ph8YFKyPg/0
Flags: needinfo?(ted)
Assignee | ||
Comment 28•9 years ago
|
||
Yes, I filed bug 1153799. I was trying to wait until we had all the Firefox builders using the API, but I haven't got the Linux ones switched over yet. We could fix that bug anyway, it'd be a simple matter of sticking an else after the ifdef SOCORRO_SYMBOL_UPLOAD_TOKEN_FILE block to wrap the ssh upload block.
Flags: needinfo?(ted)
Assignee | ||
Comment 29•9 years ago
|
||
https://hg.mozilla.org/integration/mozilla-inbound/rev/64e5241743a0
Assignee | ||
Comment 31•9 years ago
|
||
Resummarizing, Firefox desktop builds are fixed. Will file a follow-up to fix up stragglers.
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Summary: Use Socorro symbol upload API for uploading crash reporter symbols → Use Socorro symbol upload API for uploading Firefox desktop crash reporter symbols
Whiteboard: [leave open]
Updated•9 years ago
|
status-firefox40:
--- → fixed
Target Milestone: mozilla39 → mozilla40
Comment 32•9 years ago
|
||
https://hg.mozilla.org/releases/mozilla-beta/rev/c782301a8065 https://hg.mozilla.org/releases/mozilla-beta/rev/8f1875e7015b https://hg.mozilla.org/releases/mozilla-beta/rev/b27cde03edb0
Flags: in-testsuite-
Comment 33•9 years ago
|
||
(In reply to aleth [:aleth] from comment #16) > https://hg.mozilla.org/comm-central/rev/ce064c0d550b I can't see where TB39 lives these days. c-a is TB40 and c-b in TB38 AFAICT.
Flags: needinfo?(aleth)
Comment 34•9 years ago
|
||
https://hg.mozilla.org/releases/mozilla-esr38/rev/5727f63e5baa https://hg.mozilla.org/releases/mozilla-esr38/rev/54a4b86c1041 https://hg.mozilla.org/releases/mozilla-esr38/rev/799ad48300e8 https://hg.mozilla.org/releases/mozilla-esr38/rev/47d264aec889
status-firefox-esr38:
--- → fixed
Comment 35•9 years ago
|
||
https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/382fce69981f https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/eb08cce5390d https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/aaf365578245 https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/6212532a6bdb
status-b2g-v2.2:
--- → fixed
Comment 36•9 years ago
|
||
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #33) > (In reply to aleth [:aleth] from comment #16) > > https://hg.mozilla.org/comm-central/rev/ce064c0d550b > > I can't see where TB39 lives these days. c-a is TB40 and c-b in TB38 AFAICT. Iirc TB38 is in c-r now, and TB39 is in c-b, but rkent will know for sure.
Flags: needinfo?(rkent)
Updated•9 years ago
|
Flags: needinfo?(aleth)
Comment 37•9 years ago
|
||
TB-38 is in comm-esr38 as of yesterday. TB-39 is in comm-beta, although its status is confused at the moment because we are still transitioning comm-beta from TB 38 to TB 39. But landings in comm-beta default are clearly Thunderbird 39. We'll ship a Thunderbird 39 beta in about two weeks, and at that point comm-beta will be used. I am confused though about your expectations. We are in the process of doing our initial release builds for Thunderbird 38 this week. Is there some expectation that Thunderbird 38 needs to have the patches from this bug? Or just starting at Thunderbird 39?
Flags: needinfo?(rkent)
Comment 38•9 years ago
|
||
(In reply to Kent James (:rkent) from comment #37) > I am confused though about your expectations. We are in the process of doing > our initial release builds for Thunderbird 38 this week. Is there some > expectation that Thunderbird 38 needs to have the patches from this bug? Or > just starting at Thunderbird 39? Afaik in all repos it has to match what the corresponding builders expect?
Flags: needinfo?(ryanvm)
Comment 39•9 years ago
|
||
Not sure why you're asking me, I didn't write these patches. I just saw a c-c landing and figured it was worth a ping.
status-b2g-master:
--- → fixed
Flags: needinfo?(ryanvm)
Comment 40•9 years ago
|
||
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #39) > Not sure why you're asking me, I didn't write these patches. I just saw a > c-c landing and figured it was worth a ping. OK, thanks! rkent: Uplifting these patches will be necessary if you see corresponding check-sync-dirs.py build failures when TB38 is built from comm-esr38. Comment 34 makes this seem likely.
Comment 41•9 years ago
|
||
comm-central and comm-esr38 have a variety of issues in this area. I've filed Bug 1171218 to get the changes that are needed to support symbols in Windows comm-central (which looks to me like has not worked for awhile). For OSX and Linux I'll need to port some small changes into comm-esr38.
Assignee | ||
Comment 42•9 years ago
|
||
I filed bug 1170253 on switching Thunderbird over to the new crash-stats API for upload. I...didn't notice comment 16, but that checkin is bad and should not have happened.
Comment 43•9 years ago
|
||
https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/c1706c0deb29 https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/0b291c349e0a https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/65a897f5af2e https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/3245f53b0a22
status-b2g-v2.1:
--- → fixed
Comment 44•9 years ago
|
||
https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/3ab2e8d85b3f https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/91f0ca7a2b32 https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/c837ff2ac9dd https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/fe6658406c21
status-b2g-v2.0:
--- → fixed
Comment 45•9 years ago
|
||
https://hg.mozilla.org/releases/mozilla-esr31/rev/82b33eda8bb7 https://hg.mozilla.org/releases/mozilla-esr31/rev/57f83f42424c https://hg.mozilla.org/releases/mozilla-esr31/rev/a03ba2592753 https://hg.mozilla.org/releases/mozilla-esr31/rev/765a5c6b1f73
status-firefox-esr31:
--- → fixed
Comment 46•9 years ago
|
||
https://hg.mozilla.org/releases/mozilla-b2g34_v2_1s/rev/c1706c0deb29 https://hg.mozilla.org/releases/mozilla-b2g34_v2_1s/rev/0b291c349e0a https://hg.mozilla.org/releases/mozilla-b2g34_v2_1s/rev/65a897f5af2e https://hg.mozilla.org/releases/mozilla-b2g34_v2_1s/rev/3245f53b0a22
Comment 47•9 years ago
|
||
https://hg.mozilla.org/releases/mozilla-b2g32_v2_0m/rev/3ab2e8d85b3f https://hg.mozilla.org/releases/mozilla-b2g32_v2_0m/rev/91f0ca7a2b32 https://hg.mozilla.org/releases/mozilla-b2g32_v2_0m/rev/c837ff2ac9dd https://hg.mozilla.org/releases/mozilla-b2g32_v2_0m/rev/fe6658406c21
status-b2g-v2.0M:
--- → fixed
Updated•9 years ago
|
status-b2g-v2.1S:
--- → fixed
Assignee | ||
Comment 48•9 years ago
|
||
Attachment #8570472 -
Attachment is obsolete: true
Attachment #8618418 -
Flags: review+
Assignee | ||
Comment 49•9 years ago
|
||
Updated•6 years ago
|
Product: Core → Firefox Build System
You need to log in
before you can comment on or make changes to this bug.
Description
•