Closed Bug 1085557 Opened 10 years ago Closed 9 years ago

Use Socorro symbol upload API for uploading Firefox desktop crash reporter symbols

Categories

(Firefox Build System :: General, defect)

defect
Not set
normal

Tracking

(firefox39 fixed, firefox40 fixed, firefox-esr31 fixed, firefox-esr38 fixed, b2g-v2.0 fixed, b2g-v2.0M fixed, b2g-v2.1 fixed, b2g-v2.1S fixed, b2g-v2.2 fixed, b2g-master fixed)

RESOLVED FIXED
mozilla40
Tracking Status
firefox39 --- fixed
firefox40 --- fixed
firefox-esr31 --- fixed
firefox-esr38 --- fixed
b2g-v2.0 --- fixed
b2g-v2.0M --- fixed
b2g-v2.1 --- fixed
b2g-v2.1S --- fixed
b2g-v2.2 --- fixed
b2g-master --- fixed

People

(Reporter: ted, Assigned: ted)

References

Details

Attachments

(1 file, 1 obsolete file)

I think this will be mostly build config work (as automation just calls "make buildsymbols") but we will likely need to at least configure an access token for builders to use.
Depends on: 1119238
Blocks: 1119387
No longer blocks: 1119387
Depends on: 1135163
Depends on: 1135700
Attached file MozReview Request: bz://1085557/ted (obsolete) —
/r/4423 - bug 1085557 - Add Socorro symbol upload token file to mozconfigs

Pull down this commit:

hg pull review -r 3c62d97274352feb330eaa8721c0c364e2dbfc8c
Attachment #8570472 - Flags: review?(coop)
Obviously this patch needs to wait to land for bug 1119238 to go live.
Assignee: nobody → ted
Attachment #8570472 - Flags: review?(coop) → review+
Blocks: 1138617
Blocks: 1097207
Depends on: 1142713
Landed directly on central so we can retrigger nightlies and see if it works:
https://hg.mozilla.org/mozilla-central/rev/b2e71f32548f
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #4)
> Landed directly on central so we can retrigger nightlies and see if it works:
> https://hg.mozilla.org/mozilla-central/rev/b2e71f32548f

sorry had to back this out for nightly bustage like https://treeherder.mozilla.org/logviewer.html#?job_id=1197057&repo=mozilla-central
Status: RESOLVED → REOPENED
Flags: needinfo?(ted)
Resolution: FIXED → ---
Thanks, looks like the token file didn't actually get put on all the machines.
Flags: needinfo?(ted)
The fact that this broke on the Windows builders looks to be my fault:
10:45:53     INFO -  c:/builds/moz2_slave/m-cen-w64-ntly-000000000000000/build/src/obj-firefox/_virtualenv/Scripts/python.exe c:/builds/moz2_slave/m-cen-w64-ntly-000000000000000/build/src/toolkit/crashreporter/tools/upload_symbols.py 'dist/firefox-39.0a1.en-US.win64.crashreporter-symbols-full.zip'
10:45:53     INFO -  Error: SOCORRO_SYMBOL_UPLOAD_TOKEN_FILE "/c/builds/crash-stats-api.token" does not exist!

The path needs to be in native Windows format, I forgot that we have Python code for loading the mozconfig nowadays so msys path translation isn't going to happen.

The Linux bustage just looks like the file isn't where it's supposed to be.
On Mac we got a different error, maybe we need a network flow opened up?

https://treeherder.mozilla.org/logviewer.html#?job_id=1197061&repo=mozilla-central

08:32:03 INFO - Uploading symbol file "dist/firefox-39.0a1.en-US.mac.crashreporter-symbols-full.zip" to "https://crash-stats.allizom.org/symbols/upload"...
08:32:03 INFO - Error: HTTPSConnectionPool(host='crash-stats.allizom.org', port=443): Read timed out. (read timeout=60)
08:32:03 INFO - make[1]: *** [uploadsymbols] Error 1
The Linux issue was taken care of in bug 1119238, it turns out the file needed to be added to the mock chroot environment. Callek looked into the network flow in bug 1119238 comment 17 and it appears to be open (and he added a test to that effect), so I might just try bumping the timeout up and see if that helps. I'm still kind of suspicious that anything takes more than 60 seconds, but I guess we will find out.
Trying again:
https://hg.mozilla.org/mozilla-central/rev/e3d23172f0fe

Also bumped the timeout to 120s to see if that fixes things:
https://hg.mozilla.org/mozilla-central/rev/db0409de517a
Status: REOPENED → RESOLVED
Closed: 9 years ago9 years ago
Component: General Automation → Build Config
Product: Release Engineering → Core
QA Contact: catlee
Resolution: --- → FIXED
Target Milestone: --- → mozilla39
Backed out the mozconfig changes for bustage again.
https://hg.mozilla.org/mozilla-central/rev/5330c6f461a4
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
So on the plus side the Windows and OS X builds worked fine this time. The Linux builds are still failing not being able to find the token file:
10:27:31 INFO - /builds/slave/m-cen-lx-ntly-0000000000000000/build/src/obj-firefox/_virtualenv/bin/python /builds/slave/m-cen-lx-ntly-0000000000000000/build/src/toolkit/crashreporter/tools/upload_symbols.py 'dist/firefox-39.0a1.en-US.linux-i686.crashreporter-symbols-full.zip'
10:27:31 INFO - Error: SOCORRO_SYMBOL_UPLOAD_TOKEN_FILE "/builds/crash-stats-api.token" does not exist!
I split the patch in two and re-landed just the Windows/Mac mozconfig changes. I'll wait on the Linux mozconfig until coop sorts out the Linux builder issue.
Whiteboard: [leave open]
Comment on attachment 8570472 [details]
MozReview Request: bz://1085557/ted

/r/4423 - bug 1085557 - Switch symbol upload to use Socorro production server

Pull down this commit:

hg pull -r 14fbb3eab6cc5217f078280241353e3771c9c76a https://reviewboard-hg.mozilla.org/gecko/
Attachment #8570472 - Flags: review+ → review?(rhelmer)
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #17)
> Comment on attachment 8570472 [details]
> MozReview Request: bz://1085557/ted
> 
> /r/4423 - bug 1085557 - Switch symbol upload to use Socorro production server
> 
> Pull down this commit:
> 
> hg pull -r 14fbb3eab6cc5217f078280241353e3771c9c76a
> https://reviewboard-hg.mozilla.org/gecko/

We just shipped bug 1151435 to make our lives easier when this goes live, going to make sure everything worked fine tomorrow and will r+ if so.
Comment on attachment 8570472 [details]
MozReview Request: bz://1085557/ted

This lgtm, symbol upload is working fine on stage and content-type for .sym files is now "text/plain" for convenience:

https://s3-us-west-2.amazonaws.com/org.mozilla.crash-stats.staging.symbols-public/v1/browsercomps.pdb/8427296CD2F446C0B8BE00027F8C00522/browsercomps.sym
Attachment #8570472 - Flags: review?(rhelmer) → review+
Looks like b2g device builds aren't getting SOCORRO_SYMBOL_UPLOAD_TOKEN_FILE set, so no upload to S3 for them. There may be other types of builds also missing out too.
Nick, I'd love your help tracking down what we need to do to switch over all remaining builds that are uploading symbols currently. Getting the Firefox builds switched over was obviously highest-priority (and was enough of a project on its own), but the endgame here is that we want to turn off the SSH upload endpoint, so we'll need to clean up the stragglers.
Commenter's remorse.

The bit of the b2g build harness you want is at http://hg.mozilla.org/build/mozharness/file/default/scripts/b2g_build.py#l599, so whatever that uploadsymbols target is in the b2g build system.

We'll be calling that on any job that uses one these configs - http://mxr.mozilla.org/build/search?string=build-symbols&find=%2Fmozharness%2F&findi=&filter=^[^\0]*%24&hitlimit=&tree=build

Looks like a lot of device and emulator builds from a quick skim, but there isn't a good mapping to buildbot or Treeherder job names. I can hack/grep something up if that'd help.
Blocks: 1153799
Does this need to land for 2.2 and lower?
Flags: needinfo?(ted)
Flags: needinfo?(rhelmer)
We haven't sorted out any B2G-related symbol uploading with this yet. I'm not concerned about any branches for which Mozilla isn't running automated builds that need symbol uploads, FWIW. Anything outside of our infra our partners are likely to be using the upload API directly and not the build system integration for it.

We will need to uplift this to aurora/beta/release before we can turn off SSH uploading.
Flags: needinfo?(ted)
Flags: needinfo?(rhelmer)
Hi,

I am working on Bug 1148630, which adds symbol upload for Aries. Since Aries is built using Taskcluster infrastructure, it needs to use the new Socorro API, but as ssh uploads are still on [1], "./build.sh uploadsymbols" fails [2]. Is there a filed bug to turn ssh off?

[1] http://hg.mozilla.org/integration/b2g-inbound/file/b592396987d7/Makefile.in#l259
[2] https://tools.taskcluster.net/task-inspector/#DFlZplm4R6Kf6ph8YFKyPg/0
Flags: needinfo?(ted)
Yes, I filed bug 1153799. I was trying to wait until we had all the Firefox builders using the API, but I haven't got the Linux ones switched over yet. We could fix that bug anyway, it'd be a simple matter of sticking an else after the  ifdef SOCORRO_SYMBOL_UPLOAD_TOKEN_FILE block to wrap the ssh upload block.
Flags: needinfo?(ted)
Blocks: 1148630
Resummarizing, Firefox desktop builds are fixed. Will file a follow-up to fix up stragglers.
Status: REOPENED → RESOLVED
Closed: 9 years ago9 years ago
Resolution: --- → FIXED
Summary: Use Socorro symbol upload API for uploading crash reporter symbols → Use Socorro symbol upload API for uploading Firefox desktop crash reporter symbols
Whiteboard: [leave open]
Blocks: 528092
Target Milestone: mozilla39 → mozilla40
(In reply to aleth [:aleth] from comment #16)
> https://hg.mozilla.org/comm-central/rev/ce064c0d550b

I can't see where TB39 lives these days. c-a is TB40 and c-b in TB38 AFAICT.
Flags: needinfo?(aleth)
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #33)
> (In reply to aleth [:aleth] from comment #16)
> > https://hg.mozilla.org/comm-central/rev/ce064c0d550b
> 
> I can't see where TB39 lives these days. c-a is TB40 and c-b in TB38 AFAICT.

Iirc TB38 is in c-r now, and TB39 is in c-b, but rkent will know for sure.
Flags: needinfo?(rkent)
Flags: needinfo?(aleth)
TB-38 is in comm-esr38 as of yesterday. TB-39 is in comm-beta, although its status is confused at the moment because we are still transitioning comm-beta from TB 38 to TB 39. But landings in comm-beta default are clearly Thunderbird 39.

We'll ship a Thunderbird 39 beta in about two weeks, and at that point comm-beta will be used.

I am confused though about your expectations. We are in the process of doing our initial release builds for Thunderbird 38 this week. Is there some expectation that Thunderbird 38 needs to have the patches from this bug? Or just starting at Thunderbird 39?
Flags: needinfo?(rkent)
(In reply to Kent James (:rkent) from comment #37)
> I am confused though about your expectations. We are in the process of doing
> our initial release builds for Thunderbird 38 this week. Is there some
> expectation that Thunderbird 38 needs to have the patches from this bug? Or
> just starting at Thunderbird 39?

Afaik in all repos it has to match what the corresponding builders expect?
Flags: needinfo?(ryanvm)
Not sure why you're asking me, I didn't write these patches. I just saw a c-c landing and figured it was worth a ping.
Flags: needinfo?(ryanvm)
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #39)
> Not sure why you're asking me, I didn't write these patches. I just saw a
> c-c landing and figured it was worth a ping.

OK, thanks!

rkent: Uplifting these patches will be necessary if you see corresponding check-sync-dirs.py build failures when TB38 is built from comm-esr38. Comment 34 makes this seem likely.
comm-central and comm-esr38 have a variety of issues in this area. I've filed Bug 1171218 to get the changes that are needed to support symbols in Windows comm-central (which looks to me like has not worked for awhile). For OSX and Linux I'll need to port some small changes into comm-esr38.
I filed bug 1170253 on switching Thunderbird over to the new crash-stats API for upload. I...didn't notice comment 16, but that checkin is bad and should not have happened.
Attachment #8570472 - Attachment is obsolete: true
Attachment #8618418 - Flags: review+
Blocks: 311977
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.