Closed Bug 702337 Opened 8 years ago Closed 7 years ago

Stop uploading try symbols to symbol server

Categories

(Release Engineering :: General, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ted, Assigned: nthomas)

References

Details

(Whiteboard: [good first bug][automation][tryserver])

Attachments

(1 file)

Currently we upload symbols from Windows try builds to build.mozilla.org, and serve them via HTTP as a symbol server. We keep running out of space on dm-wwwbuild01, which sucks.

I think we should stop uploading symbols to build.mozilla.org, and instead make the "full symbol package"[1] get uploaded[2] to FTP along with the build. This way developers could simply download their symbols along with the build, and we'd only have to maintain one storage volume for tryserver builds. Plus, the retention policy for builds+symbols would be identical. (As a bonus, we could get debug symbols for non-Windows builds, which we don't currently get.)

1. http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/package-name.mk#149
2. http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/packager.mk#869
Priority: -- → P3
Whiteboard: [automation][tryserver][triagefollowup]
Ok, the solution that I see:

**m-c code** change the |make upload| code to switch between SYMBOL_FULL_ARCHIVE_BASENAME and SYMBOL_INDEX_NAME based on an environ var, suggest "MOZ_FTP_UPLOAD_FULL_SYMBOLS" (so its clear this is different than upload server)
c.f. http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/packager.mk#869

Then, in buildbot-configs:
Change upload_symbols for the try config from True to False
Add to try config, env: {MOZ_UPLOAD_FULL_SYMBOLS: "1"}

Then:
Success

Unfortunately its pretty hard to test without both buildbot and m-c code changes.

caveats:

Short-term we would hurt for anyone pushing anything non-m-c to try if they need symbols, since they wouldn't get full symbols uploaded, unless the m-c side gets approvals.

Also 1.9.2 based try builds would also hurt on symbol stuff; But on the bright side we do treat try as m-c supported other branches by luck :-)

I am willing to do this work, but its not the highest on my priority list, this comment is to guide if anyone wants to take on some extra work.

Adding "Good First Bug" based on our collective comments here
Whiteboard: [automation][tryserver][triagefollowup] → [good first bug][automation][tryserver][triagefollowup]
(In reply to Justin Wood (:Callek) from comment #1)
> Ok, the solution that I see:

> Then, in buildbot-configs:
> Change upload_symbols for the try config from True to False
> Add to try config, env: {MOZ_UPLOAD_FULL_SYMBOLS: "1"}

Err meant: MOZ_FTP_UPLOAD_FULL_SYMBOLS
(In reply to Justin Wood (:Callek) from comment #1)
> I am willing to do this work, but its not the highest on my priority list,
> this comment is to guide if anyone wants to take on some extra work.

I'm happy to review and help get things tested.
Assignee: nobody → bugspam.Callek
Whiteboard: [good first bug][automation][tryserver][triagefollowup] → [good first bug][automation][tryserver]
I would like to see a diskspace budget before we go ahead with this. A full set of try builds currently takes up ~1400MB, of which ~230MB is the smaller symbols file. 
Example file sizes
 linux32 opt:   26 -> 192MB
which would push the contribution of linux32 from 245MB to 410MB. Multiply that up by the number of jobs try does in a day, and the desire to keep try builds for a number of days, and I'm guessing it's going to be a big number. (Should take into account that not all jobs do all builds). 

I've also looked at the number of hits on the try symbol server (elsewhere) and it was essentially zero, which makes me wonder if all that space would be usefully consumed.
Slightly alternate proposal: stop uploading to the symbol server, but add an opt-in in try syntax to get the full symbol packages uploaded. That way only users that explicitly request them will get them, but it will at least be possible.
Pushing to nick, since he had concerns [needing data] in c#4 with the approach I suggested in c#1
Assignee: bugspam.Callek → nrthomas
Assignee: nrthomas → nobody
Component: Release Engineering → Release Engineering: Automation (General)
QA Contact: catlee
This is currently using build.mozilla.org, which we're trying to retire as a web host in bug 604688.
Blocks: 604688
Also, worth noting that http://build.mozilla.org/tryserver-symbols handed out nothing but 404's so far in August, per my grep:

  grep /tryserver-symbols access_2012-08-* | grep -v 404

so perhaps this is moot, or Ted's suggestion in comment 5 could be very effective?
I've seen a bunch of people complain that they wanted to debug their debug builds from Try, but we don't currently make the symbols available. I think we're probably not serving developers very well with the current setup.
I noticed that the 404's don't start with /windows, but the only direntry in /symbols is windows/.  So that may be broken.
Ping?  Can someone confirm that these aren't available, so we can stop storing them?
Wouldn't the solution be to fix how they're being served rather than delete them because people can't access them?
Not if they're unneeded..
Summarizing the situation:
 * symbols on relengweb cause that volume to change size quickly, causing nagios alerts
 * symbols on relengweb are not accessible to anyone who might need them
 * symbols on relengweb cause a lot of data churn, which causes minor problems with the storage backend
 * relengweb is not intended for uploads - we have other, better tools for that

Suggested steps:
 * now: stop uploading to relengweb (since it's a lot of wasted bits and causes pain)
 * later: as suggested in comment 0, start uploading to either symbolpush or stage.m.o, either of which is designed for uploads and fast-churning data.
 * bonus points: from comment 5, only upload in the (apparently quite rare) case that symbols are actually needed, as specified via try syntax
Blocks: 819159
Goosed /vol/releng_web up another 50g in response to a warning alarm.
Most recent month was +30g growth.
Here's my new proposal:
1) I add a new thing that developers can put in mozconfigs, like UPLOAD_FULL_SYMBOLS=1. This would make us upload the full symbol package to FTP.
2) We stop uploading symbols to the try symbol server.

Then developers can opt-in to get symbols if they want them. Does that sound reasonable?
Sounds fine to me.
Depends on: 863715
Part 1 is in bug 863715. Once we get that landed I think we just fix this bug and call it a day.
Summary: Stop uploading try symbols to symbol server, upload full symbol package to ftp → Stop uploading try symbols to symbol server
So I just pushed that bug to mozilla-inbound. Right now it requires adding this change to a try push:
http://hg.mozilla.org/users/tmielczarek_mozilla.com/mq/raw-file/b44faf6cd177/enable-full-symbols

It might be nice to add a way to trigger this to trychooser syntax, but I don't think that's a requirement. I'll add some documentation somewhere on how to use this, and I think you can just stop uploading symbols.
dump_master output looks good for this - removed uploadsymbols steps on 32bit pgo and non-pgo, and the three vars removed from environment of several steps.
Assignee: nobody → nthomas
Status: NEW → ASSIGNED
Attachment #742877 - Flags: review?(rail)
Attachment #742877 - Flags: review?(rail) → review+
Comment on attachment 742877 [details] [diff] [review]
Disable symbol upload

https://hg.mozilla.org/build/buildbot-configs/rev/a9873b0abd24

Will get picked up in the next reconfig.
Attachment #742877 - Flags: checked-in+
Live.
Verified we're not trying to upload symbols to build.m.o any more.
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.