702337 - Stop uploading try symbols to symbol server

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[(not currently active) Ted Mielczarek]

Currently we upload symbols from Windows try builds to build.mozilla.org, and serve them via HTTP as a symbol server. We keep running out of space on dm-wwwbuild01, which sucks.

I think we should stop uploading symbols to build.mozilla.org, and instead make the "full symbol package"[1] get uploaded[2] to FTP along with the build. This way developers could simply download their symbols along with the build, and we'd only have to maintain one storage volume for tryserver builds. Plus, the retention policy for builds+symbols would be identical. (As a bonus, we could get debug symbols for non-Windows builds, which we don't currently get.)

1. http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/package-name.mk#149
2. http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/packager.mk#869

Comment 1

[Justin Wood (:Callek)]

Ok, the solution that I see:

**m-c code** change the |make upload| code to switch between SYMBOL_FULL_ARCHIVE_BASENAME and SYMBOL_INDEX_NAME based on an environ var, suggest "MOZ_FTP_UPLOAD_FULL_SYMBOLS" (so its clear this is different than upload server)
c.f. http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/packager.mk#869

Then, in buildbot-configs:
Change upload_symbols for the try config from True to False
Add to try config, env: {MOZ_UPLOAD_FULL_SYMBOLS: "1"}

Then:
Success

Unfortunately its pretty hard to test without both buildbot and m-c code changes.

caveats:

Short-term we would hurt for anyone pushing anything non-m-c to try if they need symbols, since they wouldn't get full symbols uploaded, unless the m-c side gets approvals.

Also 1.9.2 based try builds would also hurt on symbol stuff; But on the bright side we do treat try as m-c supported other branches by luck :-)

I am willing to do this work, but its not the highest on my priority list, this comment is to guide if anyone wants to take on some extra work.

Adding "Good First Bug" based on our collective comments here

Comment 2

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #1)
> Ok, the solution that I see:

> Then, in buildbot-configs:
> Change upload_symbols for the try config from True to False
> Add to try config, env: {MOZ_UPLOAD_FULL_SYMBOLS: "1"}

Err meant: MOZ_FTP_UPLOAD_FULL_SYMBOLS

Comment 3

[Chris Cooper [:coop] (he/him)]

(In reply to Justin Wood (:Callek) from comment #1)
> I am willing to do this work, but its not the highest on my priority list,
> this comment is to guide if anyone wants to take on some extra work.

I'm happy to review and help get things tested.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

I would like to see a diskspace budget before we go ahead with this. A full set of try builds currently takes up ~1400MB, of which ~230MB is the smaller symbols file. 
Example file sizes
 linux32 opt:   26 -> 192MB
which would push the contribution of linux32 from 245MB to 410MB. Multiply that up by the number of jobs try does in a day, and the desire to keep try builds for a number of days, and I'm guessing it's going to be a big number. (Should take into account that not all jobs do all builds). 

I've also looked at the number of hits on the try symbol server (elsewhere) and it was essentially zero, which makes me wonder if all that space would be usefully consumed.

Comment 5

[(not currently active) Ted Mielczarek]

Slightly alternate proposal: stop uploading to the symbol server, but add an opt-in in try syntax to get the full symbol packages uploaded. That way only users that explicitly request them will get them, but it will at least be possible.

Comment 6

[Justin Wood (:Callek)]

Pushing to nick, since he had concerns [needing data] in c#4 with the approach I suggested in c#1

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

This is currently using build.mozilla.org, which we're trying to retire as a web host in bug 604688.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Also, worth noting that http://build.mozilla.org/tryserver-symbols handed out nothing but 404's so far in August, per my grep:

  grep /tryserver-symbols access_2012-08-* | grep -v 404

so perhaps this is moot, or Ted's suggestion in comment 5 could be very effective?

Comment 9

[(not currently active) Ted Mielczarek]

I've seen a bunch of people complain that they wanted to debug their debug builds from Try, but we don't currently make the symbols available. I think we're probably not serving developers very well with the current setup.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

I noticed that the 404's don't start with /windows, but the only direntry in /symbols is windows/.  So that may be broken.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Ping?  Can someone confirm that these aren't available, so we can stop storing them?

Comment 12

[Chris AtLee [:catlee]]

Wouldn't the solution be to fix how they're being served rather than delete them because people can't access them?

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Not if they're unneeded..

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Summarizing the situation:
 * symbols on relengweb cause that volume to change size quickly, causing nagios alerts
 * symbols on relengweb are not accessible to anyone who might need them
 * symbols on relengweb cause a lot of data churn, which causes minor problems with the storage backend
 * relengweb is not intended for uploads - we have other, better tools for that

Suggested steps:
 * now: stop uploading to relengweb (since it's a lot of wasted bits and causes pain)
 * later: as suggested in comment 0, start uploading to either symbolpush or stage.m.o, either of which is designed for uploads and fast-churning data.
 * bonus points: from comment 5, only upload in the (apparently quite rare) case that symbols are actually needed, as specified via try syntax

Comment 15

[Greg Cox [:gcox]]

Goosed /vol/releng_web up another 50g in response to a warning alarm.
Most recent month was +30g growth.

Comment 16

[(not currently active) Ted Mielczarek]

Here's my new proposal:
1) I add a new thing that developers can put in mozconfigs, like UPLOAD_FULL_SYMBOLS=1. This would make us upload the full symbol package to FTP.
2) We stop uploading symbols to the try symbol server.

Then developers can opt-in to get symbols if they want them. Does that sound reasonable?

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Sounds fine to me.

Comment 18

[(not currently active) Ted Mielczarek]

Part 1 is in bug 863715. Once we get that landed I think we just fix this bug and call it a day.

Comment 19

[(not currently active) Ted Mielczarek]

So I just pushed that bug to mozilla-inbound. Right now it requires adding this change to a try push:
http://hg.mozilla.org/users/tmielczarek_mozilla.com/mq/raw-file/b44faf6cd177/enable-full-symbols

It might be nice to add a way to trigger this to trychooser syntax, but I don't think that's a requirement. I'll add some documentation somewhere on how to use this, and I think you can just stop uploading symbols.

Comment 20

[(not currently active) Ted Mielczarek]

Added docs:
https://wiki.mozilla.org/ReleaseEngineering/TryServer#Getting_debug_symbols

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Disable symbol upload

dump_master output looks good for this - removed uploadsymbols steps on 32bit pgo and non-pgo, and the three vars removed from environment of several steps.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 742877 [details] [diff] [review]
Disable symbol upload

https://hg.mozilla.org/build/buildbot-configs/rev/a9873b0abd24

Will get picked up in the next reconfig.

Comment 23

[Armen [:armenzg]]

Live.

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Verified we're not trying to upload symbols to build.m.o any more.