Closed Bug 684251 Opened 14 years ago Closed 10 years ago

Figure out a better way to store Breakpad symbols

Categories

(Socorro :: General, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ted, Unassigned)

References

Details

Storing symbols as flat files on a NFS mount was a great idea back when we had 3 machines uploading symbols. Now that we have tons of project branches and tons of release builds it's unmaintainable. Every few months we get an out-of-space bug, like bug 684245, and I have to scramble around to figure out what to do. I'd like to figure out a smarter way to store our symbols. Maybe we can leave the symbol files themselves on disk, but put metadata in some kind of data store, so that we can at least more easily figure out where all our space is going.
How long do you keep the files shouldn't this be aligned with the dump retention policy or something similar ?
We keep release symbols indefinitely right now, so I don't think it matters that much. Since we only have one set of symbols per platform per release, it's not a huge amount of space. Most of the space is used by nightly builds, since we have one set of symbols per platform per nightly per branch/project branch, which adds up to a huge amount.
Component: Socorro → General
Product: Webtools → Socorro
Daniel and I discussed this a while ago, and Laura and I just discussed this again yesterday. I think we want to put these all in HBase (or maybe just HFS?). Our ideal migration path would be something like: 1) Put symbol files as-is into hbase/hfs, use FUSE to mount it onto the processors as a normal filesystem, so we don't have to rewrite minidump_stackwalk right away. 2) Rewrite minidump_stackwalk to talk to Hbase/HFS and ditch FUSE. Some tricky bits in step 1: * Making symbol uploads continue to work. Right now all our build slaves simply scp a zipfile up to symbols1.dmz.phx1.mozilla.com. Helpfully, they do run a currently-noop post-upload Python script, so we have a hook to add more complexity. We'd have to make hbase accessible from that symbolpush1 host, but that's in PHX. * The symbol server needs to continue to work. Currently this is just Apache serving flat files from a webhead. We'd either need to switch that to use FUSE (if that's stable enough to work) or a custom webapp that serves the files out of Hbase.
Depends on: 789493
Depends on: 1071724
We put symbols in S3, life is better.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.