849808 - Figure out an archive strategy for old release symbols

Status: RESOLVED INCOMPLETE

(Infrastructure & Operations Graveyard :: WebOps: Socorro, task)

Description

[(not currently active) Ted Mielczarek]

We currently never remove symbols from release builds. This is not a great long-term strategy, especially in the modern rapid-release world. However, I can't bring myself to say that they ought to be deleted.

I'd like to figure out a way to archive symbols for old release builds to some kind of long-term storage so we can remove them from the NetApp storage. I don't care if it's tape backups or Amazon Glacier or what, as long as we can automate it at some point.

Comment 1

[Sean Rich [:srich]]

We discussed Glacier as a possible back-up solution (alternatively, pushing to S3 and then using archive rules to push to Glacier...mmm, automation).  Feels like SLAs would be okay.  No concerns around backing up private symbols because we're supposed to be monitoring to prevent them.  Additionally, we discussed whether we could use NetApp's de-dup functionality, but the store may be too big for that (currently 6.4TB).  We could go through object hashes, find duplicates and rationalize duplicate objects but that may be tough given the data store size.

Comment 2

[Chris Turra [:cturra]]

i have some ideas about -> s3 - lifecycle rule -> glacier. grabbing this.

Comment 3

[Lonnen :lonnen]

I consulted with ted and it turns out we do get at least one set of symbols on purpose that are not public. That may change our options.

Comment 4

[(not currently active) Ted Mielczarek]

FYI we were using NetApp de-dupe last I knew, but that may have changed when we moved storage backends.

Comment 5

[Chris Turra [:cturra]]

is there a way to identify private symbols? if so, we can archive only public symbols and leave private ones in netapp.

Comment 6

[Lonnen :lonnen]

The private symbols are easy to identify. How important is their inclusion in the backups? Do we expect to have more of them in the future?

Comment 7

[(not currently active) Ted Mielczarek]

It's a single directory in the symbol store. If it's a problem I'd say just leave them out of the backup. We will continue to get new symbol files there for the forseeable future, but I'm not overly worried about backing them up.

Comment 8

[Chris Turra [:cturra]]

i spent some time late last week/over the weekend playing with this and have come up with a strategy i'm pretty happy with. however, before pushing terabytes of data out to s3, i thought i would run my solution by the group for a review.

i have created a small shell script on sp-admin01.phx1 that looks through all the symbols_* directories and will push files which have been changed in the last day up to an s3 bucket. it of course will exclude any of the private directories we identify.

once the files are in s3, i have configured a lifecycle rule to archive objects into Glacier after 1 day and delete them from the s3 bucket after 7 days. the following docs discuss how to restore an object:

  http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOSTrestore.html


i would be happy to hand over the access keys for the service account i created so a recovery function, etc can be written into the symbols tools.


how does this all sound?

Comment 9

[(not currently active) Ted Mielczarek]

I don't think that's what we want to do. The majority of symbols on the symbol store are for nightly builds and they get deleted after ~30 days. Archiving them is a waste of space. This bug as filed is for archiving symbols from release builds, of which there are way fewer.

You could archive just release symbols by doing something like:
1) Gather all -symbols.txt files whose version does not contain "a1" or "a2". The version is the second piece of the filename if you split it on '-'.
2) Gather all files listed in the contents of those -symbols.txt files.
3) Archive that entire set of files.

Comment 10

[(not currently active) Ted Mielczarek]

The current cleanup script (that skips release builds) is here:
http://hg.mozilla.org/build/tools/file/tip/buildfarm/breakpad/cleanup-breakpad-symbols.py

Comment 11

[Chris Turra [:cturra]]

putting this back into the unassigned queue now that i am leaving mozilla.

Comment 12

[:Atoll]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #9)
> I don't think that's what we want to do. The majority of symbols on the
> symbol store are for nightly builds and they get deleted after ~30 days.
> Archiving them is a waste of space. This bug as filed is for archiving
> symbols from release builds, of which there are way fewer.
> 
> You could archive just release symbols by doing something like:
> 1) Gather all -symbols.txt files whose version does not contain "a1" or
> "a2". The version is the second piece of the filename if you split it on '-'.
> 2) Gather all files listed in the contents of those -symbols.txt files.
> 3) Archive that entire set of files.

Ted, could you write a shell script or similar that does steps one and two, and emits a set of filenames that we could backup? That would provide you certainty that we're selecting the correct files and save us from several rounds of "is this right" "no, this actually" back and forth dev time.

(Still working on the actual storage/backup side, but it'd be one less thing blocking forward progress.)

Comment 13

[(not currently active) Ted Mielczarek]

Yes, this should be a fairly straightforward modification of the existing cleanup script. I'll take a crack at it.

Comment 14

[Shyam Mani [:fox2mike]]

Time out, please re-open when you have the updated script. Thanks!