379757 - cleanup script for breakpad symbols on stage

Status: VERIFIED FIXED

(Release Engineering :: General, defect, P3)

Description

[(not currently active) Ted Mielczarek]

Attachment: symbol cleanup script

We need this script to clean up old nightly symbols on stage.  When symbols are uploaded, a text file is uploaded along with them that contains a list of the symbol files for that build.  the text file is named <appname>-<version>-<OS_TARGET>-<build id>-symbols.txt, for example:
firefox-3.0a4pre-WINNT-2007040605-symbols.txt

This script groups those files by product, branch, OS, and allows a fixed number of builds per group (set to 30 right now).  It then deletes all the symbols for those builds.

Comment 1

[(not currently active) Ted Mielczarek]

I also don't know how we will run this script, since the Firefox symbols are owned by ffxbld.

Comment 2

[J. Paul Reed [:preed]]

Comment on attachment 263759 [details]
symbol cleanup script

Some general comments:

-- I'd *really* like to have a style for Python (we already sorta have one, for new stuff, in Perl land), that is consistently used. Personally, I'm not a fan of _'s, and rhelmer and I talked about doing CamelCase, since that *seems* more prevalent in the Python community. I'd also like to see things like constant in caps. I wouldn't block the script on this alone, but we should really have this conversation, especially since we're starting to do everything in Python.

Let's not use os.system() for this sort of thing; Perl has a File::Find module; Python should have something equivalent, I think.

What makes these system() calls particularly scary is the use of xargs, which if you ever have files with spaces in, things start barfing. 

It's more robust to just handle this internally.

Comment 3

[Benjamin Smedberg]

Yeah, you want os.walk or os.path.walk

Comment 4

[(not currently active) Ted Mielczarek]

(In reply to comment #2)
> (From update of attachment 263759 [details])
> Some general comments:
> 
> -- I'd *really* like to have a style for Python (we already sorta have one, for
> new stuff, in Perl land), that is consistently used. Personally, I'm not a fan
> of _'s, and rhelmer and I talked about doing CamelCase, since that *seems* more
> prevalent in the Python community. I'd also like to see things like constant in
> caps. I wouldn't block the script on this alone, but we should really have this
> conversation, especially since we're starting to do everything in Python.

Sounds reasonable.  The variables up top are sort of configuration options, except that this script is pretty much for internal use only.  I just don't feel comfortable hardcoding values in there.  Should I just treat them as constants in the scope of the file?

Comment 5

[(not currently active) Ted Mielczarek]

This isn't going to work right on OS X, since the symbols' debug identifiers are idempotent.  For example, all of the uploaded symbols from the nightlies so far have the same ID for libnspr4.dylib.

Comment 6

[(not currently active) Ted Mielczarek]

Attachment: updated to use os.walk

Updated a bit.

Comment 7

[Ben Turner (not reading bugmail, use the needinfo flag!)]

(In reply to comment #5)
> This isn't going to work right on OS X, since the symbols' debug identifiers
> are idempotent.  For example, all of the uploaded symbols from the nightlies so
> far have the same ID for libnspr4.dylib.
> 

Not anymore!

Comment 8

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Ahem, I misunderstood. Linux and OS X are now going to use MD5 to generate the signature so libraries like NSPR (that are pulled from a tag and thus identical each time) will have the same signature. Windows apparently incorporates the timestamp in its uuid.

Comment 9

[(not currently active) Ted Mielczarek]

FWIW, we're up to 44% of our allotted storage for symbols.  We should get this script working soon.

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found in triage - any update on this?

Comment 11

[(not currently active) Ted Mielczarek]

I think bsmedberg misplaced his rewrite, so either I'll have to fix my script here, or convince someone else to do it. :-) We're at about 50% utilization, I won't really be worried until we start tossing symbols for Moz2 builds on top of this.

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #11)
> I think bsmedberg misplaced his rewrite, so either I'll have to fix my script
> here, or convince someone else to do it. :-) We're at about 50% utilization, I
> won't really be worried until we start tossing symbols for Moz2 builds on top
> of this.
> 
Found in triage - any update on this?

We've now opened up mozilla-central, so should we be worried now? Or is this something we can defer to "Future" until someone gets time to look at it?

Comment 13

[(not currently active) Ted Mielczarek]

500G  379G  122G  76% /mnt/netapp/breakpad

We're now at 76% utilization, so we're going to have to make this happen very soon. We could probably buy a little bit of time by just mass deleting symbols from the Firefox 3 alphas/pre-alpha nightlies.

Comment 14

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Thanks for the update. Unless someone else whats jump in and help, we'll have
to deal with this early next month, as we've got our hands full right now.

Comment 15

[(not currently active) Ted Mielczarek]

I'll take this for lack of any better options, since we're at 82% now.

Comment 16

[Aravind Gottipati [:aravind]]

This is starting to be a priority, so if you guys don't have the bandwidth, I
can help out here.  Either way let me know.  If you want me handle this, could
someone explain the logic I would use.

Comment 17

[(not currently active) Ted Mielczarek]

Attachment: rewritten

Ok, I've rewritten the previous script, and it handles the OS X/Linux case of symbols being included in multiple builds just fine now. It will delete all but the last 30 nightlies from each branch, as well as any symbols from nightlies > 6 months old. The individual symbol files from each build are reference counted, such that they will only be deleted if all the builds referencing them are to be deleted. I also added a --dry-run argument to the script so you can run it to test without deleting anything.

The script takes one argument, the symbol directory to process. You'll need to run it once per symbol directory, that's just the way it's written right now, so:
cleanup-breakpad-symbols.py /mnt/netapp/breakpad/symbols_ffx/

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Nagios sez:

DISK WARNING - free space: /mnt/netapp/breakpad 45465 MB (8% inode=89%):

Dropped about 11MB in the last 14 hours.

Comment 19

[Reed Loden [:reed]]

(In reply to comment #18)
> Dropped about 11MB in the last 14 hours.

By 11MB, I hope you mean 11GB. :)

Comment 20

[(not currently active) Ted Mielczarek]

Aravind is running this in production.

10.253.0.11:/vol/socorro/symbols
                      500G  216G  285G  44% /mnt/netapp/breakpad

much better!

Comment 21

[Aravind Gottipati [:aravind]]

Thank you!

Comment 22

[Chris Cooper [:coop] (he/him)]

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.