1014147 - Old Nightly symbols still on symbol mount

Status: RESOLVED INVALID

(Socorro :: Symbols, task)

Description

[Vladan Djeric (:vladan)]

The symbols for a Nightly from December 26th are still available on the symbol mount:

/mnt/netapp/breakpad/symbols_ffx/...

xul.pdb/1DA9424349774537A4C1E0D22847355F2/
firefox.pdb/0AB19400F8944C4E94E77FA55F047D192/
mozjs.pdb/A75EB655E0B34A9CAA90E02A91912D382/

Additionally, bsmedberg says they're not listed in any of the index files.

Comment 1

[Benjamin Smedberg]

Ted, Selena, does that make sense? I thought we deleted Firefox symbols that didn't appear in any of the text files.

Do we need a global-GC of symbol files?

Comment 2

[(not currently active) Ted Mielczarek]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #1)
> Ted, Selena, does that make sense? I thought we deleted Firefox symbols that
> didn't appear in any of the text files.

We don't do that, no. The cleanup script basically does reference counting: grab all the index files, increment each symbol file that appears in them, figure out which builds we want to remove, decrement the symbol files whose builds are being removed, remove index files and symbol files with zero references.

http://hg.mozilla.org/build/tools/file/tip/buildfarm/breakpad/cleanup-breakpad-symbols.py

Comment 3

[Selena Deckelmann :selenamarie :selena]

Is it important to remove those old symbols soon?

Comment 4

[Benjamin Smedberg]

No, but I do wonder whether we're orphaning a bunch of nightly symbols or other unused symbols by accident and could potentially pick up a bunch of disk space by doing a global GC. I'll take a stab at figuring out how big the problem is and coming up with a list of candidate symbols.

Comment 5

[(not currently active) Ted Mielczarek]

I don't think it would be difficult to modify the existing cleanup script to do this, it'll just take forever to run because it will need to trawl all the files in the directory.

Comment 6

[Selena Deckelmann :selenamarie :selena]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5)
> I don't think it would be difficult to modify the existing cleanup script to
> do this, it'll just take forever to run because it will need to trawl all
> the files in the directory.

Wasn't the longer term plan to move our symbols management to use the API that :peterbe made exclusively, so then we'd have a database record of submissions?

Comment 7

[(not currently active) Ted Mielczarek]

Yes. This would just be a one-off run to see if we're wasting significant space in this way.

Comment 8

[Benjamin Smedberg]

Attachment: all-symbols.results

Here's a dump of the files that aren't listed in a .txt file for a total of 507G of symbol files. There's some interesting patterns here:

XUL/{randomid}/XUL.dSYM.tar.bz2

We're getting .dSYM files for some builds? That surprises me, although it might be useful for local debugging. Sounds like we should either list these in the .txt or remove them. They are 442MB each. NI?ted in case he knows what these are for.

There are similarly some

libxul.so/92A5586D1E53018165EE17504B30B14D0/libxul.so.dbg.gz    236764186

f2ce6551911cd59a656a5cef57d76b0055d690d5-firefox-16.0a1.en-US.mac.crashreporter-
symbols-full.zip        133890048

There are a collection of these. Sounds kinda like something went wrong in the middle of a symbol upload operations. Probably needs straight-cleanup.

Then there are a bunch of real .sym files. I wonder if we can create a list of all the debug IDs, poke through the last week of crash reports to see if any of those symbols were actually used, and delete any which weren't? Slightly complicated mapreduce job, but not too terrible.

Comment 9

[(not currently active) Ted Mielczarek]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #8)
> XUL/{randomid}/XUL.dSYM.tar.bz2
> 
> We're getting .dSYM files for some builds? That surprises me, although it
> might be useful for local debugging. Sounds like we should either list these
> in the .txt or remove them. They are 442MB each. NI?ted in case he knows
> what these are for.

Yes, we upload .dSYM files for Mac builds and .dbg files for Linux builds. There's a Python script you can use to fetch the matching native debug symbols for a build:
http://hg.mozilla.org/users/jwatt_jwatt.org/fetch-symbols

> 
> There are similarly some
> 
> libxul.so/92A5586D1E53018165EE17504B30B14D0/libxul.so.dbg.gz    236764186

This is expected, per above.

> 
> f2ce6551911cd59a656a5cef57d76b0055d690d5-firefox-16.0a1.en-US.mac.
> crashreporter-
> symbols-full.zip        133890048
> 
> There are a collection of these. Sounds kinda like something went wrong in
> the middle of a symbol upload operations. Probably needs straight-cleanup.

This is not, sounds like something got broken.

> Then there are a bunch of real .sym files. I wonder if we can create a list
> of all the debug IDs, poke through the last week of crash reports to see if
> any of those symbols were actually used, and delete any which weren't?
> Slightly complicated mapreduce job, but not too terrible.

We actually did a query on "all symbols that were used in a day" not long ago in bug 981021. You could cross-reference with the data there.

Comment 10

[Benjamin Smedberg]

We expect the .dSYM and .dbg.gz files to be cleaned up per the normal expiration policy, right? There are currently 3700 .dSYM files. They should be listed in the appropriate .txt file along with everything else?

Comment 11

[(not currently active) Ted Mielczarek]

Correct:

[[e-mail]1 symbols_ffx]$ grep XUL firefox-29.0-Darwin-20140421221237-x86-macosx64-symbols.txt
XUL/DA82349E22963F01AD30933063017ADA0/XUL.sym
XUL/208989F9A1343946B3A77D8ACF460EB70/XUL.sym
XUL/208989F9A1343946B3A77D8ACF460EB70/XUL.dSYM.tar.bz2

There's one .sym per architecture, and one unified .dSYM.

Comment 12

[Peter Bengtsson [:peterbe]]

From the Portland work week, September 2016;

1) We concluded that the S3 cost isn't significant enough to warrant tight control of excess files. 
2) We added a S3 life-cycle event that deletes things older than 2 years.