465622 - fix symbol cleanup script to handle security releases properly

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[(not currently active) Ted Mielczarek]

The script from bug 379757 turns out to not handle security releases correctly. Specifically, it tries to group versions by "branch", but the regex it uses is "^(\d+(\.\d+)*)", which just grabs all the numeric part of the number, so like "3.0.5pre" becomes "3.0.5". THe problem is that we then keep 30 days worth of builds *per branch*, so we have 30 days of builds hanging around for 3.0pre, 3.0.1pre, 3.0.2pre... We should be able to save a bunch of disk space by fixing this to lump all the 3.0*pre builds together. (Which is good, because the symbol store is down to 10% free.)

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: --dry-run output

This change 
   # RE to get the version number without alpha/beta designation
  -versionRE = re.compile("^(\d+(\.\d+)*)")
  +versionRE = re.compile("^(\d+\.\d+)")
gets the --dry-run output in the attached file. There's 3.0.*pre builds from May 22 up to October 20, plus the 3.1b2 builds from 30 days ago which will be expired next time the current script runs - both look fine to me.

Ted, do you recall why the regexp is more complicated ? I guess it matches both 3 and 4 digit version schemes rather than relying on 3, but we're only pushing release symbols for 2.0.0.x and they get skipped anyway. Unfortunately "^(\d+(\.\d+)*)\.\d+" doesn't work for 3.1b2pre, returning "3".

Comment 2

[(not currently active) Ted Mielczarek]

That sounds more sane. No, I don't know why I went with that. This regex looks good, we'll wind up lumping all the "3.0" branch builds together, and all the "3.1" branch builds together.

I was probably trying to do something fancy, and it backfired on me. I'm going to put this script in a hg repo somewhere, then apply your change.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Pushed the original script (attachment 326473 [details] [diff] [review]) to 
  http://hg.mozilla.org/build/tools/rev/9a87d1edfad0
as 
  http://hg.mozilla.org/build/tools/file/9a87d1edfad0/buildfarm/breakpad/cleanup-breakpad-symbols.py

Pushed the change in comment #1 to 
  http://hg.mozilla.org/build/tools/rev/828d1865f7b9

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Aravind, please pull the update the cleanup script to the copy in this repo
  http://hg.mozilla.org/build/tools/
It's at buildfarm/breakpad/cleanup-breakpad-symbols.py in there.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Once again, in English ...

Aravind, please update the breakpad cleanup script to the copy in this repo
  http://hg.mozilla.org/build/tools/
It's at buildfarm/breakpad/cleanup-breakpad-symbols.py in there.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

I ran the updated symbol script against symbols_ffx and didn't get a huge improvement in free space, still 93% used. What other dirs does it normally run against ?

Doing some du runs to figure out what's using all the space.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

All in MB:
76340  symbols_sbrd
31014  symbols_ffx
27556  symbols_xr
14393  symbols_sea
 7308  symbols_tbrd
  927  symbols_os
  504  symbols_solaris

There are txt files for sunbird (sbrd) going back to August '07, so no cleanup is being done there at the moment ?

These numbers don't quite make sense though. If you add them up you get 158042MB, which is well shy of what df reports:
# df -m /mnt/netapp/breakpad/
Filesystem           1M-blocks      Used Available Use% Mounted on
10.253.0.11:/vol/socorro/symbols
                       1536000   1370815    165186  90% /mnt/netapp/breakpad

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Gavin pointed this out
  http://www.cyberciti.biz/tips/freebsd-why-command-df-and-du-reports-different-output.html
Could other parts of Socorro be holding these files open, so that the disk space isn't freed ?

Comment 9

[Aravind Gottipati [:aravind]]

You only see the symbols space in this mount point.  Socorro dumps also live in the same file system (they are just not exposed to you).  The rest of the space is taken up by those.  I know they are filling up steadily, SS and some of the other release drivers are aware of the problem.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

The low-water mark is steadily going down
dm-symbolpush01:disk - /mnt/netapp/breakpad is CRITICAL: DISK CRITICAL - free space: /mnt/netapp/breakpad 90702 MB (5% inode=17%)
while oscillating between 5 and 10% during a day (not to mention the 2 hourly alerts getting monotonous).

Any progress ?

Comment 11

[Samuel Sidler (old account; do not CC)]

Yes, but not enough yet. In the mean time, I told Aravind he could slowly go down day-by-day, with Socorro dumps, as needed.

Comment 12

[Aravind Gottipati [:aravind]]

Updated the script and added a cleanup crontab for symbols_sbrd (calbld user) on surf.