431657 - in the name of saving disk space, mozilla2 buildbot nightlies should delete their trees at the end of a build

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[[e-mail] (:bhearsum)]

As we add more branches to the Mozilla2 Buildbot disk space on the slaves is going to become a larger issue. Currently, we keep 2 trees for each branch, one for dep builds, one for nightlies. (At some point we should look at having nightly builds done off of the dep builder, but that's a much bigger issue.) We can easily save a bunch of disk space by simply having the nightly builder clobber its build directory (tree + symbol directories) after uploading the build, symbols, etc. 

IIRC it used to be a big deal to keep thy symbol directories around on fx-win32-tbox but I'm not sure if this has changed with the symbol server. CC'ing folks in case there's objections.

Comment 1

[(not currently active) Ted Mielczarek]

The reason the tinderboxes kept builds/symbols around was in case network problems (or some other issue) caused the upload to fail, it would be fairly easy to go back and re-upload the bits. The bits were stored outside of the objdir though, so you could still blow away the source/objdir without problems.

Comment 2

[Benjamin Smedberg]

As long as we result-check the symbol upload process (something tinderbox never did) I think we're fine to blow everything away. If we wanted to be paranoid, we could archive the symbols in a separate package and ship them with the nightly... ted, how big would that package be?

Comment 3

[(not currently active) Ted Mielczarek]

Looks like for a Windows build the zip package is about 35Mb, the Linux/Mac archives should be smaller.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

The zip file of Fx3 symbols is 6.3/10/35 MB for linux/mac/win32.

Tinderbox also keeps the packages separately from the objdir. The required space per clobber is then it's 24/43/61 MB.

Comment 5

[[e-mail] (:bhearsum)]

If people feel like it's important to keep the symbols around we may as well just continue doing that. We'd have to setup a cron job or go around and cleanup the symbols manually from time to time if we go that route.

Comment 6

[(not currently active) Ted Mielczarek]

Currently tinderbox saves something like one week of packages. I definitely think it's important to cache the symbols/build locally if the upload fails, as otherwise they'll get lost.

Comment 7

[[e-mail] (:bhearsum)]

OK. How about this:
* Add a BuildStep that deletes 7+ day old symbols (this way we can't forget to add a cronjob on new machines)
* Clobber only the tree (src+objdir) at the end of a build

Comment 8

[[e-mail] (:bhearsum)]

Attachment: [checked in] delete old symbols at the start of a nightly & delete src+objdir at the end

I haven't heard any objections to this plan so I will plan to do this on Monday, May 5th with some other maintenance. If anyone *does* have an objection please speak up before then.

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 319006 [details] [diff] [review]
[checked in] delete old symbols at the start of a nightly & delete src+objdir at the end

changeset:   44:70fc3dc84c08

Comment 10

[[e-mail] (:bhearsum)]

Triggered a nightly to give it one last test.