464103 - Running all different branches in the slaves is going to make us run out of space

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Armen [:armenzg]]

You can look at bug 464093 and see how the macs actually need manual intervention

Since we added unittest and l10n repackages the inodes and the disk usage of the mac slaves has gone crazy high and hopefully it won't give us problems on windows and linux any time soon

Comment 1

[[e-mail] (:bhearsum)]

We did some calculations during the onsite and we *can* support unittests and the Firefox 3.1 maintenance branch without bumping up disk sizes. However, l10n might push us over the edge.

What's the action item here, Armen?

Comment 2

[Armen [:armenzg]]

The l10n nightly work basically has the "mozilla-central" repo and a subset of the "l10n-central" depending on how many locales a specific slave dealt with and a maximum of the whole "l10n-central" repo if all locales were done by the same slave.

NOTE: The installers are normally deleted in each iteration since they are moved from the folder "dist" to "dist/upload" which is deleted in the first step of each iteration

If we want to minimize the disk usage we could:
1) Delete every time "mozilla-central" and "l10n-central" - Checking out both for each locale will be a high performance hit specially with "mozilla-central"
2) Delete every time the "l10n-central" - It will be a performance hit but will keep the size the "l10n-central" folder the size reduced from "size of all repos" to the size of "only one locale"


NOTE: I believe that the major problem that the mac slaves were hiting was that the iNodes capacity was full

Comment 3

[[e-mail] (:bhearsum)]

Yes, I understand the problem here. I don't understand what you want us to _do_ about it. Are you asking for disks to be added to the Macs, and the disks on the VMs to be increased?

Comment 4

[Armen [:armenzg]]

I think this is more of "are we going to hit this on production?" and "can we assess the impact on staging?"

If we are running out of space in staging we have to be able to see if there is any changes we have to do like the one if being commented in 464093#c9 to don't hit the same problem.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Some numbers would really help here.

Comment 6

[Chris AtLee [:catlee]]

on bm-xserve19, sizes of directories in /builds/moz2_slave (in MB)

25	mozilla-central-macosx-nightly
38	tracemonkey-macosx-nightly
1707	mozilla-central-macosx-debug
13357	mozilla-central-macosx
14735	tracemonkey-macosx

on moz2-linux-slave15:
8	tracemonkey-linux-nightly
15	mozilla-central-linux-nightly
699	mozilla-central-linux-l10n-nightly
909	mozilla-central-linux-unittest
2001	mozilla-central-linux-debug
3568	tracemonkey-linux
3574	mozilla-central-linux

on moz2-win32-slave11:
200     mozilla-central-win32-nightly
257     tracemonkey-win32-nightly
2109    mozilla-central-win32-debug
2259    tracemonkey-win32
4870    mozilla-central-win32

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

One thing I noticed on win32 is that the size on disk can be bigger than the sum of all the file sizes (thanks FAT!). Eg mozilla-central-win32/ has Size of 4.37GB, and Size on disk of 5.61GB (according to the properties dialog for that dir). Similarly 2.0/3.22GB for tracemonkey-win32, 1.82/3.02GB for debug, nightly about 4.56/5.79GB.

Comment 8

[Chris AtLee [:catlee]]

(In reply to comment #7)
> One thing I noticed on win32 is that the size on disk can be bigger than the
> sum of all the file sizes (thanks FAT!). Eg mozilla-central-win32/ has Size of
> 4.37GB, and Size on disk of 5.61GB (according to the properties dialog for that
> dir). Similarly 2.0/3.22GB for tracemonkey-win32, 1.82/3.02GB for debug,
> nightly about 4.56/5.79GB.

Yes, this is going to be a big problem actually.  I looked at one of the windows slaves in staging yesterday, it had 15 GB of data, but 25 GB of used disk space.  That's 10 GB of filesystem overhead!

We'll definitely need to expand the size of these drives.

Comment 9

[Chris AtLee [:catlee]]

I think we also need a script that can be run before each build that can clean out other builds.  Something smarter than what's used in #464093.

It should be able to free up a configurable amount of space, e.g. 6 GB for a windows build.  It should delete builds in the following order:
- Old nightlies
- Old release builds
- Dependent builds in order from oldest to newest

This should stop builds from failing because of insufficient disk space.  We could end up clobbering dependent builds too frequently, thus slowing things down, but that's better than failing to complete!

Comment 10

[Chris AtLee [:catlee]]

Attachment: python script to delete other build directories

Comment 11

[Chris AtLee [:catlee]]

Attachment: Fetch and call purge_builds.py in build factories

Comment 12

[Chris AtLee [:catlee]]

Attachment: Updates to mozilla2-staging to use the new purge_builds script

Comment 13

[Chris AtLee [:catlee]]

Attachment: Create a MozillaBuildFactory base class that runs build cleanup steps

Comment 14

[Chris AtLee [:catlee]]

Attachment: Updates to mozilla2-staging to use the new purge_builds script

Comment 15

[[e-mail] (:bhearsum)]

Comment on attachment 351384 [details] [diff] [review]
python script to delete other build directories

>+    parser = OptionParser(__doc__)
>+

According to the OptionParser reference page you should pass this with the argument name. eg, description=__doc__


Other than that, this looks great.

Comment 16

[Chris AtLee [:catlee]]

Attachment: python script to delete other build directories

uses win32file module on windows, since os.statvfs doesn't work

Comment 17

[Chris AtLee [:catlee]]

Attachment: python script to delete other build directories

Comment 18

[Chris AtLee [:catlee]]

Attachment: Create a MozillaBuildFactory base class that runs build cleanup steps

Comment 19

[[e-mail] (:bhearsum)]

Comment on attachment 352119 [details] [diff] [review]
Create a MozillaBuildFactory base class that runs build cleanup steps

>Index: factory.py
>===================================================================

>+class MozillaBuildFactory(BuildFactory):
>+    ignore_dirs = []

We should be ignoring the release directories - does it make sense to put that here and append others from the subclasses?

>+            self.addStep(ShellCommand,
>+             command=['bash', '-c', 'hg pull -u'],
>+             description=['update', 'build tools'],
>+             workdir='tools',
>+            )

No need to run this through bash, just ['hg', 'pull', '-u']

>-class UnittestBuildFactory(BuildFactory):
>-    def __init__(self, platform, config_repo_url, config_dir, branch, **kwargs):
>-        BuildFactory.__init__(self, **kwargs)
>-
>+class UnittestBuildFactory(MozillaBuildFactory):
>+    def __init__(self, platform, config_repo_url, config_dir, branch,
>+            buildToolsRepo, buildSpace, **kwargs):

style nit: indent the second line up to the start of the first lines arguments, eg:
def __init__(foo, bar,
             blah, crap)


This looks fine overall - I assume it's been running well in staging?

r-'ing because release directories aren't being excluded.

Comment 20

[[e-mail] (:bhearsum)]

Comment on attachment 351417 [details] [diff] [review]
Updates to mozilla2-staging to use the new purge_builds script

>diff -r 529a2eaf2c09 mozilla2-staging/config.py
>--- a/mozilla2-staging/config.py	Tue Dec 02 14:04:52 2008 -0500
>+++ b/mozilla2-staging/config.py	Thu Dec 04 14:48:56 2008 -0500
>+BRANCHES['mozilla-central']['platforms']['linux']['build_space'] = 5
>+BRANCHES['mozilla-central']['platforms']['linux64']['build_space'] = 5
>+BRANCHES['mozilla-central']['platforms']['win32']['build_space'] = 6
>+BRANCHES['mozilla-central']['platforms']['macosx']['build_space'] = 15

Now that we have DWARF support we can lower this to 10.

>+BRANCHES['mozilla-central']['platforms']['linux-debug']['build_space'] = 5
>+BRANCHES['mozilla-central']['platforms']['win32-debug']['build_space'] = 6
>+BRANCHES['mozilla-central']['platforms']['macosx-debug']['build_space'] = 15

These builds only use 1.5GB of space. Let's bump this to 2GB to be safe.

>diff -r 529a2eaf2c09 mozilla2-staging/release_master.py
>--- a/mozilla2-staging/release_master.py	Tue Dec 02 14:04:52 2008 -0500
>+++ b/mozilla2-staging/release_master.py	Thu Dec 04 14:48:56 2008 -0500

Now that we have BUILD_TOOL_REPO_URL can you switch the rest of this file to use it and drop the buildToolsRepo entry in release_config.py?

This is fine overall. r=bhearsum with the changes mentioned above.

Comment 21

[Chris AtLee [:catlee]]

Attachment: python script to delete other build directories

Comment 22

[Chris AtLee [:catlee]]

Attachment: Create a MozillaBuildFactory base class that runs build cleanup steps

Comment 23

[Chris AtLee [:catlee]]

Attachment: Updates to mozilla2-staging to use the new purge_builds script

Comment 24

[[e-mail] (:bhearsum)]

Comment on attachment 352181 [details] [diff] [review]
Create a MozillaBuildFactory base class that runs build cleanup steps

>Index: factory.py
>===================================================================
>+class MozillaBuildFactory(BuildFactory):
>+    ignore_dirs = [
>+            'info',
>+            'mac_build',
>+            'mac_verify',
>+            'mac_update',
>+            'linux_build',
>+            'linux_verify',
>+            'linux_update',
>+            'win32_build',
>+            'win32_verify',
>+            'win32_update',
>+            'l10n_verification',
>+            ]
> 

Some of these are wrong. You don't want win32_verify or win32_update - you want win32_update_verify. Also added: tag, source, updates, final_verification

>+class MercurialBuildFactory(MozillaBuildFactory):
>+    def __init__(self, env, objdir, platform, branch, sourceRepo, buildToolsRepo,
>+                 configRepo, configSubDir, profiledBuild, mozconfig,

'mozconfig' is probably a remnant from my staging patches - it's not currently an argument to this factory. Please remove it.


r=bhearsum with those changes.

Comment 25

[[e-mail] (:bhearsum)]

Comment on attachment 352182 [details] [diff] [review]
Updates to mozilla2-staging to use the new purge_builds script

>diff -r b76bbdd5af57 mozilla2-staging/config.py
>--- a/mozilla2-staging/config.py	Mon Dec 08 09:39:30 2008 -0500
>+++ b/mozilla2-staging/config.py	Tue Dec 09 16:03:07 2008 -0500
>@@ -9,16 +9,18 @@ STAGE_BASE_PATH = '/home/ftp/pub/firefox
> STAGE_BASE_PATH = '/home/ftp/pub/firefox'
> STAGE_GROUP = None
> STAGE_SSH_KEY = 'ffxbld_dsa'
> AUS2_USER = 'cltbld'
> AUS2_HOST = 'staging-stage.build.mozilla.org'
> DOWNLOAD_BASE_URL = 'ftp://ftp.mozilla.org/pub/mozilla.org/firefox'
> GRAPH_SERVER = 'graphs-stage.mozilla.org'
> GRAPH_SELECTOR = 'server'
>+BUILD_TOOLS_REPO_URL = 'http://hg.mozilla.org/users/bhearsum_mozilla.com/tools'

This should be http://hg.mozilla.org/build/tools. (Obviously you need to set it to this locally until the script is checked in there, though.)

>+BRANCHES['mozilla-1.9.1']['platforms']['macosx']['build_space'] = 10

We still need 15GB for this and tracemonkey - they don't have dwarf support yet.


> 
>-from buildbot.process import factory
>+from buildbotcustom.process import factory

Please import/reload this module.

r=bhearsum with those changes.

Comment 26

[Chris AtLee [:catlee]]

Attachment: Create a MozillaBuildFactory base class that runs build cleanup steps

Comment 27

[Chris AtLee [:catlee]]

Attachment: Updates to mozilla2-staging to use the new purge_builds script

Comment 28

[Chris AtLee [:catlee]]

Attachment: Create a MozillaBuildFactory base class that runs build cleanup steps

Updated against latest changes to factory.py

Comment 29

[Nick Thomas [:nthomas] (UTC+12)]

FWIW, Tracemonkey merged from m-c at Wed Dec 10 15:41:24 2008 PST, picking up Ted's dwarf fixes and the followups

Comment 30

[Chris AtLee [:catlee]]

Attachment: Updates to mozilla2, mozilla2-staging to use the new purge_builds script

This should apply cleanly on top of Ben's changes.

Comment 31

[[e-mail] (:bhearsum)]

Comment on attachment 352612 [details] [diff] [review]
Updates to mozilla2, mozilla2-staging to use the new purge_builds script

This is fine, I'll push it tomorrow along with my change.

Comment 32

[[e-mail] (:bhearsum)]

Comment on attachment 352589 [details] [diff] [review]
Create a MozillaBuildFactory base class that runs build cleanup steps

Checking in factory.py;
/cvsroot/mozilla/tools/buildbotcustom/process/factory.py,v  <--  factory.py
new revision: 1.59; previous revision: 1.58
done

Comment 33

[[e-mail] (:bhearsum)]

Comment on attachment 352179 [details] [diff] [review]
python script to delete other build directories

changeset:   31:9a6f40559579

Comment 34

[[e-mail] (:bhearsum)]

Comment on attachment 352612 [details] [diff] [review]
Updates to mozilla2, mozilla2-staging to use the new purge_builds script

changeset:   595:d9b530399c44

Comment 35

[[e-mail] (:bhearsum)]

Are we all done here?

Comment 36

[Chris AtLee [:catlee]]

Yup, done for now.  Possible future work would be notifying us when dependent build directories are getting deleted too often, which will slow them down considerably.

Comment 37

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Fix typo in ignore list for dir cleanup

I noticed a win32_update_verify dir getting removed, traced it back to 
  ... -n mac_buildwin32_update_verify ...
in the call to purge_builds.py, and to this missing comma.

new revision: 1.76; previous revision: 1.75