486765 - Change linux mounts for the build filesystems to mount with noatime

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Aravind Gottipati [:aravind]]

afaik, all of the linux vms mount stuff with the default ext2/ext3 settings.  This means that we update atime everytime a file is accessed (even for reads).  This is really not very useful for like 90% of the applications.  I think the make in general depends on the mtime of files for compilation, so turning off atimes shouldn't impact that.

In short, I'd like you folks to investigate changing the ext2/3 build mount points to defaults,noatime.  This should give us some performance improvements and reduce load on the eql storage arrays.

Comment 1

[Chris AtLee [:catlee]]

I've updated the staging machines to have /, and /builds mounted with noatime.  Let's see how it goes.

Comment 2

[Chris AtLee [:catlee]]

Also updating /, /var, and /builds on staging-master

Comment 3

[Aravind Gottipati [:aravind]]

How do these look so far?  Any improvements on the cycle times?

Comment 4

[Chris AtLee [:catlee]]

(In reply to comment #3)
> How do these look so far?  Any improvements on the cycle times?

Yes, and no.  The following are based on data since april 1st.

For l10n nightlies, our average build time decreased from 65 to 54 seconds.
For debug builds, our average build time decreased from 2061 to 1905 seconds.

For depend builds, our average build time increased from 2411 to 2900 seconds.
For our nightly builds, our average build time increased from 3623 to 3999 seconds.

I'm willing to ignore the increase in nightly build time, since there's not a lot of data there (only 6 builds before the change, and 6 after).

The increase in depend build time is a bit confusing though.  I'd like to keep this running in staging for a few more days to see if times improve.

Another option would be to enable this on a few production slaves, and compare their timings to the timings of the slaves without it enabled.

Comment 5

[Aravind Gottipati [:aravind]]

Yeah, the increase in build times doesn't make sense.. A few more days of tests with machines that do not have the mount options enabled should give us some reasonable estimates.

Comment 6

[Chris AtLee [:catlee]]

Excluding failed builds from the data results in the following:
L10n builds decreased from 81s to 66s
Debug builds decreased from 2402s to 1857s (wow!)
Depend builds decreased from 3028s to 2938s
Nightly builds increased from 3623s to 3999s

Comment 7

[Chris AtLee [:catlee]]

Unittest builds go from 7768s to 6686s.

Comment 8

[Aravind Gottipati [:aravind]]

Looks like it's mostly improvements?  The nightly build thing doesn't bother me as much, mostly because it's only once every night?  These look like conclusive numbers to enable those options on all the Linux systems?

Comment 9

[Chris AtLee [:catlee]]

Haven't seen any significant change of the times now, with a few more days of data.

In particular, the nightly builds still take around the same time.  My suspicion is that the 6 builds we had data for from before the noatime change were abnormally fast for some reason.

Also, the average time on production-master for nightly builds is 4313s.

I recommend we go forward and enable this change on some or all of the production slaves.

Comment 10

[Chris AtLee [:catlee]]

All the production linux slaves have been updated (including moz2-linux-slave20-25).

Let's let this bake a little while longer, then update the ref image.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

I set noatime on try-linux-06 thru 09.

Comment 12

[Chris AtLee [:catlee]]

So far so good.

Depend builds on linux now average 2314 seconds, nightlies average 3576, debug builds average 1706 and unittests average 6831.

Let's update the ref image with this.

Comment 13

[Chris AtLee [:catlee]]

Ref image updated with noatime.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #11)
> I set noatime on try-linux-06 thru 09.

Also did try-linux-slave01 thru 05 for uniformity across the slave pool.