Change linux mounts for the build filesystems to mount with noatime

RESOLVED FIXED

Status

P3
normal
RESOLVED FIXED
10 years ago
5 years ago

People

(Reporter: aravind, Assigned: catlee)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

10 years ago
afaik, all of the linux vms mount stuff with the default ext2/ext3 settings.  This means that we update atime everytime a file is accessed (even for reads).  This is really not very useful for like 90% of the applications.  I think the make in general depends on the mtime of files for compilation, so turning off atimes shouldn't impact that.

In short, I'd like you folks to investigate changing the ext2/3 build mount points to defaults,noatime.  This should give us some performance improvements and reduce load on the eql storage arrays.
(Assignee)

Updated

10 years ago
Assignee: nobody → catlee
Priority: -- → P3
(Assignee)

Comment 1

10 years ago
I've updated the staging machines to have /, and /builds mounted with noatime.  Let's see how it goes.
(Assignee)

Comment 2

10 years ago
Also updating /, /var, and /builds on staging-master
(Reporter)

Comment 3

10 years ago
How do these look so far?  Any improvements on the cycle times?
(Assignee)

Comment 4

10 years ago
(In reply to comment #3)
> How do these look so far?  Any improvements on the cycle times?

Yes, and no.  The following are based on data since april 1st.

For l10n nightlies, our average build time decreased from 65 to 54 seconds.
For debug builds, our average build time decreased from 2061 to 1905 seconds.

For depend builds, our average build time increased from 2411 to 2900 seconds.
For our nightly builds, our average build time increased from 3623 to 3999 seconds.

I'm willing to ignore the increase in nightly build time, since there's not a lot of data there (only 6 builds before the change, and 6 after).

The increase in depend build time is a bit confusing though.  I'd like to keep this running in staging for a few more days to see if times improve.

Another option would be to enable this on a few production slaves, and compare their timings to the timings of the slaves without it enabled.
(Reporter)

Comment 5

10 years ago
Yeah, the increase in build times doesn't make sense.. A few more days of tests with machines that do not have the mount options enabled should give us some reasonable estimates.
(Assignee)

Comment 6

10 years ago
Excluding failed builds from the data results in the following:
L10n builds decreased from 81s to 66s
Debug builds decreased from 2402s to 1857s (wow!)
Depend builds decreased from 3028s to 2938s
Nightly builds increased from 3623s to 3999s
(Assignee)

Comment 7

10 years ago
Unittest builds go from 7768s to 6686s.
(Reporter)

Comment 8

10 years ago
Looks like it's mostly improvements?  The nightly build thing doesn't bother me as much, mostly because it's only once every night?  These look like conclusive numbers to enable those options on all the Linux systems?
(Assignee)

Comment 9

10 years ago
Haven't seen any significant change of the times now, with a few more days of data.

In particular, the nightly builds still take around the same time.  My suspicion is that the 6 builds we had data for from before the noatime change were abnormally fast for some reason.

Also, the average time on production-master for nightly builds is 4313s.

I recommend we go forward and enable this change on some or all of the production slaves.
(Assignee)

Comment 10

10 years ago
All the production linux slaves have been updated (including moz2-linux-slave20-25).

Let's let this bake a little while longer, then update the ref image.
I set noatime on try-linux-06 thru 09.
(Assignee)

Comment 12

10 years ago
So far so good.

Depend builds on linux now average 2314 seconds, nightlies average 3576, debug builds average 1706 and unittests average 6831.

Let's update the ref image with this.
(Assignee)

Comment 13

10 years ago
Ref image updated with noatime.
Status: NEW → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → FIXED
(In reply to comment #11)
> I set noatime on try-linux-06 thru 09.

Also did try-linux-slave01 thru 05 for uniformity across the slave pool.
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.