afaik, all of the linux vms mount stuff with the default ext2/ext3 settings. This means that we update atime everytime a file is accessed (even for reads). This is really not very useful for like 90% of the applications. I think the make in general depends on the mtime of files for compilation, so turning off atimes shouldn't impact that. In short, I'd like you folks to investigate changing the ext2/3 build mount points to defaults,noatime. This should give us some performance improvements and reduce load on the eql storage arrays.
I've updated the staging machines to have /, and /builds mounted with noatime. Let's see how it goes.
Also updating /, /var, and /builds on staging-master
How do these look so far? Any improvements on the cycle times?
(In reply to comment #3) > How do these look so far? Any improvements on the cycle times? Yes, and no. The following are based on data since april 1st. For l10n nightlies, our average build time decreased from 65 to 54 seconds. For debug builds, our average build time decreased from 2061 to 1905 seconds. For depend builds, our average build time increased from 2411 to 2900 seconds. For our nightly builds, our average build time increased from 3623 to 3999 seconds. I'm willing to ignore the increase in nightly build time, since there's not a lot of data there (only 6 builds before the change, and 6 after). The increase in depend build time is a bit confusing though. I'd like to keep this running in staging for a few more days to see if times improve. Another option would be to enable this on a few production slaves, and compare their timings to the timings of the slaves without it enabled.
Yeah, the increase in build times doesn't make sense.. A few more days of tests with machines that do not have the mount options enabled should give us some reasonable estimates.
Excluding failed builds from the data results in the following: L10n builds decreased from 81s to 66s Debug builds decreased from 2402s to 1857s (wow!) Depend builds decreased from 3028s to 2938s Nightly builds increased from 3623s to 3999s
Unittest builds go from 7768s to 6686s.
Looks like it's mostly improvements? The nightly build thing doesn't bother me as much, mostly because it's only once every night? These look like conclusive numbers to enable those options on all the Linux systems?
Haven't seen any significant change of the times now, with a few more days of data. In particular, the nightly builds still take around the same time. My suspicion is that the 6 builds we had data for from before the noatime change were abnormally fast for some reason. Also, the average time on production-master for nightly builds is 4313s. I recommend we go forward and enable this change on some or all of the production slaves.
All the production linux slaves have been updated (including moz2-linux-slave20-25). Let's let this bake a little while longer, then update the ref image.
I set noatime on try-linux-06 thru 09.
So far so good. Depend builds on linux now average 2314 seconds, nightlies average 3576, debug builds average 1706 and unittests average 6831. Let's update the ref image with this.
Ref image updated with noatime.
Status: NEW → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → FIXED
(In reply to comment #11) > I set noatime on try-linux-06 thru 09. Also did try-linux-slave01 thru 05 for uniformity across the slave pool.
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.