874039 - Implement noatime and nodiratime on all databases

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Sheeri Cabral [:sheeri]]

Definitely for MySQL; also for postgres?

Comment 1

[Brandon Johnson [:cyborgshadow]]

Should we expand on this? There's definitely other options we should be using for each specific file systems where we're using them.

Comment 2

[Sheeri Cabral [:sheeri]]

Indeed we can. I filed this as a placeholder so it would eventually get done. Feel free to list what options you recommend on the mysql/postgres datadirs.

Comment 3

[Brandon Johnson [:cyborgshadow]]

ext4:

errors=remount-ro,noatime,nodirtime,data=writeback,barrier=0,nobh,dioread_nolock 0 1

xfs:

nobarrier,noatime,nodirtime

Comment 5

[Sheeri Cabral [:sheeri]]

some notes from a previous bug:

e.g.:

    mount {
        "/":
            device => "/dev/mapper/VolGroup00-LogVol00",
            ensure => mounted,
            options => "rw,noatime",
            remounts => true,
    }

Jakem does this via puppet for hbase servers, so when the time is right, we can talk to him.

Comment 6

[Sheeri Cabral [:sheeri]]

putting [2012q3] as a whiteboard because we should make this part of the MySQL 5.6 upgrades.

Comment 7

[Brandon Johnson [:cyborgshadow]]

Working on these now. fwiw, I'm doing them all manually as I move through, and I don't see a way to improve upon it currently. I tried to puppetize it and already lost half a day failing. 

What jakem does for the mount {} in puppet comes from the puppet module nfsclient and is designed exclusively for new nfs mounts. The problem we have on our systems is that the majority of db servers are writing to the / mount.

Comment 8

[Brandon Johnson [:cyborgshadow]]

I also had to removed the data=writeback option because we're on the root filesystem. New mount parameters look like this for ext4:

ext4 root partition:
errors=remount-ro,noatime,nodirtime,barrier=0,dioread_nolock 1 1

ext4 non-root data partition:
errors=remount-ro,noatime,nodirtime,data=writeback,barrier=0,dioread_nolock 0 1

Comment 9

[Brandon Johnson [:cyborgshadow]]

noatime and nodirtime are set in puppet. Some of the servers have been rebooted as per bug 917859. 

Tree closing servers will be rebooted in the maintenance window on 10/12 per bug 917856. 

Non-tree closing servers that weren't rebooted will be rebooted during their next mysql upgrade.

Comment 10

[Sheeri Cabral [:sheeri]]

tl;dr - do we actually need to reboot the servers?

Committed revision 76239 adding dev in scl3 and phx servers.

Interesting point:

####################
All the dev servers (all 4, dev1/2 in scl3 and phx1) have the datadir on / and are using /dev/sda3 and ext4. For example:

[[e-mail]3 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             218G   11G  197G   6% /
tmpfs                  12G     0   12G   0% /dev/shm
/dev/sda1              97M   77M   16M  84% /boot
[[e-mail]3 ~]# mount  | head -1
/dev/sda3 on / type ext4 (rw)
[[e-mail]3 ~]#

####################
It looks like puppet remounts the directory when it runs (note the "scheduling refresh of mount":
Info: Applying configuration version '76239'
Notice: /Stage[main]//Node[dev12.db.scl3.mozilla.com]/Mount[/]/device: device changed 'UUID=5945e272-4b11-4985-b40e-4818ce7567f0' to '/dev/sda3'
Notice: /Stage[main]//Node[dev12.db.scl3.mozilla.com]/Mount[/]/options: options changed 'defaults' to 'errors=remount-ro,noatime,nodiratime,barrier=0'
Notice: /Stage[main]//Node[dev12.db.scl3.mozilla.com]/Mount[/]/pass: pass changed '1' to '0'
Notice: /Stage[main]//Node[dev12.db.scl3.mozilla.com]/Mount[/]/dump: dump changed '1' to '0'
Info: FileBucket adding {md5}cd4e3d902836a686b22be2b4d75359e5
Info: /Stage[main]//Node[dev12.db.scl3.mozilla.com]/Mount[/]: Scheduling refresh of Mount[/]
Info: /Stage[main]//Node[dev12.db.scl3.mozilla.com]/Mount[/]: Scheduling refresh of Mount[/]
Info: /Stage[main]//Node[dev12.db.scl3.mozilla.com]/Mount[/]: Scheduling refresh of Mount[/]
Info: /Stage[main]//Node[dev12.db.scl3.mozilla.com]/Mount[/]: Scheduling refresh of Mount[/]
Info: Mount[/](provider=parsed): Remounting
Notice: /Stage[main]//Node[dev12.db.scl3.mozilla.com]/Mount[/]: Triggered 'refresh' from 4 events
Info: /Stage[main]//Node[dev12.db.scl3.mozilla.com]/Mount[/]: Scheduling refresh of Mount[/]

####################
And, here's the result afterwards.
[[e-mail]3 ~]# mount  | head -1
/dev/sda3 on / type ext4 (rw,noatime,nodiratime,errors=remount-ro,barrier=0)

####################
Given this information, do we actually need to reboot the servers?

Comment 11

[Sheeri Cabral [:sheeri]]

Committed revision 76245 for metrics dmz, scalebase, jenkins and paas mounts.

Comment 12

[Sheeri Cabral [:sheeri]]

Committed revision 76249 for jenkins mount.

Comment 13

[Sheeri Cabral [:sheeri]]

intranet2.stage.db.corp.phx1.mozilla.com uses ext3; any recommendations? Can I use the same as ext4?

Comment 14

[Brandon Johnson [:cyborgshadow]]

No reboot necessary per comment 10, and ext3 is equivalent to ext4 in these mount parameters.

Comment 15

[Sheeri Cabral [:sheeri]]

Committed revision 76253 for developer2 mount.
committed revision 76265 for devtools2, generic2 in scl3 and puppetagain mounts.
revision 76273 for webdev2.

Comment 16

[Sheeri Cabral [:sheeri]]

revision 76281 for stage scl3 mounts. (this was at the end of yesterday, I'm not playing with mount points on a Friday!)

Comment 17

[Sheeri Cabral [:sheeri]]

Saturday work (in lieu of rebooting)
revision 76360 "adding mountpoints for tbpl and sentry"
revision 76363 "adding mountpoints for generic1,2,3 in phx1 and generic1 in scl3"
revision 76366 "adding mountpoints for buildbot and builder-addons"
revision 76368 "updating sumo for mount points"
revision 76369 "setting mount point for metrics"
revision 76370 "adding mount points for bugzilla stage VMs and datazilla"
revision 76371 "adding mounts for devtools1, developer1, intranet1/2.db.corp.phx1"
revision 76372 "adding mount points for sumotools and bugzilla in scl3"
revision 76373 "adding mount points for intranet and getpersonas (all scl3)"
revision 76374 "adding mount points for itutils and bouncer (phx)"
revision 76375 "adding mount points for plugins and snippets"

Comment 18

[Sheeri Cabral [:sheeri]]

r76639 bugzilla stage
r76641 addons stage
r76650 intranet corp stage
r76651 engagement
r76654 engagement stage (Seamicros) - this was tricky because each one has its own mount, I used a case statement though.
r76655 addons versioncheck
r76657 mysql1.stage.metrics

Comment 19

[Sheeri Cabral [:sheeri]]

r76659 puppetagain servers

This is all in the spreadsheet, but everything has noatime and nodiratime except:

marketplace (not under our puppet control)
marketplace payments (not under our puppet control)
identity db's (not under our puppet control)
backup servers (will do that at the end)
tooldev1 (not puppetized for mysql)

Comment 20

[Sheeri Cabral [:sheeri]]

Added mount point for /dev/cciss/c0d0p1 on backup1 in phx revision 77059 and 77060. Ran puppet, the remount happened with no problems once I got the kinks worked out (had to make it before mysql2::backups, not before mysql2::server, since mysql2::server isn't called, and the datadir is in /data).

Added mount points & fixes for the rest of the servers in revisions 77066, 77069-77071.

Comment 21

[Sheeri Cabral [:sheeri]]

backup servers have been done; closing this as there's nothing more that we manage from our side.