The MySQL settings are still being tweaked, but for Google's benefit this is what I've done so far:
- /tmp was formatted as mkfs.xfs -l size=64m /dev/sdb1, and mounted with rw,auto,nouser,async,dev,exec,noatime,nodiratime,logbufs=8 as the mount options.
- The RAID-10 was formatted with mkfs.xfs -l size=128m -d agcount=90 /dev/sdc1 and the same mount options as above.
- I've changed /proc/sys/vm/swappiness (also found in vm.swappiness in sysctl.conf) to 15 from its default of 60. This value is subject to adjustment as needed.
- I've set all the disks to use the deadline I/O scheduler. A convenient way to turn that on on a running system is echo "deadline" > /sys/block/(dev)/queue/scheduler. Add elevator=deadline to your kernel's boot parameters to do it on startup.
- renice -20 your mysql server process.
- Finally, remember kids: if at all possible, adjust all your MySQL settings to avoid swap if at all possible. Hitting the swap unnecessarily will send your load through the roof and generally make things slower.