Dealing with a really large database and keeping your web-based service scalable can be quite a challenge.
One thing you might want to consider in order to improve performance is to prevent Unix from updating atime
(last access time) at every read/write.
This is done by adding the noatime
flag when mounting the file system associated with the database and/or files being accessed alot.
Adding this flag will give you a tremendous boost in read and write capacity, so why not give it a try and see what happens?
Reference: Building Scalable Web Sites by Cal Henderson.