Troubleshooting for High Load php-Driven Site

Hello, everyone,

Time pasts so fast, it has been over 2 years since last blog post. Now we are back and will use this blog regularly. It’s good way to communicate with each other, as you, LiteSpeed users, may also wonder what we are doing.

Our daily life at LiteSpeed is very busy. Trouble shooting problematic server is a routine: if it is LiteSpeed related, we usually will fix it in a day or two and have a pre-release build for a particular client; many times it is not LiteSpeed bug, but configuration/tuning related.

We would like to post here and hopefully benefit others. Back to today’s story, one client report urgently:
> Server Load is very high

> Server configration : Opteron 8350 16 cores , 8gb ram , 300gb SA-SCSI 15.000 RPM .

> Look at top command output : Cpu idle %80 and MYSQL cpu very low %10 because database is a INNODB and very stable optimized.

> But load averega too high. and I can see Litespeed admin console — Realtime Stats too many EAWaitQ process (300-500 )

Logged to that server, did the following steps

Top shows at the moment Server load is 75.
Check what php is busy with.
- ps -ef | grep php
- find a php process with long CPU, for e.g. 2767.
- strace -p 2767
- from output, it is spending most of time cleaning up session files.
Look at php session files, under /tmp/sess_*, too many cannot be displayed by ls directly.
- find . | grep sess_ | wc, return count > 190K
- check df, enough space left on file system.
- Check phpinfo output, session auto start is off, no problem.
- Move session path to tempfs:create a directory under /dev /shm/phpsess
  modify .htaccess under public_html/ of the problematic client, add php saved path /dev/shm/phpsess
In a few minutes, got session write error.
- Hit the limit of files that can be created under one directory of tempfs
- use 2 directory level of php session files. (see details here)
In a few minutes, got error again: “No space left on device”.
- df -k , only use 1% of file system
- df -i, usage is 100%, used up all the inode.
- Increase the inode limit:
  mount -o remount,nr_inodes=1G /dev/shm
  modify /etc/fstab mount, so next time server reboot will keep the change.
Server load is around 3, no more errors.