PHP Troubleshooting: Too Many Sessions Causes Bottleneck

A client came to us with server loads of 200-400 during peak times. (He has a 64 core 32 GB server and a 4-CPU LSWS license.) Detailed observation revealed that PHP processes were eating up his processing power. But why? What were they doing? Taking out the trash.

PHP processes can perform a task called “session garbage collection.” Essentially, garbage collection searches through session files for files that are no longer being used and removes those files. By default, PHP processes are set with a 1% probability of performing garbage collection every time they run. Normally, this isn’t a problem. But our client has a very busy site with a lot of PHP processes. During peak times, his server serves over 1000 requests per second, with over 100 new PHP sessions created per second. These session files are all saved under one directory in the /dev/shm partition (memory-based file system). With a session time out of 24 minutes, that session directory soon became monstrous. (When I first came in, it had over 170,000 files.) So each time a PHP process performed garbage collection, it had to search through all 170,000 files. Any operation on that directory was going to be sluggish, and with so many PHP processes running, there were always more than a few performing garbage collection. Processes were bumping into each other, getting locked and unlocked in the middle of tasks, and sometimes finding that the files they had just decided to delete had now already been deleted by another process. And all this time these processes were not getting to the tasks they were designed to run.

Once I figured out the problem, the solution was simple. I created an extra layer of 16 directories (/0 – /f) below the session directory. I then set session.save_path = “1;/dev/shm/phpsess” in php.ini. As explained in the PHP manual, setting that first numerical argument to “1” shuts off automatic garbage collection and specifies that session files should be spread around in one directory level below /dev/shm/phpsess (those new directories I just made). I then set up a cron job (*/5 * * * * root find /dev/shm/phpsess/ -type f -mmin +25 -delete 2>/dev/null in /etc/crontab) to clear these directories every five minutes. Now the PHP processes are doing what they are supposed to, the session files are in more manageably-sized directories, and the trash is being taken out. The server has around 20-30 peak load with 80% idle CPU.