While most people stay with their families and relax on Christmas Eve (well, the holiday season), cyber attackers often take advantage of this time to prove their existence.
On this Christmas Eve, a client who has a busy site hosting forum and blog which has an average of 55,000 unique visitors daily, came to us for urgent help regarding mitigating DDoS attacks. We at LiteSpeed worked extra late hours to set up the LiteSpeed web server with advanced anti-DDoS service for him. The attacks were thus defended against and our client’s site has gone through the Christmas weekend with no issues. Here is the story.
The site setup is as follows:
Intel Dual Xeon E5520 Nehalem (8 cores)
2 x 146 GB SAS 15K in RAID 1
Linux CentOS (64 bit)
MySQL 5.0.77 — Database
Nginx 0.7.66 — Front-end proxy and serving static contents
Apache 2.2.14 — Upstream web server serving dynamic content, only listening on internal IP
WordPress 3.0.3 (cache plug-in not enabled due to conflict with another plug-in.)
The site was unable to sustain under this round of DDoS attacks.
Admittedly, the server had a pretty nice customized LAMP setup: Nginx as content accelerator sits in front of Apache – which is often recommended by experienced webmasters as the most optimized LAMP setup, and it certainly did help, and survived two previous round of DDoS attacks.
However, this round of DDoS attacks brought the server to its knees. The DDoS was a typical HTTP GET attack from botnet with about one thousand bots, sending GET requests to hit the blog landing page which in turn triggered db access. The site admin had to manually block attackers’ IPs, which is very inefficient. Nevertheless It could not help this situation: CPU load average was around 150-250 and it took 30-60 seconds to load a page with a browser, while memory usage was about 93% (out of 12GB). The site owner got very anxious and came to LiteSpeed asking for help.
LiteSpeed to the rescue
On Christmas Eve, LiteSpeed support staff got this installation ticket and access to that server. Our staff began immediately working on it. While under the attack, the server was too slow to work with; remote SSH access constantly got dropped, downloading files took a long time, and compiling PHP took forever. Eventually, we got permission from site owner to shutdown Apache temporarily to speed things up.
In order to make its built-in anti-DDoS feature work as designed (http://www.litespeedtech.com/how-tos.html#qa_dos), LiteSpeed had to take the front line, replacing Nginx and Apache all together. LiteSpeed is able to detect the IPs (bot) that abuse servers and drop all the connections from detected bots. Once setup, more than a thousand of IPs from botnets were detected and blacklisted in a few minutes by LiteSpeed. The server load dropped to 40-50 and memory usage was down to 50%. The site was responsive again, but it still took 10 seconds to load a page. This was because although LiteSpeed was able to immediately drop the connections from detected IPs, the detected bots were still able to reach the server, and therefore still consumed a considerable amount of system resources.
Fortunately the site owner also ordered our Advanced Anti-DDoS service at the same time, which can intelligently detect the bot IPs and blacklist them at the firewall level. Finally, web pages could then be loaded in 2~5 seconds — the web site was usable again!
The server was able to throughput: 4.3MB/sec with 4300+ requests/sec (out of 100Mbps uplink).
2-3 hours later: 1548 IPs were blocked and the server cpu load average was down to less than 2.0. Page load time was ~1-2 seconds.
The server was back to normal and ran through Christmas weekend without any worries.
Keep in mind that we have not yet enabled any cache mechanism at the application level (vBulletin and WordPress) — that’s another interesting topic we’ll talk about in our future blogs.
LiteSpeed can defend servers from DDoS (Distributed DoS attack) quite efficiently. In this case, we see that LiteSpeed successfully replaced Nginx + Apache and shutdown attacks through its built-in Anti-DDoS feature along with LiteSpeed’s advanced Anti-DDoS service. Compared to long hours of work with limited progress under a heavily loaded server, LiteSpeed’s solution made a quick turnaround and quieted down the attacks in just a few hours – now that is impressive! Obviously, LiteSpeed achieved something that a site owner dreams about. So how do you go through a holiday season without worrying about attacks on your websites? With LiteSpeed, you can surely relax with peace of mind!