LiteSpeed against thc-ssl-dos

October 26th, 2011

thc-ssl-dos script was published two days ago, and it has been posted all-over the internet. You may wonder if LiteSpeed is vulnerable to this attack. The short answer is

NO

in our lab, we tried it against LSWS 4.1.6 release.

thc-ssl-dos-1.4> src/thc-ssl-dos [target_ip] 443 --accept
     ______________ ___  _________
     \__    ___/   |   \ \_   ___ \
       |    | /    ~    \/    \  \/
       |    | \    Y    /\     \____
       |____|  \___|_  /  \______  /
                     \/          \/

http://www.thc.org

          Twitter @hackerschoice

Greetingz: the french underground

Waiting for script kiddies to piss off................
The force is with those who read the source...
Handshakes 0 [0.00 h/s], 1 Conn, 0 Err
Handshakes 0 [0.00 h/s], 181 Conn, 0 Err
Handshakes 0 [0.00 h/s], 181 Conn, 0 Err
Handshakes 0 [0.00 h/s], 181 Conn, 0 Err
Handshakes 0 [0.00 h/s], 231 Conn, 0 Err
Handshakes 0 [0.00 h/s], 231 Conn, 0 Err
Handshakes 0 [0.00 h/s], 231 Conn, 0 Err
Handshakes 0 [0.00 h/s], 231 Conn, 0 Err
Handshakes 0 [0.00 h/s], 231 Conn, 0 Err
Handshakes 0 [0.00 h/s], 231 Conn, 0 Err
Handshakes 0 [0.00 h/s], 231 Conn, 0 Err
Handshakes 0 [0.00 h/s], 231 Conn, 0 Err
Handshakes 0 [0.00 h/s], 231 Conn, 0 Err
Handshakes 0 [0.00 h/s], 233 Conn, 0 Err
Handshakes 0 [0.00 h/s], 233 Conn, 0 Err
Handshakes 0 [0.00 h/s], 233 Conn, 0 Err
Handshakes 0 [0.00 h/s], 233 Conn, 0 Err
Handshakes 0 [0.00 h/s], 233 Conn, 0 Err
Handshakes 0 [0.00 h/s], 233 Conn, 0 Err
Handshakes 0 [0.00 h/s], 233 Conn, 0 Err
Handshakes 0 [0.00 h/s], 233 Conn, 0 Err
Handshakes 0 [0.00 h/s], 233 Conn, 0 Err
ERROR: Target has disabled renegotiations.
Use your own skills to modify the source to test/attack
the target [hint: TCP reconnect for every handshake].

Can we block it? you may wonder.

YES

Just set "Connection soft Limit" and "Connection hard limit" under "Server"->"Security"->"Per client throttling", we set soft limit to 20, hard limit to 30, and did the test again.

 

thc-ssl-dos-1.4> src/thc-ssl-dos [target_ip] 443 --accept
     ______________ ___  _________
     \__    ___/   |   \ \_   ___ \
       |    | /    ~    \/    \  \/
       |    | \    Y    /\     \____
       |____|  \___|_  /  \______  /
                     \/          \/

http://www.thc.org

          Twitter @hackerschoice

Greetingz: the french underground

Waiting for script kiddies to piss off................
The force is with those who read the source...
Handshakes 0 [0.00 h/s], 1 Conn, 0 Err
SSL: error:00000000:lib(0):func(0):reason(0)
SSL: error:00000000:lib(0):func(0):reason(0)
SSL: error:00000000:lib(0):func(0):reason(0)
SSL: error:00000000:lib(0):func(0):reason(0)
SSL: error:00000000:lib(0):func(0):reason(0)

LiteSpeed immediately blacklisted the IP and rejected SSL connections from that IP. Actually, normal HTTP connections will be rejected as well.

How about the form of attack in their private release, which repeatedly reestablish new HTTPS connections instead of using SSL negotiation?

We have an answer as well, Stay tuned!

To be continued…

Speed up vBulletin sites through LiteSpeed Built-in Cache

January 28th, 2011

LiteSpeed has provided a built-in cache functionality (an advanced feature for Enterprise 2+CPU licenses) since version 4.0 (March, 2009), and keeps improving this feature through subsequent 4.0.x releases. Many of you maybe still not aware of the powerfulness of this feature.  This article will show a sample usage for the popular vBulletin  forum with detailed steps, and you are welcome to post your feedback on your real-life exciting results.

LiteSpeed cache has similar features as Apache mod_cache but implemented in a more efficient way, and works like Varnish. It is an output cache for dynamic contents, so the usage does not limit to PHP pages.  Unlike  Varnish, it is built into the web server, reducing one layer of reverse proxy. Hence for static contents it is more efficient without one extra layer.  The uniqueness of LiteSpeed cache is that you can use rewrite rules (either in configuration files or .htaccess) to control its behavior, which gives you maximum flexibility.

Sample Use Case for vBulletin forums (vB 3.8.x or vB 4.0.x)

Here we give one example of how to use LiteSpeed cache with rewrite rules to speed up vBulletin forums. We would like to cache the pages for guest users, which generate the majority of the traffic.

1. Install vBulletin plug-in – Boost Product XML “product-boostv1.xml” from vBulletin admin panel – manage plugin.

You can download from here. It is a good article elaborating about why output cache is essential for high traffic sites.  The purpose of this plug-in is to determine if the user is a guest user or a logged in member. You do not need boost.php, that’s for Varnish only.

2. Enable the cache function. You have to go to the LiteSpeed admin console to set this up, mod_cache config for “CacheRoot” will be ignored.

a. Go to admin console->Server->Cache

b. Cache Storage Settings->Storage Path: set to a fast local disk, for e.g.: /tmp/diskcache (you can also set it to a ramdisk if you have enough extra memory to speed up further)

c. Manually create the directory from shell, and give permission for litespeed/lshttpd processes (assume running as "nobody" here):

#mkdir /tmp/diskcache
#chown nobody:nobody /tmp/diskcache
#chmod 700 /tmp/diskcache

3. Set vhost Cache policy

a. For natively configured vhost (for e.g. myforum) in admin console:

i.  Go to admin console->Virtual Hosts->myforum->Cache

ii.  Disable the cache globally (will be enabled through rewrite rules under condition)

Enable Cache:No

iii. Set cache policy

Cache Request with Query String:Yes
Cache Request with Cookie:Yes
Cache Response with Cookie:Yes
Ignore Request Cache-Control:Yes
Ignore Response Cache-Control:Yes

b. For imported Apache vhost:

There is no need to use mod_cache directive to control cache. You do not need to do anything for the vhost cache policy. It will assume the same default settings as the above natively configured vhost. However, you do need to set the server level Cache storage from the Litespeed web admin console stated in step 2; and put rewrite rules in the .htaccess file as in step 4.

4. Add rewrite rules to cache guest pages for 2 minutes

Assuming vBulletin is installed at $DOCUMENT_ROOT/forum

## following redirect will reduce cached files:
## otherwise domain.com/abc.php and www.domain.com/abc.php will create 2 cached files.
RewriteCond %{HTTP_HOST} !^www\.domain\.com$
RewriteRule /forum/(.*)$ http://www.domain.com/forum/$1 [R=301,L]

## select which pages to cache
RewriteCond %{HTTP_COOKIE} !bbimloggedin=yes
RewriteCond %{HTTP_COOKIE} !bbuserid=
RewriteCond %{HTTP_COOKIE} !bbpassword=
RewriteCond %{REQUEST_METHOD} ^HEAD|GET$
RewriteCond %{QUERY_STRING} !s=[a-fA-F0-9]{32}
RewriteCond %{QUERY_STRING} !product=vbnexus
RewriteCond %{REQUEST_URI} !^/forum/(login|register|usercp|private|profile|cron|image) \.php$
RewriteCond %{REQUEST_URI} !^/forum/admincp
RewriteRule /forum/(.*\.php)?$ – [L,E=Cache-Control:max-age=120]

You can add the rewrite rules in either of the 3 locations:

  • .htaccess  (adding a line at beginning: “RewriteEngine on”)
  • Apache config file vhost section
  • Litespeed natively configured vhost: admin console->Virtual Hosts->myforum->Rewrite

5.  Apply changes by restarting LiteSpeed

6.  Optional:  Set a cron job to clean up cache directory /tmp/diskcache

*/10 * * * * root find /tmp/diskcache -type f -mmin +8 -delete 2>/dev/null

Every 10 minutes, delete cached files that are more than 8 minutes old. Since we set the cache expire time as 120 seconds (2 minutes),  it's safe to delete those old files.

7. Verify if pages are served from the cache

Since 4.0.19, LiteSpeed will output a response header “X-LiteSpeed-Cache hit” if the content is served from its internal cache.

8. Optional: If you want to log cache hit data for future analytics, you can use the environment variable “CACHE_HIT=1” (introduced in 4.0.19) in your in customized access log with format string “%{CACHE_HIT}e”.

For example: in the httpd.conf vhost section, add

CustomLog /usr/local/apache/domlogs/domamin.com-cache_log "%h %t \"%r\" %>s %b %{CACHE_HIT}e"

Reference:

  1. How does the output cache work? vBulletion forum
  2. Big boost: related posts at our forum.

LiteSpeed beats Nginx + Apache over DDoS attack

January 5th, 2011

While most people stay with their families and relax on Christmas Eve (well, the holiday season), cyber attackers often take advantage of this time to prove their existence.

On this Christmas Eve, a client who has a busy site hosting forum and blog which has an average of 55,000 unique visitors daily, came to us for urgent help regarding mitigating DDoS attacks. We at LiteSpeed worked extra late hours to set up the LiteSpeed web server with advanced anti-DDoS service for him. The attacks were thus defended against and our client’s site has gone through the Christmas weekend with no issues. Here is the story.

The site setup is as follows:

Hardware:
———————–
Intel Dual Xeon E5520 Nehalem (8 cores)
12GB RAM
2 x 146 GB SAS 15K in RAID 1
100Mbps Uplink

Software
———————-
Linux CentOS (64 bit)
MySQL 5.0.77 — Database
Nginx 0.7.66 — Front-end proxy and serving static contents
Apache 2.2.14 — Upstream web server serving dynamic content, only listening on internal IP
vBulletin 4.0
WordPress 3.0.3 (cache plug-in not enabled due to conflict with another plug-in.)

The site was unable to sustain under this round of DDoS attacks.

Admittedly, the server had a pretty nice customized LAMP setup: Nginx as content accelerator sits in front of Apache – which is often recommended by experienced webmasters as the most optimized LAMP setup, and it certainly did help, and survived two previous round of DDoS attacks.

However, this round of DDoS attacks brought the server to its knees. The DDoS was a typical HTTP GET attack from botnet with about one thousand bots, sending GET requests to hit the blog landing page which in turn triggered db access. The site admin had to manually block attackers’ IPs, which is very inefficient. Nevertheless It could not help this situation: CPU load average was around 150-250 and it took 30-60 seconds to load a page with a browser, while memory usage was about 93% (out of 12GB). The site owner got very anxious and came to LiteSpeed asking for help.

LiteSpeed to the rescue

On Christmas Eve, LiteSpeed support staff got this installation ticket and access to that server. Our staff began immediately working on it. While under the attack, the server was too slow to work with; remote SSH access constantly got dropped, downloading files took a long time, and compiling PHP took forever. Eventually, we got permission from site owner to shutdown Apache temporarily to speed things up.

In order to make its built-in anti-DDoS feature work as designed (http://www.litespeedtech.com/how-tos.html#qa_dos), LiteSpeed had to take the front line, replacing Nginx and Apache all together. LiteSpeed is able to detect the IPs (bot) that abuse servers and drop all the connections from detected bots. Once setup, more than a thousand of IPs from botnets were detected and blacklisted in a few minutes by LiteSpeed. The server load dropped to 40-50 and memory usage was down to 50%. The site was responsive again, but it still took 10 seconds to load a page. This was because although LiteSpeed was able to immediately drop the connections from detected IPs, the detected bots were still able to reach the server, and therefore still consumed a considerable amount of system resources.

Fortunately the site owner also ordered our Advanced Anti-DDoS service at the same time, which can intelligently detect the bot IPs and blacklist them at the firewall level. Finally, web pages could then be loaded in 2~5 seconds — the web site was usable again!

The server was able to throughput: 4.3MB/sec with 4300+ requests/sec (out of 100Mbps uplink).

2-3 hours later: 1548 IPs were blocked and the server cpu load average was down to less than 2.0. Page load time was ~1-2 seconds.

The server was back to normal and ran through Christmas weekend without any worries.

Keep in mind that we have not yet enabled any cache mechanism at the application level (vBulletin and WordPress) — that’s another interesting topic we’ll talk about in our future blogs.

Conclusion:

LiteSpeed can defend servers from DDoS (Distributed DoS attack) quite efficiently. In this case, we see that LiteSpeed successfully replaced Nginx + Apache and shutdown attacks through its built-in Anti-DDoS feature along with LiteSpeed’s advanced Anti-DDoS service. Compared to long hours of work with limited progress under a heavily loaded server, LiteSpeed’s solution made a quick turnaround and quieted down the attacks in just a few hours – now that is impressive! Obviously, LiteSpeed achieved something that a site owner dreams about. So how do you go through a holiday season without worrying about attacks on your websites? With LiteSpeed, you can surely relax with peace of mind!

LiteSpeed, Nginx, Lighttpd, Apache Show Down

February 3rd, 2010

After we posted recent benchmark test results, we got some interests and challenges. On our side, we can only say that we just posted untwisted results under our test environment.

One WebHostingTalk user offered his server and opened a topic there. Mistwang spent time and tested LSWS 4.1RC2 and gave full results. However, no full results on other web servers. So it ends up nowhere. If you are experienced in configuring other web servers, you are welcome to continue the test.

Benchmarking takes a lot of time and is hard to get comprehensive. We’ll do more benchmark tests after we release LSWS 4.1.

This year we will also have major feature enhancement on LSLB, since more and more people started using it and requested new features.

Benchmark Comparison on Serving Simple PHP: LiteSpeed vs Apache vs Nginx

January 6th, 2010

To perform a fair benchmark requiring a lot of preparation, as you need to know the detail of each product and set appropriate configuration. Recently, we got chance to do some internal testing. It’s experimental, however the result is good enough to make some points and may be useful to our users, so we would like to share it in a series of blog posts first. Feel free to provide any feedback, so we can improve in our formal benchmark test later on.

This time we will compare web server performance on serving simple hello.php. Comparing simple php files is standard practice for benchmark, as this can reveal web server performance instead of php engine being the bottleneck. We will compare LiteSpeed, Apache and Nginx with different configurations.

Software version:  LiteSpeed 4.1RC2, LSAPI 5.0b2, PHP 5.3.0, Apache 2.2.14, Nginx 0.7.61.

Test results:

Configuration Non-Keep-Alive Keep-Alive
Server Mode Average of 4 tests Rating Average of 4 tests Rating
LiteSpeed LSAPI 1 CPU 4741.78 305 6488.52 326
Apache mod_php worker 1554.69 100 1992.34 100
event 1736.26 112 2217.04 111
prefork 1844.85 119 2501.67 126
Nginx 1 worker 1213.67 78 1226.19 62
Apache mod_fcgid worker 1012.43 65 990.87 50
event 970.53 62 973.11 49
prefork 939.67 60 942.61 47
Apache mod_fastcgi worker 832.76 54 825.22 41
event 792.64 51 779.92 39
prefork 795.51 51 793.18 40
Apache suPHP worker 89.63 6 90.09 5

Conclusion
LiteSpeed + LSAPI apparently have big advantage over other configurations. LiteSpeed performs more than 2 times faster than Apache mod_php, which is best configuration for Apache setup. There is little performance impact for LiteSpeed to run in suEXEC mode; therefore it can achieve security benefit of suPHP without losing performance while CGI-based suPHP is the biggest loser in this test.

There are almost no differences between keep-alive and non-keep-alive for Nginx, mod_fcgid, mod_fastcgi and suphp.

Apache mod_php is second to LiteSpeed: about 52% (non-keep-alive) / 103% (keep-alive) faster than Nginx, about 83% (non-keep-alive) / 152% (keep-alive) faster than mod_fcgid, about 120% (non-keep-alive) / 207% (keep-alive) faster than mod_fastcgi, 18 (non-keep-alive) / 22 (keep-alive) times faster than suPHP.

Apache prefork performs mod_php better than worker and event modes; Apache worker performs mod_fcgid and mod_fastcgi just a little better than event and prefork modes.

Apache mod_fcgid is about 20% faster than mod_fastcgi; either mod_fcgid or mod_fastcgi lose keep-alive ability that mod_php has.

Read the rest of this entry »

Benchmark Comparison on Serving Small Static Files: LiteSpeed vs Apache vs Nginx

December 29th, 2009

To perform a fair benchmark requiring a lot of preparation, as you need to know the detail of each product and set appropriate configuration. Recently, we got chance to do some internal testing. It’s experimental, however the result is good enough to make some points and may be useful to our users, so we would like to share it in a series of blog posts first. Feel free to provide any feedback, so we can improve on our formal benchmark tests later on.

This time we will compare web server performance on serving small static files. Comparing small files is standard practice for benchmark, as this can reveal web server performance instead of network being saturated by large files first. We will compare LiteSpeed, Apache and Nginx with different configurations on serving small static files.

Software version: LiteSpeed 4.1RC2, Apache 2.2.14, Nginx 0.7.61.

Test Result

Round Apache
(worker)
Apache
(event)
Apache
(prefork)
Nginx
(1 worker)
LiteSpeed
(1 CPU)
Non-Keep-Alive
1 2570.63 3192.56 3342.65 6616.52 9217.25
2 2555.7 3150.34 3275.96 7098.36 9474.72
3 2562.29 3159.37 3309.69 9609.46 9526.48
4 2534.73 3181.72 3392.23 6525.97 9550.62
Average 2555.84 3171.00 3330.13 7462.58 9442.27
Rating 100 124 130 292 369
Keep-Alive
1 4957.55 6571.35 6668.33 13862.73 27487.78
2 4609.48 6482.42 6595.71 13729.05 28333.26
3 4321.93 6709.53 6655.88 12319.22 27872.33
4 4384.76 6323.04 6633.4 13123.33 28788.51
Average 4568.43 6521.58 6638.33 13258.58 28120.47
Rating 100 143 145 290 616

Conclusion

LiteSpeed is apparently a big winner in this test. Apache worker and event model does not perform better than prefork model in serving small files. We are also surprised with LiteSpeed performance over Nginx, especially on keep-alive test.

Read the rest of this entry »

LiteSpeed + LSAPI vs. Apache + Passenger vs. Nginx + Passenger

December 16th, 2009

To perform a fair benchmark requiring a lot of preparation, as you need to know the detail of each product and set appropriate configuration. Recently, we got chance to do some internal testing. It’s experimental, as all tests were run within 1 box. We bind LSWS to 1 CPU, and leave another CPU to ab. The result here is good enough to make some points and may be useful to our users, so we would like to share in a series of blog posts first. Feel free to provide any feedback, so we can improve in our formal benchmark test later on.

In our last test, we studied the memory consumption of Ruby vs Ruby Enterprise Edition using Typo as a test case. Since Typo itself is a bottleneck when cache is disabled, performance among different web servers cannot be revealed.

This naturally leads to our next interested topic “Comparing the performance of the interface between web server and Ruby engine”.

This time, we focus on performance of the interface between web server and Ruby engine by using the simplest “Hello World” without database. We tested 3 configurations, all based on Ruby Enterprise 1.8.7: LiteSpeed/4.1RC2 + REE, Apache/2.2.14 (worker MPM) + REE + Phusion Passenger and Nginx/0.7.61  + REE + Phusion Passenger.

Test method is still based on http://www.rubyenterpriseedition.com/comparisons.html

Software Version: LSWS 4.1RC2, Nginx/0.7.61, Apache/2.2.14(Worker MPM), Ruby Enterprise Edition 1.8.7, Rails 2.2, Passenger 2.2.5

Test Result

Configuration Req/Sec Memory Usage
LiteSpeed + REE + LSAPI 280 146M ~ 158M
Nginx + REE + Passenger 212 153M
Apache(worker MPM) + REE + Passenger 200 178M

Conclusion

Based on above simple tests, LiteSpeed + REE + LSAPI are the preferred configuration.

Read the rest of this entry »

Litespeed + Ruby Enterprise Edition

December 9th, 2009

We haven’t done benchmarks since LSWS 2.0 due to various reasons… People may discount what we say because this is a vendor version… The main reason is that we are really busy with other high-priority things, like new feature enhancement, bug fixes and support… We treat every user’s feedback seriously. So to answer those people who request new benchmark, YES, we know it is OVERDUE, and it is on our TODO list after 4.1 stable released.

To perform a fair benchmark requiring a lot of preparation, as you need to know the detail of each product and set appropriate configuration. Recently, we got chance to do some internal testing. It’s experimental, as all tests were run within 1 box. We bind LSWS to 1 CPU, and leave another CPU to ab. The result here is good enough to make some points and may be useful to our users, so we would like to share in a series of blog posts first. Feel free to provide any feedback, so we can improve on our formal benchmark test later on.

In official site of Ruby Enterprise Edition, it compares REE with LiteSpeed 3.3.11 standard version (4-25-2008 released). According to the comparison, LiteSpeed is highest on memory usage. That test was flawed because each Ruby process occupies a lot of memory, and the default setting in LiteSpeed spawns 10 Ruby processes which are higher than that in other test cases. So the memory usage graph is basically illustrated how much memory Ruby processes used. Also in order to reduce the memory in real-time, the default setting also tries to destroy Ruby process whenever it is idle for a short time. This is not good for benchmark testing, as the server has to spend extra CPU in creating/destroying Ruby processes as this is not real-world scenario.

However, what we are interested here is “How about LiteSpeed + Ruby Enterprise Edition?

Test method is based on http://www.rubyenterpriseedition.com/comparisons.html

1. REE Saves Memory

LiteSpeed + Ruby Enterprise 165M
LiteSpeed + regular Ruby 264M
Memory Saving with REE (264 – 165) / 264 = 37.5%

This is close to “33% less memory on average” claimed by REE web site.

2. REE’s Copy-on-write Works with LiteSpeed

In order to test this, we manually enable/disable copy-on-write function by modifying lsws/fcgi-bin/RailsRunner.rb:

GC.copy_on_write_friendly = true|false

LiteSpeed + REE with Copy-on-Write 165M
LiteSpeed + REE without Copy-on-Write 234M
Memory Saving with Copy-on-Write (234 – 165) / 234 = 29~30%

Conclusion

Based on above simple tests, we recommend using REE with LiteSpeed.

Next, we’ll touch on another interesting topic “LiteSpeed vs Apache + Passenger vs Nginx + Passenger”.

Read the rest of this entry »

We did go to Hostingcon 09

August 28th, 2009

This is a delayed post. Yes, we did go to Hostingcon in DC this year. We first attended it on 07 at Chicago, when LSWS 3.0 released. Last year, we didn’t make it. This year, since in DC so close, we drove down directly.

Because of the conference, we had to delay our 4.0.10 release for a few days. We did not join the last evening session and went back directly to continue working on those 503 errors. Finally, after serveral round of build and test run on end users’ environment, 4.0.10 seems quite stable for now. We feel a little bit relief.

It was great pleasure that we were able to meet some LiteSpeed users there, all are very sharp, young and energetic guys. It’s nice feeling to work together with smart users. Well, I have to say all LiteSpeed users are smart …, they know that by using LiteSpeed actually save a lot for them, not only on support, maintenance, but also worries and sleepless night.

National Harbor has very good facility, still under development. While in DC, most places will have a name with “national”. We did not go around this time, since most places in DC we are pretty familiar. We just went out dining once at a local Thai restaurant, which kind of compensated all other junk food meals. Talking about food, George like Thai food a lot, of course, the best one we like still Chinese food, not American Chinese food, but genuine Chinese food. That’s why we already got our tickets to “Just EAT” on Sept. 21 in NYC – International Chinese Culinary Competition. You can watch them cook and taste at same time. If anyone in metropolitan area like authentic Chinese food, come and join us for fun!

Back to work, we will finish up a few to-dos for 4.0.x, and then will fully focus on 4.1RC2 release. Yes, new features will be added for RC2, but I’ll leave that for later post.

Going to switch this blog to wordpress

July 29th, 2009

Now it’s time to be more active on this blog site and put more useful stuff here, we decide to switch to wordpress.

WordPress is easier and more popular than current RoR based Typo. Besides that, wordpress.org and wordpress.com still use LiteSpeed.

One funny site if you need a short break: icanhascheezburger.com

If you like gadgets, you may also like slashgear.com.

They are both LiteSpeed powered wordpress based sites. Some people may get confused by server header, you can read more info here. I guess this may due to no free version of LSLB.