MM: Cache Warmup for Large Stores

Welcome to another installment of Magento Monday!
Today’s Topic: LiteMage Cache Warmup for Large Stores

Disclaimer: The information contained in this post is accurate for LiteMage v1.3.7 [release log] and Magento 1.x. If you are using a newer version of the extension, some details may have changed. Please refer to our wiki for the latest!

LiteMage for Magento 1.x is equipped with a crawler function. The crawler travels its way throughout the backend, refreshing pages in the cache. The purpose is to keep the cache as fresh as possible while minimizing visitor exposure to uncached content.

Our wiki instructions for setting up the LiteMage crawler are detailed and extensive, and if you’ve never enabled your site’s crawler before, we recommend you start there. If you own a large store, you may find the crawler’s duration to be problematic, so today we’re going to discuss how to configure the crawler for stores with many thousands of products.

What works for small-to-medium sized stores may not work for larger stores. If you have thousands of pages for the crawler to traverse, you may need to adjust the crawler’s settings to make it less intrusive into your store’s daily operation.

How Long Does Your Crawler Take?

Before you can begin to determine the optimal configuration for your store’s crawler, you will need to know how long it actually takes to complete.

You can find that out, along with other useful crawler statistics, in the LiteMage Crawler Warm-up Status section. From the Magento Admin Panel, navigate to System > Cache Management. It should be at the bottom of that page.

An Example Scenario

• Shop configuration: A book shop with 15 thousand products
• Public Cache TTL (Time to Live): 86400 (24 hours)
• Crawler Warmup Interval: 57600 (16 hours)
• Crawler Duration: 8 hours

Fifteen thousand product pages are set to expire in the cache after 24 hours, so every 16 hours the crawler begins refreshing the pages. This process takes about 8 hours to complete, and consumes enough resources to slow the site down considerably.

Crawling the site takes 8 hours, and it runs every 16 hours. Essentially half of the time that this store is up, it is experiencing crawler-induced lag.

This is not ideal. However, you don’t want to give up the warm cache, so what can you do?

A Possible Solution

LiteMage’s crawler function is flexible, and with a few adjustments, you can tune it to work well for your store.

The first thing to consider is the TTL. How often do your products change? If it’s less frequently than “every day,” you don’t need a TTL of 24 hours. How about a week? If it’s reasonable to assume that your products are not likely to change within seven days, you can set your TTL to 604800 seconds (1 week).

Once your TTL is changed, you can adjust your warmup interval. We have a handy rule-of-thumb for that:

Warmup Interval <= Public Cache TTL – 2 X Warmup Duration

Let’s say the TTL is set to 604800 (7 days).

We know our warmup duration is about 8 hours, or ⅓ day.

The Warmup Interval we choose should be less than or equal to 7 days minus 2 times ⅓ day

In other words, the Warmup interval should be less than 6 ⅓ days.

Let’s set the Warmup Interval to 6 days (or 518400 seconds). This means that instead of your system slowing down every day for 8 hours, it will slow down every six days for 8 hours. That’s a big improvement.

You can also enable the delta crawler. The delta crawler will allow you to confidently set a one-week TTL, knowing that if a product changes during that time, it will be recrawled within 5 minutes.

In Conclusion

As you can see, with the LiteMage crawler, you have options. If your products are fairly static, you might even choose to set a one month TTL, or longer!

If you have any questions about this, leave a comment. We’ll be happy to help!

P.S. Perhaps your large store could benefit from our ADC solution. Check out LiteSpeed Web ADC for details.

—

Have some of your own ideas for future Magento Monday topics? Leave us a comment!

If you enjoyed this, here are a few other things you can do:

• Subscribe to the Magento Monday RSS feed
• Download LiteMage Cache for Magento 1
• Download LiteMage Cache for Magento 2 from the GitHub Repository
• Learn more about the plugin on our website