Thread: Looking for Caching Software

I am working on a project where I need to cache static HTML copies of dynamic pages. I need to dramatically cut down on requests from a remote database.

It needs to runs on Linux and I need to be able to set a lengthy timeout, such as 8 to 12 hours.

A PHP solution is preferred.

Does anyone have any recommendations for a freebie software option that is easy to learn and install?

I'm looking for a recommendation for a system that someone here has used. Source Forge is a good source for free software, but a lot of it is crap with long learning curves. Unfortunately, I don't really have the time to test a half-dozen caching systems.

There are a lot of caching options out there, but I am really looking for one that is easy to use and works well.

For WordPress I use WP Super Cache which is easy to use and works well.

For my own code such as where I might be sourcing an RSS feed, I find it fairly easy to code a cache. It entails saving the feed to a text file and also saving a time stamp value for the last update (which could be got from the attributes of the saved file) or to the database. Then the script compares the stored time to the current time to check if the feed should be refreshed.

Writing this time stamp to a text file is not much of a time burden since MySQL uses text files to store data on a server anyway.

If you had a custom script developed, I don't think it is much work to get the cache feature added i.e. it shouldn't cost much extra to pay your programmer to implement it.

Sorry, I don't know of any specific software solutions for this. I think that the admin code would be far more complex than a custom, hard-coded solution.

This is not a WordPress project and I am the programmer.

This is an existing dynamic PHP site that uses the .html extension for the pages. We need to cut down on the requests from a remote XML database. The data is not in a local database. Caching for extended time periods appears to be the best option.

Thanks for the help, but I am really looking for a recommendation from someone who has already been down this path.

Originally Posted by TopDogger

We need to cut down on the requests from a remote XML database. The data is not in a local database.

Are you sure that page level caching is the right solution? If you're using a decent XML database, it should have something like MySQL's Query Caching built in. Which XLM database are you using? As examples, both Sedna and Ozone offer caching, although the implementation details are beyond my skill level.

Alternatively, you could also set up Squid in front of your web servers. It's the oldest and most respected caching proxy in existence. I've used Squid and I've competed against it in support of a commercial product -- it's good stuff.

It is not a local database, so I have no control over database caching. I'm trying to reduce requests from Amazon's live XML database.

Pulling all of the data into a MySQL database is not a good option, because there are millions of products and prices can change often. Plus, I don't want to slow the site down with a behemoth database.

It appears that page caching is the best method for reducing requests. For example, a popular page might request the same data 500 times in a day, but only once or twice if I cache the page for 12 hour durations.

I have heard of Squid, but have not looked into it. Is it an option for my scenario?

Originally Posted by TopDogger

It is not a local database, so I have no control over database caching. I'm trying to reduce requests from Amazon's live XML database.

You can implement caching when calling a remote database -- that's actually when it makes the most sense. But... the database client has to support it.

I don't know jack about Amazon's system. What is your interface to it?

Originally Posted by TopDogger

Pulling all of the data into a MySQL database is not a good option, because there are millions of products and prices can change often. Plus, I don't want to slow the site down with a behemoth database.

Oh yeah, definitely agree.

Originally Posted by TopDogger

It appears that page caching is the best method for reducing requests. For example, a popular page might request the same data 500 times in a day, but only once or twice if I cache the page for 12 hour durations.

I have heard of Squid, but have not looked into it. Is it an option for my scenario?

Squid will sit in front of the web servers and serve pages for the web servers. Because the web servers will never see the page requests, they will not make requests to the database server. Of course, that can screw with your stats.

Originally Posted by Will.Spencer

You can implement caching when calling a remote database -- that's actually when it makes the most sense. But... the database client has to support it.

I don't know jack about Amazon's system. What is your interface to it?

I use REST to retrieve an XML file with Amazon.

I suspect Amazon does not support caching. It is hard to get any answers out of them. They advise against using caching because they want the data displayed to be fresh. However, their XML DB servers can slow down considerably during the Christmas season and it frustrates visitors to affiliate sites. I'm trying to speed up my sites for the holidays, which means I need something I can set up quickly.

Originally Posted by Will.Spencer

Squid will sit in front of the web servers and serve pages for the web servers. Because the web servers will never see the page requests, they will not make requests to the database server. Of course, that can screw with your stats.

It sounds like you are talking about clustering software. Is that how squid works?

Can that be used with my VPS server?

Are you using it yourself?

Originally Posted by TopDogger

I use REST to retrieve an XML file with Amazon.

Ahhh... now it makes sense.

RESTWiki has bad things to say about caching support in Amazon's implementation of REST.

Here's someone caching Amazon REST requests in PERL.

Originally Posted by TopDogger

It sounds like you are talking about clustering software. Is that how squid works?

Squid isn't a cluster, it's more of a reverse proxy when used as a proxy cache.

Originally Posted by TopDogger

Can that be used with my VPS server?

I think so, although I have only ever used dedicated Squid servers.

Originally Posted by TopDogger

Are you using it yourself?

I have in the past, but am not currently. I was using Squid for a very different purpose back then. We were trying to optimize bandwidth utilization over limited WAN circuits.