I think we have at least two problems with slow planet updates. One, sometimes planet is taking forever to update (an hour or more) and two, the html page sometimes lags the feed. Two could be because of a cache in front of the html page, but I'm concerned about one the most. Is this something we can look into or do we need IT support?
Planet updates every 15 min... if you see a delay, it's because the netscaler is caching content. I can force index.html not to cache, but I don't believe I can force non-caching on the feed files without some config work on the netscaler.
Justin, can you advise here? Planet's updating every 15 minutes but people often aren't getting that result in the feed or the html page. I'd like both to be a lot closer to 15 minutes than they currently seem to be. Can we just not use the netscaler on planet?
Am I the only one who thinks updating every couple hours is more than fast enough if faster updates required fiddling with cache behavior?
I don't think a couple of hours is fast enough. There's lots of time-sensitive content that gets out via Mozilla blogs and the planet aggregator. Before filing this bug I was going to file one asking for the time to be bumped to 5 minutes :-)
Stop me if you've heard this one: explicitly invalidate when planet changes? Or set the netscaler to revalidate every 5 mins -- surely planet can handle a hit every 5 mins, even if it's running on Justin's iPhone. :)
Planet is generating static html iirc. Planet doesn't get anywhere near enough traffic to cause trouble on it's own in the past few years that I'm aware of. I've included tons of images that point directly to my server and have never felt a burden. I can understand the point of being behind netscaler just in case of digg/slashdot... but I don't see why it would be problem to drop to <= 5 min. It's not that it's pulling in all the feeds every pageview, or even hitting a database.
however, there's enough feeds on it now that it takes close to five minutes (maybe more) to pull and aggregate them all every time the cron job fires. The Netscaler honors whatever Apache tells it though. All you need is a little love in the .htaccess file or the vhost config for mod_expires.
Is it pulling serially, or are there just some slow ones in there?
(In reply to comment #8) > Is it pulling serially, or are there just some slow ones in there? Both... It pulls one at a time, but there are also some slow ones. It caches every post it gets, so it's just looking for new content every run. There are 207 blogs on planet, so it's not unexpected that it takes a while to visit all of them.
That reminds me When you get a moment, can you put me back on the cron spam so I can see when stuff is craping out? Seem to have fallen off sometime early this year.
If anyone still needs my OK - I am fine changing this to 5 min, especially if it's a static file. As dave said this can be done with an htaccess file.
Sending venus/config.ini Adding venus/venus/themes/mozilla/.htaccess Sending venus/venus/themes/mozilla/config.ini Transmitting file data ... Committed revision 13141. Cache of planet.m.o now expires every 5 min. and the time to process all 207 blogs has been divided by 5. Cronjob is still running every 15 min., so if this works out ok, we can change that to something less like 5 or 10 min.