[wp-hackers] should (can?) wp-cache be adopted into the core?

Wed Apr 18 15:22:47 GMT 2007

We were handling decent on 4 cores, we just upgraded to 8 because of a great
deal we got, plus we have 2 more sites going online (1 WP, 1 Drupal) that
will most likely push us to over 1,000,000 hits a day (these are all
political sites and as next year's elections get closer, the hits
skyrocket). 

We also have a lot of media stuff that gets downloaded (generally 2
terabytes a week), so that adds a lot into things.

The site we use HS on never really causes problems. The biggest problem
comes from the site that has all the WP commenting on (which gets about 1/2
the hits).

We are running EAccelerator on it and that has helped out a lot. We keep it
so the cache only expires once every 12 hours. I just go through and
manually dump it if needed (through the included web interface). We are also
using MySql's caching mechanisms. 

Jamie Holly
http://www.intoxination.net

> -----Original Message-----
> From: wp-hackers-bounces at lists.automattic.com [mailto:wp-hackers-
> bounces at lists.automattic.com] On Behalf Of Computer Guru
> Sent: Wednesday, April 18, 2007 11:03 AM
> To: wp-hackers at lists.automattic.com
> Subject: RE: [wp-hackers] should (can?) wp-cache be adopted into the
> core?
> 
> Jamie, that's really weird.
> 
> For the month of March, I was averaging 375k hits a day on my one
> server, on
> a single-core P4 2.8GHz, with about 260k of those going to WP, the rest
> to
> various DB-intensive things like Gallery2 and the forum.
> (http://neosmart.net/ && http://neosmart.net/blog/)
> 
> This is just a single server, tons of plugins, MySQL, IIS, URL
> Rewriters, no
> AJAX, etc.
> 
> Like I said in my previous email, I'm not using any SQL-caching
> techniques.
> Disable unneeded/unused PHP extensions (or if on *nix, recompile
> without
> them). Are you using APC/eAccelerator?
> 
> You shouldn't need 8 cores for < 400k hits/day, esp. seeing as you're
> offloading comments to HS.
> 
> If you want to discuss this further, feel free to hit me off-list.
> 
> Computer Guru
> NeoSmart Technologies
> http://neosmart.net/blog/
> 
> 
> > -----Original Message-----
> > From: wp-hackers-bounces at lists.automattic.com [mailto:wp-hackers-
> > bounces at lists.automattic.com] On Behalf Of Jamie Holly
> > Sent: Wednesday, April 18, 2007 4:33 PM
> > To: wp-hackers at lists.automattic.com
> > Subject: RE: [wp-hackers] should (can?) wp-cache be adopted into the
> > core?
> >
> > I agree 1000%. I do the tech work on two sites. One is averaging
> > 250,000
> > hits a day and the other about 120,000 hits a day. Without running
> wp-
> > cache,
> > our 8 core server dies. We also aren't running that many plugins, and
> > the
> > ones we are have been run through the mill on profiling and not
> causing
> > any
> > significant load. We also have a small shell script that runs, which
> > downloads the front page and generates an index.html for it to keep
> > loads
> > down (which is a must for us), as well as using feedburner to handle
> > the
> > additional 100,000+ feed subscribers for both sites.
> >
> > One thing I think would help out a lot is to re-examine the filters.
> > The
> > larger blog, we are using Haloscan. The smaller one is using WP-
> > Comments,
> > and that one gets anywhere from 200-600 comments per post (with 8-10
> > posts
> > per day). On that site I changed the comments so they run off AJAX to
> > help
> > with the load. I have a refresh comments script that reloads the
> newer
> > comments (which does not load the wp-core, but rather contains all
> the
> > filter functions within the single file and generates the output). It
> > also
> > does comment posting via AJAX, but I still load the core in that to
> > handle
> > posting better.
> >
> > With all that said, it seems that we might be wasting a lot on
> comment
> > filtering. For every comment on every page we look at these filters
> > being
> > loaded:
> >
> > add_filter('comment_author', 'wptexturize');
> > add_filter('comment_author', 'convert_chars');
> > add_filter('comment_author', 'wp_specialchars');
> >
> > add_filter('comment_email', 'antispambot');
> >
> > add_filter('comment_flood_filter', 'wp_throttle_comment_flood', 10,
> 3);
> >
> > add_filter('comment_url', 'clean_url');
> >
> > add_filter('comment_text', 'convert_chars');
> > add_filter('comment_text', 'make_clickable', 9);
> > add_filter('comment_text', 'force_balance_tags', 25);
> > add_filter('comment_text', 'wpautop', 30);
> > add_filter('comment_text', 'convert_smilies', 20);
> >
> > I have been toying with ideas in my mind to call those before the
> > comment is
> > saved and then don't call it when the page is viewed. The big problem
> > is
> > that we have close to 1,000,000 comments in the database right now,
> so
> > I
> > need to come up with a way for Wordpress to tell if the comment has
> > been
> > filtered (possible Boolean added to each comment - is_filtered), and
> > then
> > using a cron job to filter out X amount of comments every Y minutes
> > until
> > everything is caught up.
> >
> > I did see a ticket awhile back for the same filter rearrangement to
> > take
> > place on posts also and it seemed to be a very popular idea.
> >
> > Something else I saw Matt mention in this thread was making WP more
> > modular
> > to that it only loads what is needed versus the entire core. This is
> > something else that can help out a lot and I would like to see plugin
> > authors take the same approach. Some plugins require a lot of coding
> > for the
> > admin code then very little for the actual page generation code.
> Having
> > PHP
> > load and compile that admin code on every page view is a waste. It
> > would be
> > better to move plugins to their own directory and do something as
> > simple as
> > having the main plugin file load then if the user is on an admin
> page,
> > load
> > the file containing all admin related code.
> >
> >
> >
> > Jamie Holly
> > http://www.intoxination.net
> > > -----Original Message-----
> > > From: wp-hackers-bounces at lists.automattic.com [mailto:wp-hackers-
> > > bounces at lists.automattic.com] On Behalf Of Viper007Bond
> > > Sent: Wednesday, April 18, 2007 8:37 AM
> > > To: wp-hackers at lists.automattic.com
> > > Subject: Re: [wp-hackers] should (can?) wp-cache be adopted into
> the
> > > core?
> > >
> > > Couldn't agree more. I'd love to see a output caching system built
> in
> > > (or
> > > even better, as a bundled plugin to make it modular). While
> WordPress
> > > can
> > > run great under most circumstances, some blogs are just too dang
> > > popular to
> > > run without a super computer. Having something that is essentially
> > > core, but
> > > easily loadable / unloadable (a plugin) would solve this problem
> and
> > > make
> > > WordPress even more feasible / desirable for large bloggers.
> > >
> > > Case in point: I have a WP powered site that gets around a million
> > hits
> > > a
> > > month. While I know I could run it without WP-Cache, I'd rather put
> > the
> > > CPU
> > > to better use (like handling the site's forums which get around
> > another
> > > million or two hits a month). Not having to hack WP-Cache to work
> and
> > > knowing for sure it'd support future versions would be a godsend.
> > >
> > > On 4/17/07, Matt Mullenweg <m at mullenweg.com> wrote:
> > > >
> > > > David Chait wrote:
> > > > > Opinions?  Matt?  (since I think staticize was your baby
> > > originally,
> > > > no?)
> > > >
> > > > Having a built-in output cache could be very powerful for the
> > growing
> > > > number of WordPress users who are climbing to the top of the
> > > blogosphere.
> > > >
> > > > There are a lot of issues to work out, as have been mentioned on
> > this
> > > > thread, so this is a good example of a project that needs a
> > champion
> > > to
> > > > take ownership of the issue and coordinate what needs to happen
> to
> > > get
> > > > the plugin or functionality into a release like 2.3 or 2.4.
> > > >
> > > > --
> > > > Matt Mullenweg
> > > >   http://photomatt.net | http://wordpress.org
> > > > http://automattic.com | http://akismet.com
> > > > _______________________________________________
> > > > wp-hackers mailing list
> > > > wp-hackers at lists.automattic.com
> > > > http://lists.automattic.com/mailman/listinfo/wp-hackers
> > > >
> > >
> > >
> > >
> > > --
> > > Viper007Bond | http://www.viper007bond.com/
> > > _______________________________________________
> > > wp-hackers mailing list
> > > wp-hackers at lists.automattic.com
> > > http://lists.automattic.com/mailman/listinfo/wp-hackers
> >
> > _______________________________________________
> > wp-hackers mailing list
> > wp-hackers at lists.automattic.com
> > http://lists.automattic.com/mailman/listinfo/wp-hackers
> 
> _______________________________________________
> wp-hackers mailing list
> wp-hackers at lists.automattic.com
> http://lists.automattic.com/mailman/listinfo/wp-hackers