[l008com]

If I was goona run a popular forum like this, and a vBulletin one at that. Where would you say the server load would be? Is it split between apache and mysql, or does apache do very little and mysql is where most of the server load goes?
I'm just trying to plan ahead for a site of mine.

[Arkham_c]

I'd guess that Apache will be a relatively small part of the load. Rendering the pages is pretty quick once you have the data together.

[Camelot]

Originally posted by Arkham_c:
I'd guess that Apache will be a relatively small part of the load. Rendering the pages is pretty quick once you have the data together.

Not so. The work is all in the web server.

Any decent BBS system is going to use caches to minimize database impact so only writes (new posts, message counts, etc.) and idle threads (that aren't already in the cache) are going to hit the database.

The web server, on the other hand, has to parse the page template, work out what data to insert and where, while accepting connections and serving existing users. If written properly, most of the data it needs to build most pages is already in server RAM.

I'm running a site with a 6:1 ratio of front-end web servers to back-end databases servers, and they're all running with similar load averages. Admittedly, it's not vBulletin, so I can't speak for how well vB handles caching.

[MacOSR]

Originally posted by Camelot:
Not so. The work is all in the web server.

Any decent BBS system is going to use caches to minimize database impact so only writes (new posts, message counts, etc.) and idle threads (that aren't already in the cache) are going to hit the database.

The web server, on the other hand, has to parse the page template, work out what data to insert and where, while accepting connections and serving existing users. If written properly, most of the data it needs to build most pages is already in server RAM.

I'm running a site with a 6:1 ratio of front-end web servers to back-end databases servers, and they're all running with similar load averages. Admittedly, it's not vBulletin, so I can't speak for how well vB handles caching.

What kind of page views are you seeing???? We're serving up ~30,000,000 page views from a two node cluster. The machines are hardly doing anything and it is HEAVY database and PHP.

Also, what are you using for the back-end database and are you running a cluster? We are getting ready to put into place a new database cluster (we run MySQL). We have two Dell 2600's each with dual 3.06 Xeon, 6Gb RAM, U320 Internal RAID and U160 shared storage. We are heavy in our writes (1 write for every 3 reads) so we are only gonig to have writes go to the cluster and the cluster will replicate to other servers for reads.

Our current server averages 229 questions per second and during peak times it sees around 2000 questions per second.

However, a couple of our concers are split-brained syndrome and when a node fails what is in the cache? We are going to put into place three heart-beat methods so we should be fine with the split-brained syndrome and we are also going to setup the cluster to power cycle a node when it fails to avoid problems with the cache.

What are your experiences?

[l008com]

I run a 400 MHz G3 server that does it all, including a vB with no users yet, and mysql. It gets about 1000+ page views a day and averages at about 5% CPU unless I do something like do a traffic stats run.

[Camelot]

Originally posted by MacOSR:
What kind of page views are you seeing???? We're serving up ~30,000,000 page views from a two node cluster. The machines are hardly doing anything and it is HEAVY database and PHP.

Also, what are you using for the back-end database and are you running a cluster? We are getting ready to put into place a new database cluster (we run MySQL). We have two Dell 2600's each with dual 3.06 Xeon, 6Gb RAM, U320 Internal RAID and U160 shared storage. We are heavy in our writes (1 write for every 3 reads) so we are only gonig to have writes go to the cluster and the cluster will replicate to other servers for reads.

Our current server averages 229 questions per second and during peak times it sees around 2000 questions per second.

However, a couple of our concers are split-brained syndrome and when a node fails what is in the cache? We are going to put into place three heart-beat methods so we should be fine with the split-brained syndrome and we are also going to setup the cluster to power cycle a node when it fails to avoid problems with the cache.

What are your experiences?

I don't know the actual page counts we run - we run sites for other companies and the counts vary by client. I don't deal with the clients, just make sure the servers are running. If I had to guess I'd say 200 million page views a month.

This is all based on Sun hardware (typically quad-processor boxes with 4GB ram each for the front-end servers, and quad or 8-way processor systems for the back-end using three-way replication (so there's always 2 current copies of each live database).

Everything we do is JSP (java) based, I don't know if that's helping my CPU load or not. I've never worked with PHP to know how it compares, but typically my systems are running under 20% CPU load. Some of our clients are very spiky and we factor in a 4x overhead for that. We can't afford to have client A go slow just because client b is having a busy day.

I do question your database traffic, though. Our database servers are peaking under 1,000 queries/second. I know we did a lot of tweaking in the code to cache updates. For example things like message view counts are held on the app server and updated to the database every 15 seconds rather than making every page view generating an update in the database to reflect the message's count. You might want to look at your code to see if you can do anything there.
This might be where JSP has an edge over PHP. JSP runs in a servlet engine which can maintain state. Does PHP do the same, or is each page essentially disctinct from each other page (meaning two users who use the same page will generate two sets of identical lookups in the database)?
In our setup, each servlet engine has as much as 1GB of RAM for caching content from the database. It makes a huge difference.

As for clustering, synchronization and cache coherence is a major issue when trying to spread the load over two or more servers. Most of our setups are in a primary/standby model where all traffic goes to the primary box unless it's down in which case everything fails over to the standby box (typically with a 5 to 15 second failover). In the cases where this isn't possible, we use a lot of smart load balancing to hint different forums transparently towards different servers. This means that all users browsing forum 1 are on server 1 while those browsing forum 2 are on server 2. In that way server 1 caches everything in forum 1 and doesn't need to worry about anything in forum 2. This prevents two users browsing the same forum from seeing different content because the servers' caches aren't in sync.
In a failover situation, the standby machine's caches are empty, so there's a brief period of slowness while the data is loaded from the database, but it soon passes.

[MacOSR]

Originally posted by Camelot:
I don't know the actual page counts we run - we run sites for other companies and the counts vary by client. I don't deal with the clients, just make sure the servers are running. If I had to guess I'd say 200 million page views a month.

This is all based on Sun hardware (typically quad-processor boxes with 4GB ram each for the front-end servers, and quad or 8-way processor systems for the back-end using three-way replication (so there's always 2 current copies of each live database).

Everything we do is JSP (java) based, I don't know if that's helping my CPU load or not. I've never worked with PHP to know how it compares, but typically my systems are running under 20% CPU load. Some of our clients are very spiky and we factor in a 4x overhead for that. We can't afford to have client A go slow just because client b is having a busy day.

I do question your database traffic, though. Our database servers are peaking under 1,000 queries/second. I know we did a lot of tweaking in the code to cache updates. For example things like message view counts are held on the app server and updated to the database every 15 seconds rather than making every page view generating an update in the database to reflect the message's count. You might want to look at your code to see if you can do anything there.
This might be where JSP has an edge over PHP. JSP runs in a servlet engine which can maintain state. Does PHP do the same, or is each page essentially disctinct from each other page (meaning two users who use the same page will generate two sets of identical lookups in the database)?
In our setup, each servlet engine has as much as 1GB of RAM for caching content from the database. It makes a huge difference.

As for clustering, synchronization and cache coherence is a major issue when trying to spread the load over two or more servers. Most of our setups are in a primary/standby model where all traffic goes to the primary box unless it's down in which case everything fails over to the standby box (typically with a 5 to 15 second failover). In the cases where this isn't possible, we use a lot of smart load balancing to hint different forums transparently towards different servers. This means that all users browsing forum 1 are on server 1 while those browsing forum 2 are on server 2. In that way server 1 caches everything in forum 1 and doesn't need to worry about anything in forum 2. This prevents two users browsing the same forum from seeing different content because the servers' caches aren't in sync.
In a failover situation, the standby machine's caches are empty, so there's a brief period of slowness while the data is loaded from the database, but it soon passes.

With the type of industry we serve we need to update a vast majority of our records throughout the day with new data. (Real Estate Listing Data)

RE JSP vs PHP: From my experience JSP is harder on cpu than PHP but it is a very good language to use

Part of the reason we are seeing so many queries/sec peak is that we rebuid some search tables every 2 hours to cut down load on our database server. We are also doing some things poorly in regards to how we are currently handling searches. This is about to be changed however

Most of our hardware is normal Intel stuff.

We are running Dell PowerApp Web 120's (single PIII 933) as our controllers for the web cluster. The live controller typically sit at 2% cpu useage.

We are running Dell PowerEdge 1650's (dual PIII 1.4's) as our web servers. We currently running 1-2Gb RAM in these servers. Our cpu useage on these sits around 20%.

For our database server we are currently running a single Dell PowerEdge 2650 (dual 2.4 Xeon's with HT turned on) with 6Gb RAM and an Ultra 160 RAID. We are getting ready to put into place Dell PowerEdge 2600's (dual 3.06 Xeon's with HT turned on) with 6Gb RAM and an Ultra 320 RAID. We are also going to only perform writes to the 2600's and perform reads off of 2650's that are replicated from the 2600's. We will be doing our own load balancing of the 2650 database servers with our own scripting to help distribute load and only hit servers that are up and running. Our biggest headache here is MySQL's lack of support for active/active servers but we are working around it.