How to host a busy forum with 20.000 unique daily visitors in shared hosting for $29.99
Okay it sounds implausible I know, but it CAN be possible if you know what to do. I know its possible, because I do it with one my own boards.
It is true that this is really very hard to achieve. If you have 20.000 daily unique visitors, it means you have a pretty large board with a lot of content, a huge DB size, a lot of members, high bandwidth consumption and all these forces you to host your site in a dedicated server. I know that even the sites that has half or even quarter of this traffic are on dedicated server with $100-500 USD monthly fee. So how can that be possible?
First you should know what forces a site to be on a dedicated server. In other words when does your hosting company ask you to leave their shared (=cheap) hosting service and says it is about time for you to jump into a dedicated (=expensive) server. There are 3 factors that hosting companies care and monitor to decide on this:
Each hosting company can have a different policy and method to decide when your site should leave their shared hosting. Most puts certain restrictions or limits on the server/hostage package and checks if your site goes over them. Some other reviews your site manually and decides according to different custom criteria they have.
- Your bandwidth consumption.
- The space your site takes in the server
- The CPU process your site devour. (That is, how much your site makes the server busy)
But the bottom line and most crucial point is that: If you manage to reduce your bandwidth consumption, the server space you take and CPU time you consume, your site will not have a bad effect on their server so they'll not ask you to leave their shared hosting package regardless of how many daily visitors you have. So the question lies how can you reduce your site's bandwidth consumption, server space and CPU usage with a heavy traffic forum based site. Let's review each item one by one:
1- REDUCING YOUR BANDWIDTH CONSUMPTION:
Bandwidth is the number of data your visitors transfer each time they visit your site. When a visitor loads a web page in your site, every data (images, downloaded files, texts) that is in that web page (and also in your server) costs you bandwidth. The more visitors you have, naturally, the more bandwidth you consume. Likewise the longer and more crowded your pages are, the more bandwidth you'll spend.
If you have a typical forum based site, you can expect your site to consume around 400-500 Gb. monthly consumption if you have around 15.000-20.000 daily visitors. (Obviously this changes from site to site, this is just an average estimate!) This is too much for any shared packages and if you go over 100 Gb., most web hosts will ask you to leave their shared hosting package. This limit obviously changes from company to company but it is really very hard, if not impossible, to find yourself a shared hosting page which endures your site that consumes 500 Gb. monthly. So one of the first thing you need to do is reduce your bandwidth consumption if you want to stay in your shared hosting package. Let's see things which eats your bandwidth mercilessly:
a) IMAGES : Image files are the main reason your site consumes bandwidth. If a user posts a text joke in to the forum, it will take less than 1 Kb. of bandwidth each time it is loaded but if same user posts the same joke as an comical image that is attached to the forum post, it can lead up to 100-200 Kb. consumption and this is for every time it is loaded by a visitor! So if 10.000 visitor reads that joke in your forum that month, text joke will cause you 10 Mb. monthly consumption and image joke will cost you 2000 Mb. (=2 Gb) Nice ha? So with a SINGLE attachment you allowed in your forum, you lost %2 of your allowed bandwidth consumption to get kicked out of your shared hosting package if it went into a popular and frequently read thread. Have 50 more of them in a month and your hosting company will show you the door even if you have just these 50 posts in your board that month. So one thing you do is to cut them as much as possible and instead of allowing users to post such images as attachments in forum posts, encourage them to use remote servers like imageshack.us which allows your board to display images without costing you any bandwidth.
Images are not the problem only when they are attached to forum posts. If your site has a graphically heavy web design/skin, it is likely that you are spending about 2-3 times more bandwidth. Having a text based, humble, simple, image-wise reduced design will also help you a lot to save bandwidth.
b) ATTACHMENTS: Of course everything told for images also apply for other file attachments (except .txt attachment which are quite short). If your board is hosting too many zip files, doc files, pdf files, wav files or especially videos, you'll be spending incredible amount of bandwidth monthly. You should at least make sure, such attachments can not be lurked by guests (=non-members) and if you can do it, you should also consider disallowing attachment uploading in your board altogether. If your board is about file sharing or graphical design, this might not be possible but what's the point of encouraging users to use attachments if your site subject is say "medicine", "law", "mathematics" etc. You should make a site administration policy to encourage text content in your board and discourage any attachments where possible. This will save you incredible amount of bandwidth.
c) GUEST LURKING : Do you allow non-members to lurk your site content without being a member? Think again! I wouldn't suggest turning off displaying of all forum content to guests as IMO this is really a bad marketing practice, but you should at least make sure non-members get entirely "text based" information from your site, no other information at all! For instance don't display avatars to guests, replace image buttons like "edit post" "send pm" "send email" in the postbit with text links, don't allow guests to view attachments (let alone download them!), even consider not parsing smilies for guest views. So a guest's reading your board content should always be dramatically different than a member's reading the same content. A member should have full access but a guest should always get a minimum text content and should be asked to register if they want to see the actual design. Most of your bandwidth consumption is always consumed by non-members so you should definitely deal with them properly to reduce your consumption.
You should also consider using a server based compression to reduce bandwidth consumption like gzip.
Of course your forums is not the only source of your consumption so you should also make sure your sites other pages and content do not have heavy pages, large graphics, high number of downloadable files all of which will increase your consumption.
2- SPARING SERVER SPACE:
If you are running a forum based site, your database will probably be the number one culprit of your server space eater. Especially if you have a large board, you'll likely to have a large database as well. If your database (and other files in your server) takes much space in the server and exceeds the limits your hosting company set, you'll be asked to move to a dedicated server so you must make sure your site does not occupy too much space if you want to stick with your cheap shared hosting package.
The easiest way to do reduce your DB (hence server) size would again be getting rid of attachments! Just login to phpmyadmin and check which tables take most space in your database and if you allow attachments in your board, you'll likely see that more than %50 of your database space is taken by attachment table. If your users are crazy about attaching files, you can even see %80-%90 of your database space is consumed by attachment table. If you could get rid of it, your database would shrink incredibly! So consider doing this by again disallowing attachments or at least getting easy about the usage of them.
After the attachments you can focus on search index tables. If you are using 3.5.x versions of vbulletin, it allows you to use MYSQL Fulltext search index over vbulletin's default search index algorithm. If you are able to use it, use it. It will save you space about %20-30 (let alone relieve your server load in busy forums).
Although it does not a major issue, you can also take a look at custom avatar and custom profile picture tables. If they take too much space, you can also consider disallowing usage of them (or at least restrict it). For instance you can allow regular users to select only preloaded avatars and allow only paid members to use custom avatars. If you have a large member database (like over 20.000 members), this can again save you some space.
Another issue to take care of is your private message tables. If you have an old board with a lot of members, you'll surprised to see how much space private message table takes! Warn your members that it will be pruned in certain intervals and do so. If it fills too fast, restrict number of Private messages regular members can keep in their inboxes. Give large quotas to only paid members, not every member. You'll also save a lot of space.
SAVING CPU PROCESS YOUR SERVER DEVOUR:
This would be the hardest part to optimize but don't worry it is also the least important of all elements mentioned above. It is also hard for hosting companies to keep a track of which site uses how much CPU time in shared hosting solutions so just a few monitors them very closely. However this can cause a problem, if your site is frequently increasing server load and effecting entire server. This is usually indication of using badly written hacks too so if your board is unhacked or hacks applied are carefully and wisely selected, you will not bother server much.
But some hosting companies can still put some restrictions on server to force you move to a dedicated if you have too many visitors at a time. The most important of these restrictions is "max_user_connections" which prevents a site to make more than set number of connections to mysql database simultaneously. So if your site gets too many visitors at once and passes this limit, it starts producing "User ... has already more than 'max_user_connections' active connections" error and goes down. If your hosting company has this restriction and set it as a small number, sorry tough luck for you! There is not much you can except going dedicated or finding another hosting company who has a higher setting.
But if your hosting company does not restrict you on this with too small numbers and if you don't go crazy in hacking and devour entire cpu power of the server with self-absorbed code modifications, you won't have much problems in your board with CPU work.
REAL LIFE EXAMPLE:
okay so after general information, let me give you a real life example from one of my own boards: (please don't ask my hosting company or my board's URL and, no, it is obviously not worldwideknights.com)
It has 15.000-20.000 daily visitors, 100.000 forum posts, 300-400 online users at all times and over 500 online users at peak hours (cookie time set to 20 minutes) but it is still on shared hosting. Its database takes 542 Mb., (gzipped backup file is 80 Mb.) the entire server space it consumes is less than 1 Gb. and my hosting company set "max_user_connections" to 500 which is handling my online users fine (at least for now). Most of my online users are guests too as my board has VERY STRICT moderation on new user sign ups and out of 400 register attempt everyday, only 20-25 are accepted according to our member criteria. We consume around 70 Gb. bandwidth every month.
This is how I configured my board to achieve this stats:
And until now, we didn't need to upgrade to dedicated. It is only natural that one day we will as we have a 10% increase rate every month and regardless of how much we optimize the board, it will go beyond the limits of shared hosting. But I know many sites which are forced to go on dedicated with 2000-3000 daily visitors so our still sticking with shared hosting is a big money saver for us (especially considering we are not a commercial board and except Google ads which pays bills, we don't even accept advertisement or sponsorships, nor have paid membership or even donation programs). We are paying $29.99 monthly so this is $359.88 yearly. If we were forced to move to a dedicated 3 years ago with a $200 monthly bill, it would cost us $7200 but keeping our shared hosting cost us $1079.64 which is around 6100 USD saving for 3 years.
- Attachment usage has never been enabled in its 10 year history. Not even administrators can use attachments and my attachment table is empty.
- Members can choose from 500 pre-defined avatars but they can not use custom avatars.
- PM quota is set to 30 for regular members.
- It has a light skin with a few images only, it is mostly text based.
- All forums and threads are fully open to guest views but it is highly optimized for them: If visitor is a guest, they can't see avatars or user signatures in posts. Irrelevant images and elements in page disappears (like edit this post, reply this post, send pm to member etc.). Guests has a smaller "Preview Text" option with hacking. Postbit elements like "Number of Posts", "Location", "user reputation", "user title" disappear for guests. Even popup page in postbit (which opens a popup page when you click posters name) is removed for guest viewing. So what guests get is pretty much a simpler and very light version of thread content but it still has the full text information in thread.
So if you ask me, this optimization really worths it.
This article is written to help spread the knowledge and with the hope of its being useful to someone who needs this information. If you find this article informative and useful, join "Spreading the knowledge with Articles Campaign!" and write an article about anything you know so that you can also help someone just like this article helped you! This article is written as a part of WorldWideKnight.com's very own Spreading the Good project.