What do you mean you don’t have a backup?

Discussion in 'Articles' started by Ted S, Jan 25, 2005.

  1. Ted S

    Ted S Tazmanian Master

    6,204
    735
    +22
    Running a forum takes time, energy and money no matter how small the site or what its intentions are. Yet for all the time and energy people put into their sites many never think to protect them from a database crash or system failure. While this may shock some it shouldn’t be all that surprising. After all, most sites haven’t had a major crash or don’t consider themselves big enough to need any backups. At the end of the day most people just don’t spend the time dealing with pesky tasks like backups but when things go wrong they wish they had!

    Backing up for small sites, why bother?
    So your forum has 50 members and 500 posts, it’s small, there aren’t even people on most of the time. So why would you need a backup, after all, it’s not like the data is going to corrupt and if it does, it’s not a big deal. For most forums starting is the hardest part and it’s even been said that once you get past a certain mark any decline, any lack of activity will mean the instant death of the forum. If your site has members and has posts you’ve already started succeeding and to loose all the time and effort, even if it only has amounted to 500 posts is a huge waste. Furthermore, while your post count may not have crossed the million mark yet, your forum is growing and restarting it from zero is going to be nearly impossible.

    Backing up for non-profit sites, who cares and who pays?
    You have a successful site with a fair number of posts but you don’t make much money off it and you may even be loosing money in the long run, your users don’t pay you for the forum so why on earth would you put the time or money into a backup. Making money is not the goal of all communities and while no one wants to loose funds, not having cash should never be a reason to let your forum teeter on the edge of destruction. Backups don’t require funds or expensive programs, just disk space and if you don’t have enough space to make a backup every now and then, you probably are going to out grow your host every, very soon. If your site is trying to become profitable but has yet to reach that point and you don’t back it up you are basically running a business without a safety net – loosing something you invested hours and hours and possibly thousands of dollars into for the sake of saving a little time here and a little money there is not a wise move on any level.

    Backing up for large sites, it’s too hard!
    You have a big site and by big we’re talking hundreds of thousands of posts, tons of attachments and more users then you can count to but with all that data backups take a ton of hard drive space and require some serious time so you only do them when you’re really bored and sometimes not even then. It’s true, backing up a 2gig database is a pain no matter how you cut it, especially if your server is already being pushed to its limits for resources but that’s all the more reason to do the backup. Large forums are the most susceptible to database corruption and crashes because of the resources they consume and the number of concurrent connections and inserts. There are very few big forums which have not gone down for a day, had a server crash or lost some amount of data at some point yet many still do not think to run backups on a consistent basis because of the effort they think it involves.

    The truth… you have to backup.
    The truth is no matter how big or small your site is you have put time and money into it, users of some count visit your site to be a part of a community and your site is inherently valuable to someone, probably you. Loosing your site because of data corruption is not just a nightmare, it can and does happen all the time. In December of 2004 alone ScubaBoard’s databases crashed twice and we lost 2,000 posts, a bunch of new members and a bunch of repeat members from 3 days of downtime. The only lucky part was that it all happened of the holidays when almost no one was online and we had 5 backups sitting there waiting for us. When we finally identified the problem it turned out that our post table had been completely wiped due to some serious corruption. Had the backups not worked we would have lost 850,000 posts in the blink of an eye and it can happen to any forum at any time.

    The truth… backs take almost no time.
    If you’ve ever run a manual backup either through shell or even clicking around in your control panel you know that the problem with backups is they take time and more importantly, require you to remember to perform them. For years my version of a backup meant running a few commands in ssh that I had saved to a text file. The system worked and was even used to perform a recovery once but as time went on I began realizing that my backups would often be spaced out by 3 or 4 days or even as much as a week. This meant that had the site crashed I could have lost thousands of posts (when the forum was younger hundreds of posts but the effect would have been just as bad).

    The truth… automated backups.
    When I finally realized how important backups were it became obvious that the only solution was an automated solution. Machines don’t forget to perform backups, they don’t go out of town and they don’t mind waiting. Every forum program out there has a backup solution, internal, addon or external. For vBulletin there is a free backup hack by Brad.loo which you can find at http://www.vbulletin.org/forum/showthread.php?t=62149. This hack isn’t fancy but it uses vBulletin’s internal cron system to perform the backup without the need for shell access or any advanced systems. For phpBB TerraFrost has written a similar hack which can be found at http://www.phpbbhacks.com/download/2170/. Most other forums have their own backup programs or hacks that you can find at their site.

    The alternative solution to these addons is to use a completely external cron script to go in, backup the database and perform any related tasks. For larger sites or sites on small hosting accounts a major factor in backups is space and to save on space I asked my system admin to come up with a simple backup solution that should work for just about any forum. This script (source below) logs in to any sql database and backs it up to a compressed gz file using the gzip program found on nearly all linux machines. Beyond simply creating backups this script also goes out and prunes old backups leaving only as many as you want (I like to keep 5).
    The code:
    Code:
    #!/usr/bin/perl
    
    $BACKUPPATH="/usr/backup";	#path do backup directory
    $BACKUPPREFIX="backup";		#name to begin our backups with
    $DBNAME="FORUM DATABASE";		#name of database
    $DBUSER="ROOT";		#user with access to database
    $DBPASS="PASSWORD";		#path to db
    $MYSQLDUMP = "/usr/local/bin/mysqldump"; # path to mysql dump binary program
    $LSOPTS="-tr";			#options to sort in reverse by creation date
    
    $KEEP=5;			#number of backups to keep
    
    #$DOM=`date +%d%b%S`;		#replace this line with the line below it
    				#to get rid of the seconds at the end of the
    				#filename (used for testing)
    $DOM=`date +%d%b%S`;		
    
    
    #---------------------shouldn't have to change anything below here
    
    chomp $DOM;
    
    $FILENAME = "$BACKUPPREFIX-$DBNAME-$DOM.sql";
    
    #========== deal with local backups ==========#
    #get existing backups
    @backups=`ls $LSOPTS $BACKUPPATH/$BACKUPPREFIX-$DBNAME-*`;
    
    #count their number
    $count = $#backups;
    $count++;
    
    #over desired number, kill the oldest
    if ($count >= $KEEP){
    	chomp @backups[0];
    	system "rm @backups[0]";
    }
    
    #housekeeping out of the way, let's do a backup
    system "$MYSQLDUMP --databases $DBNAMEs --add-locks --extended-insert -u $DBUSER -p$DBPASS | gzip > $BACKUPPATH/$FILENAME.gz";
    
    Copy this code to your favorite text editor (ultraedit, textpad, etc…), configure the variables up top with your backup path, database login settings, mysql dump folder and file name and save it as backup.pl. Upload backup.pl anywhere on your server and create a new file named cron.txt with the contents:
    Code:
    0 2 * * * perl /path/to/backup.pl
    
    Use the shell command “crontab cron.txt” to load the file and the cron will be set.
    This will run the backup script every morning at 2am, if you want to generate a more advanced cron or change the options see http://www.htmlbasix.com/crontab.shtml

    Going the extra distance with off site backups.
    In an ideal world you do not only backup to your local box but also to an external tape drive, secondary server or home machine. In any event having backups in two places is always better then having them in just one, especially if that one is your main server. After all, if the server crashes because of a bad hard drive with all the backups on it, you may as well have no backups. Luckily many server providers (like ThePlanet.com) and hosts now provide tape or NAS backups for free or a low fee. If your host provides these services take use of them, in many cases the support staff will set it all up for you and my script can easily interface with any mounted drive to make a second set of backups, just ask me for help.

    The alternative, doing nothing.
    Perhaps you still think backups aren’t something you need to deal with. Maybe you have the teenager mindset and believe your forum is invincible, perhaps you don’t have the disk space for backups or maybe you just don’t care. No matter what the reason if you decide not to backup data you may as well stop working on the site. Sooner or later your data will have corruption and even a little corruption can cause a whole table to crash. If it isn’t corruption your server will crash, your host will fail, a hacker will get in or something else will happen. This isn’t a pessimistic view, it’s a realistic view – every site has a problem at some time and unless you don’t mind restarting the only way to recovery from that problem is to be proactive and keep backups.

    Got questions?
    Post them here or feel free to contact me. I will gladly assist anyone with getting backups running on their site. Please note, my script is an example and I take no liability for your backups failing for any reason. When you run your first backup, test it before assuming the system works!
     
    • Like Like x 1
  2. Morganna

    Morganna Mistress of Mayhem

    24,322
    1,797
    +4,823
    Excellent article - the amount of people who I've spoken to and they've complained about losing their forum, then look at me blankly when I state "well, didn't you do a backup".... lol
     
  3. Ted S

    Ted S Tazmanian Master

    6,204
    735
    +22
    Yup yup. We "lost" our photo gallery in a botched upgrade. It took a week for photopost to decide that the best fix was to revert (rather than trying to write a script to rebuild the database from the data files). 45 minutes later we were back online but sadly, another site with the same problem is back at zero because they did not have a backup :(
     
  4. PacMan

    PacMan Tazmanian

    4,051
    792
    +199
    Ted, I was going to start a thread on forum backup strategies, but you beat me to it! My VB database is 360 megs with 240 megs being attachments and I rely on my host for backups (!).

    I want to move my attachments into the file system in order to make the database backup smaller. I was thinking of backing up the db on a daily basis and doing a complete backup once a week of the db and attachments. I want to get another hosting account from a different host and use it to store my backups. I guess I could write a shell script to FTP the db to the other server?
    Does this sound like a viable backup solution?

    I would like other admins to share their strategies with taking backups.
     
  5. Ted S

    Ted S Tazmanian Master

    6,204
    735
    +22
    PacMan,
    Don't let me stop you! My article is more about why you have to backup then how you should do it (yes I provide a few ways and an example script but thats far from the complete story).

    The option you preposed sounds great. A nighly backup of the critical data which is the most important stuff to have and the easiet to loose (databases corrupt more than attachments and users complain more about no posts then they do about a few missing links)! I recommend you automate the process as much as possible to insure it actually gets done. For me this means an automatic backup every night and the big backup weekly. Ontop of that I do some serious mysql repairs every Sunday night but that really turns into every other Sunday or every Sunday I have time which is why automated = better!
     
  6. Hot_Water

    Hot_Water Participant

    51
    6
    +0
    What can I add to my home computer to use as backup.

    I was thinking I could just add another hard drive to my existing computer and use it only for backups. Is that possible? If so what size hard drive would I need
    and do I need to upgrade the rest of my computer to do it?

    This is info of my computer:
    HP Pavilion
    Intel Pentium 4
    CPU 2.93 Ghz
    504 MB Ram
     
  7. JoshB

    JoshB TalkSTAR

    2,034
    0
    +9
    You just need a small hard drive, for even the largest forums I would say 2GB is the max. size needed for the db and attachments. So you are only going to need around 10GB to keep the last few backups on. Your current system should be able to support this.
     
  8. Hot_Water

    Hot_Water Participant

    51
    6
    +0
    THANK YOU VERY MUCH! This is the concrete information I have been looking for. Now I can actually go and get this done.:) :tiphat:
     
Verification:
Draft saved Draft deleted