An example of why not to use beta software on a live site

[mysiteguy]

I won't post any links, as it really isn't relevant, it's a client's site and I want to respect their privacy.

Anyway, this client had upgraded from VB to XF 2.0 a few months ago. After they saw massive traffic drops they realized they weren't redirecting old urls to XF. They hired someone to fix this, and it really didn't help. I went in and found the other developer had put in a few dozen manual redirects for forums, a few prominent threads, but there weren't any means to do global redirects of all URL types properly.

After implementing XF 2's VB redirection addon, .htaccess rewrites for vBSEO and a custom PHP file to handle certain types of vBSEO urls .htaccess couldn't handle, the site started seeing a recovery in less than a week and traffic has grown steadily on a daily basis since then.

So, the site was working well from that point.

Yesterday I received a somewhat panicked message from them on Skype. They upgraded to XF 2.1 beta, and it broke the theme in major ways. Some of their addons were also causing errors. I checked, and none of the issues could be resolved without significant (ie, costly) time working on the theme, XF 2.1's templates and PHP bug fixes. I asked them if they made a backup before they did this, and found their last full backup was from over a month ago.

I don't chastise when someone is stuck like that. We've probably all made mistakes like that at one time or another (and now I'm in the process of implementing a daily automatic backup), plus no one wants to be kicked when they are down.

They were faced with a decision:

1. Roll back a month to their full XF 2.0 backup, and lose a month's worth of posts - not a viable option since its a busy board.
2. Do step 1, and pay me to manually move all new users, threads, and posts from 2.1 back to 2.0. They didn't want the expense of this.
3. Disable several useful addons, switch to XF 2.1's default theme and wait it out while addon developers and the theme provider produced updates.

They went with option 3. The theme provider says they will have an update in a week, but no word yet on all the addons.

So, if you feel you must run beta software on a live site, my suggestions:

1. Backup everything
2. Backup everything
3. Backup everything
4. Did I mention backing up everything?
5. Ask all addon and theme providers if they are XF 2.1 compatible.
6. Try it first on a staging/test site before going live. This includes testing for proper operation of all addons.

For me, it's just too much hassle. It's not going to hurt a site to wait a month, or maybe a few, for XF 2.1 to go gold before upgrading.

 

[we_are_borg]

I think most have done that upgrade and make no backups.

 

[mysiteguy]

I think most have done that upgrade and make no backups.

True. Unfortunately, often we learn lessons the hard way. I've had my fair share of mistakes. Biggest hard lesson I recall: Early in my Linux experience I learned how important it is not to be root, and to sudo anything which requires root. I was in the base directory ( / ), did an rm -fr * thinking I was in a different directory and proceeded to wipe out half the drive!

 

[zappaDPJ]

I doubt I've ever upgraded anything without taking a backup first but I did learn the hard way to ensure that I can actually restore from backup when it really matters. Not all backup routines are equal

 

[R0binHood]

Happens. Every. Single. Time.

XF should add an extra product to their services offering available to purchase directly from within the Admin Control Panel:

Fix your site after you ignore our repeated warnings not to install betas on a live forum: $50 to $5k depending on how bad you managed to bugger it up

 

[Alpha1]

I have used xenforo 1.x beta versions live many times. After testing those on staging first. Everything is always backed up online and in such cases also locally.

But it seems that with XF2 that is a different matter. Its not ready for production and in this case beta really means beta.

 

[mysiteguy]

Happens. Every. Single. Time.

XF should add an extra product to their services offering available to purchase directly from within the Admin Control Panel:

Fix your site after you ignore our repeated warnings not to install betas on a live forum: $50 to $5k depending on how bad you managed to bugger it up

I hear the data recovery business is very profitable. Hmmmm, maybe I'm in the wrong line of work? Actually, I would hate it. I can't stand seeing someone struggling with a potentially huge loss. Business or personal data, it's gotta suck.

 

[R0binHood]

True, but you also get the opportunity to be the hero if you're able to jump in and save the day, plus earn some cash if you're good at it.

Even though you weren't able to fix everything overnight, I'm sure the site you dealt with was incredibly grateful that you could successfully analyse the situation, figure out what was wrong and produce a plan of action, even if some of that action means waiting on others to update add ons on their end. I'm sure you still came across as a hero to a certain extent, you go the site working again and made the patient stable. Just need to wait for the 3rd party meds to arrive and they'll be back up and running a full speed soon enough, with a tough lesson learned.

I think many admins treat these forum betas too cavalier these days. It doesn't matter which vendor you choose, these community suites are getting more and more complex compared to forums of 10 years ago, especially when you could be running dozens or hundreds of add ons. It's not a good idea to run betas live unless you're a dev of very comfortable with the entire admin back end system, someone that know's how to properly troubleshoot and fix any problems you encounter, even then it's a bad idea not to test it first.

Not that I really blame anyone for doing it. It's really hard to resist after some of the simpler betas of old.

But we need to remember, these aren't iOS Betas where you can wipe your phone and restore from a daily iCloud backup.

Set up a duplicate of your site, upgrade that and get your dedicated users on it to test it and break it before going anywhere near your live site. If you're not at a skill level to create a mirror of your site for testing, or ensure that you have a backup from the last 24 hours, then you certainly shouldn't be even considering upgrading your live site to a beta. At least wait for a Release Candidate if you're that impatient (even then that still wont catch 3rd party plugin or theme issues ahead of time that are much better caught and tested on a demo site).

Thinking about it more, it's just a plain terrible idea to upgrade a live site to a beta these days, unless you're running the software completely stock, or willing to disable any/all 3rd party add ons/themes until those devs have updated and tested with the latest version.

If you're not willing to run the software completely stock, themes and all, then it's your duty as a responsible admin to setup a demo site and upgrade that and test everything thoroughly.

 

[Chris D]

I have used xenforo 1.x beta versions live many times. After testing those on staging first. Everything is always backed up online and in such cases also locally.

But it seems that with XF2 that is a different matter. Its not ready for production and in this case beta really means beta.

Most of the issues mentioned seem related to issues we won’t be resolving through the beta process ourselves.

We can’t fix outdated templates for people and release new styles for people and we can’t resolve add-on code problems for people.

So really your extreme generalisation and assumption is just not useful.

 

[Alpha1]

I see I could have worded that better. It should have said that xf 2.1beta is not ready for production. I am using xf2 in production on one site.

 

[we_are_borg]

I hear the data recovery business is very profitable. Hmmmm, maybe I'm in the wrong line of work? Actually, I would hate it. I can't stand seeing someone struggling with a potentially huge loss. Business or personal data, it's gotta suck.

I did a few weeks ago data recovery for a friend, 3 days of my life gone.

 

[maksim]

Sorry to say this but you have to be an amateur or a moron to run beta software on a live money site.

And yes... daily backups. In today's age with dirt cheap off site storage and built in cpanel backups... there is ZERO reason you should not have daily backups locally on the server and offsite on Amazon S3.

NONE... ZERO excuses. There is no need to know for any coding as they are built in click options.

 

[eva2000]

not all betas are the same with varying degrees of stability.

But it seems that with XF2 that is a different matter. Its not ready for production and in this case beta really means beta.

but yes i track xenforo bugs forum in my custom slack channel amongst other software I rely on, and xf 2.x still has alot more bug reports (2.1 beta seems to make up alot of these now) compare to xf 1.5 - more than i am comfortable with, so stick stickying with xf 1.5 myself and waiting for more addons to catch up.

slack is awesome

Biggest hard lesson I recall: Early in my Linux experience I learned how important it is not to be root, and to sudo anything which requires root. I was in the base directory ( / ), did an rm -fr * thinking I was in a different directory and proceeded to wipe out half the drive!

There's a little trick i devised for this - never run rm command until you actually change into (cd) the directory where the intended file/directory you want to delete is located or directory you're trying to remove and running list (ls) command to verify the files. Make it habit and instinctive. In 18+ yrs of running root, never have I mistakenly deleted something. Also always use full path to what you want to delete so rm -rf /path/to/*

so if you want to delete /home/testdir contents

cd /home/testdir
ls -lah
rm -rf /home/testdir/*

or

cd /home/testdir
pwd
ls -lah
rm -rf /home/testdir/*

also instead of just pwd i have have an alias for pwdh (centmin mod lemp has it setup out of the box) it lists hostname and current working directory in full path format so if you're working with multiple ssh session windows for multiple servers you can be sure which server you're working on

cd /home/testdir
pwdh
rm -rf /home/testdir/*

example pwdh command output

pwdh
host.domain.com /home/testdir

and my ssh command prompts always add hostname too i.e.

[09:26][root@host.domain.com testdir]#

Make it habit and instinctive and you'd be less likely to wrongly delete something as even non-root users can wrongfully delete files/directories they own themselves.

My AUD$0.02

I doubt I've ever upgraded anything without taking a backup first but I did learn the hard way to ensure that I can actually restore from backup when it really matters. Not all backup routines are equal

Yeah true a staging test server helps with backup restoration verification too. I usually never upgrade without a backup - for major version updates, i wouldn't even do the upgrade on live site/server until I tried it on staging server with copy of the files/databases first. Then even on live, it would always be upgrade on a copy of the file/databases while leaving original files/database untouched and intact. It's a force of habit from my vBulletin days from vB 1.x to 5.x. It's always been this way

It's not going to hurt a site to wait a month, or maybe a few, for XF 2.1 to go gold before upgrading

Yeah I'd be looking at xf 2 bug reports forum to determine when I'd be upgrading myself along with the addons I need. But can't blame your client, XF 2.1 looks a lot more enticing for me than XF 2.0 feature wise The xf 2 bug report forum tracking/notifications in slack keep my urges to upgrade prematurely in check

 

[Chris D]

not all betas are the same with varying degrees of stability.



but yes i track xenforo bugs forum in my custom slack channel amongst other software I rely on, and xf 2.x still has alot more bug reports (2.1 beta seems to make up alot of these now) compare to xf 1.5 - more than i am comfortable with, so stick stickying with xf 1.5 myself and waiting for more addons to catch up.

slack is awesome

View attachment 52214



There's a little trick i devised for this - never run rm command until you actually change into (cd) the directory where the intended file/directory you want to delete is located or directory you're trying to remove and running list (ls) command to verify the files. Make it habit and instinctive. In 18+ yrs of running root, never have I mistakenly deleted something. Also always use full path to what you want to delete so rm -rf /path/to/*

so if you want to delete /home/testdir contents

cd /home/testdir
ls -lah
rm -rf /home/testdir/*

or

cd /home/testdir
pwd
ls -lah
rm -rf /home/testdir/*

also instead of just pwd i have have an alias for pwdh (centmin mod lemp has it setup out of the box) it lists hostname and current working directory in full path format so if you're working with multiple ssh session windows for multiple servers you can be sure which server you're working on

cd /home/testdir
pwdh
rm -rf /home/testdir/*

example pwdh command output

pwdh
host.domain.com /home/testdir

and my ssh command prompts always add hostname too i.e.

[09:26][root@host.domain.com testdir]#

Make it habit and instinctive and you'd be less likely to wrongly delete something as even non-root users can wrongfully delete files/directories they own themselves.

My AUD$0.02



Yeah true a staging test server helps with backup restoration verification too. I usually never upgrade without a backup - for major version updates, i wouldn't even do the upgrade on live site/server until I tried it on staging server with copy of the files/databases first. Then even on live, it would always be upgrade on a copy of the file/databases while leaving original files/database untouched and intact. It's a force of habit from my vBulletin days from vB 1.x to 5.x. It's always been this way



Yeah I'd be looking at xf 2 bug reports forum to determine when I'd be upgrading myself along with the addons I need. But can't blame your client, XF 2.1 looks a lot more enticing for me than XF 2.0 feature wise The xf 2 bug report forum tracking/notifications in slack keep my urges to upgrade prematurely in check

I don’t disagree with you ever, but this is a shortsighted way of looking at it.

It doesn’t factor in things like the severity of issues, nor the fact that the XF 1.x code base is essentially 8 years more mature than XF 2.x.

If you’re looking to wait until XF 2.x reaches the same quantity of bug reports as 1.x then you might still be running 1.x in 2022.

If you base the right time to upgrade on severity rather than quantity, you could have upgraded a year ago and you probably still wouldn’t have come across any bugs that would have significantly affected you.

 

[eva2000]

I don’t disagree with you ever, but this is a shortsighted way of looking at it.

It doesn’t factor in things like the severity of issues, nor the fact that the XF 1.x code base is essentially 8 years more mature than XF 2.x.

If you’re looking to wait until XF 2.x reaches the same quantity of bug reports as 1.x then you might still be running 1.x in 2022.

If you base the right time to upgrade on severity rather than quantity, you could have upgraded a year ago and you probably still wouldn’t have come across any bugs that would have significantly affected you.

Maybe true though I'd still will be waiting on addons compatibility and maturity to catch up a year ago. So it isn't just the number or severity of the bugs that is a factor. Either way for me it's alot of testing on staging servers first

 

[mysiteguy]

There's a little trick i devised for this - never run rm command until you actually change into (cd) the directory where the intended file/directory you want to delete is located or directory you're trying to remove and running list (ls) command to verify the files. Make it habit and instinctive. In 18+ yrs of running root, never have I mistakenly deleted something. Also always use full path to what you want to delete so rm -rf /path/to/*

I started doing something similar after that first painful lesson around 1995. Though I'd been accessing Unix based systems for years via dialup and telnet, it was to play games, connect to distant file archives via connecting to outside dialup, and some Usenet & Gopher. I had been so used to DOS, CP/M variants, etc. and was not terribly proficient with Linux. Fortunately, the data loss wasn't catastrophic but it was a major pain to reinstall Slackware, spanning dozens of floppy disks.

 

[Paul M]

Regardless of upgrading (to beta or otherwise) you should still be backing up your site, ideally daily.

 

[southernlady]

My entire server is set to back up daily.

But I have done the upgrade (not to beta tho) without backing up on a live site. Other than smacking my forehead for being stupid, it finally worked out.