Anyone familiar with the vBulletin.com community knows George Liu (AKA eva2000), the server optimisation guru. George has helped countless people fix server issues and get their sites running at maximum performance. I asked George to do an interview with us and he cranked out this awesome writeup in just a couple of days. Thanks George!
Can you tell us a little about yourself - a short bio?
I'm pretty much a private person when it comes to my personal life so pretty much at odds with the phenomenon that is Facebook - posting every little tidbit about oneself online for folks to see. But short bio:
- My sister and I are Chinese and were born in England and spent quarter of our lives in England and the rest of the time now in the country I call home, Australia.
- My parents are from Hong Kong but both spent two thirds of their lives overseas in England and Australia as well.
- I speak English and Cantonese (can't read though).
- Due mainly to my parents work, between England and Australia, I've lived at and moved homes a total of 10 times and prior to university, attended a total of 8 schools, including 4 different high schools in two cities/states, Sydney, New South Wales and Brisbane, Queensland.
What is your educational experience?
Bachelor of Commerce with minor electives in Political Science (PACS - Peace & Conflict Studies) and Psychology from University of Queensland. Due to financial reasons at the time, I had to turn down an invitation into the Political Science department's PACs Honours program. Psychology and PACs knowledge armed me well for my role in technical support dealing with sometimes irate customers and conflict resolution etc.
What is your fulltime job?
Working for vBulletin.com since 2001 and also doing private server optimisation consulting (for vB clients and non-vB clients).
Take us through your schedule on a typical work day.
There's no real typical day as hours at vBulletin.com are very flexible. For private consult work can come and go hard and fast with lots of work in a short span, or nothing for weeks. The last 6-8 months have been pretty busy though. For vBulletin.com work, over the past few years it has steered me more towards working with vBulletin customers to providing basic courtesy server optimisation advice on forums and making sure the server environments that vB customers run on is setup optimally.
When I'm not specifically working on a task, I'm constantly keeping up to date with web server end technologies, best practices - reading newsgroups, mailing lists, forums, online articles and technical books. Spare time is also dedicated to setting up local virtualised servers for local performance testing or various web technologies i.e. Varnish cache, Litespeed & Nginx web servers, playing with various MySQL server forks such as MariaDB and Percona and alternative MySQL Replication technologies like Tungsten Replicator based MySQL replication. Basically, all things to do with improving a forum server’s performance and scalability.
How did you get to be so knowledgeable about servers?
It all started 12 years ago in 2000 through trial and error when I wanted to migrate my growing Ezboard hosted forum to my own domain name/hosting. I purchased my first copy of vBulletin v1.x from Jelsoft after evaluating my choices between vB and UBB at the time. It was also my first time on a dedicated server with Rackspace.com and I was a Linux and dedicated server noob - I didn't even know how to backup my database at first - mysqldump? Never even heard of it at the time LOL.
My vBulletin v.1 forum (which I'd like to keep the name private) was growing with up to 1,000+ users online and 3,500+ new posts per day. Basically, the Rackspace.com server (mainly MySQL server related) was crashing every few hours or days, with days on end of downtime and support calls to Rackspace.com. Luckily, vB staff member at the time, Chris Schreiber was there to guide me through the sleepless nights, constant hair pulling and answering all the noob Linux and server related questions I had. I vaguely recall one time my server was down a full week and another time I had to ring Rackspace.com for emergency tech support 5+ times in a span of a few hours! In the same year, another helpful person was Joe Eng who wasn't even a vBulletin customer or staff member who saw the problems I was having and explained a lot of the basics related to MySQL server administration.
Using what I learnt through experience, I ended up hanging out on vBulletin.com support forums a lot - I recall 12+ hours of constant posting on many occasions. In turn, I used my knowledge to help other vBulletin customers voluntarily. In 2001, I was officially invited to join vBulletin /Jelsoft as an official tech support staff member and have been doing this ever since. Over the years, the focus has migrated away from vBulletin software itself more towards vB server related optimisations and performance tuning. You also learn even more through helping others deal with their specific issues as well.
So basically, it's my early painful experiences with severe server downtime and performance issues, which now drive my constant quest to find and learn the new and improved ways to optimise and scale web and MySQL servers for forum performance. I’m no coder, so I have squeeze out the most performance from server and database end itself.
What is your favourite server hardware and software?
Server software stack wise, there's a few combinations I like, the favourites being:
- Operating System: CentOS 6.x mainly but started to play with Debian 6.x as well
- Web servers: Litespeed web server (my preferred web server) & Nginx web server
- MySQL server: MariaDB 5.2, 5.3 & 5.5 or Percona 5.5 as drop in replacement for MySQL.com community releases. The latest MySQL community release 5.5 is also shaping up well. Currently, just starting to test newly released MariaDB 5.5 MySQL server which is based on MySQL 5.5
- Caching Acceleration: Varnish cache, Litespeed cache, APC and Xcache, Memcached servers. Memcached caching does wonders to reduce vBulletin MySQL related loads for example two clients I worked with are saving up to 2,500 – 3,500 cache hits per second. I’ve seen forums with APC or Xcache installed reducing PHP loads by as much as 600 – 1,000 cache hits per second.
- Server hardware: Anything Intel based in the last 4yrs from Nehalem, Westmere, Sandy Bridge and latest generation of Sandy Bridge-E and Ivy Bridge architectures. Currently, have my eyes on Intel's new Xeon E5 Sandy Bridge-E 16xx / 26xx series with its new Intel IIO and DDIO technologies and the new E3-12xx Sandy Bridge and E3-12xx v2 Ivy Bridge processors. Also, you can not forget about Solid State Disks (SSDs) and PCI-E based SSDs which are changing disk, web and MySQL server performance for the better.
How many servers do you figure you've optimized over the years?
For personally delved into server type optimisations, easily hundreds of servers. If you include optimisation advice provided on the forums, it would probably run up close to thousands mark. What I like most about forums, is when you post info or advice you are not only helping the original thread poster, but also any visitor or reader who happens to come across the forum thread. I always keep this point in mind when I’m replying to threads on public forums.
How much of a performance boost is generally associated with an optimization?
It can vary for different server setups. But I'd say for optimisations, 50% of the time it's correcting non-optimal configurations and settings for server software and hardware. Trying not to allocate more server resources (i.e. memory) than you have available or making sure what server hardware you have ordered and installed is working as it's intended to. A common one is related to hard drive configurations (Raid) and whether they're configured properly and perform as expected. I've seen folks jump onto cloud based servers thinking they'd get much better performance, only to find out hard drive disk I/O performance in cloud/SAN array environment was very bad causing vBulletin forum owners never ending grief. The other 50%, is squeezing out every drop of performance you can with what server software and hardware you have available and/or within your budget.
Basically, it's first making sure what software and hardware you already have, works as intended and is optimally configured. Then build on that base to squeeze out as much performance you can.
For vBulletin forum owners, the most dramatic performance boost some of them usually notice is when they correct that first 50% in making sure everything is working as it's intended to. I see this often when vBulletin forum owners complain that they upgraded their server hardware to a much more powerful configuration but their performance is the same or worse than before. It’s that first 50% that makes all the difference.
Without regard to cost, how would you configure the ideal server for a busy vBulletin Community?
For very large and heavy concurrency based vBulletin forums, you'd usually end up with multiple dedicated web and database server setups, but if it's a single ideal server setup and cost was not an issue then below configuration would be what I'd like to build:
Note: I would like to point out, that the above server is indeed overkill for most folks and you can opt for more cost effective configurations to do the job. I thought I’d mention this as I don’t want readers wasting their money on such a configuration. But you did ask if cost wasn’t an issue
- CPU: Dual Intel Xeon E5-2687 3.1-3.8Ghz (standard-turbo boost) Sandy Bridge-EP (32nm) based processors totalling 16 cpu cores / 32 cpu threads which can in theory support 750GB max Quad Channel based memory up to DDR3-1600 speeds and have Intel IIO and Intel DDIO technology support.
- MEMORY: As much as it can support which can be allocated to ramdisks and various caching acceleration technologies like Varnish cache, Memcached servers etc. That's part of the remaining 50% in terms of optimisation
- Disks: SSD or PCI-E SSD based, holy grails would be an 800GB SLC SSD based Virident FlashMax PCI-E device which supports replaceable NAND modules in a raid 7 design or 1.2TB SLC SSD based Fusion ioDrive2 Duo PCI-E device.
- Network: Full 10Gbe (Gigabit) based network connected / interconnected server(s).
- Software: Litespeed web server (LSAPI PHP 5.3.x) to replace Apache web server, APC Cache for PHP opcode caching, MariaDB 5.3/5.5 as drop in replacement for MySQL.com community based server and Memcached servers.
Tell us about your websites.
My private sites, I prefer to keep separate and private. But work related my focus is on the following:
vbtechsupport.com - my personal/semi work blog I started last year mainly to document what I have learnt server/forum performance wise. The domain name pays homage to what I have been doing for the past 11+ years and so was glad it was available to be registered.
centminmod.com - also started last year. Centmin Mod is a menu based Nginx web server auto installer script which I extended upon from original Centmin script. You can use Centmin Mod to completely setup a working server software stack on CentOS based server consisting of Nginx web server, PHP-FPM, MariaDB 5.2 based MySQL server, APC or Xcache, Memcached server, CSF Firewall, Siege Benchmark tool as well as updating CentOS to Python 2.7.2. Over the past year, quite a few vBulletin forums small to very large have migrated their forum servers to Centmin Mod installed and setup servers.
mysqlmymon.com - this site was born out of what I learnt about bash shell scripting while working on Centmin Mod script. The site hosts mysqlmymonlite.sh tool which I wrote to help server and forum owners to quickly gather all there server environment and stats info for server optimisation and analysis. It takes less than <12 seconds to gather all the info outlined on the site at http://mysqlmymon.com/.
Are you involved in any other projects?
As already mentioned, centminmod.com and mysqlmymon.com are main focus right now.
I am also using my bash shell scripting knowledge to write up all kinds of useful server administration tools to make my life a lot easier - including various forms of MySQL backup tools such as mysqlmybackup.sh as well as tools and status scripts, myreplicator.sh for Tungsten Replicator based MySQL replication (page 4). I hope to extend myreplicator.sh to allow me to do some basic setup, install and management for Tungsten Replicator based MySQL replication as I can not afford the Tungsten Replicator Enterprise version which has a much wider range of already inclusive administration tools. It’s all good practice in furthering my skills in this area.
What Boards do you Admin/SuperMod/Mod?
Besides private personal forums, there's quite a few I am an Admin on but only in capacity for my server optimisation work which takes up most of my time these days. As such, you may find me listed as an Admin on hundreds of vBulletin forums online!
What's the significance of your username eva2000?
In short, eva refers to Japanese Anime series called Neon Genesis Evangelion (NGE). Eva was short for Evangelions mecha units which are only piloted by children. The 2000 was basically the year I got into the NGE series but is also coincidently the same year in the series which is central to the storyline, that in the year 2000 the "Second Impact" occurred which destroys most of Antarctica and leads to the deaths of half of all humanity! If that isn't enough to entice new NGE fans in, check out some Youtube NGE clips - always happy to convert new NGE fans. Yes, I am a Japanese Anime fan
The 3 Eva units 00, 01 and 02 from my collection
What are the common mistakes you see admins making?
If you role in forum owners interchangeably with admins (forum admins & system admins), there a few common mistakes I see:
- Sticking with aging and very out dated server hardware out of plain fear in migrating forums to new servers or web hosts. One recent client I was working with had an aging Pentium 4 server and was paying monthly server rental fees which were nearly 8-10x times what the going rate for that old server fetches. They were constantly having performance issues with their vBulletin forum which affected their advertising revenues. I was brought into consult and analyse their current vBulletin forum usage profile, advise and migrate them to a more appropriate server hardware and web host. Monthly costs wise, they ended up paying 5-10% more than their old Pentium 4 server but got a 10x times boost in performance for their vBulletin forums along with the addition of a sorely missing proper dual backup procedure in place for their forum data. The client now has the server hardware and web host who are equipped to handle their future growth easily with the added peace of mind that their forum data is adequately backed up.
- On the other side of the coin, there’s forum owners forking out massive amounts of money on very high end hardware which is overkill for their forum usage requirements or might not perform any better than a properly matched and selected server hardware. One example is server processor selection, 64x cpu threads running at 1.8Ghz vs 16x to 24x cpu threads running at 3.6Ghz.
- Then there’s middle of the road situation where the forum owners have all the adequate resources needed to meet their forum usage requirements, but are under utilising these resources due incorrect or non-optimal settings or hardware configurations. Three examples, having a 30GB sized forum MySQL database and 64GB of memory installed but only allocating 1GB of memory for MySQL server usage and wondering why MySQL server performance is poor. Or having a 4x hard drive raid array setup, and not verifying that the raid array was correctly setup and performing as it should be. In one case I had a client who’s web host didn’t correctly configure their raid array and only one of the hard drives was actually being used in a non-raid standalone manner while the other hard drives were sitting idle. Last example where a client had paid for dual Intel Xeon processors, and only 6 months into using this server and until I was hired to consult for their performance issues did I find they were only given a single Intel Xeon processor and that it was a lower model than the one ordered. So effectively they were paying for processors they never got to use! This goes back to that first 50% in terms of optimisation in making sure what server hardware and resources you have, are working optimally and as intended.
- For system and forum admins in general, not properly monitoring hard drive and partition space usage. Do you know your rate of hard drive and partition space consumption? How much time do you have until you need to start looking into hard drive capacity upgrades? One of the worse cases would be having MySQL database corruption and having not enough free hard drive space on any partitions to perform MySQL database repair and recovery procedures.
What advice do you have for Admins just starting out?
Purchase and download VMWare or VirtualBox virtualisation platforms, so you can create new local server instances on your own home computer to test out different flavours of Linux, CentOS, Debian or Ubuntu etc. Both allow snapshots to save whatever stages of work you have so you can revert to these snapshots for whatever reason. Play around, make mistakes and then learn from those mistakes.
Next step up grab a small cheap low end VPS server with SSH root access to play with. My Centmin Mod script installs on CentOS based VPS with as low as 256MB memory minimum and hovers around 100-130MB usage.
Then as always brush up on your Google-fu skills, a lot of answers are out there online just waiting for you to find and they’ll lead you to a lot of online resources, question & answer type sites, forums, mailing lists, newsgroups and sites which will hold a special place in your browser bookmark folders.
One forum (besides theadminzone.com), that you’d want to bookmark is webhostingtalk.com, you’re guaranteed to learn something new every day if you follow threads where folks are having server or hosting related issues and other ‘in the know’ system admins and web hosts voluntarily offer advice and help.
Where do you look for answers when you have questions?
Well my Google-fu is pretty polished these days
What do you know now that you wish you'd known 5 years ago?
The price of Gold and how it keeps on rising and rising! 5 year trend for Gold went from under AUD$800 an ounce to AUD$1,800+ an ounce while 10 year trend saw it rise from the lowly AUD$600 an ounce! I still remember when Gold was as low as AUD$200 an ounce! But there's quite a lot of things I wish I knew 5 years ago. What I've learnt regarding server optimisation and tuning and software etc in the past 12-18 months, I wish I knew 5yrs ago. Then again some of what I learnt in the last 12-18 months, didn't even exist 5yrs ago!
What offline activities do you enjoy in your spare time?
I'm heavily into computer hardware testing, reviewing & overclocking scene and have had gear provided by various manufacturers including Intel, Asus, Gigabyte, Corsair, G.Skill and OCZ (use to be a beta tester for their products as well). So anything related to latest computer hardware and overclocking or tweaking and tuning computer hardware and software to squeeze out the best performance peeks my interest. I have the following computers overclocked using various methods of water cooling on the processors - Intel 980X Gulftown @4.8Ghz, Intel 2700K Sandy Bridge @5Ghz, Intel Core i7 3930K Sandy Bridge-E @4.8Ghz and my main work computer is Intel Xeon W3550 Nehalem @3.6Ghz. Can’t wait to try out the new Intel 3570K & 3770K Ivy Bridge and Intel E5-26xx Hexa and Octa core Sandy Bridge-EP based processors.
See the pattern? I just love tweaking away to get the most of computer gear or web servers. If I could do both – tweak an overclocked web server, then I’d be really happy Besides computer hardware, I'm also into photography with my Pentax K-7 DSLR and many DSLR lenses and Japanese Anime and avid television watcher.
What Websites do you visit for fun and relaxation?
Majority are computer hardware and technology review and news sites along with usual list of web hosting and web tech (WHM/Cpanel, Apache, MySQL, Varnish cache) related forums and web sites to keep on top of latest happenings.
What are your favourite books? TV shows? Music? Foods? Beverages?
Been ages since I read a book for fun (besides for learning), but any books by John Grisham would be something I'd go out and buy. TV shows too many to list, but a few in the western world include The Walking Dead, Falling Skies, Supernatural, Dexter, House, Tru Blood, Suits and White Collar. I also get satellite TV from Hong Kong via TVB Australia so there’s also a few interesting Hong Kong tv series coming out that I may follow particularly if it’s action / martial arts related. One of my favourite series was The Master of Tai Chi – sample on [ame="http://www.youtube.com/watch?v=zHkLNvKgJUA"]Youtube[/ame]. Then there's anything Star Trek and Japanese Anime related
Music wise, nothing in particular I just listen to whatever I feel sounds good at the time. I’m currently watching the Australian version of The Voice and I’m liking the music that one of the contestants, Lakyn Heperi is performing. His battle round song has currently broken through to the Top 10 on Australia’s Aria Charts. Probably, my favourite is Lakyn’s rendition of [ame="http://www.youtube.com/watch?v=63MWgzl8ef8"]I Need a Dollar[/ame] and [ame="http://www.youtube.com/watch?v=al84roZbliE"]Red Hot Chilli Pepper’s Under The Bridge[/ame]. Lakyn inspired me to buy a guitar – now I need to learn how to play
Food and beverages wise, I love my pastas, cheese cakes, danishes and any dishes with chicken and a chilled glass of milk or water. I’ve never drunk any alcohol in my life.
Do you have any pet peeves?
Quite a few, rudeness and general lack of manners is probably at the top of my list. If I say good morning to someone, I’d expect the same in reply. If I answer a phone call, I’d expect the person on the other end to say who they are and why they’re calling first. Ever gotten a phone call and the first words out of the line is, ‘can I speak to…?’
What does the future hold for George Liu?
I wish I was a fortune teller, unfortunately I’m not. I’d definitely like to see how far I can take and develop my above mentioned projects and interests. Lately, been dabbling more in multi-server load balanced server setups for clients and myself, so I’m sure this will develop more in the future. Then there’s learning how to play the guitar!