A look into the terrible interfaces of the two most popular online A/B testing solutions, and how their designers fail miserably at both UX and usability.
VWO, Adobe Target and Terrible A/B Testing UX Design
Note: This article was originally posted on my Medium.com account here, and rejected from at least one publication for being too negative.
As I’m sure you’re all aware, good UX design is not exactly common place in today’s software world. Whethers it in social media sites or games, mobile apps or enterprise solutions, almost everything product and industry seems to be littered with poorly designed, user hostile solutions that feel like they’re cxplicitly waste everyone’s time and money.
It’s pretty sad really, especially given that the worst offenders are products like Skype and LinkedIn, both of which have billion dollar corporations behind them and whole teams of experts who should have the ability to make something great instead.
Yet as bad as they are, nothing compares to the level of hell that’s called A/B testing. Why? Because as I’ll discuss today, both the major options in this space are horribly designed scripts and services that make even the worst of Google and Microsoft’s blunders look well thought out by comparison, and show a complete disregard for everyone’s time, money and general sanity.
So let’s look at them shall we? Starting with VWO, which may in fact be the better of the two.
If better means ‘doesn’t crash on a regular basis’, since VWO is still an absolute trainwreck in every other way that matters.
Which in turn becomes immediately apparent the moment you try to add extra media to a test created in the tool’s code editor. Why? Because for some reason I can’t begin to fathom, VWO has decided that anyone using the code editor should not have any way to upload extra images or other types of media at all.
No way to upload images exists in the code based editor.
Yes really. Despite the visual editor letting you click an image to upload and replace it with a new one, the code editor has nothing of the sort. This means that the best solution for replacing images here is to open a dummy visual editor built test in one window, the code based one you’re working on in another, and to manually copy the address from one to the other once its finished uploading.
It’s basically a whole bunch of extra work to get around the fact VWO’s developers forgot how to add a file uploader.
But the woes don’t end there. Oh no, do you wonder why I told you to open a new window just now?
Yeah, I guessed you might. Well it turns out VWO also doesn’t like you opening different projects at the same time, and will flat out refuse to let you switch between ‘accounts’ in the same browser session.
And it’s not exactly subtle about it either, oh no, the tool will loudly yell at you should you dare to switch accounts with an editor window open at all.
Why do you care about this?
Plus it’s quite forceful about other contexts too; closing the window with the preview button will also close the actual preview window/tab, despite the fact the latter has absolutely nothing to do with the former. Pretty stupid really, especially given it must have required extra coding to add in this very specific piece of behaviour.
The tool also has serious problems with dropdowns too. These are kept in tiny boxes that get longer whenever you open one, but revert the minute you remove focus from the select box. This means that scrolling down to certain options is a nightmare, and becomes increasingly obnoxious the more conditions you add.
However, all of these pale in comparison to the tool’s biggest problem. One so dumbfounding you genuinely have to wonder how it got through the design stage at all.
Namely, you can’t preview a test, without saving it.
Why is this an issue?
Well, two reasons really:
- There’s no draft option, meaning you have to setup the targeting, code and goals to actually save anything for testing purposes.
- And worse still, the toolactually suggest you put the test live before you’ve had the chance to preview it in a real web browser!
This message should scare the hell out of anyone who’s a programmer, web developer or software engineer
Yeah, we’re not joking about that. VWO is explicitly designed to encourage developers to make their tests live before checking they work properly, especially given that the built in editor likely doesn’t match how it appears in a web browser at all.
It’s like a game making tool which suggests you ‘upload to Steam’ before checking if the game’s code compiles or not. Forget bad design, this is downright dangerous, irresponsible design, and that may well be ten times worse.
Either way, it’s obviously a pretty bad system, and one which should logically be unsurped by a better competitor any minute now. Problem is, that won’t happen, since said competition is even worse. Like Adobe Target, which has its own fair share of UX blunders…
Such as the complete inability to upload media at all. You can replace anything you want, but you need to host it yourself, making things awkward for external agencies needing to work on a site with it.
Still, that may be because Adobe’s Marketing ‘Platform’ is aimed squarely at ‘enterprise’ users, with even basics like support being something you explicitly have to authorise in order to use. Got a problem with the tool? Work with an agency or external development company?
Yeah you’re basically just screwed, since Adobe won’t even talk to you, and will ignore your emails, calls and support tickets altogether.
And that applies to pretty much anyone in the company too. If you’re not explicitly allowed to contact Adobe about this, they have no interest in helping you. Presumably even if you’re the company CEO. Or tech lead.
The actual program ain’t much better. For one thing, it loves giving you random errors like this one at annoying times:
This gets increasingly likely to happen the more changes you’ve made in the window, and is virtually guaranteed to do so if you’ve used Adobe’s visual editor in any way whatsoever.
Get used to seeing messages like this a lot
In fact, said visual editor is so buggy and inefficient that heavily edited tests will often flat out refuse to load altogether. For example, one of my early projects relied heavily on the editor, and is now so unstable that the editor errors out 9/10 its loaded.
As a result, you quickly realise the visual editor is a bloody liability, and avoid using it for anything other than the most trivial tweaks.
Still, given the code it generates, you’d be wise to avoid it most of the time anyway. No joke, the code Adobe’s tool creates is so obtuse and so ridiculously specific that a div being moved about half a pixel on the page will cause the whole thing to crash down like a house of cards.
That’s because instead of doing things the sane way and targeting ids or classes like this:
It instead tries to use generic selectors with nth-of-type sprinkled liberally throughout. Like this:
document.querySelector(‘div:nth-of-type(3) > div:nth-of-type(1) > div:first-of-type’)
Here’s a picture showing how bad one of Adobe’s ‘selectors’ is
Which kills the whole test whenever anything is changed whatsoever.
And it’s not like you can avoid it by writing your own code either. No, your goals in Adobe Target can’t be set like in VWO with a custom class name or ID, but have to be ‘selected’ through a visual interface.
This generates the exact same kind of crap code as the visual editor does, and means your goals will fall apart if one or more pages has a slightly different structure compared to the rest.
Want to fix them? Tough luck, there’s no text editor here at all, so you’re stuck with whatever crap Adobe generates (likely while wishing horrible things on Adobe’s development staff).
Still, it’s not like the editor itself is any better outside of the selectors. Oh no, it’s a UI trainwreck in every way.
For starters, even loading the damn thing is a nightmare. You see, Adobe sucks at security. Like, really sucks at security.
And they seemingly don’t have a clue how to load content in a secure way. This means that to even enable the editor, you basically have to disable all forms of tracking protection in your browser, to the point of often having to install a browser add on to disable CORS protection. Otherwise you get to this screen and that’s as far as you go:
When your documentation says to edit about:config preferences, you know you’ve screwed up
Getting past it brings you to the actual editor, which (as mentioned already) is about 30% likely to error out without loading anything. If you do load it though, it’s got its own usability issues.
Like the setup for adding custom code. You’d think it’d save once you clicked the save button, right?
It’s blue and its says save. Logically that means it saves something.
No, that just… sort of reloads the preview with the changes applied? Yeah, actually saving requires clicking the arrow next to the blue button, then selecting save from there.
Yeah, this is not a logical place to look to save your work
This is pretty confusing, and the former setup means you’re pretty likely to lose all your work by random chance anyway. I mean, any page load is chance for Target to drop down dead after all.
So the practical solution you fall back to is writing all your code in an editor offline, then only copying it into Target when you’re not worried about losing it. Though to be fair, you probably should be doing that anyway, since actually keeping track of lengthy bits of code in Target’s tiny text boxes is going to drive you insane anyhow.
And setting up the targeting isn’t much better. Oh no, your options at running tests on certain pages are limiting to say the least. For instance, you can’t choose to run a test based on multiple and/or conditions at once. Only ‘if this condition is true and/or these other conditions are all true’:
All conditions after the first one in the second section are ‘AND’ conditions.
It’s pretty limiting like that. Fortunately, you can get around it by adding ‘additional pages’ for the test to run on, but if you do…
Well it turns out the extra pages don’t copy over the code added previously. This means you have to manually set up the JS and CSS changes you made on every page you want the page to run on.
It’s slow, its clunky, and in an error prone tool like Adobe Target, absolutely inexcusable in every way.
All in all, Adobe Target is somehow even worse than VWO is, despite being by a team with significantly more resources and expertise behind it. How that’s possible only Adobe knows.
So in conclusion, both VWO and Adobe Target are terrible systems that fail miserably at providing a good user experience, and which clearly went through very little testing in their own right. If you have any choice in the matter whatsoever, avoid both of them at all costs, and give your money to a company that knows what UI, UX and usability design means at all.
Thanks for reading.