Why I’m Switching Back to Crossbrowsertesting (vs. Browserstack)

This is a follow up to an older post I did a couple of years back where I detailed out why I had made the decision to move from Crossbrowsertesting.com over to Browserstack.

The guys over at CBT got in touch with me and offered a few free hours to go back in and test out the upgrades they’d made to their site, and I’ve recently had the opportunity to do a little testing there.

To cut right to it, CBT has greatly improved their interface. Not only does it look sharp, like a modern day web app, but it’s pretty fast…something I can’t always say about Browserstack anymore. In fact, Browserstack is down right slow much of the time.

I often operate on a Verizon MiFi Jetpack, aka over a cellular connection, and I live full-time on the road, so I’m also quite often in remote areas with little bandwidth, so I’ve just been forgiving of Browserstack and assumed it was my connection. Today though, while testing both platforms on a Comcast cable connection, right in a big city, Browserstack still lost out.

a screenshot of the CrossBrowserTesting interface
Screenshot of Crossbrowsertesting.com’s new interface.

Testing Time to Render a Page

Crossbrowsertesting
Fastest: 10 seconds
Slowest: 15 seconds
Connection Issues: None
Browserstack
Fastest: 8 seconds
Slowest: 36 seconds
Connection Issues: Several times I would just get an error message after 20 seconds or so, at which point I’d need to start all over.

Overall though, it’s not that big of a difference. The dropped connections are definitely something Browserstack should work on, but unlike issues I’d had with CBT back in 2012, I’ve never had another complaint with Browserstack, and never had any trouble using their interface. Some might argue it’s a bit outdated by today’s standards, but I’m not one of them. Still, CBT is looking sharp…

The biggest issue for me, and I’ll assume with other freelancers like myself, is pricing. While $19.95 / month certainly isn’t a huge factor, it’s just another bill. Cell phone and internet connections. Adobe Creative Cloud. Time tracking software, hosting, domain names, SSL certificates…all of this stuff adds up.

$19.95 doesn’t exist anymore, though. That was 2010 pricing with Crossbrowsertesting, and even my current $19 / month subscription with Browserstack doesn’t exist for new customers either. Both companies have raised their prices significantly. It used to be hard to just come right out and say who was providing the better value. Crossbrowsertesting is way cheaper, even at current pricing. The issue with them previously was that you had a certain number of minutes to do browser testing with them–and I am adamantly against any type of minutes or throttling of a service, be it cell phone minutes or data connections.

So the big reason why I’m moving back to Crossbrowsertesting is that they’ve removed this “minutes” strategy.

Add to that my previously wonderful experiences with their customer support and I’m once again sold.

For those interested in what you get from each service, here’s the breakdown:

Cost of Service

Crossbrowsertesting
Monthly Price: $29.95 for unlimited live browser testing*
Extra Fees: None*
Total Monthly Cost: $29.95
Browserstack
Monthly Price: $39 for unlimited live browser testing
Extra Fees: $99 / month if you want automation, another $39 / month if you want screenshots captured instead of doing it live. I don’t pay extra for those services with my plan, but I also don’t use them.
Total Monthly Cost: $177 for everything.

* Note that this is for live browser testing. They still have a “minutes” concept in place for Automated Screenshots, as well as Selenium and Javascript testing. I don’t use these services all that often, and am typically quite happy just to test in Internet Explorer(s) via the live testing mode.

Quite frankly, all of those numbers sound crazy to me. I’m very thankful that I got in on both company’s early pricing models before they raised them, and both companies did the right thing by grandfathering early adopters in. Crossbrowsertesting even allowed me to go back to them at my old pricing, though I hadn’t been a customer in nearly a year.

I don’t know the details of what it takes to keep a company like this successful. There are employees, machines to keep healthy, storage and office space for those machines and employees… Obviously, if a company can’t stay profitable, I would consider it worse to subscribe to their services only for them to go out of business rather than complain about said pricing.

But…look at it this way, if you purchased the devices yourself (and no, not every device they offer, but the basics), you’d be looking at something like the following. Again, I’m basing this off of my own needs. I use a Macbook Pro for all of my design and development, so that is a cost I have even now, and so the price isn’t included, though I’ve listed it for clarity of browser coverage. Same with my iPhone.

Macbook Pro
Safari, Firefox and Chrome on a Mac
Cheap PC, $200
Internet Explorer, plus the other three browsers on a PC
Used Android Phone, $150
Android default browser and Chrome
iPhone
Safari, Chrome
Used Windows Phone, $200
Mobile IE
Used iPad Air, $300
Safari
Used Android Tablet, $200
Android default browser, Chrome

That gives you a total of $1050. Those devices could theoretically last for two years, especially if you don’t use them for anything but browser testing. So the price then becomes $525 / year. Of course, anytime a device broke or just became obsolete (for example, an OS upgrades and your version of a device doesn’t support the upgrade, which does happen with Apple devices every couple of years).

$525 compared to Crossbrowsertesting’s $359 / year, and the $468 that Browserstack wants (not including any services except live testing). Owning your own devices that you can hold in your hand and which would possibly even become cheaper as time goes on (say if you could manage to get a Windows machine to last longer than two years, as if) begins to look a little more realistic. And those prices above for buying all of your own devices are also generous towards the browser testing services, realistically you could find cheaper versions of devices or scrap the tablets altogether for emulators, which is what Browserstack uses. Crossbrowsertesting does pride itself on using real devices whenever possible.

Here’s how Mike Bowen (of CBT) described it:

The bulk of our mobile offerings are real, physical devices. We have banks of iOS and Android phones and tablets that we host at our facility. When you test on those, you are connecting to a real device, not an emulator. We do also offer some simulators and those are clearly marked as such. It is a big differentiator for us. That one price gets you access to all of our services including access to real, physical devices.

He went on to clarify that Browserstack only offers emulators, and that comes at an additional cost as well.

For my needs, Crossbrowsertesting is cheaper. I also don’t have a ton of room with my lifestyle to lug around extra computers, etc. It’s also faster than Browserstack, so I lean once again toward it being the better route. Customer service is still stellar. Browser testing is just not as necessary and time consuming as it used to be. The browsers have gotten pretty close to one another. Inspectors make it easy to figure out what’s wrong, what any differences are being caused by.

Do yourself a favor, if you’re on Browserstack right now (and don’t have a grandfathered monthly price), head over to CBT and give their free trial a go.

I think you’ll find the experience just that much smoother that you’ll become a customer yourself.

Up Next: Made in Canada SVG