Two of the most exciting things in the world - load testing and the US census – recently came together to provide us with an interesting case study in launch preparation. I’m talking of course about the new (and free) website on the 1940 census that crumbled under huge traffic last week.

The story is interesting on a number of fronts (particularly, how it was paid for), but for the sake of this blog, I want to stay focused on the load requirements put in place prior to the site’s launch. It’s something we see here at uTest quite frequently: companies simulate what they consider to be an absurd amount of traffic, only to have that figure exceeded after launching. Not the worst problem to have, but a problem nevertheless. This can be caused by a huge media pickup or, as was the case with Inflection, becoming a trending topic on Twitter.

So how well prepared were the operators of the site? Here were the contractual requirements in terms of load testing, according to msn.com:

• “When browsing from one image to another, each image should be presented to the user in 3 seconds or less.”
• “When moving from the standard rendered image to each zoom level (e.g. zoom 1x, 2x, 3x), the reformatted image should be rendered in 2 seconds or less.”
• “Support up to 10 million hits per day while providing response times of less than three seconds for keyword searches of the descriptive metadata.”
• “Support up to 25,000 concurrent users.”

And how far off were they? Inflection’s general manager was quoted as saying, “We were expecting a flood, but we got a tsunami.” Here were the hard numbers:

Read more…

That’s right, if your mobile site doesn’t load within THREE seconds 60% of visitors will abandon it. And, seeing how we’re not a very patient society anymore, 57% of mobile users will only give your site an extra two seconds (bring the grand total up to five) to load before aborting the mission. (According to data pulled from a Compuware survey). And if your site is slow just once it doesn’t matter, the damage has already been done. Need some more convincing? Check out this infographic to find out why load testing is so important.

And remember, your mobile site doesn’t need to load well only in the lab. It has to work out in-the-wild … because apparently 67% of people use their smartphones on a date (and they’re probably in the middle of a crowded restaurant). From Tatango:

It used to be that people visiting your website were sitting in front a computer – traffic to your site may peak on specific days, but they still needed to be at a computer. Load testing for that scenario isn’t good enough any more. Now your site must be prepared for even more traffic because in today’s mobile world people can access the web-anywhere, anytime. Want to snag that Black Friday sale but still in a turkey coma? No problem, grab your smartphone and buy that Blu-ray player from the comfort of your bed. Stuck in a board meeting and realize you forgot to send your sweetheart flowers for Valentine’s Day? No problem, tap tap tap and it’s done. That is, if the site you’re ordering from was prepared for the holiday traffic.

Time and time again companies fail to prepare their websites for high traffic days. I’m not talking “it’s payday and I’m going to buy a new pair of shoes,” I’m talking about Black Friday, major holidays, extensively advertised product launches, and now, Valentine’s Day.

Last November several e-tail sites crashed under the weight of the Black Friday shopping rush. Target underestimated the popularity of its new, highly advertised, Missoni for Target line and left customers frustrated after its site crashed repeatedly. How many Coca-Cola polar bear commercials did you see during the Super Bowl? Enough to think that Coke would be prepared for heavy web traffic. They weren’t (and neither was Acura or the site for the film Act of Valor).

After all these high profile site crashes you’d think companies would begin to do extra load testing leading up to especially high traffic events. And you’d think sites focused on jewelery, chocolate and flowers would take extra care to prepare for Valentine’s Day. Alas, they still haven’t gotten the message, and it’s hurting their sales. From pingdom:

Valentine’s Day is a great day for any vendor selling flowers. Over the years, a large number of websites selling flowers have sprung up, and as you might expect, many of these websites are flooded by eager shoppers on February 14 wanting to buy flowers and gifts for their loved ones.

This is big business. Americans are expected to spend $18.6 billion on Valentine’s Day gifts this year.

Now here is the catch. Every year, some of these websites won’t be prepared to handle the increase in visitor traffic and as a result they slow down significantly, or even crash under the pressure. …

Read more…

For those who frequent the uTest website, blog or forums, you may notice a few changes around here. Yes, we had a little work done… nothing major, just going from focusing on one testing service (functional) to a providing whole range of testing services that span the dev lifecycle (functional, security, load, localization and usability testing).

For our customers, this means they can find the testing expertise they need, no matter where they are in their SDLC. And for testers, it means provide more earning opportunities for those individuals with expertise in areas like security testing, performance engineering, or localization validation. Like I said, nothing major. </sarcasm>

In all seriousness, these are exciting times around the halls of uTest. We’ve spent the past 12 months trialing new types of testing services with select beta customers. And now, we’re ready to offer them to any and all companies, on demand. A quick introduction to uTest’s new suite of testing services:

Functional testing services to help ensure your applications function as intended. Our services related to functional testing include exploratory testing, test case execution, test case creation and writing automated test scripts.

Security testing services to help you avoid launching products with common security- and privacy-related vulnerabilities. Our services include tools-based static and dynamic security testing, as well as manual penetration from trusted, white hat security testers.

Load testing services to make sure your app is ready for peak traffic, and that performance won’t degrade under heavy load. Our services include live load, simulated load and a hybrid load offering that combines cloud-based load testing with live testers.

Localization testing to validate that your app is saying what you think it’s saying. Services include translation validation from native speakers who live in-market, as well as full L10N testing that covers content translations, currency, taxes, shipping options and more.

Usability testing to help you launch products that are intuitive, clean and achieve high conversions. Services include surveys-based testing with targeted focus groups (by age, gender, education, hobbies, location, etc) or usability audits from one of our UX experts.

Special thanks to our friends at Stein + Partners for all their help with our rebranding, as well as an epic month of late nights from the amazing uTest crew. And finally, a word of thanks to our testers for their help in this launch, and the dozens of customers who helped us learn so much about each of these new types of testing. If you’d like more info about any of these new services, drop us a note.

We’ve got more on the way in the coming months. We’re not going to rest until we’ve completely reinvented the way testing services are provided in this ever-evolving apps universe.

Have a comment? Want to tell us you hate/love the new look? Drop us a comment and let us have it!

Update: Mike Butcher over at TechCrunch just took this news prime time. Seems we’re not the only ones who recognize the need for better app security testing.

One thing I’ve learned in life is that it’s important to test everything to prepare for worst-case scenarios — whether it’s software, emergency response, or simply your new snow tires in a desolate parking lot.

My in-laws are recent transplants to the New England area. I don’t think they were prepared for snowfall depths measured in Boston Celtics players, but they certainly got it. My father-in-law was reluctant to buy snow tires for his four-wheel-drive SUV, but after several convincing articles, he quickly purchased and installed them.

The next time I saw him I asked him if he had “tested” them yet. He gave me a quizzical look, and I knew I’d have to take him to an empty parking lot and pretend we were teenagers doing doughnuts all over again. My point was that he’d never know the capabilities and limitations unless he tested them in a (relatively) controlled environment.

The same goes for load testing.

Yesterday, we were hit with an unusually messy, two-part storm that involved precipitation in every liquid state. Morning commuters knew this would impact Boston’s public transportation system’s (the MBTA) train schedule. In an effort to avoid standing outside in rain/snow/sleet, everyone logged onto the MBTA’s website, which inconveniently wasn’t able to handle the demand and the site was brought to its knees.

In a most critical time when people needed updates the most, the site was unable to help. This is a prime example of why it’s important in the software testing world to test in controlled environments. Had the MBTA done more, they could have flushed out potential errors in advance and served their customers better.

(It’s worth noting that the Mass DOT & MBTA are far from strangers to technology and do a wonderful job utilizing Twitter. You can follow them @MassDOT & @mbtaGM.)

In part II of our interview with SOASTA’s Dan Bartow, we get his thoughts on why load testing often gets neglected; advice for test tool selection; the challenge of load testing for mobile apps; SOASTA’s plans for 2011; fights between flying sharks vs. flying crocodiles and more. If you missed our previous segment, you can read Part I here. Enjoy!

uTest: True or false: Load and performance testing is often one of the most neglected phases of software quality. Please explain why this is (or is not) the case.

DB: True!  The time allotted for QA in the software development lifecycle has always been the first thing to get squeezed when a project gets behind.  Traditional software methodologies such as Waterfall essentially go from requirements to development and ultimately QA at the tail end.  When the project development activities get behind then the only thing left to cut from and still deliver a product on time is the QA cycle.  Performance still isn’t a part of many project plans today (this is almost a separate topic in itself), but when it is in the project plan it usually gets a slice of the QA time which is already too short in most cases.

Now we live in an agile development world and while agile functional QA is catching up we still don’t have agile performance testing as an industry standard.  The reason for this is that the dominant product in lab performance testing, HP LoadRunner, requires you to write code for your performance tests that is more complex than the actual web application code you’re testing.  If you have to write your test cases in C and it takes two weeks to write an end-to-end scenario on a finished web app then you have dead weight in your dev lifecycle.  As a result of these weaknesses companies have lost confidence in the value of performance testing their apps.  The way to reinstate this confidence is with a modern testing tool and a modern approach to testing.

uTest: How important is tool selection when it comes to load and performance testing? Are testing failures a result of this or something else, like personnel?

DB: Tool selection is very important for overall success although testing failures can be because of people, processes and/or technology.  You need the right tool for the job and you need the right people to use them in a process that’s set up for success.  Just like QA isn’t a one size fits all shoe neither is performance testing.  Personally though I think most testing failures are a leadership and execution problem and not because of the tools being used or the processes.  Quality comes from the top down.  The companies out there delivering the highest quality offerings are the ones that build quality in from the CEO all the way through the company.  Probably every tester reading this knows what its like to be a QA Engineer at a company that doesn’t seem to actually care about quality.  How ironic!  I said tool selection was very important, but I really don’t even want to focus on that here because tools are just tools.  Before you worry about whether or not you have the right tools time should be spent on making sure you have the right attitudes on your team and the right players.  Once you have a good enough team that is pushing the capabilities of your toolset then I think you’ve got a foundation for success and you can start driving higher.

uTest: How does the expansion of mobile apps and devices impact load testing? Is this a game-changer? Or something current load testing is well suited for?

Read more…

Our Testing the Limits guest this month is Dan Bartow, Vice President & CloudTest Evangelist at SOASTA – an industry leader in load testing, cloud testing and performance intelligence. You’d be hard-pressed to find someone with more knowledge of these matters than Dan, whose career has included stints at Intuit, Turbo Tax, Neiman Marcus, ATG and several start-ups. You can keep up on all the latest trends in cloud computing, load testing and other topics by following Dan on Twitter, or by checking out the SOASTA blogs (here and here).

In Part I of our interview, we get Dan’s thoughts on the increasing number of DDOS attacks on commercial sites; load testing solutions for cash-strapped start-ups; a possible shortage of load test engineers; misconceptions of the cloud and other topics. When you’re done with one, be sure to read Part II.

uTest: It seems that DDOS attacks on commercial websites (Visa, MasterCard, etc.) have been in news a lot lately thanks to the Wiki leaks drama. Is it safe to say that preventative measures related to load testing are on the minds of companies now more than ever? Why or why not?

DB: Yes, it is definitely fair to say that load related concerns are hot on the minds of technology leaders… right now more than ever.  On the DDOS front I’ve seen a marked increase over the past year of companies searching for security testing offerings.  More importantly on the load and capacity front it’s hard to go a single day without reading about an outage or experiencing one yourself on a site that you like to visit.  The reasons why companies are taking preventative measures are mostly self-evident.  Crashing means losing revenue and impacting your brand.  In the online world customers have lots of choices so if they have a poor experience on a site there’s usually another alternative that’s just a click away.

Testing to and above projected peak load in the production environment is the new minimum bar of entry for deployed applications.  In the next few years I think you will be hard pressed to find companies that are not doing this as part of their software development lifecycle and regular operations.  Google routinely pulls the plug on entire data-centers as part of their monthly operations.  The New York Stock Exchange used to go down at close of business on Friday and performance test the trading floor systems all weekend.  Then they would clean it up and have it ready to go again Monday morning – and this was in the eighties!  It’s time we got with it in terms of testing our mission critical web applications for peak load.

uTest: What advice do you have for smaller companies who think that load testing is out of their budget?

DB: An outage for a small company during a peak time can be catastrophic.  I recently saw a software-as-a-service startup company get the opportunity to be integrated into the web site of one of the world’s largest online retailers for their biggest day of the year.  Sure enough they crashed… impacting not only their reputation with said retailer and damaging that relationship, but also impacting every customer that visited the retailer’s site.  Bad stuff.

I don’t consider performance testing optional if you’re making money online.  Here is how I would think about it:  “What is the cost of an outage to my company?”  Think about it in terms of dollars lost as well as customers lost.  What are the short-term impacts and the long-term impacts of going down?  Companies with good ideas have faded away entirely because of poor performance (a earlier competitor to MySpace comes to mind).  Once you understand the dollar cost of an outage then it’s easier to make a case to management for budget to spend on testing.  Also it’s been my experience that all of the sudden a testing budget materializes out of thin air when you actually do have a crash in production.  The challenge is getting people to proactively test before something bad happens in production.

Read more…

Last week, a company called Skyfire launched an app that made it possible to watch Flash movies on an iPhone or iPad. Their app uses servers hosted in the cloud to convert Flash videos to HTML5, making them viewable on iOS devices where Flash is not available.

Now we’ve heard before that nobody wants Flash on their phones, so you would understand that Skyfire had a slow first day after their launch. And by slow first day I mean their servers were so overloaded by the demand that they couldn’t provision them fast enough. Eventually they had to pull their app from the app store and declare it “sold out.”

We here at uTest think that among the things in this world that could be called “problems,” having so much demand that your servers can’t keep up is definitely a good problem to have.  Skyfire did, after all, make over $1 million in their first weekend on the app store.

On the other hand, nobody wants their servers to crash because of excessive load. The trouble is that properly planning for heavy load is often something that can vary from one application to another. (The load planning for Skyfire is probably very different from Facebook.) Still, there are some very simple load planning tips that almost everyone should know:

Read more…

It’s been a busy few weeks around the hallowed halls of uTest (ok, they’re really more of a beige).  Somewhere between delivering the keynote at the QUEST conference, interviewing testing guru Scott Barber, and launching v3.0 of our platform, we found time to sign a partnership with SOASTA.

For those who don’t know them, SOASTA leverages the cloud to offer on-demand load & performance testing, which enables companies to test the real-world performance of their web apps.  In fact, the Wall Street Journal recently named them one of the top 50 VC-backed companies in all the land.  So what’s a crowd company doing partnering with a cloud company?  A lot, actually.

For starters, this partnership expands upon uTest’s recently announced load testing services.  As our customers and their apps become more sophisticated, they’re turning to uTest to provide them with real-world load testing.  And this partnership with SOASTA enables us to offer a unique package of on-demand load testing and advanced analytics to help our customers identify bottlenecks and points of failure between their apps and their users.

Read more…

It’s back to the basics for the US Census. After several failed attempts to “modernize” the national headcount with hand held computers, officials are now blaming the resulting confusion, delays and costs on – wait for it – software bugs. You can read more about it here. To save you some time, I’ve summarized the progress made along the way with some telling quotations from those in the know:

Phase I: “Critical software errors are increasing; system performance is still lagging and testing continues to be compressed. A shortfall in testing portends potentially significant technical problems in the field.” – John Thompson, former associate director of the Census Bureau

Phase II: “The testing of the system is continuing to reveal critical defects.” – Todd Zinser, Commerce Department Inspector General

Phase III: “The performance of this system is not taking the load we’d like. It’s not going to accept the load that we will need to get it in about a month or so.” – Robert Groves, Census Director

Results: “They have prepared to do the follow-up using pencil and paper, and they’re very good at that.” – John Thompson (again)

Read more…