Blast from the Past: Load Testing the 1940 Census

Two of the most exciting things in the world - load testing and the US census – recently came together to provide us with an interesting case study in launch preparation. I’m talking of course about the new (and free) website on the 1940 census that crumbled under huge traffic last week.

The story is interesting on a number of fronts (particularly, how it was paid for), but for the sake of this blog, I want to stay focused on the load requirements put in place prior to the site’s launch. It’s something we see here at uTest quite frequently: companies simulate what they consider to be an absurd amount of traffic, only to have that figure exceeded after launching. Not the worst problem to have, but a problem nevertheless. This can be caused by a huge media pickup or, as was the case with Inflection, becoming a trending topic on Twitter.

So how well prepared were the operators of the site? Here were the contractual requirements in terms of load testing, according to msn.com:

  • “When browsing from one image to another, each image should be presented to the user in 3 seconds or less.”
  • “When moving from the standard rendered image to each zoom level (e.g. zoom 1x, 2x, 3x), the reformatted image should be rendered in 2 seconds or less.”
  • “Support up to 10 million hits per day while providing response times of less than three seconds for keyword searches of the descriptive metadata.”
  • “Support up to 25,000 concurrent users.”

And how far off were they? Inflection’s general manager was quoted as saying, “We were expecting a flood, but we got a tsunami.” Here were the hard numbers:

Continue Reading

Essential Guide to Mobile App Testing

60% of Users Expect Mobile Sites To Load in 3 Seconds or Less

That’s right, if your mobile site doesn’t load within THREE seconds 60% of visitors will abandon it. And, seeing how we’re not a very patient society anymore, 57% of mobile users will only give your site an extra two seconds (bring the grand total up to five) to load before aborting the mission. (According to data pulled from a Compuware survey). And if your site is slow just once it doesn’t matter, the damage has already been done. Need some more convincing? Check out this infographic to find out why load testing is so important.

Mobile Web Infographic

And remember, your mobile site doesn’t need to load well only in the lab. It has to work out in-the-wild … because apparently 67% of people use their smartphones on a date (and they’re probably in the middle of a crowded restaurant). From Tatango:

 Smartphones Usage By Event

Essential Guide to Mobile App Testing

For the Love of Holidays – Load Test!

Not Load Testing Your Site Can Ruin Valentine's DayIt used to be that people visiting your website were sitting in front a computer – traffic to your site may peak on specific days, but they still needed to be at a computer. Load testing for that scenario isn’t good enough any more. Now your site must be prepared for even more traffic because in today’s mobile world people can access the web-anywhere, anytime. Want to snag that Black Friday sale but still in a turkey coma? No problem, grab your smartphone and buy that Blu-ray player from the comfort of your bed. Stuck in a board meeting and realize you forgot to send your sweetheart flowers for Valentine’s Day? No problem, tap tap tap and it’s done. That is, if the site you’re ordering from was prepared for the holiday traffic.

Time and time again companies fail to prepare their websites for high traffic days. I’m not talking “it’s payday and I’m going to buy a new pair of shoes,” I’m talking about Black Friday, major holidays, extensively advertised product launches, and now, Valentine’s Day.

Last November several e-tail sites crashed under the weight of the Black Friday shopping rush. Target underestimated the popularity of its new, highly advertised, Missoni for Target line and left customers frustrated after its site crashed repeatedly. How many Coca-Cola polar bear commercials did you see during the Super Bowl? Enough to think that Coke would be prepared for heavy web traffic. They weren’t (and neither was Acura or the site for the film Act of Valor).

After all these high profile site crashes you’d think companies would begin to do extra load testing leading up to especially high traffic events. And you’d think sites focused on jewelery, chocolate and flowers would take extra care to prepare for Valentine’s Day. Alas, they still haven’t gotten the message, and it’s hurting their sales. From pingdom:

Valentine’s Day is a great day for any vendor selling flowers. Over the years, a large number of websites selling flowers have sprung up, and as you might expect, many of these websites are flooded by eager shoppers on February 14 wanting to buy flowers and gifts for their loved ones.

This is big business. Americans are expected to spend $18.6 billion on Valentine’s Day gifts this year.

Now here is the catch. Every year, some of these websites won’t be prepared to handle the increase in visitor traffic and as a result they slow down significantly, or even crash under the pressure. …

Continue Reading

Essential Guide to Mobile App Testing

Introducing The New & Improved @uTest: Now With Security & L10N Testing

For those who frequent the uTest website, blog or forums, you may notice a few changes around here. Yes, we had a little work done… nothing major, just going from focusing on one testing service (functional) to a providing whole range of testing services that span the dev lifecycle (functional, security, load, localization and usability testing).

For our customers, this means they can find the testing expertise they need, no matter where they are in their SDLC. And for testers, it means provide more earning opportunities for those individuals with expertise in areas like security testing, performance engineering, or localization validation. Like I said, nothing major. </sarcasm>

In all seriousness, these are exciting times around the halls of uTest. We’ve spent the past 12 months trialing new types of testing services with select beta customers. And now, we’re ready to offer them to any and all companies, on demand. A quick introduction to uTest’s new suite of testing services:

Functional testing services to help ensure your applications function as intended. Our services related to functional testing include exploratory testing, test case execution, test case creation and writing automated test scripts.

Security testing services to help you avoid launching products with common security- and privacy-related vulnerabilities. Our services include tools-based static and dynamic security testing, as well as manual penetration from trusted, white hat security testers.

Load testing services to make sure your app is ready for peak traffic, and that performance won’t degrade under heavy load. Our services include live load, simulated load and a hybrid load offering that combines cloud-based load testing with live testers.

Localization testing to validate that your app is saying what you think it’s saying. Services include translation validation from native speakers who live in-market, as well as full L10N testing that covers content translations, currency, taxes, shipping options and more.

Usability testing to help you launch products that are intuitive, clean and achieve high conversions. Services include surveys-based testing with targeted focus groups (by age, gender, education, hobbies, location, etc) or usability audits from one of our UX experts.

Special thanks to our friends at Stein + Partners for all their help with our rebranding, as well as an epic month of late nights from the amazing uTest crew. And finally, a word of thanks to our testers for their help in this launch, and the dozens of customers who helped us learn so much about each of these new types of testing. If you’d like more info about any of these new services, drop us a note.

We’ve got more on the way in the coming months. We’re not going to rest until we’ve completely reinvented the way testing services are provided in this ever-evolving apps universe.

Have a comment? Want to tell us you hate/love the new look? Drop us a comment and let us have it!

Update: Mike Butcher over at TechCrunch just took this news prime time. Seems we’re not the only ones who recognize the need for better app security testing.

Essential Guide to Mobile App Testing

A Lesson In Load Testing Your Snow Tires

One thing I’ve learned in life is that it’s important to test everything to prepare for worst-case scenarios — whether it’s software, emergency response, or simply your new snow tires in a desolate parking lot.

My in-laws are recent transplants to the New England area. I don’t think they were prepared for snowfall depths measured in Boston Celtics players, but they certainly got it. My father-in-law was reluctant to buy snow tires for his four-wheel-drive SUV, but after several convincing articles, he quickly purchased and installed them.

The next time I saw him I asked him if he had “tested” them yet. He gave me a quizzical look, and I knew I’d have to take him to an empty parking lot and pretend we were teenagers doing doughnuts all over again. My point was that he’d never know the capabilities and limitations unless he tested them in a (relatively) controlled environment.

The same goes for load testing.

Yesterday, we were hit with an unusually messy, two-part storm that involved precipitation in every liquid state. Morning commuters knew this would impact Boston’s public transportation system’s (the MBTA) train schedule. In an effort to avoid standing outside in rain/snow/sleet, everyone logged onto the MBTA’s website, which inconveniently wasn’t able to handle the demand and the site was brought to its knees.

In a most critical time when people needed updates the most, the site was unable to help. This is a prime example of why it’s important in the software testing world to test in controlled environments. Had the MBTA done more, they could have flushed out potential errors in advance and served their customers better.

(It’s worth noting that the Mass DOT & MBTA are far from strangers to technology and do a wonderful job utilizing Twitter. You can follow them @MassDOT & @mbtaGM.)

Essential Guide to Mobile App Testing