Software Testing Blog

We here at uTest sometime find that other blogs write stuff a heck of a lot better than we can. And nobody writes lists of things better than the fine folks at Cracked. We especially enjoyed their recent post: 6 Disasters Caused by Poorly Designed User Interfaces.

Read it and learn how poor UIs resulted in the USS Vincennes shooting down a passenger airline, the Herald of Free Enterprise sinking due to a door left open, and Three Mile Island releasing excess radiation during a partial meltdown.

Click here to read the whole thing.

As I scan the software testing stories of the day, I’m amazed at the frequency of certain misconceptions. While there are too many to list, I wanted to share five of the most common testing myths (in my brief experience). The first three I find to be prevalent in mainstream news articles, while the other two are more common within the tech industry in general.

Take a look and see if you agree with me.

Myth 1. Testing is boring: It’s been said that “Testing is like sex. If it’s not fun, then you’re doing it wrong.” The myth of testing as a monotonous, boring activity is seen frequently in mainstream media articles, which regard testers as the assembly line workers of the software business. In reality, testing presents new and exciting challenges every day. Here’s a nice quote from Michael Bolton that pretty much sums it up:

“Testing is something that we do with the motivation of finding new information.  Testing is a process of exploration, discovery, investigation, and learning.  When we configure, operate, and observe a product with the intention of evaluating it, or with the intention of recognizing a problem that we hadn’t anticipated, we’re testing.  We’re testing when we’re trying to find out about the extents and limitations of the product and its design, and when we’re largely driven by questions that haven’t been answered or even asked before.”

Myth 2. Testing is easy: It’s often assumed testing cannot be that difficult, since everyday users find bugs all the time. In truth, testing is a very complex craft that’s not suited for your average Joe. Here’s Google’s Patrick Copeland on the qualities of a great tester:

Read more…

Two of the most exciting things in the world - load testing and the US census – recently came together to provide us with an interesting case study in launch preparation. I’m talking of course about the new (and free) website on the 1940 census that crumbled under huge traffic last week.

The story is interesting on a number of fronts (particularly, how it was paid for), but for the sake of this blog, I want to stay focused on the load requirements put in place prior to the site’s launch. It’s something we see here at uTest quite frequently: companies simulate what they consider to be an absurd amount of traffic, only to have that figure exceeded after launching. Not the worst problem to have, but a problem nevertheless. This can be caused by a huge media pickup or, as was the case with Inflection, becoming a trending topic on Twitter.

So how well prepared were the operators of the site? Here were the contractual requirements in terms of load testing, according to msn.com:

• “When browsing from one image to another, each image should be presented to the user in 3 seconds or less.”
• “When moving from the standard rendered image to each zoom level (e.g. zoom 1x, 2x, 3x), the reformatted image should be rendered in 2 seconds or less.”
• “Support up to 10 million hits per day while providing response times of less than three seconds for keyword searches of the descriptive metadata.”
• “Support up to 25,000 concurrent users.”

And how far off were they? Inflection’s general manager was quoted as saying, “We were expecting a flood, but we got a tsunami.” Here were the hard numbers:

Read more…

When asked what they enjoy most about being a member of the uTest community, testers regularly cite the variety of projects and the pay that comes with them. For proof on both fronts, you can check out the Paid Projects threads in the uTest Forums.

Unfortunately, variety and compensation aren’t always the good parts about being a tester. In fact, they can be the worst parts of the job. Case in point: The gaming industry. Specifically, video game testers.

What may seem like a dream job for many (wait, you’re going to pay me to play video games?) is actually something of a nightmare according to Jimmy Thang, a writer for ign.com. In his recent article The Tough Life of a Games Tester, Thang interviews game testers who’ve decided to break their NDAs and speak out against the tedious tasks, low pay, lack of respect and other plagues of their profession.

I highly encourage you to read the entire article,  but here are a few quick excerpts that I found interesting:

Read more…

Perhaps you’ve read about Google’s Project Glass – a set of augmented reality glasses that will provide users with real-time information right before their eyes. Literally.

After waiting awhile for an appending “April Fool’s” announcement that never came, we can now safely call your attention to this project’s unique testing challenges. Before we do that, however, here’s a good description of the project from TechCrunch:

To call these things glasses may be a bit of a stretch — early rumors noted that glasses bore a striking resemblance to a pair of Oakley Thumps, but the demo images on Project Glass’s Google+ page don’t look a thing like them. Rather, they appear to be constructed of a solid metal band that runs across the brow line, with a small heads-up display mounted on the right side.

The New York Times‘ Nick Bilton, who broke the Project Glass story today, went on to say that the prototype model seen in the images is just one of the potential designs currently in testing. Among others, one of the potential designs for Project Glass is (thankfully) meant to be attached to a person’s existing pair of glasses.

It’s also worth noting that as downright magical as these things could be, there’s still very little insight into how they would actually work. Bilton’s early write-up notes that the glasses will be capable of establishing a 3G or 4G wireless connection, but how exactly Google will shoehorn those components (just to name a few) into a comfortable headset is still up the air.

After watching the demo/concept video below, it’s clear that in-the-wild testing will have to play a major part in the quality of this product (should it ever come to fruition). And judging from the comments on their Google+ page, finding real-world beta testers will not be an issue. But that’s the easy part. Here’s a look at some heavy testing challenges they are likely to encounter:

Read more…

When asked to think about the happiest job in America, does your own job come to mind? What job do you think is the happiest job? When asking this question, did these positions come to mind: Customer Service Representative, Accountant, Bank Teller or even Warehouse Manager?  Well, according to a survey posted on Forbes.com those jobs are listed in the top 20 of “The Happiest Jobs in America.”  The study took nearly a year to compile and coming in at #1 for the Happiest Job in the US is the Software Quality Assurance Engineer!

“Since we tend to spend more waking hours working than doing anything else, our work happiness is a huge factor in our overall happiness,” says CareerBliss’ chief executive, Heidi Golledge.

Well quoted by Heidi Golledge, as this statement is very true.  I spend more time in the office and with my coworkers than I do my own home and family.  So if I work with people who share the same interests as me, as well as enjoy the work I do, work isn’t necessarily work anymore, it becomes more of a second home.

CareerBliss also found that many people appreciate their jobs more in a down economy. “As the job market is improving every day, we see that employees are looking to evaluate if they are happy in their current position and if their company is providing the type of culture they identify with,” Golledge says. “This year will be a very important year for employers as employees look at a possible career or job change to improve their satisfaction at work.”

More than 100,000 workers took part in the survey and rated factors such as workplace happiness and environment, job resources, co-worker relationships and daily tasks on a sale of 1 to 5. In the end, Software Quality Assurance Engineers came out on top.

With an index score of 4.24, software quality assurance engineers said they are more than satisfied with the people they work with and the company they work for. They’re also fairly content with their daily tasks and bosses. …

Golledge says, “In past studies, we have noted that the long hours and intense demands on software engineers’ time caused them to rank as less than happy.  However, we are happy to report that software quality assurance engineers feel rewarded at work, as they are typically the last stop before software goes live and correctly feel that they are an integral part of the job being done at the company.”

Read more…

You can do just about anything online these, so much so that it feels like an inconvenience if you can’t complete a task online. But some things are just best left the old fashioned way.

Take, for example, the act of voting. I’m not talking about voting for American Idol (which you actually can do online now), I’m talking about voting in a major, official election. While paper absentee ballots may seem outdated, voting has proved to fragile and tamper-tempting to be shifted online. We wouldn’t know that though without some good, solid security testing.

A few years ago an e-voting system was created for Washington, D.C. and in 2010 its developers reached out to security testing experts to put the system through its paces. It failed miserably. The story is surfacing again now because the processes and results of the testing were recently officially published. The testers didn’t find some exceptionally complicated flaw only detectable with a lot of out-of-the-box thinking, they were able to completely infiltrate and manipulate the program. There’s The H with some details:

“Within 48 hours of the system going live, we had gained near complete control of the election server”, the researchers wrote in a paper that has now been released. “We successfully changed every vote and revealed almost every secret ballot.” The hack was only discovered after about two business days – and most likely only because the intruders left a visible trail on purpose. …

The security experts investigated common vulnerable points such as login fields, the virtual ballots’ content and file names, and session cookies – and found several exploitable weaknesses. Even the Linux kernel used in the project proved to have a well known vulnerability. They were also able to use the PDFs generated by the system to trick the encryption mechanism, while unsecured surveillance cameras provided additional insights into the infrastructure. While the open source nature of the code made their work somewhat easier, they believe that attackers would have been able to make quick headway even if the system had been proprietary.

Read more…

We all know what happens when you rush through testing and push a new product out to market too early (hint: crashing and burning comes to mind). In the spirit of not releasing software with major security flaws, functional defects or usability missteps Traq Software has highlighted “7 Important Principals for Test Management.” Full disclosure: Traq Software sells QA management software, but the tips are good too keep in mind non-the-less. (Numbers 4 and 5 are my favorites!)

1. Make sure you have a repeatable process. A good process helps you see where you are and where you are going.

2. Don’t cut corners prior to a release. When the delivery date is getting closer there is, naturally, a tendency to want to skip some low priority test management tasks. In doing so you hope to get the product out on time. Resist this temptation.

3. Know the metrics. Defect find rates, cases executed and lines of code changed. All these metrics help you argue the case for why the product may, or may not, be ready for release.

4. Listen to the testers. Software testers are your projects headlights. They light the road at night and help you read the map. They are trying to help you get to your destination in the shortest amount of time. Ignore their advice and you can be sure you’ll end up taking the longest route to your destination.

5. Employ a good QA manager. The QA manager is like the pilot of the plane guiding the product to touch down. … good QA manager has an eye for balancing the demands of time, quality and features. He or she is worth listening too carefully.

6. Get the customer involved. The sooner your customer starts giving you feedback the sooner you can correct issues.

Read more…

For the second year in a row, uTest will be making an appearance at SXSW, the world-famous music/film/interactive conference in Austin, Texas. Unlike last year – where we spent most of our time eating, drinking and schmoozing with storm troopers – we have  some new, big plans in store.

The obvious difference is that we’ll be cruising around Austin in the RVIP Lounge, hitting up hotspots, giving rides, singing karaoke (poorly) and playing host to SXSW attendees throughout the week. More to come on that, but you can follow @InTheWildTest for deets on our adventures, and real-time locations if you’re at SXSW..

The other difference is that, instead of just talking about the merits of in-the-wild testing, we decided to show a real-world demonstration. So, over the last 36 hours, we assembled a select group of US-based testers to put the official SXSW mobile apps through their paces. In-the-wild testing means live testers, real devices, imperfect connectivity… basically, true real-world conditions. So we went to work testing SXSW’s official apps for iOS, Android, Windows Phone and BlackBerry. For iOS and Android, we also included tablet testing, to bring the comparison total to six.

Below are some top-level results (note that each category ranged from 1-5):

 ********

% of Total Bugs	17.7%	18.3%	18%	6.6%	23%	16.4%
Overall Score	4.1	4.0	4.6	4.7	3.8	4.2
Usability & Design	4.2	4.1	4.6	4.8	4.1	4.2
Features & Functionality	4.1	4.1	4.7	4.7	3.2	3.9
Application & Performance	3.3	3.2	3.9	4.0	3.5	3.7

********

Of course, these figures only tell part of the story. As the apps were tested in terms of functionality, performance, design, connectivity and other factors, several issues popped up on more than one occasion. Here were a few areas where some notable bugs were uncovered:

• Incorrect time displays
• Sync issues with registration and deleted items
• Crashes on various tablet OS versions
• Issues with installation
• Social media integration
• Issues with rating and uploading photos

It should be noted that despite these issues, the overall reaction from our community was positive for each of these applications. In fact, the overall ratings you see above are substantially higher than the industry norms, so kudos to the respective dev teams.

Anyway, if you’re at SXSW and want to learn more about In-The-Wild Testing, be sure to stop by the RVIP Lounge. If you’re not able to attend, then head on over to inthewildtesting.com.

Last month the Department of Homeland Security announced it will begin adopting agile development methods to increase the pace if its IT projects. Richard Spires, DHS CIO, said he hopes agile development will help the department complete projects more quickly and on budget – something he acknowledges the DHS has struggled with in the past. Testing is a large part of the DHS’ new approach. From InformationWeek:

It’s clear that DHS CIOs believe that agile is definitely the development modality of the future for many scenarios. “Historically, we have had a contract that has some huge chunk of functionality included, and then five years later we don’t understand why we’re not there yet,” said Charlie Armstrong, CIO of U.S. Customs and Border Protection. “We really think this is going to be our future, to help our programs be more successful.”

DHS is looking to tailor its enterprise software development policy specifically for agile development to account for how significantly agile development differs from traditional development. For example, DHS deputy CIO Margie Graves said, the agency will need to build testing into the short, few-week “sprint” development cycles for which agile development is known. These sprint cycles are designed to deliver functionality quickly, so testing must be a part of each sprint, according to Graves.

DHS is also looking at rethinking how it contracts for IT services to support more agile development, according to Spires. That thought process is playing out at an agency level as well as a component organization level.

Other government departments have also begun embracing – or at least discussing – agile development, including the FBI and the Department of Defense:

FBI CIO Chad Fulgham is using agile development to complete the agency’s Sentinel case management system, which had fallen two years behind schedule in the hands of contractor Lockheed Martin. That project continues to have its setbacks, but the delays are measured in months, not years, and the project remains within budget.

Read more…