Software Testing Blog

We recently wrote about the need for security testing on medical equipment, but it looks like an even larger virus threat has come to light – on U.S. Predator and Reaper drone weapons systems.

While an unofficial source said they suspect it’s benign, they also added, “But we just don’t know”.  The thought of an attack drone being hacked is a chilling to say the least.  Jalpnik has a nice write-up of some of their historic missions (and the virus) but this seems to reinforce the hypothesis that the United States is entering a “Code War”.

Here’s the crux:

The virus, first detected nearly two weeks ago by the military’s Host-Based Security System, has not prevented pilots at Creech Air Force Base in Nevada from flying their missions overseas. Nor have there been any confirmed incidents of classified information being lost or sent to an outside source. But the virus has resisted multiple efforts to remove it from Creech’s computers, network security specialists say. And the infection underscores the ongoing security risks in what has become the U.S. military’s most important weapons system.

“We keep wiping it off, and it keeps coming back,” says a source familiar with the network infection, one of three that told Danger Room about the virus. “We think it’s benign. But we just don’t know.”

For those interested, we have a new whitepaper on Software Security Testing.

Every few weeks, it seems like there’s another major security breach to the website, gaming system or native app of a big global brand.  And that doesn’t even include the hundreds (thousands?) of hacks into the properties of smaller enterprises, SMBs and startups that consumers may (or may not) hear about.

In fact, a few months ago we wrote about The Top Security Hacks of 2011, and referenced that the attacks on Playstation were estimated to have cost Sony $24 billion dollars– nearly 10x their revenue for the same period.

So here’s the point: Would you rather look back and say your company overshot and used too many systems for security testing?  Or get that nauseaus, sinking feeling in your gut when your CIO wakes you at 2:00am and says the company has spent too little?

That’s why– as the cornerstone of uTest’s showstopping announcement yesterday– we announced the launch of uTest Security Testing that leverages the talents of new and existing white hat security professionals within our crowdsourced community.  Since we now offer the first crowdsourced, real-world security testing in the world…there’s a new kid in town to join the collective effort to protect your company, and customers’, private data.

Moreover, we’ve joined forces with industry leader Veracode to provide seamless access to their complementary, cloud-based application security verification services.  Veracode has scalable, policy-driven application risk management programs that help identify and eradicate numerous vulnerabilities by leveraging best-in-class technologies from vulnerability scanning to penetration testing and static code analysis.

As a result, companies will have access to a cost-effective, powerful combination of automated (Veracode) and real-world (uTest) testing that mitigates security risks across the entire software development lifecycle.

We’re thrilled, honored and excited to be partnering with Veracode.  And we’re certain that our joint offering– as a complement to organizations’ in-house security testing– will offer tech executives peace-of-mind at a price with infinitely fewer zeroes than $24,000,000,000.

For those who frequent the uTest website, blog or forums, you may notice a few changes around here. Yes, we had a little work done… nothing major, just going from focusing on one testing service (functional) to a providing whole range of testing services that span the dev lifecycle (functional, security, load, localization and usability testing).

For our customers, this means they can find the testing expertise they need, no matter where they are in their SDLC. And for testers, it means provide more earning opportunities for those individuals with expertise in areas like security testing, performance engineering, or localization validation. Like I said, nothing major. </sarcasm>

In all seriousness, these are exciting times around the halls of uTest. We’ve spent the past 12 months trialing new types of testing services with select beta customers. And now, we’re ready to offer them to any and all companies, on demand. A quick introduction to uTest’s new suite of testing services:

Functional testing services to help ensure your applications function as intended. Our services related to functional testing include exploratory testing, test case execution, test case creation and writing automated test scripts.

Security testing services to help you avoid launching products with common security- and privacy-related vulnerabilities. Our services include tools-based static and dynamic security testing, as well as manual penetration from trusted, white hat security testers.

Load testing services to make sure your app is ready for peak traffic, and that performance won’t degrade under heavy load. Our services include live load, simulated load and a hybrid load offering that combines cloud-based load testing with live testers.

Localization testing to validate that your app is saying what you think it’s saying. Services include translation validation from native speakers who live in-market, as well as full L10N testing that covers content translations, currency, taxes, shipping options and more.

Usability testing to help you launch products that are intuitive, clean and achieve high conversions. Services include surveys-based testing with targeted focus groups (by age, gender, education, hobbies, location, etc) or usability audits from one of our UX experts.

Special thanks to our friends at Stein + Partners for all their help with our rebranding, as well as an epic month of late nights from the amazing uTest crew. And finally, a word of thanks to our testers for their help in this launch, and the dozens of customers who helped us learn so much about each of these new types of testing. If you’d like more info about any of these new services, drop us a note.

We’ve got more on the way in the coming months. We’re not going to rest until we’ve completely reinvented the way testing services are provided in this ever-evolving apps universe.

Have a comment? Want to tell us you hate/love the new look? Drop us a comment and let us have it!

Update: Mike Butcher over at TechCrunch just took this news prime time. Seems we’re not the only ones who recognize the need for better app security testing.

Another month, another stellar guest for our Testing the Limits series. This time, we shoot some questions back-and-forth with testing expert Jim Sivak. Jim has been in the computer technology field for over 35 years, including a recent four-year stint as the Senior QA Manager at McAfee.  His career as a tester began with the Space Shuttle and over the years has encompassed warehouse systems, cyclotrons, radars, operating systems and now security software.  He is a Senior member of the ASQ and is certified as a Software Quality Engineer (CSQE).

In part one of our interview, we get his thoughts on the dangers of ignoring security testing; the false sense of security in mobile apps and devices; the evolution of malware; managing QA expectations; the meaning of SWAG and much more. Be sure to check back tomorrow for Part II.

**************

uTest: We noticed that you recently joined Unidesk after four years at McAfee. First off, what does Unidesk specialize in? And what are you looking forward to most in this new role?

JS: Unidesk is in the Virtual Desktop space. Our product allows companies to utilize virtual desktops that truly have the look, feel and capability of a hardware desktop. Due to our technology, desktop personalizations are easily managed. Virtual Desktops can become the IT department’s best friend in that changes and patches only have to be rolled out to one system, which then gets replicated automatically to every associated desktop.

Because Unidesk is a startup, I have the opportunity to really define the QA processes and goals, determining both the tactical and strategic visions. Being able to drive this work, using new techniques and past experience is really what brings me to my desk every day.

uTest: Your time at McAfee must have given you great insight into the web’s dark underbelly (i.e. security threats). Looking back over the last few years, what’s surprised you the most about the way businesses and consumers deal with security measures?

JS: Great question, Mike. The biggest surprise is the whole ostrich “head in the sand” attitude that exists. The tools and techniques are there, the information is readily available, but security still takes a lower priority until an incident happens.  Just look at the breaches that appear on an almost daily basis.  In the home, how many emails do people open and respond to that say ‘you have a credit card application ready for you”?

uTest: It seems safe to assume that users are more aware of threats on the web, as opposed to mobile? In your view, how does the explosion of mobile apps, social media and third-party integrations affect security?

JS: It is the sheer volume of opportunity for security lapses and breaches with these new avenues that is really frightening.  Just look at the incidents that have happened because someone sent a malicious link to their networked friends unbeknownst to them. Or applications that contain malware that just get downloaded and incorporated on these devices. People just assume that their phone is secure or that their tablet is unhackable. Again, software providers need to take security seriously and not wait until a major incident happens. It all comes down to the fact the users are human and we take a lot for granted.

uTest: Which evolves faster: security threats (viruses, malware, etc.) or the technology used to combat them? Why is this the case and what are the implications for end-users going forward?

JS: Unfortunately, I think that malware is winning.  Although there is research in trying to get ahead of the bad guys, most technology is reactive—the threat exists first and a solution/detection comes after.

uTest: This might seem like a job interview question, but what were some of the biggest testing challenges you faced at McAfee and how were you able to overcome them?

Read more…

In part II of our Testing the Limits interview with Mozilla QA Director Matt Evans, we get his thoughts on mobile immaturity; the worst bug ever submitted by a Mozilla community member; the so-called “skills shortage” in Silicon Valley; skepticism for all things open-source; the next great browser innovation and more.

If you missed Part I, do yourself a favor and catch up here.

*******

uTest: In many ways, mobile is still playing catch up to the web. Is there one area in particular where you see the most room for improvement? If so, where?

ME: Well, there are some obvious platform deficiencies around inconsistent UI and whether Flash is going to be fully supported across mobile devices or not. But this is a testing blog, so let’s talk about that. As I mention elsewhere in this interview, mobile is just a really tough testing challenge. The big problem is that there is very little support for cross-platform mobile device test automation. I suspect most of mobile device and application testing is done 100% manually. If any environment needed more test automation, it is mobile. At Palm, we rolled our own test harness that ran on the Pre. This became extremely important for endurance testing and finding memory leaks in the Pre applications.

Mobile software companies have an uphill battle since developing automated system tests for every platform is very costly, both in time and resources. However, reliance on mostly manual testing has lots of quality risks. If the quality of mobile devices and software is to rise about what it is now, we need automated test tool support that works well across all device platforms.

uTest: What’s the best (and by that, we mean the worst) bug ever submitted by one of your community members?

ME: Recently, Alex Miller, a Mozilla community member, discovered a very critical security bug and was awarded $3000 for finding and reporting the bug. He’s been hard at work finding and discovering other security flaws in Firefox, too, and was even given clearance access to all Mozilla security-related bugs reported in Bugzilla. Very few people have this access.  Oh, I forgot to add a little fact about Alex: he’s only 12 years old. That’s an awesome accomplishment by a really smart kid. This exemplifies the opportunity Mozilla provides to the community: an incredible technology playground where anyone that spends the time to learn can participate at any level no matter who you are or what your background is. The more you prove what you can do, the more you will be encouraged and acknowledged for that effort. Finding bugs is a good place to start for anyone who wants to participate. Certainly, not everyone is going to develop the expertise to discover deep level security bugs, but believe me there is plenty of testing folks can really help us out with. If you are so inclined, we will welcome you with open arms. Please visit us here.

uTest: We keep reading about the skills shortage in Silicon Valley. Are you seeing this at all, particularly when it comes to software testers? If so, what do you suspect is the reason?  And how do you overcome this dearth of top-shelf talent?

Read more…

News has been all over the web the past few days about the AT&T and iPad security breach.  If you haven’t heard the details, in short a group of hackers discovered a vulnerability in AT&T’s private web APIs where one could send the ICC-ID from an iPad SIM card and AT&T’s servers would send back the corresponding owner’s email address – no authentication required. Since the ICC-IDs for the iPad are somewhat predictable, it was trivial for the hackers to send in thousands of semi-random guesses and collect any email addresses that came back. Some of those addresses were for people with addresses from domains like faa.gov and us.army.mil.

The hackers claim they reported the flaw to AT&T before sending their discovery to the fine folks at Gawker. AT&T, on the other hand, was not pleased to see their security problems appear in a popular tech blog at all, and had this to say in an email to their iPad customers:

On June 7 we learned that unauthorized computer “hackers” maliciously exploited a function designed to make your iPad log-in process faster by pre-populating an AT&T authentication page with the email address you used to register your iPad for 3G service.

…

The hackers deliberately went to great efforts with a random program to extract possible ICC-IDs and capture customer email addresses.

So who’s to blame for a problem like this? Is it AT&T, or do the hackers themselves deserve some of the blame for the public way they handled their disclosure? Give us your thoughts below.

Before I get into the story of this fascinating bug, I wanted to take a moment to introduce you to T.W.I.T. We liked the “bug-iversary” concept so much here at uTest that we decided to make it a recurring column, called T.W.I.T. or This Week In Testing (also noting the happy coincidence that the word “twit” is synonymous with “fool” and “dope,” words that characterize many of these bug follies ).

But I digress! So, this week in testing brings us an interesting heart device bug discovered March 12, 2008.

A team of computer security researchers were able to gain wireless access to a combination heart defibrillator and pacemaker. According to the New York Times,

[The researchers] were able to reprogram it to shut down and to deliver jolts of electricity that would potentially be fatal. The researchers said they had also been able to glean personal patient data by eavesdropping on signals from the tiny wireless radio embedded in the implant as a way to let doctors monitor and adjust it without surgery.

Full report and more after the bump!

Read more…

SCMagazine reported this week that researchers in Malta have discovered a decade-old vulnerability, present in all versions of Windows since 2000.  This bug can cause PCs to crash instantaneously and without warning, as well as reeling the compromised machine into a distributed denial-of-service (DDoS) attack.  This exploit is only dangerous if the user is duped into running an app with the malicious code (according to Paul Gafa, CTO of 2X Software).

The bug was discovered while Gafa was writing a software testing app:

“You can be the least privileged user on the system and still crash it,” Gafa said. “I believe it is very easy for Microsoft to sort it out. They just need to validate arguments passed to Windows APIs.” (source: SC Magazine)

Microsoft is currently aware of the defect and responded with this insight:

“Our initial assessment of the report is that malicious code would have to already be running or a user would have to be able to run a specially crafted application to cause the system to crash. In either case, the system has already been compromised or the user has rights to logon to the system.”

I’m curious to hear if anyone has other stories of old bugs causing new problems or vulnerabilities?

Darkreading.com published an article yesterday about a new proposal that could hold software developers accountable for security bugs. Not the “my bad” type of accountable – the legal kind. With support from some high-profile public and private entities, the proposal would likely require developers to make their software free of the CWE/SANS Top 25 Most Dangerous Programming Errors before it’s shipped. Needless to say, such a measure would drastically affect the day-to-day responsibilities of testers.

Stanton blogged about the Top 25 list around this time last year, noting that although it was comprehensive, it lacked meaningful context for testers. It appears that his feedback was incorporated into the 2010 version. Writes Kelly Jackson Higgins:

SANS’ annual list had been criticized by security experts as more of a laundry list rather than offering a solution, but this year the list came with so-called “focus profiles” that broke the programming errors into groups based on categories of weaknesses and also provided mitigation information. The list is in order of priority this year, with failure to preserve Web page structure (think cross-site scripting) as No. 1, and race condition mistakes as No. 25.

Not surprisingly, the proposal has sparked a lively debate among industry participants – testers, developers and consumers. Here’s how the pros and cons boil down:

Read more…

As a way to extract the collective wisdom of the uTest community, we decided to experiment with a Guest Blogger program beginning in April. To say that it’s been a success would be an understatement, but we’ll say it anyway (the number of page views don’t lie!). Having covered a wide range of topics – including mobile app testing, tester overconfidence, security testing and more – the series has become a big hit within the community — and a great way for testers to get published in front of a large audience.

Here are a some of the highlights from our 2009 guest blogger program.  Stay tuned for an even bigger 2010!

Who is the User? – by Lucia Maldonado:  In what ways is software similar to architecture? And how can this help steer testers in the right direction? In this post, Lucia Maldonado takes an in-depth look at user accessibility standards, and offers a number of essential tips for testers in this field.

Security Testing Tips (from a Bug Battle Winner) – by Bernard Shai Lelchuck:  When it comes to security testing, few can match the expertise of Bernard Shai Lelchuck – one of uTest’s first (and finest) QA professionals. In this post, Bernard covers the basics methods of security testing, including tips for  information gathering, logical attacks and injection attacks. Oh, and here’s Part II.

Respect the Defect: Advice That Will Change the Perception of  Testing – by Joseph Ours:  Testers need to reconsider they way they report bugs – this was the position taken by Joseph Ours in his first (and hopefully not last) uTest blog post. Challenging testers to demonstrate their value by writing more clearly about the bugs they uncover (among other tactics), Joesph has sparked an interesting debate among our community. Visit the comments section to see for yourself.

Step Away from the Simulator: Putting Mobile Applications Into a Tester’s Hands – by Brad Sellick:  What makes mobile testing different from conventional software testing? For one, the simulators and emulators are far less reliable. In this post, uTester Brad Sellick – a self-made expert on mobile app testing and development – explains the dangers of relying on these tools while performing mobile app testing.

What You Need to Know About Writing Effective Test Cases – by Valerie Dale:  Despite all evidence to the contrary, test case design is often seen as work with no real value – a remedial task with no significant ROI. One would think that with the added pressures to launch a quality product on schedule, test case design and planning would be a top priority. It’s not. At best, there is minimal attention paid to the practice. At worst, it’s non-existent. In this post, Valerie Dale makes a great defense of  this beleaguered practice.

Your Overconfidence is Your Weakness: Lessons from a “Crash Specialist” – by Pradeep Soundararajan:  In our most-popular guest post to date, noted blogger Pradeep Soundararajan explains why finding lots and lots of bugs isn’t necessarily a good thing. Reliving his days as a “crash specialist” Pradeep examines how a tester’s ego can get in the way of their objective. His advice is as funny as it is useful. Simply put: a must read.

Software Testers: The “Eyes of the Battlefield” – by Brian Rock:  Our testers come from all sorts of backgrounds, including the armed forces. Brian Rock – a former Sgt. for Combat Arms Forward Recon Team in the U.S Army – is a great example. In this post, Brian makes analogizes testers with cavalry scouts. That is, they are the “eyes of the battlefield.”  Advocating exploratory software testing (especially for those in the uTest community) this post will make you rethink the role of testers.

You’re a Professional Mobile Tester (you just don’t know it yet) – by Bernard Shai Lelchuck:  As the title would imply, this post makes the case that anyone with a mobile phone and an inquisitive mind can become a successful mobile tester. It worked for Bernard Shai Lelchuck! Here Beranrd explains the rise in mobile applications, how he himself broke into the field and some basic tips for those who would like to get started in this growing (and highly lucrative) field.

Question the Connection: Tips for Diagnosing User Login Failures – by Sherry Chukpa:  Forget the sweeping generalizations about software testing “best practices.” This post by uTester Sherry Chupka gets right to the point on a very specific issue: user login failures. If you’ve ever been pitted against this problem in the testing lab, Sherry feels your pains, and has some invaluable advice for you as you move forward.

It’s been a great year, with some terrific insights into the world of testing, but our Guest Blogger program is just getting started. So if you have an opinion to express, a tip to share or a bone to pick, we’re always eager to share the thoughts of our tester community. Email us your ideas at marketing@utest.com.