Software Testing Blog

You’d be hard-pressed to find a more vocal proponent of crowdsourced software testing than yours truly (uTest). Over the last few years, we’ve seen first-hand just how successful community-based testing can be with regards to functional, security, load, localization and usability testing. But what about testing military systems? Could crowdsourced testing play a role in verifying the functionality of hi-tech weapons systems?

DARPA says yes.

Here with the story is informationweek.com:

The Department of Defense wants to create computer games that will crowdsource the complex process of verifying software for weapons systems.

The Defense Advanced Research Projects Agency (DARPA), the DOD’s research arm, through a project called Crowd Sourced Formal Verification (CSFV), aims to provide a “fun” way for the public to take part in software verification, a software engineering process to ensure an application satisfies its requirements, according to an agency announcement posted on FedBizOpps.gov.

The way DARPA sees it, if software verification was turned into a computer game that was fun for anyone to play, it could test the properties of software on a wider audience to ensure it will achieve its desired outcomes, according to the announcement.

Of course, one could argue that what DARPA is envisioning here is not really software testing as much as it is software checking – an important distinction made by many in the field, including James Bach. Nevertheless, there’s something to be said for the most advanced military on the planet looking to leverage the crowd for more accurate, more in-the-wild testing results. And if you recall, this is not DARPA’s first foray into crowdsourcing.

Read more…

Oh Black Friday – that joyous occasion when sleep-deprived, turkey-charged shoppers do battle at unholy hours of the morning. Smartphones and tablets are again at the top of many holiday wish lists and Black Friday is the day that promises excellent discounts on these pricey items. But before you wrap that new purchase (or lose the receipt) take a quick look at this list of “Dirty Dozen Smartphones” to make sure you’re not getting a bad deal:

• Samsung Galaxy Mini
• HTC Desire
• Sony Ericsson Xperia X10
• HTC Wildfire
• Samsung Epic 4G
• LG Optimus S
• Samsung Galaxy S
• Motorola Droid X
• LG Optimus One
• Motorola Droid 2
• HTC Evo 4G

Those 12 phones pose the highest security and privacy risks for users, according to Bit9, a company focused on software end-point protection. Interestingly, all 12 poor performers are Androids. Harry Sverdlove, Bit9 CTO, told PCWorld that the reason Android poses more of a risk than iOS is because of the wide-spread nature of Android over manufactures, models and carriers. Here’s what Harry had to say to PCWorld about how the study was done and why the results are what they are:

In compiling the list, Bit9 researchers looked at three things: the market share of the smartphone, what out-of-date and insecure software the model had running on it and how long it took for the phone to receive updates.

Read more…

Happy Thanksgiving everyone! We know you’re busy watching football, eating turkey and…um…(what else is there?), but we wanted to take a moment and list off a few things uTest is thankful for this holiday season. Here they are, in no particular order:

1. Our community. Seriously, there isn’t a testing project on the planet that you guys can’t handle.
2. Our customers. Without your dedication to quality and in-the-wild testing, none of this would be possible.
3. Testing experts. The craft has some truly brilliant minds and we’re thankful for the opportunity pick those brains as part of our Testing the Limits interview series.
4. Engineers. You’re developing amazing apps at a torrid pace. Thanks for keeping us busy and continually blowing our minds with your creativity and drive.
5. Our partners. Testing complex applications around the globe can’t happen without collaboration at the highest levels. And our partners are some of the top tools companies on the planet.
6. Our investors. Hey, I said it was in no particular order! Thanks for having the courage and vision to see that in-the-wild testing is a massive growth opportunity. Oh, and thanks for the cash, too.
7. Our staff. Big ups to our project managers, our community team, dev & product – and yes, you too, sales and marketing crews.
8. The media. Your coverage of crowdsourced testing and our recent expansion into security, localization, usability and load testing has shed light on a new way to test apps under real-world conditions – and the world is taking notice.
9. Doubters. You’re skepticism drives us to work even harder to deliver top-notch results and help our customer create better apps (and delighted end users).
10. Our fans. Whether you’re a tester, a customer or just a casual blog reader, we thank you. Stay tuned, the best is yet to come.

What are YOU thankful for this holiday season? Comments welcome.

Once upon a time, the internet was something that lived on your desktop computer. Only accessible from that one – stationary – place, just where you left it … maybe even in two places if you had an internet connected computer at home AND at work. Those days are long gone. Now a growing number of people can access the internet from their computers (both desktop and laptop, almost always from home AND at work), their smartphones and, increasingly, tablets. These new devices have drastically changed the way we connect to the internet and has turned us into what comScore is calling “digital omnivores.” comScore recently released a study looking at how exactly we consume digital media via internet connected, non-computer devices. This is what they found:

An analysis of ten selected global markets in August 2011 showed a notable percentage of Internet traffic (measured as browser-based page views) coming from non-computer devices. Singapore led among those markets with more than 7 percent of all Internet traffic in the country coming from smartphones, tablets and other connected devices.

While mobile phones continue to be the main driver of non-computer traffic around the world, it is interesting to see tablets contributing a sizeable amount of traffic for a few mature technological markets. In Canada, tablets drove nearly 40 percent of all non-computer traffic

One of the most interesting points in the study is that though 43% of U.S. consumers own a smartphone and only 11% own tablets, tablets account for almost 2% of all internet traffic in the U.S. That may not sound like a lot but it’s fairly significant considering how relatively new tablets are versus smartphones. Plus, that 2% is almost entirely attributable to the iPad (the Kindle Fire, which is hot on the iPad’s heels, hadn’t been released at the time of the study). Overall, when you combine the i-trifecta (Pad, Touch and Phone) iOS comes out as the dominate non-computer based browser.

Read more…

Thomas Suarez seems like your normal 6th grader from California… until you watch his presentation during a recent TEDx speech.  I’m extremely impressed with his app and presentation skills, but what I like more was his commentary on two items; first, that apps and mobile technology can be a huge asset to teachers, and second, that we need to increase the accessibility of technology to students who are interested.

If you have a four minutes of free time, I highly recommend watching this video (after the break).  Congratulations Thomas, and just let us know when you’re ready for a free test cycle for “Bustin Jieber“!

In part II of our Testing the Limits interview with Noah Sussman, we ask him some questions on Etsy’s testing challenges; assembling a testing team; localization testing and more. Did you miss part I? You can catch up here.

uTest: Total job interview question: What’s been the biggest testing challenge at Etsy and how have you been able to overcome it?

NS: Total job interview answer: concurrency has been a big challenge for us as we scaled our CI cluster. Most xUnit test runners aren’t designed to deliver massive concurrency out of the box. Eg if you want to run all your PHPUnit tests concurrently, you have to build a special branch of the test runner.

And even then, most tests are not written with concurrency in mind either. So a lot of design, debugging and rewriting of legacy code have been required in order to get to a point where we could run our tests in as concurrent a fashion as the available hardware would allow.

uTest: A previous interview guest once wrote a lengthy post on the “victims of fake agile.” In your view, is there a danger of a half-assed move into the agile methodology? If so, what are some of the major consequences?

NS: I can only speak from personal experience and I don’t know how typical my experiences are. I agree with the author of that post in that adopting Agile won’t fix a dysfunctional team, and it won’t help an organization to learn to accept its limitations and work within them.

uTest: What types of traits/qualities/skills does Etsy look for when hiring testers?

NS: On my team there are two roles for which I hire. These aren’t formal roles and we switch them up sometimes.

I hire software engineers who value quality and who are interested in how complex systems fail. These are the people who customize our mocks and fixtures, manage our coding standards, build static analysis tools, develop Jenkins plugins, design the CI system in general other things of that nature.

Then there is another equally important set of people whose skillset is generally described as “hardcore QA chops and a deep connection to the Etsy community.” The people I’ve hired here have many years of formal QA experience in high-risk industries like finance. These are people who are willing to use what they know about formal process, to help Etsy avoid the need for formal processes.

Here I look for talented QA analysts who are longtime Etsy users and deeply involved in the Etsy community. These are the people who develop functional testing tools, manage Selenium integration with CI and work with others in the organization to formulate test plans and resolve bugs. They also design and improve the process by which we triage bugs found in production.

uTest: How does Etsy – which has a global user base – deal with testing issues related to localization?

Read more…

Our Testing the Limits guest this month is Noah Sussman, test architect at Etsy. If you recall, we chatted with Noah last month as part of our STPCon video interview series, which you can watch here. Noah’s professional background includes a stint at Barnes & Noble, where he was the Tools and Automation Specialist, in addition to being a technical adviser to several startups and organizations. To learn more about Noah, you can follow him on Twitter, or check out his posts on github or Quora.

In part one of our interview with Noah, we get his thoughts on how Etsy tests software; why engineers should take more responsibility for testing their code; testing at startups and much more.

uTest: When it comes to testing, there’s certainly no shortage of collective knowledge to draw from. Who are some of your software testing role models – in terms of both companies and individuals – that you strive to emulate?

NS: As far as people, I work with a great team at Etsy. I feel really lucky to work with this many talented and knowledgeable people. Check out our engineering blog, http://codeascraft.etsy.com, as well as http://github.com/Etsy. You’ll find a large number of people with a wide range of ideas, all of which I believe are interesting. Once you’ve read their blog posts on Code As Craft, you should check out their personal blogs too, because those are fascinating as well.

Sebastian Bergmann and Jason Huggins have also been very helpful to me as I tried to figure out the best way to make CI work with Etsy’s unique deployment process.

As to companies, Blizzard, Amazon.com and Facebook all are using the Internet to build new kinds of communities and businesses. All three are essentially collections of Web services. But they’re able to manage their enormous communities and are constantly adding new features that their users pretty much love.

uTest: You’re an established figure in the testing space now, but you’ve spent much of your career on the development side of things. What (if any) were some of the biggest challenges you faced transitioning from dev to testing? And conversely, what advantages has your prior dev experience given you in your testing career?

NS: When I began making Web sites in 1999, I manually tested each site I built. Sometimes I worked with a QA team and sometimes not, but I have always checked my own work before delivering it, regardless.

As the Web grew, my projects got more complex. Tasks like making sure there were no broken links on a site quickly became infeasible to do manually. Soon even manually viewing every page on a site became unworkable as well.

So I learned some Perl and suddenly I could do static analysis. Then I learned to parse and alert on the output from tools like HTMLTidy, the W3 CSS Validator and later JSLint. After that, the number of bugs in my code dropped dramatically.

At first most people thought I was a little nuts for taking this approach. In 2001 almost no one cared if a site was robust. Web sites were cheap, throwaway things — “brochureware.” But a few years later along came Google Maps and GMail and now Web sites were first-class applications, expected to perform well and last a long time.

After that I just found myself more and more involved in helping teams produce quality Web sites. I think once everyone realized testing Web sites was important, I just naturally got drawn into that because I was interested in the domain. There aren’t a whole lot of Web developers who are really deeply passionate about testing. Or if there are, I haven’t met most of them yet!

uTest: What’s the biggest difference between testing at a startup versus testing at a larger company like Etsy?

Read more…

Testers and programmers are two groups of people who should get along, but often don’t. It’s a sad fact of life that testers (by virtue of what they do) often bring bad news. And programmers, by virtue of what they do, are the source of the defects that create the bad news. Rather than both accepting that this is a reality of life and working together, they allow the relationship to become acrimonious.

James Bach is no stranger to this problem, and his latest blog post is a blueprint for making that relationship more productive and professional. Titled A Tester’s Commitments, James starts by writing:

Dear Programmer,

My job is to help you look good. My job is to support you as you create quality; to ease that burden instead of adding to it.

What follows are twelve commitments a tester should make to their programmers. They include things like:

• I provide a service. You are an important client of that service. I am not satisfied unless you are satisfied.
• I will learn the product quickly, and make use of that knowledge to test more cleverly.
• I will not carelessly waste your time. Or if I do, I will learn from that mistake.

But James is not in usual form unless he invites controversy, and that first bullet struck quite a chord with some of his readers. Testers provide a service!? Since when?

Read more…

Leading up to (and following) the release of the iPhone 4S, Apple was heavily pushing Siri – hands down the most innovative new feature since the touch screen. I don’t really remember seeing commercials leading up to other iPhone releases but I’ve definitely seen this one more than a few times.

Exclusive new features are a major draw when you’re choosing one phone over another and Siri is something you can only get one place … for now.

One month after the iPhone 4S was released, mobile app developer Applidium cracked the Siri protocol and discovered exactly how it works. PCWorld summarizes Applidium’s process (emphasis added):

Applidium achieved this crack by using its very own HTTPS server and faked SSL certificates (which sounds like a nasty security issue to us). And along the way, the French software makers found out a few tidbits about how the voice service works. Applidium claim that Siri sends compressed audio data over the network to the server. … According to Applidium, theorhetically, “anyone could now write an Android app that uses the real Siri.”

 PCWorld goes on to point out that just cracking the protocol isn’t enough to enable developers to convert Siri to Androids and other phones (they’d still need an authentic, unique Apple UDID).

Circumventing Siri’s protocol is a “nasty security issue” on Apple’s part. But according to Applidium, there are significant security measures in place:

Read more…

Tell me if this sounds familiar:

1. Company X dreams up a new app (or a new version)
2. Product management specs the requirements; engineering builds it, and QA tests it
3. As soon as it launches,  Twitter and Facebook blow up with user complaints about how the app doesn’t work; is confusing to use; is slow; or some other flaw
4. Everyone blames engineering and QA for launching something that sucks

So what went wrong?  We need better test automation… we need to outsource testing… we need to partner our testers with our engineers… we need better product documentation… we need to move from SCRUM to Kanban.  We invest thousand of hours and millions of dollars trying to create apps that work as designed — not just inside our firewall, but in the hands of users.

Problem is, we focus all of these good intentions on stuff that occurs inside the lab (whether it’s our in-house QA lab, our test automation, or a vendor’s test lab 12 timezones away).  And users don’t consume our app under the sterile conditions of a lab environment. So how do companies ensure that their apps work as well under real-world conditions as they do under lab conditions? There are two ways:

1. Running a beta program
2. Push a portion of their testing out into a distributed community

And as MG points out over at TechCrunch in a recent post about Apple’s launch of Siri, the use of the “beta” label no longer lets companies off the hook with regards to app quality:

Read more…