Software Testing Blog

As a way to extract the collective wisdom of the uTest community, we decided to experiment with a Guest Blogger program beginning in April. To say that it’s been a success would be an understatement, but we’ll say it anyway (the number of page views don’t lie!). Having covered a wide range of topics – including mobile app testing, tester overconfidence, security testing and more – the series has become a big hit within the community — and a great way for testers to get published in front of a large audience.

Here are a some of the highlights from our 2009 guest blogger program.  Stay tuned for an even bigger 2010!

Who is the User? – by Lucia Maldonado:  In what ways is software similar to architecture? And how can this help steer testers in the right direction? In this post, Lucia Maldonado takes an in-depth look at user accessibility standards, and offers a number of essential tips for testers in this field.

Security Testing Tips (from a Bug Battle Winner) – by Bernard Shai Lelchuck:  When it comes to security testing, few can match the expertise of Bernard Shai Lelchuck – one of uTest’s first (and finest) QA professionals. In this post, Bernard covers the basics methods of security testing, including tips for  information gathering, logical attacks and injection attacks. Oh, and here’s Part II.

Respect the Defect: Advice That Will Change the Perception of  Testing – by Joseph Ours:  Testers need to reconsider they way they report bugs – this was the position taken by Joseph Ours in his first (and hopefully not last) uTest blog post. Challenging testers to demonstrate their value by writing more clearly about the bugs they uncover (among other tactics), Joesph has sparked an interesting debate among our community. Visit the comments section to see for yourself.

Step Away from the Simulator: Putting Mobile Applications Into a Tester’s Hands – by Brad Sellick:  What makes mobile testing different from conventional software testing? For one, the simulators and emulators are far less reliable. In this post, uTester Brad Sellick – a self-made expert on mobile app testing and development – explains the dangers of relying on these tools while performing mobile app testing.

What You Need to Know About Writing Effective Test Cases – by Valerie Dale:  Despite all evidence to the contrary, test case design is often seen as work with no real value – a remedial task with no significant ROI. One would think that with the added pressures to launch a quality product on schedule, test case design and planning would be a top priority. It’s not. At best, there is minimal attention paid to the practice. At worst, it’s non-existent. In this post, Valerie Dale makes a great defense of  this beleaguered practice.

Your Overconfidence is Your Weakness: Lessons from a “Crash Specialist” – by Pradeep Soundararajan:  In our most-popular guest post to date, noted blogger Pradeep Soundararajan explains why finding lots and lots of bugs isn’t necessarily a good thing. Reliving his days as a “crash specialist” Pradeep examines how a tester’s ego can get in the way of their objective. His advice is as funny as it is useful. Simply put: a must read.

Software Testers: The “Eyes of the Battlefield” – by Brian Rock:  Our testers come from all sorts of backgrounds, including the armed forces. Brian Rock – a former Sgt. for Combat Arms Forward Recon Team in the U.S Army – is a great example. In this post, Brian makes analogizes testers with cavalry scouts. That is, they are the “eyes of the battlefield.”  Advocating exploratory software testing (especially for those in the uTest community) this post will make you rethink the role of testers.

You’re a Professional Mobile Tester (you just don’t know it yet) – by Bernard Shai Lelchuck:  As the title would imply, this post makes the case that anyone with a mobile phone and an inquisitive mind can become a successful mobile tester. It worked for Bernard Shai Lelchuck! Here Beranrd explains the rise in mobile applications, how he himself broke into the field and some basic tips for those who would like to get started in this growing (and highly lucrative) field.

Question the Connection: Tips for Diagnosing User Login Failures – by Sherry Chukpa:  Forget the sweeping generalizations about software testing “best practices.” This post by uTester Sherry Chupka gets right to the point on a very specific issue: user login failures. If you’ve ever been pitted against this problem in the testing lab, Sherry feels your pains, and has some invaluable advice for you as you move forward.

It’s been a great year, with some terrific insights into the world of testing, but our Guest Blogger program is just getting started. So if you have an opinion to express, a tip to share or a bone to pick, we’re always eager to share the thoughts of our tester community. Email us your ideas at marketing@utest.com.

Several months ago, we outlined how to build an iPhone app and presented a primer for building iPhone apps.

But as demand continues to grow for top-quality iPhone apps — 100,000 different apps have been downloaded more than 2 billion times — many companies just can’t keep up.  So in the spirit of offering a build vs. buy alternative, now we’re back to share a great article from BusinessInsider about how to hire a great iPhone app developer.

Among the tips for companies looking to find a rockstar iPhone app developer include knowing what problem you want your app to solve, what it’s going to cost you and to avoid the temptation to rush your app to market (said differently, quality matters!).  It’s a good read, and definitely worthwhile if you participate (or intend to) in the rapidly growing market for iPhone apps.  Got your mobile app(s) developed, but want to know about the best ways to test your mobile apps?

Have other advice for entrepreneurs or companies who are looking to hire mobile app developers?  Drop us a comment and drop some knowledge.

Yesterday we posted Part 1 of our interview with James Bach, where he discussed tester certifications, faking test projects, his latest book and wide range of other topics (including life as a freelance sentry in a parallel universe). Today, for Part 2, we discuss tips for automated checking, what makes a good tester a great tester, his flying lessons and much more. Enjoy!

uTest:  Do you see the quality of resources in the testing field increasing or decreasing (tools, training, certs, et al)?  What do you think are some of the drivers of that change?

JB: There are many good resources out there, and yes there are resources getting better. There’s testingeducation.org and the Weekend Testers project, to name two. At the same time there are terrible things out there (such as certification and all the stupidity that goes with that). You have to be a smart consumer, because it seems to me that the bad stuff has always outweighed the good stuff by an order of magnitude or so. Maybe by two orders of magnitude.

uTest: When it comes to automated checking, what are some of the key opportunities to employ it that generally generate a positive ROI? Are there any good rules of thumb that can be used, i.e. if you plan on executing the same test 7 times, then it is a candidate (understanding of course that some assumptions need to be made to answer this)?

JB: Here’s how I think of it:

- Is the product highly controllable and observable? A command line tool that provides its output solely to the console window is inexpensive to automate, compared to an iPod touchscreen app. I want to get under the GUI.

- How expensive is the tool I’m using? I urge you not to use expensive tools, even if they work. Never let your manager buy them. Because expensive tools become something you MUST use, even if they don’t work. A free tool may be freely abandoned. This gives you flexibility.

- How well can I automate the oracle? Will the bugs be able to elude my automation because it can’t tell if a complex graphic is rendered correctly?

- What is the learning and testing value I’m giving up by using automated checks? I find that doing a test multiple times also causes me to learn and see new things in the product. Furthermore, when I re-run tests, I often run them in a different way, and that allows me to find new bugs.

- Can the automated check be parameterized and randomized, so that I get lots of similar checks for very little additional investment? I like automation more for data intensive testing, because I get new tests just by changing the database.

- Is the technology “Pyramid shaped?” In some products lot of underlying code boils up to one simple output, by placing checks on that output, we may be able to find lots of bugs. In other products, there are many different pathways, and you need a lot more checks to get decent coverage.

- How critical are the checks to the business? Is this critical functionality? Is it a common usage scenario? There are candidates for smoke testing.

- Is this part of the product especially prone to breaking? If so, that may be good for automation, UNLESS, it breaks in a way that breaks the automation.

- When I automate, I do it incrementally, in small bits.

I want automated checks for high value, highly testable parts of the product, and I want to do them in such a way as they aren’t constantly breaking or giving me false readings. I want to augment those checks periodic sapient testing as a cross-check.

uTest:  What characteristics and practices make for a good tester?  How about a great tester?

Read more…

Pundits are predicting a breakout year for Android in 2010.  The upstart mobile OS (if you can call anything Google does an ‘upstart’) is earning rave reviews from mobile app developers, and is growing rapidly.  And while the Android Market and it’s 20,000 apps have a ways to go before catch up with iPhone and it’s 100,000 approved apps, the growth has been impressive.  As we’ve seen time and time again, he who wins the hearts & minds of the development community, wins the battle.

Also helping Android’s case are some great new smartphones from manufacturers like HTC, Motorola and Samsung.

Robin Waulters from TechCrunch wrote a great piece today detailing the milestone and making some predictions for 2010:

2010 is going to be a big year for the Android operating system, with many new handsets finding their way to stores around the world (including Google’s own phone) and an increasing number of developers building tools, games and the likes for the fast-growing platform.

One way of noticing that the OS is poised for a big breakthrough at the expense of Windows Mobile, Symbian and other operating systems designed to run on various mobile devices, is the number of applications already available for download in the platform’s own application store, Android Market.

Waulters cited recently published data from AndroidLib, which showed an wildly impressive growth line for the Android Market (see the chart after the jump):

Read more…

uTest was on fire at Under the Radar Mobility this year. I think Under the Radar said it best!

For anyone looking to deploy an app across multiple mobile platforms and a gazillion different handsets, one massive problem awaits them: QA. uTest solves this problem with an army of testers across the world. Crowdsourced QA… Problem solved. (Click here to see Doron’s presentation.)

And that’s not all! Doron was able to connect with multiple partners, prospects and top media outlets, including Mashable’s Ben Parr (@benparr), editor in chief at IntoMobile.com Will Park (@willpark), ReadWriteWeb’s Dana Oshiro (@suzyperplexus), as well as participated in a couple great video interviews with bnetTV’s Michelle Sklar (@bnettv) and GoMo News’ Cian O’Sullivan (@gomonews) which are posted below!

Take a peek at the video interviews below to learn more:

Doron Reuveni-CEO of uTest speaks with bnetTV.com at the Under the Radar event.

Read more…

Late last week, our fearless leader was out in the Valley presenting at the Under The Radar Mobility 2009 event.  We first presented at UTR in April as an emerging company.  This time, we were asked back to present as a Graduate Circle company, which is reserved for those past presenters who are making a big splash in the market — quite an honor indeed.

The theme of this event was all things mobile.  The event was heavily attended by mobile OEMs, wireless carriers and all types of mobile app makers, including innovators such as Boku, Mplayit, ShoZu and Yowza!! (those are their exclamation points, not mine).  More details (and fantastic photos) after the jump.

Read more…

As promised, Google has made the slides and video presentations from GTAC 2009 (Google Test Automation Conference) available on the GTAC website and on YouTube. This year’s GTAC was a huge success! The theme was “Testing for the Web,” and now anyone can watch these leading thinkers discuss test automation strategies, tools, and the challenges desktop and mobile environments present when creating web apps.

Doron was among a select group of speakers chosen to present at GTAC, including Microsoft, smartFOCUS Digital, Sauce Labs and of course Google, where he examined the complimentary role a community of professional testers plays in mobile testing.

Check out Doron’s presentation below! All other presentations can now be seen on YouTube.

Mobile is the next great frontier for games, music, media and more.  In fact, Gartner says that  more than 139 million smartphones were sold last year.  And the phone makers aren’t slowing down:  Apple is planning to launch the iPhone into the Chinese market and a more affordable Android handset expected to hit the market by the end of ’09.  But haven’t we all learned that big markets make big targets — for VC dollars, for advertisers, for media coverage… and for hackers.

Doug Gross wrote a very interesting article over at CNN Tech about the one trend that could threaten to derail the otherwise unstoppable mobile movement.

Security analysts say they’ve already seen all of the major online threats — Trojan horses, viruses, worms — spreading on smartphones, often through e-mail attachments sent to the phones.

And as mobile apps have evolved from games and tip to calculators to company email, financial transactions and other mission-critical activities, the risk of security exploits grows considerably for mobile users and their employers.

The good news, experts say, is that phones present problems for hackers and other bad guys that traditional computers don’t.

Most viruses and other malware are designed for Windows, because that’s the most popular operating system. Since smartphones use a variety of different platforms, someone writing malicious software needs to pick and choose.

Wow, getting code to work across mobile platforms is so hard that even the hackers are having trouble!

Read more…

By now, it’s painfully obvious to all of us that the market for mobile apps is BOOMING and shows no signs of slowing down.  But I was still amazed by the stats/news that back up the hype. Within the past few months, GigaOm has reported the following mind-blowing proof points:

• Apple users downloaded 2 billion apps
• Android’s Market will come pre-loaded on Verizon phones
• Microsoft launched its Windows Marketplace for Mobile
• Apple will hit shelves in China, the world’s largest market
• Research In Motion (RIM) expanded with its new App World
• Palm introduced premium apps for its webOS
• Verizon is opening its own app store later this year

And the list goes on and on…

Read more…

With the summer sun finally paying us a visit in Boston, so too is the mobile testing space heating up out here! Tomorrow (7/16), Doron Reuveni will be speaking at the 2009 Software Industry Conference (SIC) at the Boston Marriott Quincy on how crowdsourcing is helping companies test their mobile apps.

With the new iPhone 3G S, Blackberry Storm 2, and the new Palm Pre, the smartphone apps competition is fiercer than ever! Mobile app developers are battling for media attention, consumer dollars, VC investments and more.

But, status quo testing doesn’t quite work for mobile apps. Multiple testing criteria — such as wireless carriers, handset makers and models, locations, languages, platforms — all must be taken into account. With crowdsourcing, any mobile app company can have world-class QA coverage.

In his session, Doron will examine the growing trend of crowdsourced software testing for mobile apps – highlighting its benefits, challenges and uses as compared with conventional QA options.

If you’re in town and want to meet up, shoot us a note!