Software Testing Blog

Between our office March Madness bracketology and the Bug Battle of the TV Networks rocking the headlines, it’s been an exciting week here at uTest! The Q1 Bug Battle peaked the interest of several leading pundits and bloggers, including TechCrunch’s Robin Wauters (@robinwauters), VentureBeat’s Robert Mullins (@SiliconMoon) and BostInnovation’s Kyle Psaty (@bostinnovation):

• uTest Finds 908 Bugs In Web Apps Of Major U.S. TV Networks

• uTest Rates Best TV Network Web Sites, And Catches Bugs

• uTest Finds 1,000 Bugs in Web Properties of TV Networks

• “Battle of the TV Networks” Uncovers Nearly 1,000 Bugs

Additional press coverage includes Radio Business Report, NewTeeVee, Broadcast Newsroom and VentureFizz.

Thanks again to all of our cunning uTesters for exploring these web and mobile apps voraciously and hunting down some very crafty bugs. And congratulations to all the winners of the Bug Battle of the TV Networks!

Until the next Bug Battle, happy bug hunting!

In part II of our interview with Jon Bach, we get his thoughts on testing metrics; common tester stereotypes; the merit of certifications; the testing blogosphere; inventing Twitter in 1986, as well some rapid fire Q&A. We think you’ll like it. By the way, did you miss part I of the interview?

uTest: You have some great tips on how to handle bloated testing numbers and statistics: “Any number, any statistic is like software. It can be tested.” What other tips can you give testers when it comes to having the courage, diplomacy and patience to slow things down and get to the truth?

JB:  For me, the magic words that often make me feel more courageous, diplomatic and patient are: “I have been fooled before.”

No one will argue with that because it’s true.  Scammers often confess that the hardest person to fool is somebody who says “I can be fooled.”  So many times I’ve been so sure I was right just to meet someone who convinced me differently, sometimes in a matter of seconds.  So now say “I could be wrong”, and use other safety language like: “it could be”, “it seems like”, “it looks as if” and “maybe…”  That way, I don’t feel stupid when I’m shown refuting evidence to my claim.  If you practice that, chances are good that you will appear to be a kung fu master who, after having floored your 50th assailant with your skills, slowly backs out of the room on guard for the 51st.

Remember that testing is a craft.  It involves thinking about how things might be different.  Remembering to say “I have been fooled before” is consistent with that spirit.

uTest:  Testing certs: worthwhile or window dressing?

JB:  The only thing worthwhile about them is the debate they provoke.  Window dressing is an apt metaphor because it’s only meant to enhance a window’s *appearance*.  When there’s a flood or a storm or some other strong test of the window, the dressing often gets destroyed. Outside of the flood, people may prefer the look of the dressing; I just want to be a stronger window.  Passing multiple choice tests about so-called “best practices” don’t do that for me.

Read more…

After Twitter-stalking him, making some harassing phone calls and sending threatening letters, Jon Bach (@jbtestpilot) cheerily agreed to take part in our Testing the Limits series. Much like his brother, Jon has a remarkable understanding of software testing – both in theory and in practice. Having worked for companies like Quardev, LexisNexis, HP and Microsoft, Jon is also a blogger, author and software testing consultant. An expert, in the truest sense of the term.

In the first installment of our two-part interview, we get Jon’s thoughts on sibling rivalry; the blame spiral of software development; the emergence of “agile-fall”;  testing at a startup vs. testing in the enterprise; John Schneider as Jon Bach and more.

uTest: A few months back, we asked your buddy Andy Muns who’d win a fight between you and your brother (this was a big debate in the uTest office). He said you would win hands down. Would he be right? And since you and your brother seem to share the same testing philosophy, what would do you think the fight would be about?

JB: It’s hard to fight with someone who stayed in their room for most of our childhood.  He was either reading or doing science experiments with a microscope or the chemistry set.  It got worse when we got the TRS-80 in 1980.  In fact, that’s probably the last time we fought — over who got computer time next.  My memory may be fuzzy, but just when it came to blows, he programmed a user name and password dialog? Something clever like that. Now it’s better just to learn from him and do my best to keep up — but that’s true for all younger brothers, I think.

As for modern-day fighting, sponsor me for a testing certification and let’s see what he’d do.

uTest: Say you’re named grand poobah of the QA universe… what’s your first decree?

JB: Effective today, “Quality Assurance” is now “Quality Assistance”.

(Try it.  Watch what happens when you start using it.)

Read more…

Last week, gaming enthusiasts of all varieties (fans, testers, developers, warlords, etc.) gathered in San Francisco for the annual Game Developers Conference. Among the attendees was uTest Project Manager Justin Forte – our very own man on the street. Though he failed to get the exclusive Mario Brothers interview he promised us, he was able to capture some great pictures of the festivities. We hope you like them.

The images speak for themselves, with one notable exception. You’ll notice one picture showing a booth where guests can meet Steve Wiebe, the long-time owner of the world’s top score in Donkey Kong. You may recall this author’s fascination with the never-ending battle for this top score, as chronicled in King of Kong: A Fistful of Quarters. Well, as luck would have it, just days before the conference kicked off, Wiebe was de-throned by a plastic surgeon from New York. The bar has been raised, and the barrels don’t lie, so get back to work Steve!

Read more…

How should testers respond when their bugs are rejected by clients?

Here to address this topic is Bill Ricardi of Northern Ireland, this month’s guest blogger. An active member of the uTest community (especially in the uTest forums), Bill describes himself as a “huge nature fiend, with professional ties to advertising, real estate, gambling, and writing.” For more on Bill – including the upcoming publication of his first book – check out his Google Knol page.

In the field of contract software testing, you won’t always see eye to eye with the client. What you consider a critical bug, they might see as a non-issue (or worse, a ‘feature’). What you call a major security flaw, they might consider such a remote possibility that it doesn’t even deserve a mention.

You might ask how you bridge such a gap between your level of testing and the client’s level of acceptance and understanding of product integrity and the testing process in general. The answer is simple:

You don’t.

It isn’t your job to convert the client to your way of thinking. Yes, you can contest a bug that they reject out of hand if you were technically correct to report it. Sometimes they’ll accept it as valuable feedback, but most of the time they’ll just ignore any contested bugs. This is something that you have to live with.

Read more…

Before I get into the story of this fascinating bug, I wanted to take a moment to introduce you to T.W.I.T. We liked the “bug-iversary” concept so much here at uTest that we decided to make it a recurring column, called T.W.I.T. or This Week In Testing (also noting the happy coincidence that the word “twit” is synonymous with “fool” and “dope,” words that characterize many of these bug follies ).

But I digress! So, this week in testing brings us an interesting heart device bug discovered March 12, 2008.

A team of computer security researchers were able to gain wireless access to a combination heart defibrillator and pacemaker. According to the New York Times,

[The researchers] were able to reprogram it to shut down and to deliver jolts of electricity that would potentially be fatal. The researchers said they had also been able to glean personal patient data by eavesdropping on signals from the tiny wireless radio embedded in the implant as a way to let doctors monitor and adjust it without surgery.

Full report and more after the bump!

Read more…

Well, today he’s sticking close to home in Boston. Tomorrow he’ll land in London… and before the week is out, he’ll hit Tel Aviv.

Doron starts Wednesday morning off (after his usual 10-mile run, of course!) in London with some tea and networking with friend and colleague, James Whittaker and UK partner, TCL.

Then he’s off to QCon London, an excellent conference for the enterprise software community. On Friday, 3/12 @ 2pm, he’ll be presenting at QCon re: The Mobile App Quality Challenge & How Crowdsourcing Can Help.

Doron is one of five software testing leaders chosen to present in the “How Do You Test That?” track. This track explores unique solutions created to address situations in which automated testing does not suffice.

And on the last leg of his marathon journey, Doron will present at Garage Geeks in Israel on Monday, 3/15 @ 8pm. There, Doron will be taking a deep dive into the topic of Crowdsourcing, and how smart recruiting, training and incentives can turn an unstructured, loosely assembled mob into a unified, professional community.

So, where in the world is Doron this week?  Catch him if you can!

When it comes to software development and programming, few people have been read, linked to, tweeted, quoted or plagiarized more than Joel Spolsky (@spolsky). But despite his adoring fans, the widely known blogger and entrepreneur has decided to give up the former (his wildly popular blog) to focus on the latter (his growing business).

Joel’s final farewell – Let’s Take This Offline – appeared on Inc.com a few days ago, where he discussed the fallacy of blogging as business strategy, time commitment and the common mistakes of most company blogs. Of course, he also addressed his reasons for “retiring”:

So, having become an Internet celebrity in the narrow, niche world of programming, I’ve decided that it’s time to retire from blogging. March 17, the 10th anniversary of Joel on Software, will mark my last major post. This also will be my last column for Inc. For the most part, I will also quit podcasting and public speaking. Twitter? “Awful, evil, must die, CB radio, sorry with only 140 chars I can’t tell you why.”

Read more…

Self improvement is a lousy business model. Mobile app developers understand this better than most. For every app to help you lose weight or improve your IQ, there are basically 10x as many to help you drink more, find your nearest trans-fat vendor or change the channel without standing up to get the remote. What a world we live in!

But if sloth and gluttony aren’t your thing, you can rest easy knowing that your vices have also been covered. And so to illustrate, I’ve posted an app for each of the seven sins.

Gluttony: “Happy Hours, is a free application for the  iPhone,  Android, and the mobile web. With it, you get access to some 15,000 happy hours in 30 different cities around the country. You simply load the app up, tell it where you are (which it can know automatically on the iPhone and Android phones), and let it show you happy hours close by.” (from the washingtonpost.com)

Lust: Girl Zoomer – “This application turns your iPhone camera into a pair of binoculars with 4x zoom, so you can see “the details that other people can only furtively glance at.” (from reuters.com)

Read more…

SCMagazine reported this week that researchers in Malta have discovered a decade-old vulnerability, present in all versions of Windows since 2000.  This bug can cause PCs to crash instantaneously and without warning, as well as reeling the compromised machine into a distributed denial-of-service (DDoS) attack.  This exploit is only dangerous if the user is duped into running an app with the malicious code (according to Paul Gafa, CTO of 2X Software).

The bug was discovered while Gafa was writing a software testing app:

“You can be the least privileged user on the system and still crash it,” Gafa said. “I believe it is very easy for Microsoft to sort it out. They just need to validate arguments passed to Windows APIs.” (source: SC Magazine)

Microsoft is currently aware of the defect and responded with this insight:

“Our initial assessment of the report is that malicious code would have to already be running or a user would have to be able to run a specially crafted application to cause the system to crash. In either case, the system has already been compromised or the user has rights to logon to the system.”

I’m curious to hear if anyone has other stories of old bugs causing new problems or vulnerabilities?