Software Testing Blog

A number of years ago, I took a tour of a plate glass factory. Plate glass manufacturing is pretty simple: dirt pours in one end of a factory where it’s melted in a huge furnace. The melted dirt is then poured out as a thin sheet which then cools into glass as it rolls along a mile-long conveyor belt. The process is continuous – dirt constantly pours in and glass constantly flows out in a never ending ribbon. At the very end of the factory, away from the furnace, a lonely robot slices the ribbon into panes of glass for things like windows and doors.

Periodically, a technician will take one of those glass panes back to a lab where it is broken up, melted, dissolved with chemicals, and analyzed in fine detail under a microscope. That technician is a tester – one who is testing the production of the glass to make sure it matches quality requirements. His job is very different from that of a software tester, but surprisingly there are many things a software tester can learn from him.

That may sound bizarre because software isn’t manufactured. There is no real “production” in software – every copy of an application should be exactly the same. But production testing is about more than manufacturing. It’s about managing variability – and understanding variability should be incredibly important to software testers.

Read more…

According to tech analyst firm IDC, U.S. companies paid a record $14.2 billion for paid keyword-driven contextual ads in 2009, with Google dominating 55% of that revenue, Yahoo 9% and Microsoft 6%.

More dollars = More fraudsters. Period.

The company Click Forensics just released a report on the overall click fraud rates for the paid search industry. According to SearchEngineLand, the report said click fraud was up from 17.4% last quarter to 18.6% in Q2 of 2010. Traffic across 300+ ad networks is reflected in the data.

In addition, it was found that the countries outside North America with the greatest volume of click fraud were Singapore, Pakistan, Japan, Ukraine and China respectively.

Recent research by marketing intelligence company Visual IQ came out with similar numbers earlier this month. The company estimates marketers lose an average of 16.7 percent of their pay-per-click budgets to fraud.

So why is click fraud slowly trending higher and higher? The CEO of Click Forensics, Paul Pellman, stipulates that “the main reasons appear to be the continued sophistication of botnets and malware prevalent in the fast-growing search marketing space.”

According to Inc. Magazine, click scams use the following techniques:

• Manual clicking. Workers might be paid to click to run up totals.
• Software clicks. Automated clicks.
• Bot networks. Using malware to harness unsuspecting users’ computers, criminals can create large networks of computers employing programs that imitate clicks.

Despite detection innovations, click fraud rates show no signs of slowing. Attacks are becoming more sophisticated. Criminals are making more money. So what can we do? Any advice out there on how to mitigate it?



Creating new platforms like Android and iPhone is incredibly difficult, but it’s rare to hear stories about the challenges of building them unless you’re an insider.  There are probably dozens of good tales about developing these platforms that will take years to trickle out from behind closed doors.

So to hear stories like these, we must look back in time at the great development projects of the past.  Today the Computer History Museum announced that Apple has donated the source code for the original MacPaint application so that it can be downloaded by anyone.  MacPaint was a drawing application included with the first Macintosh that by today’s standards seems very simple, but in 1984 was completely revolutionary.  Many of the graphic design tools we take for granted, like the paint bucket and lasso select, were invented in MacPaint.

For developers and testers alike, there’s a lot to learn from the development of MacPaint.  Here are a few good stories:

Read more…

Not to beat a dead horse or anything, but I wanted to briefly revisit Apple’s  “Antenna-Gate” fiasco to drive home a very important lesson for companies of all shapes and sizes: Rely too heavily on “lab-testing” and you are virtually guaranteed to get burned.

We recently learned about Apple’s “Top Secret” design and testing lab thanks to MG Seigler of TechCrunch, who was given access to the state-of-the-art facilities just days before he mysteriously disappeared (kidding).

For some, the futuristic lab has conjured up images from the movie Star Gate, although I think it looks more like the Senate floor from Star Wars (episodes I through III). Here’s Seigler with a more technical description, as well as some insight into how Apple actually uses it:

Inside Apple’s headquarters in Cupertino, CA, there are a collection of rooms that house 17 giant anechoic chambers. Basically, they’re rooms where no waves (sound or electromagnetic) can reflect off of anything, so there is absolutely no interference when it comes to wireless testing. Apple places their devices from iPhones to iPads in these chambers to ensure the performance is up to their standards.

So how do they test it? There are four stages. The first is a passive test to study the form factor of the device they want to create. The second stage is what Caballero calls the “junk in the trunk” stage. Apple puts the wireless components inside of the form factor and puts them in these chambers. The third part involves studying the device in one of these chambers but with human or dummy subjects. And the fourth part is a field test, done in vans that drive around various cities monitoring the device’s signal the entire time (both with real people and with dummies).

So where did Apple go wrong? And what can this controversy teach us about the difference between in-the-lab-testing vs. in-the-wild testing? Below the jump are four critical lessons that companies ignore at their own peril:

Read more…

Everyone wants to know what Apple’s going to say at their big press conference in a couple of hours. Will the iPhone 4 bugs prompt them to issue a recall? Will they send users a plastic case that supposedly solves the reception problems? Will they try to fix the defects with a software patch?  Will they say they’re sorry and that this will never happen again? Will they tell NY Senator Chuck Schumer to suck an egg?

We’ll have to wait and find out. But here’s one thing they’re NOT likely to say (but they should): “We should have listened to our testers!”

[Update: See this TechCrunch story for a round up of the press conference]

One of the biggest pet peeves among testers and engineers (or anyone in involved in quality assurance of technology) is not being taken seriously when a serious issue is uncovered. For most companies, it’s generally a cross-site scripting vulnerability, an SQL injection or a browser compatibility flaw in the UI.  For the iPhone 4, it was an antenna issue. As it turns out, many top executives – including Steve Jobs himself – were repeatedly warned about about the “death grip” well in advance of the product’s release. These warnings from respected internal resources were either ignored or not taken seriously. They should have listened to their testers.

But what should testers do when they find themselves in this situation? According to Bill Ricardi, they should report the bug and move on. A member of the uTest community, Bill gave his advice on this matter as part of our Guest Blogger series, writing:

You won’t always see eye to eye with the client. What you consider a critical bug, they might see as a non-issue (or worse, a ‘feature’). What you call a major security flaw, they might consider such a remote possibility that it doesn’t even deserve a mention.

You might ask how you bridge such a gap between your level of testing and the client’s level of acceptance and understanding of product integrity and the testing process in general. The answer is simple:

You don’t.

Read more…

The mobile wars are heating up! Microsoft is aggressively luring app developers for its Windows Phone 7 OS, while Android quietly gains market share. Blackberry expects big things out of OS 6, while The Big Apple deals with antenna issues, the yellow screen of death and the (remote) possibility of a recall. Interesting times indeed.

As part of our newly-launched “What Do uThink?” series (more on this shortly), we decided to ask our community which mobile OS they considered to be the best. Here are the results:

1. Android – 38%
2. RIM Blackberry – 28%
3. Apple – 16%
4. Symbian – 12%
5. Windows Mobile – 6%

“What do uThink?” is a weekly poll, where we’ll be asking the uTest community their preferences and feedback on various apps, operating systems and other technologies. To encourage voting, we’ll be awarding monthly and quarterly prizes to randomly selected participants. This quarter, for instance, we’re giving away an iPod Touch. The weekly polls open every Tuesday afternoon and voting takes place in the uTest Forums available to registered testers) as well as on our Facebook page. Got it?

Good. Now back to the mobile OS results…

Read more…

The impact of check-in services, like Foursquare, on personal privacy and security is yet again making top headlines. If you remember our most recent bug battle (The Check-In Challenge), more than 80% of respondents responded “Yes” when asked if they were concerned about how location-based services (LBS) could impact their personal privacy and safety. And 49% chose “privacy/security concerns” as the top reason they don’t use check-in services more often.

Yesterday, the security company WebRoot came out with a study discovering similar results. After surveying 1,500+ social network users with geolocation-ready mobile devices, WebRoot found that more than half (55%) of respondents fear the loss of security and privacy, and 45% are very concerned about letting potential burglars know when they’re away from home (ah yes, the now shut down PleaseRobMe experiment comes to mind).

What’s most interesting to us is that 39% of those surveyed by Webroot said they use geolocation services, but take a look at the number of people that have fallen prey to social network cyber-criminals:

• Nearly a quarter of respondents (22.4 percent) were victims of a phishing attempt to steal their social network password.
• About one in six (16 percent) reported a malware infection in the past year that originated from a social networking site.
• One in nine reported at least one of their social network accounts had been compromised or hijacked.

Even in the face of these risks, many consumers admitted to engaging in risky behaviors:

Read more…

Take that Oracle! You just let Apple capture the lead in the 2010 Bug Marathon, otherwise known as Secunia’s Half Year Report (PDF). Worth the read, the 20-page report identifies the ten largest vendors with the most vulnerabilities (in all their products) and ranks them for the first half of 2010 – great entertainment for those who like to track bugs and keep score.

I mean, the World Cup is over and nobody really cares about baseball until September, so perhaps this could help fill the competitive void in the meantime…

Here are the current “standings”:

1. Apple
2. Oracle
3. Microsoft
4. HP
5. Adobe Systems
6. IBM
7. VMware
8. Cisco
9. Google
10. Mozilla Organization

As noted earlier, this is really more of a marathon than a sprint, so it would be useful if we went back a little longer than six months to crown a winner. Thankfully, Secunia did just that as part of their key findings:

Read more…

“Certifications are a farce – they simply test your ability to cram for an exam,” cries one tester in a recent uTest forums exchange. “No way – certifications are extremely valuable and establish credibility to the testing world,” replies another tester.

And it goes on and on… As a witness to the ongoing debate, it’s clear that there may never be a meeting of the minds when it comes to certifications. It’s certainly been thought-provoking and entertaining to read through the vicious cycle of pros and cons supporting both camps. Here are several to spark more debate!

Pro-Certifications Camp:

• They provide a base level of knowledge for those interested in the field, including terminology, processes, etc.
• They help testing newbies get their foot in the door of the testing world
• Certain organizations prefer to (or only) hire certified testers
• Passing a certification means that one is serious about testing
• Having a certification differentiates you from the crowd of software testers

Anti-Certifications Camp:

• Certifications bodies take in top money to create certs that simply test your ability to cram for an exam and at best possess knowledge – as opposed to your problem solving skills and how to test
• Passing such exams does not prove anything about testing skills & should not get your foot in the door for a job
• Organizations that prefer certified testers are simply lazy about their interviewing process – looking for a piece of paper or label on the resume vs. looking at the candidates skill set
• Passing a certification simply indicates that you are willing to take the time and money to complete a task
• There are so many certified testers – so what differentiates one from another? More advanced certs? Where’s the limit?

Read more…

Imagine a world where software testers are courted and wooed like LeBron James; where online job boards are littered with “Testers Wanted” posts and where everyone can finally spell “QA” correctly. In other words, imagine a world with a shortage of software testers…

“Nonsense!” you say. “There’s plenty of software testers to go around.” Not for long, says SiliconIndia, who posits that a shortage of skilled software testers is only a matter of time. Citing various facts, figures and estimates from a recent Gartner study, the author examines the reasons behind this coming tester drought.

Pradeep Chennavajhula explains:

This shortage is now a major concern for the IT service organizations, considering that the academia is not geared up to support the program, and many of the training organizations are not geared up to meet the demand of the industry. In this scenario, the question still remains as to how is the industry planning to tackle the shortage of software testers?

Good question. Of course, we’ve dealt with these issues many times before on The uTest Blog. Here are a few posts with some answers:

Read more…