Software Testing Blog

Testing usually means looking for bugs in software. But to Sport Evac, it means ensuring everyone gets out of a stadium safely in an emergency situation. The software was designed by the National Center for Sports Safety and Security and tests every scenario imaginable to help security personnel and first responders at big sporting events prepare for an emergency. But, as it turns out, testing is testing whether you’re testing the latest mobile app or an evacuation strategy for the big game. The goal is to find as many of the nasty bugs as you can before everything goes live. FoxNews has more on Sport Evac:

The Sport Evac program trains teams for those “what if” scenarios, by creating virtual 3D stadiums drawn from actual blueprints and packing them with up to 70,000 animated human avatars designed to respond to threats as unpredictably as their human counterparts.

It’s advanced technology fans won’t see on game day — but tech that behind the scenes makes watching the big game safer.

Evacuation in the event of an emergency is a critical challenge, and rehearsing the how to move around tens of thousands of people isn’t realistic. So sports security planners turn to computers to make sure it all goes smoothly.  …

The virtual stadium allows them to simulate how fans will respond in those first few critical minutes after an attack. Stadium and team security can use the virtual stadium to practice moving players and fans to safety and to run exercises with local first responders. …

Read more…

Every once and awhile, when there’s nothing topical to blog about, I decide to go back in time and focus on a software testing classic. Today is one of those days.

With that in mind, I wanted to draw your attention to Classic Testing Mistakes by Brian Marick. Whether you’re a tester or manager, experienced veteran or wide-eyed newbie, this 25-page article outlines some of the most classic testing mishaps and offers valuable tips on how to avoid them.

So what are some classic testing mistakes? One would be not reading this document in its entirety. As for the others, here are a few clips I found interesting. Note the bold titles are mine, everything in italics in Brian’s.

Mistake: Testers Not Responsible for Usability
If usability problems are not considered valid bugs, your project defines the testing task too narrowly. Testers are restricted to checking whether the product does what was intended, not whether what was intended is useful. Customers do not care about the distinction, and testers shouldn’t either.

Mistake: Misunderstanding the Role of “QA”
A first major mistake people make is thinking that the testing team is responsible for assuring quality. This role, often assigned to the first testing team in an organization, makes it the last defense, the barrier between the development team (accused of producing bad quality) and the customer (who must be protected from them). It’s characterized by a testing team (often called the “Quality Assurance Group”) that has formal authority to prevent shipment of the product. That in itself is a disheartening task: the testing team can’t improve quality, only enforce a minimal level. Worse, that authority is usually more apparent than real.

Mistake: Bad Timing on Load Testing
Putting stress and load testing off to the last minute is common, but it leaves you little time to do anything substantive when you discover your product doesn’t scale up to more than 12 users.

Read more…

Answer: In-the-wild testing.

This according to our very own CMO Matt Johnston, who recently sat down with Rich Hand of softwaretestpro.com for an in-depth discussion on the growing importance of in-the-wild testing. This lively Q&A was then adapted into a featured article  (no word yet on its big screen debut).

Anyway, if you’re new to the concept of in-the-wild testing or want to know more about it, then I highly recommend giving it a proper read.  Here are a few clips to get you started.  Enjoy!

Does it ever seem that no matter how much time, effort and money your QA team spends improving and refining software testing processes that some sort of defect is always found in the application, website, or mobile app after launch? Surprisingly, it has little to do with your organization’s in-the-lab testing – whether in-house or outsourced, manual or automated. In fact, it’s likely due to the fact that the lab environment you’re testing in cannot adequately replicate real-world conditions (real users, real devices, diverse locations, imperfect connectivity, not to mention a range of devices, operating systems, browsers, etc.).

And thus, companies test extensively in the lab, launch their apps into the real-world conditions of users, and we’re all surprised when these products don’t perform as expected. But what kind of testing can fill this gap effectively and affordably? It was out of this persistent question that uTest and crowdsourced testing was born. And the by-product of this was a new category of testing which has become a must-have for mobile, social and local apps known as “In-The-Wild Testing.”

“In-The-Wild Testing” (ITWT) is an effort to educate tech leaders about how to help QA teams and organizations launch higher quality software, quicker, faster, and cheaper. The idea of in-the-wild testing is about providing organizations with the real-world testing data necessary to make informed decisions about releasing products to market. According to Matt Johnston, Chief Marketing Officer for uTest, “Don’t be fooled by the word ‘wild’ when it comes to testing software. When you think of the term ‘In-the-wild testing’ think of it as ‘real-world vs. laboratory conditions.’” This is not outsourcing or beta testing, and it’s definitely not suggesting you replace the QA teams or solid processes you have in place within your test lab. Rather, this is about complementing, scaling, and aligning professional testing resources with your in-house or outsourced QA team. I predict that this concept will explode in the next five years . But the first step is to understand what ITWT is (and isn’t).

Continue reading >>>

In the software business, it’s all about precision, as even the slightest coding mistake can lead to catastrophic failure. This lesson is clearly not lost on the folks over at the United Nations telecommunications agency, who are meeting as we speak to decide whether or not to abolish the leap second. That’s right, the leap second.

The Sidney Morning Herald explains how this relates to software testing:

Unlike the better-known leap year, which adds a day to February in a familiar four-year cycle, the leap second is tacked on once every few years to synchronise atomic clocks – the world’s scientific timekeepers – with Earth’s rotational cycle, which, sadly, does not run quite like clockwork. The next one is scheduled for June 30 (do not bother to adjust your watch).

The United States is the primary proponent for doing away with the leap second, arguing that these sporadic adjustments, if botched or overlooked, could lead to major foul-ups if electronic systems that depend on the precise time – including computer and cellphone networks, air traffic control and financial trading markets – do not agree on the time.

Abolishing the leap second “removes one potential source of catastrophic failure for the world’s computer networks,” said Geoff Chester, a spokesman for the US Naval Observatory, America’s primary timekeeper. “That one second becomes a problem if you don’t take it into account.”

By now, you’re probably wondering what the “debate” is all about. Is anyone voting in favor of catastrophic failure? On the other hand, how can a unit of time be abolished, even if it’s only a second? The story continues:

Read more…

A teacher in Massachusetts dedicated a computer class to developing and testing mobile apps. The Education Secretary in the UK is calling for a total program overhaul of country’s computer education curriculum. Now, the Mayor Michael Bloomberg of New York City has declared that an entire public high school will be devoted to teaching students software engineering. From Government Computer News:

“Today, far too many of our graduates are leaving without the skills they need to succeed beyond high school. Not every student wants to go to college, nor is college right for everyone. But all students should leave prepared to succeed in the next phase of their lives,” Bloomberg said. “It’s a new way of thinking about secondary school based on today’s economic realities.” …

Frank Thomas, a spokesman for the city’s Department of Education, anticipates that the school will have between 420 and 460 students by 2015, when all four grade levels are enrolled, Adrianne Jeffries reported in BetaBeat. The school will start with a ninth-grade class this year and add on another grade level for the next three years.

The city has other specialized high schools for science, math, the performing arts and other subjects, but it did not have one focused on computer science. …

Joel Spolsky, a board member of the new school, said one reason he’s a proponent of the school is that it could can train many excellent software engineers who are not currently at the top of their class academically.

“I think this is the best thing about the school,” he said in a blog post. “A lot of kids are just not interested enough in other academic subjects to get good grades, but they would make great software engineers. A lot of immigrants (especially in New York) are not yet proficient enough in English to get good grades in all their subjects, but they’re going to make great software engineers, too.”

I have to say, one instance is cool. Two instances make you raise an eyebrow. Three instances (especially when they’re consistently bigger examples) might just be the start of a trend. And this trend of focusing not only on computer basics, but on more advanced – more engaging – computer topics that can lead to lucrative, fulfilling career paths is long over due.

If you had to pick two professions where you wouldn’t want glitchy software interfering, they would have to be surgeon and pilot.  These would be the only two correct answers.

And speaking of correct answers (or lack thereof) it seems that a software glitch caused 90% of would-be pilots to record failing grades on their online exam. Why online, you ask? In response to the recently exposed “fake pilot” scam – and as an added measure to prevent forgery – the Directorate General of Civil Aviation (DGCA) decided to make candidates take the test online.

Simple enough? Here’s NDTV.com with how this plan went, um, off course:

A software glitch caused a 100-mark examination to be graded on just 50, causing over 90 per cent of the examinees to believe that they had failed.  “The paper had 50 questions worth two marks each. Every candidate had to secure a minimum of 70 per cent marks to qualify.

During evaluation, the online software wrongly assigned one mark to each question, as a result of which each candidate was examined on 50 marks. The merit list, however, showed the full marks as 100. As a result, none of the 1,000 pilots who appeared for the ALT examination managed to pass the exam. Passing the exam is mandatory for officers who are seeking licenses as commanders,” said a top DGCA source.

In the CPL category for commercial piloting licenses, only two-three percent of over 4,000 candidates managed to clear the examination.

Realising that the mass failure was a result of a software problem, DGCA chief E K Bharatbhushan wasted no time in declaring that the incident was a consequence of a software glitch, and said that a new merit list would be published this Monday.

Let’s hope the DGCA  learned their lesson on the importance of in-the-wild software testing. If not, I think it’s time to take the train.

I can’t say testing was ever easy but there definitely was a time when there were far fewer components to the testing matrix. Now a days, if you’re just trying to put together a simple website there’s a whole range of browsers to consider at the very least – not to mention the ever updating versions of those browsers.

If that wasn’t enough, now you have to make sure that website works on the miniature screens of mobile devices (which themselves offer a whole gamut of sizes). And I’m not taking “should work” or something to consider if you want to be hip and trendy … because it’s not a trend, trends go away. Instead, time spent browsing the web on mobile devices is steadily increasing. Here’s the most recent statistic from Net Applications (which has been monitoring web usage across their 40,000 websites since 2004), as reported by CNet:

If you haven’t whipped your Web site into shape for easy viewing on small-screen devices, you’d better get cracking.

That’s because the use of mobile devices reached an all-time high in December, accounting for 7.7 percent of browser usage according to Net Applications’ measurements of daily visits to its network of 40,000 Web sites. That may still be a small fraction of total Web traffic, but it’s a large and growing population in absolute numbers.

Read more…

Chances are if you’re a tester, you probably spend the bulk of your work day sitting in front of a computer. You don’t have to be the Surgeon General to know the eventual result of such a lifestyle: weight gain.

If only there were a way that one could test software and burn calories at the same time. Well, now there is. Here’s a funny story from The Register:

No self-respecting techie would ever be seen dead in a gym. If you want to get fit, just put down the doughnut and go for a walk. However for those of you who do submit to treadmill workouts but don’t want to miss a minute’s coding time, this might interest you.

Brainy old Reg friend Bill Softky has come up with an ingenious low-tech contraption for people who want to use a conventional laptop while at the gym. The CardioDesk cracks the problem of providing a stable surface, fits most treadmills and folds away to the size of a book- it’s about an inch thick.

The prototype, which you can see here, is hardwood, but with a bit of interest (and a bit of investment) we should see a much cheaper carbon fibre version to go into production.

He admits it’s “an unusual solution to a nearly unknown problem”, but some of the most successful inventions are just that.

As a side gig, I think I’ll create a promotional series that’s half testing (advice and tips from people like James Bach, Michael Bolton and others) and half exercise (workouts from Richard Simmons and Chuck Norris). I’ll call it TesterCise. It’ll make millions.

On a more serious note, if you’re looking for ways to offset the ill effects of sitting at a desk all day, here are some tips from WebMD.

Test healthy my friends.

Happy New Year!  Yes, 2012 is upon us and, if you believe the pundits (or the Mayans), we’re all gonna die in about 11 1/2 months. And while that really takes the pressure off of watching your 401k or worrying about global warming, it amps us the urgency to get that killer new app launched.

So with that in mind, here are 12 questions whose  answers will shape the app universe (and thus, the testing landscape) in 2012:

1. Will we finally find a better way to vet apps than app store ratings?
2. Is Flash really and truly dead in the mobile app space?
3. What’s the next big wave in the ever-growing sea of SoLoMo?
4. Web-enabled TVs:  here or hype?
5. Will Android keep winning such rapid market share from iOS?
6. Is this the year the mobile wallet hits the U.S. mainstream?
7. How will netizens find what they need — search or social?
8. Can developers finally forget about IE6?  How about IE7?
9. Will Amazon’s app store plans fly or flop?
10. Where do tablets go from here?
11. Which direction will the IPO and VC markets turn?
12. After watching Uber battle taxis, and AirBnB take on hotels, which mature industry will be next to get disrupted in a big way (fwiw, my money is on medical and education, though the latter may take longer)?

So what’s your take — which of these issues will have the biggest impact on devs, testers and users in 2012?  Put on your fortune telling hat and share your prediction to that question in the comments below.

And happy 2012 to us all. Let’s enjoy this next (last?) year in the apps universe!

So you just unwrapped your brand new (insert gift here). You loved the old version and can’t wait to try out the latest one with all the new bells and whistles you’ve heard so much about. But shortly after opening said gift, you realize there’s a problem. Something’s different.

If your brand new gift was a Kindle Fire, that problem can likely be summed up in one (hyphenated) word: over-engineering.

Jakob Nielsen, the “King of Usability” and former Testing the Limits guest, recently published his usability findings on the Kindle Fire – and he’s not impressed. The report covers multiple aspects of the new tablet, but one area that caught my attention was his discussion of how the Fire compared to the first generation Kindle. Here’s his take (my take after the jump):

The Fire is a heavy object. It’s unpleasant to hold for extended periods of time. Unless you have forearm muscles like Popeye, you can’t comfortably sit and read an engaging novel all evening. The lack of physical buttons for turning the page also impedes on the reading experience for fiction. On the older Kindles, it’s easy to keep a finger on the button when all you use it for is to turn the page. In contrast, tapping an area of the screen disrupts reading enjoyment, is slightly error-prone, and leaves smudges on the screen. The Fire screen also has more glare than the traditional Kindle.

For reading fiction, the older Kindle design wins.

Read more…