Archive | Crowdsourcing

Stats: Software Version Fragmentation

Software FragmentationJust how fragmented is the world of real-life, still-in-use software? More fragmented than you’d like to think.

Not only are you contending with frequently released operating systems and versions, the back log of versions you need to support isn’t shrinking as quickly as most developers would probably like. Just how old is some of the software people are still clinging to? Let’s look at some numbers.

The old standby browser, Internet Explorer, is famous for having users who aren’t the quickest to update. Despite repeated rumors of its demise, IE6 is still kicking ever so slightly (with less than 1% of IE market share), according to stats from W3Schools. Internet Explorer’s market share is mostly spread across versions 8-10, though IE 10 hasn’t quite caught up to the others. So, if you want to cover the entire fragmentation of Internet Explorer, you need to test on five different versions.

In the mobile world things aren’t any better. We’re up to the seventh major version of the Android operating systems (with several sub-versions mixed in). If developers want to support all versions with a sizable market share, they need to account for Gingerbread, Ice Cream Sandwich and Jelly Bean – all of which have more than 25% of the Android distribution. While it pales in comparison to the big players, Froyo still accounts for nearly 4% and even Eclair has almost 2%. Total: 3 major versions that definitely need support and 2 older versions that should possibly be considered (location and popular device manufacturer of your target market will be a major deciding factor).

iOS is less forth coming with their distribution stats but David Smith took a look at the users of his Audiobook app to give people a general idea of what overall version distribution might look like.

On all mobile platforms (iPhone, iPad and iPod Touch) users are overwhelmingly using OS versions 6+ (more than 80% of users). Step down a major version and you reach another 10%. Anything lower than OS 5 is pretty nominal and developers like David have stopped supporting these versions (which each grab 1-2%).

Audiobooks no longer supports iOS versions prior to 4.3.0. The population below these limits was below 4% when support was dropped. Residual usage will occur at these OS versions but no new customers are being added.

Now for the kicker. Windows XP was released 12 years ago and is still in use … by a lot of people. From CNET:

Microsoft’s XP operating system is still used on more than a third of the installations out there, according to figures from Net Applications. Is it really still that popular? …

On Friday, figures from Net Applications showed XP with a robust 37.74 percent of all Windows and Mac OS installations worldwide, down only slightly from 38.31 percent in April.

Desktop OS Marketshare

It’d be difficult to support all past versions of software and the hope is that if enough developers stop supporting it eventually these holdouts will update. In the meantime it’s important to pay attention to the version distributions in your field (be it mobile, desktop or web) and support as many versions that have significant users.

As with all data, don’t take these stats blindly. Do a little digging to figure out which versions are most important to your user base and be sure to develop and test for those. If you’re having trouble achieving test coverage in this fragmented world, crowdsourced testing can give you access to a lot more hardware/software combinations than you could ever house yourself. If you’re going to support multiple versions, do it right, don’t skip testing.

Continue Reading

Harnessing Community in the New Development Life Cycle


uTestWith consumers having a greater voice than ever in what they expect from their apps, how do you build awesome apps that not only satisfy users’ appetites but work as they expect? Over time the software development life cycle has begun to evolve. While there are more inputs into the process than before, there are also more resources.

Join uTest and Cloudspokes as they co-host a webinar on unlocking the power of community development and testing. On Thursday, April 11th at 3 pm (EST) these two companies will discuss how the software development life cycle is shifting and how to get the most of out of these changes.

Learn about leveraging the crowd and hear examples of companies that have successfully crowdsourced their development and testing efforts.

Register today!



Continue Reading

Crowdsourcing – A Solution for Individuals and Companies

crowdopolis-big-appleThose of you who follow uTest closely are well aware that crowdsourcing is an efficient approach, put to use successfully within our in-the-wild software testing services.  As the job creation in the U.S. continues to lag behind other aspects of the recovery, there is increased discussion around crowdsourcing as an under-utilized way to help increase both the skill sets and marketability of individuals while providing flexibility to companies still staggering from the 2008 collapse.

In a post on’s Global Business Hub Blog, our own Matt Johnston lends his thoughts to this very topic:

“The beauty of crowdsourcing is that, in this economy, no one would think it’s possible,” writes Johnston. “Individuals can actually take their future into their own hands, without worrying about the next round of layoffs, hours being cut, or office politics. For companies weathering the same economic challenges, it’s hard to imagine hiring more full-time employees, regardless of their qualifications.”

Johnston also points out how technology companies have been early adopters of crowdsourcing, using it “for project funding, software testing, IT services, localization and globalization.”

You can hear Johnston talk more about the subject in person tomorrow at the at the Crowdopolis conference in New York City.

To learn more about uTest’s crowdsourced community of professional software testers, follow @uTestcommunity and stay engaged via the community Facebook page.



Continue Reading

How to Successfully Integrate Outside Testing Teams

Successfully integrate outside testing teamsThere are several ways to go about software testing. Some companies test entirely in-house, some outsource all their testing, some rely on beta programs, others use crowdsourcing. Many companies use a combination of testing scenarios.

“According to a recent report from analyst company Pierre Audoin Consultants (PAC), software testing is one of the most important tasks for 91% of IT departments, and almost all believe it is crucial to outsource this activity. The report also revealed that three-quarters of companies already use service providers with onshore and offshore capabilities to provide testing services,” wrote Computer Weekly.

So if you’re not keeping all the testing in-house, how can you be sure it will see be successful and up to your standards? It starts by being prepared, knowing what you need and having continued dedication to your choice.

Here are a few particular focus areas to make sure the integration of your in-house testing and outside testing goes smoothly and ends in success.

  • Know why you’re considering outside help

There are several reasons companies seek outside testing help:

– Limited in-house testing team
– Working with tight time frames
– You need an expert you don’t have on staff
– To free up your in-house team to deal with more important testing tasks
– To cover a wider testing matrix
– To ensure your product works in-the-wild

Knowing why you’re seeking outside help is vital to choosing the correct auxiliary company, having a successful experience and successfully integrating your in-house efforts with the outside testing.

  • Clearly define roles

Figure out what you’re going to keep in house and what you’re going to move to the new team. Redundant testing and small tasks (like regression testing or bug fix verification) are ideal for an outside team and will free up vital in-house testers to work on pain points, new features and critical bug fixes. Moving a portion of testing out of house is a great way to ensure testing is comprehensive and complete while allowing your in-house team to focus on what’s most important. Clearly defining the roles of each team with help keep everyone on task and avoid unintentional work overlap.

Continue Reading →

Continue Reading

The Phases of Mobile App Testing

Mobile App TestingWith so many mobile apps on the market your app certainly won’t last long if it is flawed. Anything from bad functionality, poor usability, slow load times or subpar security can sink your app. So how do you make sure your app is up to the task? By paying attention to testing from start to finish. Here are a few particular testing steps you can take from inception to launch.

The Brilliant Idea

You have a brilliant idea and recruit some awesome developers to make it come to life. At this phase of the process you should also recruit an awesome tester. Incorporating a tester into the team before the product is built can save you a lot of pain down the road. A tester looking over plans may be able to highlight potential problem areas that will require special care, or even identify ideas that flat out won’t work as designed. Being aware of these problems before developers spend time coding the application will save you time and hassle, making the entire SDLC run smoother.

You may want to also to bring in specific testing experts to look over designs. Specialized testers like security experts or usability experts will offer similar benefits as having a general tester on the team, but on a much more micro scale. It is easier to build a secure app from the beginning than going back and trying to patch vulnerabilities after a product is built. A security expert will know the most common vulnerabilities and can help you design around them before development begins. Similarly, a UX expert has a deep understand of best usability practices and what end users tend to like and dislike. Having a UX expert review design plans before a build can ensure you’re heading down a path that will lead to happy users.

Your Product is Ready for Testing

When you get through the planning and development phases and get to the point where there’s an actual app to test there are a few things you can do that will make your life easier and your testing more complete.

Find the Right Tools
Invest in tools that take limited maintenance and have an extended return. If the effort to maintain and run a testing tool is great than the benefit the tool provides it’s not worth it. Consider a versatile tool that fills multiple needs  – and thus provides multiple benefits for a single cost.

Apphance is a mobile testing tool that will carry you from version to version and through launch, offering multiple benefits along the way. Once you add the Apphance library your app you will be able to distribute builds to testers over the air and collect crash reports, bug reports, feedback and device information. After launch, Apphance continues to offer benefits by giving users a way to provide feedback directly to you from within the app.

Continue Reading →

Continue Reading

Why You Should Consider Non-IT Professionals for QA Roles

People of all professions can be software testersOne of the fastest growing industries is undoubtedly software and application development and all the roles that go into that – including testing. But what if job applicants don’t have a computer science or engineering degree? Should they be passed over without a second thought? QA professionals say not necessarily.

Top QA specialists are increasingly embracing non-IT graduates for the roles of software tester. The broader backgrounds of these new tech professionals bring a useful and increasingly necessary ability to the position – the ability to look at an application from an end-user’s perspective. From ZDNet:

According to [Jeff Findlay, a senior solution architect at Micro Focus with an architecture degree], having a general degree provides a broader mindset for testers to draw upon when doing testing. At the same time, they are able to clearly and solely focus on the testing activities, rather than worry about why and how a bug got introduced or what the best solution to an issue might be. …

Derek Ang, CEO of Morces, said since more software, such as mobile apps, today is used by normal, everyday people, “it makes no sense why non-IT grads cannot be good software testers.” … For Ang, the ideal software testing team should comprise of IT and non-IT individuals. The IT individuals test technical aspects, while their non-IT peers test other areas such as human-computer interaction (HCI) and flow.

This phenomena is nothing new. Cem Kaner endorsed the enveloping of broader experiences into the SDLC in a Testing the Limits interview from 2010:

Let me start this by saying that almost all of the best people I know in testing have significant experience in other fields.

Still, people cannot walk in off the streets and expect to be successful testers. The experts in the ZDNet article point out that testers will likely be expected to pick-up traditional IT skills and knowledge as time goes on. They’ll also need to master at least a basic understanding of how their company in particular handles the SDLC.

Continue Reading →

Continue Reading

The Problem with Outsourced Testing

FrustrationDoing a full run of software testing in house is difficult. It takes a lot of man power, countless hours (on a tight deadline none-the-less) and covering the entire testing matrix in-house is incredibly cost prohibitive. In the face of these challenges it makes sense that many companies look for outside help. Outsourcing has been the option of choice for years now, but it’s far from perfect. Austin Engery dealt with some frustrating outsourcing issues and eventually chronicled their dumpy journey for Information Week. This story perfectly highlights some of the biggest shortcomings of outsourcing.

Communication Issues
Dealing with workers who are physically apart from the rest of your employees can be difficult, but more often than not outsourcing complicates the matter even more. The outsource office can literally be time zones away, making getting in touch with someone complicated. Turnover at outsourcing companies can be high, so you may not even be dealing with the same staff for the duration of your project. Difficult or lack of communication can drag projects out and frustrate both parties. Here’s a few examples of Austin Energy’s communication issues:

“[Austin Energy CIO Alan Claypool emailed an IBM executive after] months of correspondence between himself and his staff and IBM project managers on multiple problems.”

“My concern is the lack of progress in identifying the root cause and, frankly, the pace and quality of response from IBM.”

“They have been responsive. I would characterize it as, in their words, ‘incremental response.’ We would like to see a faster response.”

Trouble with Accountability
In an outsourcing situation, you are not the only project on the company’s plate. Your project manager could be simultaneously juggling multiple projects which means A. Their attention is divided and B. More than one company is trying to get a hold of that project manager in the few common working hours – which further delays matters if something goes wrong. Austin Energy had trouble with the quality of work, but had few options for recourse.

“We continue to be gravely disappointed at the delays and seemingly ad hoc methods toward managing this project.”

Continue Reading →

Continue Reading

Valve’s “Steam Greenlight” Lets Customers Decide What Gets Published

For anyone unfamiliar with Valve Software’s Steam platform, it is the leading digital delivery marketplace for PC games. Valve has announced Steam Greenlight, a community-based website meant to provide indie developers with feedback and the opportunity to be selected by users for distribution on Steam. Unlike sites with a reviewing panel, it will be primarily the customers themselves that will determine the future of these indie titles. Debuting August 30, the community will be able to rate projects that indie developers submit and give feedback.

Games that have a relatively high number of votes from the community, as well as considering other factors like genre and features will likely see their wares on Steam. In other words, it’s not just about getting a specific number of votes, though getting a large quantity of up votes doesn’t hurt in the slightest.

“Making the call to publish or not publish a title isn’t fun,” … “Many times opinions vary and our internal jury is hung on a decision. But with the introduction of the Steam Workshop we realized an opportunity to enlist the community’s help as we review certain titles and, hopefully, increase the volume and quality of creative submissions.” -Valve Rep

This new approach to cultivating an online marketplace is exciting not only for customers but also for small  developers who want to break into a wider audience. It may be hard to imagine other “app stores” doing this, but if a good precedent is set here, it would be interesting to see this concept spread to some of the other well established walled gardens around the industry.

Continue Reading

Crowdsourcing: From 1714 to 2012 (and beyond)

Tongal connects companies with creative people to produce official online campaigns. Kickstarter connects soon-to-be companies with people willing to fund their projects. uTest connects software companies with expert software testers. Google Chrome reached out to Johnny Cash fans to create Johnny’s final music video. Each company connects a group of people who need something to a crowd of suppliers for different reasons and to accomplish different goals. So which one of them is official crowdsourcing?

All of them.

Crowdsourcing is an ever growing trend and as it continues to expand into new areas it can be hard to define. But, as it turns out, crowdsouring to solve problems and create solutions is not a new concept.

Check out this cute video by for the history of crowdsourcing (dating back all the way to 1714) and an understanding of what crowdsourcing is today.

Continue Reading

Kickstarter and the Importance of Regression Testing

KickstarterKickstarter recently experienced a security bug that made information from 70,00 unpublished projects accessible through the API connected to their homepage. The problem? Kickstarter’s new homepage didn’t play nice with the API. From Kickstarter’s blog:

The bug was introduced when we launched the API in conjunction with our new homepage on April 24, and was live until it was discovered and fixed on Friday, May 11, at 1:42pm. The bug made accessible the project description, goal, duration, rewards, video, image, location, category, and user name for unlaunched projects. No account or financial data was made accessible.

Luckily, the majority of the breached information was accessed by a Wall Street Journal reporter who notified Kickstarter about the issue. While information from 70,000 projects was accessible, only 48 projects were accessed by people other than the reporter.

Now I don’t know Kickstarter’s testing practices, and maybe this was just one of those things that unfortunately slipped through the cracks, but it seems like a prime example of the need for regression testing. When you’re introducing a new element to your site, or software or app it’s extremely important to make sure it doesn’t break any existing components – especially security related aspects. Kickstarter’s lucky personal or financial data wasn’t leaked, but that may not be the case next time. So let this be a lesson on why regression testing is so important!

Continue Reading

Crowdsourced Games Lead to Medical Breakthroughs

An online game helps sequence DNAJournalist Jeff Howe defined crowdsourcing as “The act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call.” And what’s the best way to get people to willingly respond to that open call and perform a (at times grueling) task? Make it a game.

And, as unlikely as it sounds, that’s exactly what medical researchers have done. It turns out that: A. people are very good at finding patterns and coming up with different ways to solve puzzles and B. two minds are better than one, and there are a lot of minds on the internet. That’s why games like Phylo and Biogames’ Telpathology game are growing in popularity while helping researchers solve critical problems.

Phylo is a web-based game that “harnesses the computing power of mankind to solve a common problem: Multiple Sequence Alignments.” Here’s what Phylo has players doing and how it’s helping researchers identify potentially lifesaving DNA patterns:

Continue Reading →

Continue Reading

Please Register Your Dead Dog (and other testing stories of the day)

Last week, I received a notice from the town of Ashland informing me that I was being fined for failing to register my dog Buster. Three things struck me as odd. First, I don’t own a dog. Second, the letter was not issued to me, but rather the last occupant of my house (who really needs to change his address). Third, Buster has apparently been deceased for a number of years.

How could this have happened? If you guessed software glitch, give yourself a pat on the back.

According to a post on the town’s website today, a glitch in the Town Clerk’s computer software caused notices to be sent to dog owners whose pets might have died or moved out  of town.

“If your dog has moved or passed away, please notify us,” said Clerk Tara Ward’s post.

The post said a software crash also caused purple notices about fines for dog licenses to be sent to residents whose dogs might already be registered.

That’s one of many interesting testing stories of the day. Here were a few others that caught my attention:

Continue Reading →

Continue Reading