Software Testing Blog

If you’re in the testing business, chances are you’ve used a bug-tracking system. And if you’ve used a bug-tracking system, there’s a good chance it was Atlassian’s industry-leading product, JIRA. Well, Atlassian OnDemand now takes those same development tools you know and love, including JIRA, into the cloud providing your team with greater affordability, customizability, and connectivity.

On the heels of this news, uTest is happy to announce a new plugin that allows bi-directional integration with JIRA. As one of only fifteen plugins that have been pre-installed on Atlassian OnDemand, we’re obviously quite excited about this news and what it means for our growing base of customers, and our global community of testers.

So what does it mean, specifically? Well, previously, companies could consolidate and track their bugs in JIRA by importing reports logged in uTest’s platform.  But now, JIRA users can map uTest data fields back to data fields in JIRA, which creates a seamless, two-way system for managing bugs found in the lab or in the wild. The end result: a greatly simplified process for test case management.

Perhaps the coolest thing about our new JIRA plugin is its one-click integration with Bug Fix Verification, a robust new time-saving feature launched in Version 4.0 of our software testing platform. Since uTest’s community performs app testing in real-world conditions – working on real devices, carriers, browsers, and operating systems – it can be difficult for customers to validate that a bug fix has addressed the problem.

Read more…

As you probably know, uTest is all about the community. In fact, we built this company on the idea that testing could be greatly enhanced by leveraging a global community diverse in terms of skill sets, experience, locations and other criteria.

As we discovered, one area where the community showed great value – but posed an even greater challenge – was in the creation, distribution and execution of test cases. At first it sounds easy: You describe your application/requirements and your virtual testing team follows your guidance. But with a community of 45,000+ testers, maintaining an orderly process across so many matrix combinations (device, platform, version, carrier, etc) isn’t always easy.

So over the last 9  months we decided to take a smarter approach. We asked ourselves: What’s the best way to distribute testing assignments across a global community of testers? How can we create a community that’s optimized for test cases – one that would give any testing manager the capability to capture feedback from various teams, on different environments, at the same time?

Short answer: Partner up with open source pioneer Mozilla to launch CaseConductor!

Why uTest and Mozilla? 

Of course, Mozilla also relies on the development and testing efforts of its community members to create/enhance products and help find bugs. With dozens of products and millions of end users, managing the open source community is a challenge for Mozilla as well. So when Mozilla started leveraging the uTest community, we learned pretty fast that Mozilla shared our same test management challenges. That’s where CaseConductor comes in.

What is CaseConductor?

CaseConductor is an open source test case management system that allows in-house QA teams, outsourced testers, and even community-based testers to collaborate, create, manage and execute test case libraries.  As a result of uTest and Mozilla’s partnership announced in April of this year, CaseConductor beta is now available for download in a github.com repository, where developers and QA managers are encouraged to start leveraging this powerful tool, view source code, provide feedback on usability, and ultimately help advance the development of this TCM system..

Why CaseConductor? 

We decided that CaseConductor would be developed with three core values in mind:

Read more…

Just two months after announcing an expanded suite of testing services, uTest has now launched its next generation tester profile and ratings system (Platform v4.3). These changes come at an opportune time – customers’ testing requirements are on the rise, followed by an increasing need to optimize project invitations for our global community of testers.

First, the new Profile includes the ability to control which types of projects a tester wishes to participate in. For example, if I’m only interested in security testing, I would simply fill out that specific portion of my profile.

On the other hand, you may be interested in usability testing – specifically designing surveys for end-users to complete and analyzing the results. The new profile allows you to drill down to specific flavors of testing (e.g. designing usability surveys); not just at a high level (e.g. functional, security, load, localization, and usability testing). Lastly, you can control the amount of incoming project invitations by noting your general availability for uTest projects each week.

Moving on to the rating system, testers now have the ability to earn a badge across the five major testing types. For example, instead of being a Gold uTester, one may now be a Gold uTester in security testing and a Bronze uTester in load testing. Aside from these changes, the new Rating System also places heavier weight on the quality aspect of a tester’s participation (quality over quantity). Therefore, testers who continue to submit very valuable reports will likely witness an increase in their rating. And as usual, ratings are calculated dynamically and adjusted daily based on activity across all testing projects spanning the entire community.

If you’re a uTest customer, most of these changes are under the hood – although you will directly benefit from our enhanced rating system. But if you’re a uTester, you will need to update your profile to enjoy the benefits of fewer emails hitting your inbox and more projects that are relevant to you.

Millions of people are atwitter this week (see what I did there?) over the fact that Facebook has reorganized its news feeds.  Whether users are excited or angry about the changes, at the heart of each opinion I’ve heard is the same sentiment: we want the information we consume to be organized in a convenient and predictable way.  The trick is we may not agree on what it means to be convenient and predictable.

That’s why uTest’s platform team was particularly careful during our recent redesign of uTest Express, which we unveiled earlier this month.  With this latest version, we took the bug reports, app reviews and task results that used to be on three different screens and put them into a single activity stream – essentially a news feed of what testers are doing.  When we shared mockups of this new approach with customers, they really just seemed to “get it”, and with almost no exception, they preferred it to the previous experience of separate lists and data grids.

So why did it work?  First of all, it’s very clear what appears in the feed: it’s all the work that’s being done for that test.  In our case, we don’t have to worry about hiding what’s irrelevant because it’s all relevant.  With Express, the testers’ deliverables are especially well defined and contained, so every item deserves the customer’s attention, and the stream never gets unwieldy.

Read more…

A little more than a year ago, we launched version 3.0 of the uTest platform. It was a bold rewrite that brought a better, faster, more powerful uTest experience. Last night, we launched a new version that takes that same experience and goes even further.

Besides an improved user interface, we’re launching three big new features: Bug Fix Verification, Test Case Support, and Tiered Bug Approval. Let’s take a look at them.

Bug Fix Verification
More and more of our customers are asking us to not only find bugs in their apps, but also to check to make sure the bugs were fixed as well. It makes sense – our testers are finding “real world” bugs, often times involving unusual combinations of devices, browsers, geographies, and more. For our customers, it may be completely impossible for a developer to replicate a particular bug or verify a fix in-house.

With this new service, customers can have a tester try to recreate a bug to determine whether or not a fix successfully eliminated a defect. The process is incredibly easy – uTest already knows the details about the bug, including the steps necessary to reproduce it and the environment on which it was discovered. For our customers, validating a bug fix is as simple as clicking a button and waiting for the uTest community to do its work.

Bug Fix Verification closes the loop between our community of testers and our customers’ developers. It is an additional service for our functional testing customers, and the pricing varies depending on the needs of each customer. If you are interested in learning more, talk to your project manager or contact us for more details.

Test Case Support
Many of our customers use test cases as part of their testing process. Customers write a step-by-step set of instructions for testing their application, making sure to cover all the components that must be validated. The uTest community follows these instructions using a wide variety of operating systems, browsers, locations, and more – providing comprehensive testing coverage. Customers can quickly learn if their app is ready for launch or if it has problems.

Up until now, handling test cases on the uTest platform was cumbersome and involved a lot of extra work. With this release, we are introducing a new feature that makes it easy to create and organize test cases within the uTest platform. Customers will be able to include test cases with their projects, and then see at a glance (and in real-time) the status of their testing coverage and where problems exist.

For testers, having built-in test case support makes it easy to follow the flow and report bugs about specific steps in the process. Bug reports will be tied to the exact step of the test case, making it much simpler to explain what went wrong and how.

Tiered Bug Approval
As the uTest community grows, we are always looking for ways to better align our testers and our services with the needs of our customers. With this new version, we’re launching a new feature called Tiered Bug Approval. Going forward, when customers approve bugs in the uTest platform, we’ll be asking them to rate the value of each bug.

For testers, bugs that are more valuable for our customers will yield higher payouts while bugs that are less valuable will yield lower payouts. Bug payouts will no longer be tied to bug type or severity, but will now be based on the customer’s assessment of approved bugs. This new feature is designed to align tester payouts much more closely with the real value that each tester creates for our customers.

Wrap-Up
Besides these new features, 4.0 includes dozens of other changes including UI enhancements, bug fixes, performance improvements, and more. Our product and engineering teams have been working hard on this update, and many of their best ideas came from our customers and our testers.

Have a great idea for our future product releases?  Testers should join our testing forums and check out our Platform Feedback section. Customers can contact their project manager or drop us a line.

Two months ago, we announced uTest Express – an easy and affordable way for startups and young companies to test mobile apps. So far, it’s been a huge success, and today we’re pleased to announce a big update: uTest Express for Web. Now you can quickly and easily have your web application or web site tested by a team of professional testers under real-world conditions.

Like the mobile version, Express for Web helps you validate your app by providing you with both bugs and feedback from real people testing on real computers. With Express for Web, we’ll make sure there’s a sold cross-section of today’s most popular browsers and operating systems to give you a good feel for how your app or site behaves in today’s complicated browser landscape.

So how much does this awesome testing cost? Well, plans start at $499, but to celebrate the launch of Express for Web we’re discounting prices for the Bronze and Silver packages by 50%. Between now and the end of June, you can have your web or mobile app tested for as low as $249.

So what do you get for that amount? Well how about access to handpicked members of the uTest community from North America who best match your testing requirements. Web testers will be armed with a wide variety of browsers while mobile testers will test your app on real devices using real carriers. In both cases, your app is tested professionally by real people who provide real-world testing results and expert feedback that isn’t possible with emulators, simulators or remote access.

At the conclusion of each project, you’ll receive a list of well-documented bugs, including screenshots and videos with steps to reproduce them. You’ll also receive expert feedback from the testers about the application – including app ratings and feedback for interface design, usability, app performance and more. To learn more about how it works, watch this brief Product Tour.

Here are a few key features of the service:

• Cost-Effective: With three testing packages available, uTest Express offers mobile and web app testing to fit any startup budget;
• Easy-to-Use: A simple interface walks customers through a series of questions to identify their testing needs and facilitate the creation of the testing project;
• Professional Testers: Testers are selected from our community of 35,000+ professional testers;
• Accessible 24/7: Customers can access their testing projects on-the-go, using uTest’s downloadable app for the iPhone and iPad, or through the traditional browser-based interface.

So if you need iPhone app testing, Android app testing, iPad app testing, web app testing, and more, check out uTest Express.

Over the past few years some seriously amazing testers have joined the uTest community, and today we have one of the most talented and diverse testing communities in the world. Some of our testers have taken us by surprise with their skills in areas like load automation, usability testing, and even general leadership. It’s that last area that has us most excited and is the reason behind today’s release of the uTest platform.

Test Team Lead
For a while, we have experimented with giving select and trusted members of our community additional privileges to help make our projects run more smoothly. Testers have been quietly helping our project managers with tasks like answering questions on projects, keeping testers focused and in scope, and making sure bugs match the customer’s expectations. With version 3.7 of the uTest platform, we have formalized that experiment into a new role within the community – the Test Team Lead (or TTL for short).

The TTL is not a replacement for a uTest project manager and their role will be mostly behind the scenes from a customer point of view. However, their contributions will be instrumental in helping to deliver high-quality testing results. Project managers will rely on the TTL to handle background tasks necessary to keeping a diverse community of testers in-line and on-task. This will free up the project managers to better help customers manage their test cycles, understand their bugs, and get the most from the uTest community.

Test Team Leads are select members of the uTest community who undergo an extensive training process. Among our best testers they are the best, but it is more than just their testing skills that qualify them as a TTL. They must also demonstrate leadership, communicate clearly, and be effective at organizing and motivating other testers.

Changes in 3.7
Starting in v3.7 of our platform, uTest project managers will be able to invite a Test Team Lead to projects of their choosing. The Test Team Lead will join the project just like any other uTest project, but their role will be different. Instead of reporting bugs or completing test cases, they will be helping other testers, answering questions, and keeping the project in scope.

TTLs will have a greater degree of privilege within the uTest platform, including the ability to use Tester Messenger to communicate with other testers and the ability to use Customer Notes to leave comments for project managers and customers. A TTL will not be able to approve or reject bugs, and project managers will still be responsible for the overall success of a test cycle as well as be the primary point of customer contact. Not every test cycle will require a TTL, and project managers will use their discretion to decide when a TTL will be needed.

While version 3.7 enables this new role, this is a much bigger evolution for our crowdsourced testing platform. We’re excited that our community will be taking a more active role in organizing itself, and we think this is a great way to inspire and deliver excellence.

Wrapup
While we have some good ideas in our product pipeline, we’re always looking for more. Do you have an idea for future product releases? uTest community members can join our tester forums and check out our Platform Feedback section. Customers can contact their project manager directly or drop us a line.

With the vast majority of mobile apps being developed by startups, wouldn’t it be great if there was an easier, more affordable option for young companies to get professional testing under real-world conditions? We thought so, which is why we had the testing gnomes in our research lab cook up uTest Express.

Designed for startups, Express helps young companies validate the quality & performance of their mobile apps, as well as speed up their app store approval process. Think of Express as a time machine for mobile app developers: they can transport a month or two into the future to see what type of ratings their apps will likely receive and if there are any major defects — before they launch.

Whether you’ve built an app for iPhone, iPad, Android, Blackberry, WinMo or Symbian, Express makes it easy to get real-world testing and expert feedback that meets your needs and budget. Oh, did we forget to mention that plans start at only $499? That’s half of what you probably spent on Red Bull and Starbucks while you were building your location-based, freemium, socially-linked, caffeine-free, voice-powered, 3D, virtual reality, highly-addictive, semantic-searching, gaming-layered, hybrid-powered, native app, right?.

For each project, uTest handpicks members of its tester community from North America who best match the testing requirements and have the right mobile devices and operating systems. The customer’s mobile application is then tested professionally on real devices across real carriers, providing real-world testing results and expert feedback that aren’t possible with emulators, simulators or remote access.

At the conclusion of each project, customers receive a list of well-documented bugs, including screenshots and videos with steps to reproduce them. They also receive expert feedback from the testers about the application – including app ratings and feedback for interface design, usability, app performance and more. To learn more about how it works,  watch this brief Product Tour.

Here are a few key features of the service:

• Cost-Effective: With three testing packages available, uTest Express offers mobile app testing to fit any startup budget;
• Easy-to-Use: A simple interface walks customers through a series of questions to identify their testing needs and facilitate the creation of the testing project;
• Professional Testers: Testers are selected from our community of 35,000+ professional testers;
• Real Devices and Carriers: Testing on real devices and real carriers – no emulators, simulators or remote access
• All Major Mobile Platforms Available: uTest Express is available for all mobile operating systems including iOS, Android, Blackberry, Windows Mobile and Symbian;
• Accessible 24/7: Customers can access their testing projects on-the-go, using uTest’s downloadable app for the iPhone and iPad, or through the traditional browser-based interface.

The launch coincides nicely with our trip to CTIA Wireless – one of the largest mobile technology events in the world – so we’ll be spending the next few days talking to some of the most promising names in the business about this unique new service.

In the meantime, if you’re early-stage startup looking for an easy, affordable way of testing your latest mobile app (or your latest version), look no further than uTest Express.

Last month, we launched Reviews with version 3.4 of our testing platform, and now the reviews are in (pardon the pun). Customers love the new feature, and testers enjoy having a new way to deliver structured app feedback.

Custom Reviews
For version 3.5 of our platform, we’re taking Reviews and making them better. The biggest feature request has been to make it easy to add new review questions and customize the feedback process. Our engineers tackled that requirement with gusto, and today we’re launching a new way to modify reviews.

The process is very easy: you can remove the default questions, add new questions, or change the question order to reflect your priorities.

We think reviews are a great feature for any customer looking to get feedback from real people using their software. And of course, they’re entirely optional (and can be enabled as part of any test cycle). Talk to your project manager to learn more.

Tester Device Requirements
As our community has grown, we have gathered over 35,000 enthusiastic software testers who love technology. Many of them own more than one computer or mobile device, and a few of them own dozens. In the past, testers might be matched to a project because one of their devices made them a good fit, but they had no way to know which device to prioritize for testing. Customers or uTest project managers would have to reach out to each tester to let them know that they should, for example, use their iPhone and not their Android device to test a mobile website.

With this release, we’re introducing a new system for testers to help guide them to the right device for testing. Customers can specifically say that a tester should focus on testing with specific devices while disallowing others.

Wrapup
Have an idea for future product releases? uTest community members can join our tester forums and check out our Platform Feedback section. Customers can contact their project manager directly or drop us a line.

Work never stops when you’re building a global testing platform. We could have sat back and taken a breather after releasing our first iPhone and iPad app last week, but that’s not how we roll. Last night our engineers pushed out yet another bundle of new code – this time version 3.4 of our testing platform.

Like most of our platform releases, this one contains the usual improvements in performance, the miscellaneous back-end features that help us more efficiently run a community of 30,000+ testers, and dozens of bug fixes that were all based on testing with our community. But it also has something very new that we think is going to be a big deal – reviews.

While different customers have very different testing needs, we often hear the same question at the conclusion of a test cycle. “What did the testers think about my app?” Quite often an app hasn’t seen the light of day outside of a few developers, and QA and product managers are frequently eager to hear something – anything – about how their app was received by real users. Even a little feedback can be a good early indication of market reception.

Reviews are a way to solicit structured feedback about an app. They are optional, and can be enabled as part of any functional test cycle.

Testers will be able to evaluate apps on a number of criteria, including overall design, feature breadth, performance, and more. As part of the rating process, they will also be able to provide commentary about why they felt the way they did (something that’s hard to figure out with a typical app store rating).

Customers can ask testers for more information if they feel like a review lacks enough detail or if they have questions. And just like bugs can be approved or rejected, app reviews can be rated as helpful or not helpful.

Like many features we implement, this one was based on input from our customers. We are always interested in ways to improve our platform. Do you think you have a great idea for us? uTest community members can join our tester forums and check out our Platform Feedback section. Customers can contact their project manager directly or drop us a line.