Software Testing Blog

Ho, ho, ho!  Whoa there, Blitzen– wasn’t it just Halloween?  It sure feels that way. After all, I still have two pounds of trick-or-treat candy to pretend I’m not eating.

Unfortunately, my four-year-old has already implored me to take down the skeleton and spiders hanging in the doorway because they’re going to scare away Santa.  So, rather than arguing the salient fact that Santa shimmies down the chimney versus ringing the doorbell, I’ve officially started morphing decor from the marvelous macabre to merry old Saint Nick.  Kids: 1. Mom: 0.

Nonetheless, the fact hasn’t escaped me that we’re two weeks away from Cyber Monday (November 28th), an occasion that online retailers have been planning for months.  Since summer, global brands and independent e-tailers have been testing and re-testing their mobile apps and web sites for functionality, usability, localization glitches and possible bottlenecks in site performance that could jeopardize their revenue potential.

Moreover, the ante has been upped now that the iPad and other tablets have entered the scene.  Online retailers that spent the last few years optimizing their mobile apps and porting them to additional platforms like Android, are now going through the process from scratch with tablets.  Not only are the specs non- standardized, varying significantly by manufacturer, device and network performance like smartphones.

Read more…

I found myself deliberating on something unexpectedly the other night.  I was thinking about buying the iPad–which I’ve wanted for a long time–and it occurred to me: What’s the future of Apple?

Previously, the issue was whether I should I invest in iOS and start the conversion over from a lifetime on Windows.  After all, my dad was a 30-year IBM vet, which put food on the table and paid my tuition.  I grew up seeing mammoth mainframes, punchcards…glowing green DOS.  No Apples of any color in our Big Blue household.

But on this occasion, it wasn’t a question of brand loyalty. It was the obvious: the loss of Steve Jobs.

I still find myself processing his passing both emotionally and practically. I remember how the AP alert popped up on my phone and it literally felt like someone had punched me in the stomach.  I admired him for living authentically, taking billion dollar gambles on ideas, picking himself up after billion dollar failures, and holding steadfast (stubborn?) to his vision.

I’m convinced his near-religious zeal over every minutiae of product design stemmed from the same social ethic that led to Apple’s creation:  to make computers so easy and user-friendly that everyone could benefit from computing’s powerful potential.  Not just the technical, highly-educated and elite. Computers for Everyman.

Attention to detail.  Risk-taking. Singular focus. These are among the core values of the Apple brand. As I considered buying the iPad, I wondered:  Are these values sufficiently infused in Tim Cook and the company DNA to continue on without Steve?  Or will Apple employees slowly lose direction like followers of the North Star left without guide over too many cloudy nights?
Read more…

Just two months after announcing an expanded suite of testing services, uTest has now launched its next generation tester profile and ratings system (Platform v4.3). These changes come at an opportune time – customers’ testing requirements are on the rise, followed by an increasing need to optimize project invitations for our global community of testers.

First, the new Profile includes the ability to control which types of projects a tester wishes to participate in. For example, if I’m only interested in security testing, I would simply fill out that specific portion of my profile.

On the other hand, you may be interested in usability testing – specifically designing surveys for end-users to complete and analyzing the results. The new profile allows you to drill down to specific flavors of testing (e.g. designing usability surveys); not just at a high level (e.g. functional, security, load, localization, and usability testing). Lastly, you can control the amount of incoming project invitations by noting your general availability for uTest projects each week.

Moving on to the rating system, testers now have the ability to earn a badge across the five major testing types. For example, instead of being a Gold uTester, one may now be a Gold uTester in security testing and a Bronze uTester in load testing. Aside from these changes, the new Rating System also places heavier weight on the quality aspect of a tester’s participation (quality over quantity). Therefore, testers who continue to submit very valuable reports will likely witness an increase in their rating. And as usual, ratings are calculated dynamically and adjusted daily based on activity across all testing projects spanning the entire community.

If you’re a uTest customer, most of these changes are under the hood – although you will directly benefit from our enhanced rating system. But if you’re a uTester, you will need to update your profile to enjoy the benefits of fewer emails hitting your inbox and more projects that are relevant to you.

A little more than a year ago, we launched version 3.0 of the uTest platform. It was a bold rewrite that brought a better, faster, more powerful uTest experience. Last night, we launched a new version that takes that same experience and goes even further.

Besides an improved user interface, we’re launching three big new features: Bug Fix Verification, Test Case Support, and Tiered Bug Approval. Let’s take a look at them.

Bug Fix Verification
More and more of our customers are asking us to not only find bugs in their apps, but also to check to make sure the bugs were fixed as well. It makes sense – our testers are finding “real world” bugs, often times involving unusual combinations of devices, browsers, geographies, and more. For our customers, it may be completely impossible for a developer to replicate a particular bug or verify a fix in-house.

With this new service, customers can have a tester try to recreate a bug to determine whether or not a fix successfully eliminated a defect. The process is incredibly easy – uTest already knows the details about the bug, including the steps necessary to reproduce it and the environment on which it was discovered. For our customers, validating a bug fix is as simple as clicking a button and waiting for the uTest community to do its work.

Bug Fix Verification closes the loop between our community of testers and our customers’ developers. It is an additional service for our functional testing customers, and the pricing varies depending on the needs of each customer. If you are interested in learning more, talk to your project manager or contact us for more details.

Test Case Support
Many of our customers use test cases as part of their testing process. Customers write a step-by-step set of instructions for testing their application, making sure to cover all the components that must be validated. The uTest community follows these instructions using a wide variety of operating systems, browsers, locations, and more – providing comprehensive testing coverage. Customers can quickly learn if their app is ready for launch or if it has problems.

Up until now, handling test cases on the uTest platform was cumbersome and involved a lot of extra work. With this release, we are introducing a new feature that makes it easy to create and organize test cases within the uTest platform. Customers will be able to include test cases with their projects, and then see at a glance (and in real-time) the status of their testing coverage and where problems exist.

For testers, having built-in test case support makes it easy to follow the flow and report bugs about specific steps in the process. Bug reports will be tied to the exact step of the test case, making it much simpler to explain what went wrong and how.

Tiered Bug Approval
As the uTest community grows, we are always looking for ways to better align our testers and our services with the needs of our customers. With this new version, we’re launching a new feature called Tiered Bug Approval. Going forward, when customers approve bugs in the uTest platform, we’ll be asking them to rate the value of each bug.

For testers, bugs that are more valuable for our customers will yield higher payouts while bugs that are less valuable will yield lower payouts. Bug payouts will no longer be tied to bug type or severity, but will now be based on the customer’s assessment of approved bugs. This new feature is designed to align tester payouts much more closely with the real value that each tester creates for our customers.

Wrap-Up
Besides these new features, 4.0 includes dozens of other changes including UI enhancements, bug fixes, performance improvements, and more. Our product and engineering teams have been working hard on this update, and many of their best ideas came from our customers and our testers.

Have a great idea for our future product releases?  Testers should join our testing forums and check out our Platform Feedback section. Customers can contact their project manager or drop us a line.

Over the past few years some seriously amazing testers have joined the uTest community, and today we have one of the most talented and diverse testing communities in the world. Some of our testers have taken us by surprise with their skills in areas like load automation, usability testing, and even general leadership. It’s that last area that has us most excited and is the reason behind today’s release of the uTest platform.

Test Team Lead
For a while, we have experimented with giving select and trusted members of our community additional privileges to help make our projects run more smoothly. Testers have been quietly helping our project managers with tasks like answering questions on projects, keeping testers focused and in scope, and making sure bugs match the customer’s expectations. With version 3.7 of the uTest platform, we have formalized that experiment into a new role within the community – the Test Team Lead (or TTL for short).

The TTL is not a replacement for a uTest project manager and their role will be mostly behind the scenes from a customer point of view. However, their contributions will be instrumental in helping to deliver high-quality testing results. Project managers will rely on the TTL to handle background tasks necessary to keeping a diverse community of testers in-line and on-task. This will free up the project managers to better help customers manage their test cycles, understand their bugs, and get the most from the uTest community.

Test Team Leads are select members of the uTest community who undergo an extensive training process. Among our best testers they are the best, but it is more than just their testing skills that qualify them as a TTL. They must also demonstrate leadership, communicate clearly, and be effective at organizing and motivating other testers.

Changes in 3.7
Starting in v3.7 of our platform, uTest project managers will be able to invite a Test Team Lead to projects of their choosing. The Test Team Lead will join the project just like any other uTest project, but their role will be different. Instead of reporting bugs or completing test cases, they will be helping other testers, answering questions, and keeping the project in scope.

TTLs will have a greater degree of privilege within the uTest platform, including the ability to use Tester Messenger to communicate with other testers and the ability to use Customer Notes to leave comments for project managers and customers. A TTL will not be able to approve or reject bugs, and project managers will still be responsible for the overall success of a test cycle as well as be the primary point of customer contact. Not every test cycle will require a TTL, and project managers will use their discretion to decide when a TTL will be needed.

While version 3.7 enables this new role, this is a much bigger evolution for our crowdsourced testing platform. We’re excited that our community will be taking a more active role in organizing itself, and we think this is a great way to inspire and deliver excellence.

Wrapup
While we have some good ideas in our product pipeline, we’re always looking for more. Do you have an idea for future product releases? uTest community members can join our tester forums and check out our Platform Feedback section. Customers can contact their project manager directly or drop us a line.

Last month, we launched Reviews with version 3.4 of our testing platform, and now the reviews are in (pardon the pun). Customers love the new feature, and testers enjoy having a new way to deliver structured app feedback.

Custom Reviews
For version 3.5 of our platform, we’re taking Reviews and making them better. The biggest feature request has been to make it easy to add new review questions and customize the feedback process. Our engineers tackled that requirement with gusto, and today we’re launching a new way to modify reviews.

The process is very easy: you can remove the default questions, add new questions, or change the question order to reflect your priorities.

We think reviews are a great feature for any customer looking to get feedback from real people using their software. And of course, they’re entirely optional (and can be enabled as part of any test cycle). Talk to your project manager to learn more.

Tester Device Requirements
As our community has grown, we have gathered over 35,000 enthusiastic software testers who love technology. Many of them own more than one computer or mobile device, and a few of them own dozens. In the past, testers might be matched to a project because one of their devices made them a good fit, but they had no way to know which device to prioritize for testing. Customers or uTest project managers would have to reach out to each tester to let them know that they should, for example, use their iPhone and not their Android device to test a mobile website.

With this release, we’re introducing a new system for testers to help guide them to the right device for testing. Customers can specifically say that a tester should focus on testing with specific devices while disallowing others.

Wrapup
Have an idea for future product releases? uTest community members can join our tester forums and check out our Platform Feedback section. Customers can contact their project manager directly or drop us a line.

Work never stops when you’re building a global testing platform. We could have sat back and taken a breather after releasing our first iPhone and iPad app last week, but that’s not how we roll. Last night our engineers pushed out yet another bundle of new code – this time version 3.4 of our testing platform.

Like most of our platform releases, this one contains the usual improvements in performance, the miscellaneous back-end features that help us more efficiently run a community of 30,000+ testers, and dozens of bug fixes that were all based on testing with our community. But it also has something very new that we think is going to be a big deal – reviews.

While different customers have very different testing needs, we often hear the same question at the conclusion of a test cycle. “What did the testers think about my app?” Quite often an app hasn’t seen the light of day outside of a few developers, and QA and product managers are frequently eager to hear something – anything – about how their app was received by real users. Even a little feedback can be a good early indication of market reception.

Reviews are a way to solicit structured feedback about an app. They are optional, and can be enabled as part of any functional test cycle.

Testers will be able to evaluate apps on a number of criteria, including overall design, feature breadth, performance, and more. As part of the rating process, they will also be able to provide commentary about why they felt the way they did (something that’s hard to figure out with a typical app store rating).

Customers can ask testers for more information if they feel like a review lacks enough detail or if they have questions. And just like bugs can be approved or rejected, app reviews can be rated as helpful or not helpful.

Like many features we implement, this one was based on input from our customers. We are always interested in ways to improve our platform. Do you think you have a great idea for us? uTest community members can join our tester forums and check out our Platform Feedback section. Customers can contact their project manager directly or drop us a line.

What a year it has been for the uTest platform. In April, we relaunched our product with a ground-up rewrite. Then in the following months we added features like tester snapshots, better reporting, public tester profiles, and more. It’s been a whirlwind of activity to say the least.

Social Sharing
With version 3.3, we’re introducing a new feature to make it easy for testers to share what’s happening with their uTest accounts on Facebook, Twitter, and LinkedIn. For example, you can let the world know that you’ve just gotten a rating badge or that you’ve published your public profile. You can even add a personal message to anything you share.

Remember: You cannot say anything about specific customers outside of the uTest platform, so please be careful about what you write.

Bug Fixes
With so much development in the past 8 months, it was inevitable that some glitches got through. Fortunately, we have a community of 30,000+ testers on the watch, and their efforts gave us a good list of things that needed fixing. The holiday months of 2010 proved to be a great time for bug fixing and product tightening, and with this release we’ve fixed almost 100 bugs – nearly all of them discovered and reported by the talented testers in our community.

Closing Thoughts
Our engineers have been laying the foundations for some really amazing new features that are due to come out in the next few months. It’s been a great year, and we’re looking forward to an even more exciting 2011.

Have an idea for future product releases? uTest community members can join our tester forums and check out our Platform Feedback section. Customers can contact their project manager directly or drop us a line.

Autumn has come and gone in New England. And now that we’re done gawking at the trees, we can get back to the business of building a software testing company. Last night we launched version 3.2 of the uTest platform, and as releases go this one is pretty awesome. We’ve got some fantastic new features to tell you about:

Decline a Test Cycle
Sometimes our testers get invitations for test cycles in which they just can’t participate. Maybe it’s because they’re busy or perhaps they don’t think they’re a good match for the project. In the past, testers had no way to indicate they would not be joining a test cycle – making it hard for us to make sure enough testers were committed to a particular project to get the necessary testing coverage for our customers.

That’s why we’re now enabling testers to decline a test cycle. The process is simple. If you’re invited to a test cycle you won’t be able to join, just decline it. You can even click on a “decline” link right in the invitation email. By doing so, you let the uTest project managers know that they need to invite other testers to fill out a test cycle.

The opposite of declining a test cycle is accepting a test cycle, something testers have been able to do for a while. Remember, if you accept a test cycle, we fully expect you to participate in-scope and in the stated test cycle duration. If you aren’t sure whether or not you’ll be able to participate, we want testers to decline it so we can plan accordingly.

Add a uTest badge to your website by logging in to the uTest platform and choosing from 6 different designs.

Show off your uTest cred with your own public profile.

The new uTest platform homepage.

Tester Public Profile & Badges
For a long time, testers have wanted a way to show off their membership and rating in the uTest community (everyone likes to brag, right?). But in the past, sharing their status with the world has been difficult. Now you can show off your uTest street cred easily with a public profile. This means our beloved testers can even add a uTest badge to their personal website, blog, resume or email signature.

This feature is opt-in. That means you must turn it on to use it (we wouldn’t share your personal information without your permission.) But our testers are awesome, and we want you to show the world that you’re a uTester and all you’ve accomplished with us. To create your public profile, just log in to the uTest platform and click on your Account Settings. Choose the Public Profile tab and configure your profile however you like.

New Platform Homepage
Next time you log in to the uTest platform, you’ll also get to see a new homepage. We want to make it easy for you to keep up-to-date with all the things happening within uTest that matter to you. With this new homepage, you can see your test cycles, bugs, alerts, and messages in one convenient place. Testers will be able to see how they rank within the community, and customers will have visibility into how their test cycles have performed over time.

Office 2007 and RAR Attachments
One last thing we want to announce is that we’ve added support for attaching Microsoft Office 2007 and RAR files to bugs and test cycles, including .docx, .xlsx, and .pptx files. This was a frequently requested change by both our testers and our customers, and we’re happy to oblige.

Have an idea for future product releases? uTest community members can join our tester forums and check out our Platform Feedback section. Customers can contact their project manager directly or drop us a line.

A couple weeks ago, we rolled out v3.0.4 of our platform. And now, after a little extra time in the oven, we’ve launched another feature designed to make it quicker & easier to access your uTest account.

Social Sign-In
Let’s say you’re like millions of other people around the world and you have an account on Facebook, Twitter, or LinkedIn. And let’s also say you’re tired of keeping track of passwords, remembering to sign-in everyday to all the websites you like, and are yearning for something simpler. Starting today, we have a solution for you, at least when it comes to your uTest account (we can’t speak for anyone else). Now you can log in to the uTest platform using the credentials from your Facebook, Twitter or LinkedIn accounts.

Setting up this linkage is simple:

1. Get started by visiting our platform and clicking the button for your favorite social network(s).
2. Then sign in to your social network and confirm the connection.
3. And that’s pretty much it. The next time you visit the uTest platform, just click that same button and you’re in.

Enjoy!