Software Testing Blog

See anything shocking in the news this week about Lyndsay, Kim or Ashton? I’m guessing NOT because the paparazzi were focused on uTest’s breaking news all week! All joking aside, our $17 million D-round news (led by QuestMark) announced Monday grabbed the attention of the world’s leading journalists, bloggers and tech gurus! Here are a few outstanding pieces, but you can read all the buzz in uTest’s press room.

-Fortune: Venture Capital Deals

-TechCrunch: uTest Helps Developers Build Better Software

-VentureBeat: uTest Grabs $17 Million From QuestMark

-Boston Globe: Southborough’s uTest Completes $17M Round

-Private Equity Hub: QuestMark Leads $17M For uTest

-Xconomy: uTest, Boosted By Surge In Mobile App Dev, Inks $17M

-TechTarget: Crowdsource Test Group uTest Announces $17M Funding

-Boston Business Journal: uTest Takes Step Towards IPO with $17M

-BostInnovation: With Revenue Increasing Y-O-Y 250%, uTest Raises $17M

Celebrate with us and have a great weekend everyone!!!

In case you missed today’s news, uTest announced an exciting new partnership with Acquia, the enterprise guide to Drupal. As part of the deal – which provides their customers with unique access to uTest’s full suite of testing services – Acquia was legally obligated to write a guest post for our blog. Actually, that wasn’t part of the deal, but we were able to get a great guest post from them anyway.

Meet Stellina McKinney – Acquia’s Director of Engineering Services – who is here to discuss how Acquia leverages the uTest community. That’s right, not only is Acquia a uTest partner, they’re also a very active uTest customer. Enjoy the post!

*******

I started at Acquia 6 months ago, having previously worked for larger, process-heavy corporations that sold packaged proprietary, software with long release cycles. Our QA teams consisted of over 50 people (sometimes a lot more), and were always the long pole in the process, whether it was Agile or Waterfall.

Not so at Acquia.

At Acquia, I manage a lean QA team of 4 people (we have another team that tests usability), and we support 5 products. We work in an Agile environment, release every 3 weeks, and meet our quality goals for each sprint.

Our QA testing strategy at Acquia is to perform tests on agile user stories (akin to use-case tests or acceptance tests in Behavior Driven Development). Our goals are to:

• Define the behavior of the system, and not have a previously-coded system define the behavior for us
• Test failure cases so that they won’t affect production
• Stress systems through performance and load automation
• Mix automated and manual testing methods, as they’re complementary (machines are fast and consistent, but people have brains and are unpredictable)

We do this by:

• Listing the scenarios that must succeed for a product to be complete
• Writing automated tests to perform basic success and failure operations
• Engaging a crowd-sourced manual testing platform to examine our product in more depth

How can we do this with only a team of 4?  uTest’s crowdsourced testing platform lets us leverage over 45 testers a month, without exceeding my start-up budget.

Read more…

Just two months after announcing an expanded suite of testing services, uTest has now launched its next generation tester profile and ratings system (Platform v4.3). These changes come at an opportune time – customers’ testing requirements are on the rise, followed by an increasing need to optimize project invitations for our global community of testers.

First, the new Profile includes the ability to control which types of projects a tester wishes to participate in. For example, if I’m only interested in security testing, I would simply fill out that specific portion of my profile.

On the other hand, you may be interested in usability testing – specifically designing surveys for end-users to complete and analyzing the results. The new profile allows you to drill down to specific flavors of testing (e.g. designing usability surveys); not just at a high level (e.g. functional, security, load, localization, and usability testing). Lastly, you can control the amount of incoming project invitations by noting your general availability for uTest projects each week.

Moving on to the rating system, testers now have the ability to earn a badge across the five major testing types. For example, instead of being a Gold uTester, one may now be a Gold uTester in security testing and a Bronze uTester in load testing. Aside from these changes, the new Rating System also places heavier weight on the quality aspect of a tester’s participation (quality over quantity). Therefore, testers who continue to submit very valuable reports will likely witness an increase in their rating. And as usual, ratings are calculated dynamically and adjusted daily based on activity across all testing projects spanning the entire community.

If you’re a uTest customer, most of these changes are under the hood – although you will directly benefit from our enhanced rating system. But if you’re a uTester, you will need to update your profile to enjoy the benefits of fewer emails hitting your inbox and more projects that are relevant to you.

While everyone is preparing to take it easy and “fall back” this November, uTest is picking up more speed and springing forward! We hope you can meet up with us at one of these upcoming events or at least join us for a quick conversation on Twitter – be sure to stay connected by following the hash tags below.

Oct 11-13 ANAHEIM:  CTIA Enterprise & Apps – we’re here right now! #CTIAEnA

Oct 10-12 SANTA CLARA: Localization World – also attending now, hash tag streaming on their site! #LWSV

Oct 18-20 SEATTLE: SIG Global Leadership Summit – uTest CEO Doron Reuveni joins Google Engineering Director James Whittaker and President of Massolution Carl Esposti for “An Introduction to Crowdsourcing” on Wed,  10/19 @ 3:40pm.

Oct 24-27 DALLAS: STPCon, Software Test Professionals Conf – uTest CMO Matt Johnston keynotes the event on Tue, 10/25 presenting “In-The-Wild Testing: Your Survival May Depend On It”. #STPCon

Oct 26-27 MOUNTAIN VIEW: GTAC, Google’s Test Automation Conference #gtac2011

Oct 28-29 LONG BEACH: SoTeC, Southland Technology Conf – uTest CTO Fumi Matsumoto presents the “10 Tech Trends Altering The Testing Landscape” on 10/29 @ 9:45am.  #SoTecConf11

Nov 1-2 SAN FRANCISCO: CrowdConf – uTest’s Matt Johnston joins Amazon, LiveOps, Trada and GigaOm on Wed, 11/2 @ 2:10pm for a debate panel. #crowdconf

Nov 15-17 SANTA CLARA: E2.0, Enterprise 2.0 – Matt Johnston presents “Is this the Year Crowdsourcing Goes Mainstream?” on Wed, 11/16 @2:30pm. #e2conf

Nov 21-24 MANCHESTER (UK): EuroSTAR – uTest VP of Project Delivery John Montgomery presents the “What The Top 10 Most Disruptive Technology Trends Mean For QA And Testing” on Thurs, 11/24 @ 11:00am in the Apps & Mobile track. #esconfs

If you’d like to meet up, shoot us a quick email or tweet – see you there!

As you may have read on Monday’s blog post, uTest launched a new informational campaign to promote http://www.inthewildtesting.com.   The web site – and associated social media channels, including a Twitter profile – are intended to educate forward-thinking technology leaders about the necessity, benefits and real use cases of in-the-wild testing. 

We decided to launch it at TechCrunch Disrupt in San Francisco because the very concept of in-the-wild software testing (versus traditional methodologies) is, well…disruptive. 

Sure enough, TechCrunch Disrupt turned out to be the perfect event!  There were more than 2,600 innovative, entrepreneurial-minded techies, investors and exhibitors (35% more attendees than expected) filling the halls of the Design Concourse Center from Monday to Wednesday.  In its usual fashion, the conference itself attracted top industry-leaders such as Reid Hoffman of LinkedIn, Marissa Mayer of Google, Vinod Khosla, and even Ashton Kutcher.

uTest hosted a ton of terrific activities over the course of the event:

Read more…

Every few weeks, it seems like there’s another major security breach to the website, gaming system or native app of a big global brand.  And that doesn’t even include the hundreds (thousands?) of hacks into the properties of smaller enterprises, SMBs and startups that consumers may (or may not) hear about.

In fact, a few months ago we wrote about The Top Security Hacks of 2011, and referenced that the attacks on Playstation were estimated to have cost Sony $24 billion dollars– nearly 10x their revenue for the same period.

So here’s the point: Would you rather look back and say your company overshot and used too many systems for security testing?  Or get that nauseaus, sinking feeling in your gut when your CIO wakes you at 2:00am and says the company has spent too little?

That’s why– as the cornerstone of uTest’s showstopping announcement yesterday– we announced the launch of uTest Security Testing that leverages the talents of new and existing white hat security professionals within our crowdsourced community.  Since we now offer the first crowdsourced, real-world security testing in the world…there’s a new kid in town to join the collective effort to protect your company, and customers’, private data.

Moreover, we’ve joined forces with industry leader Veracode to provide seamless access to their complementary, cloud-based application security verification services.  Veracode has scalable, policy-driven application risk management programs that help identify and eradicate numerous vulnerabilities by leveraging best-in-class technologies from vulnerability scanning to penetration testing and static code analysis.

As a result, companies will have access to a cost-effective, powerful combination of automated (Veracode) and real-world (uTest) testing that mitigates security risks across the entire software development lifecycle.

We’re thrilled, honored and excited to be partnering with Veracode.  And we’re certain that our joint offering– as a complement to organizations’ in-house security testing– will offer tech executives peace-of-mind at a price with infinitely fewer zeroes than $24,000,000,000.

For those who frequent the uTest website, blog or forums, you may notice a few changes around here. Yes, we had a little work done… nothing major, just going from focusing on one testing service (functional) to a providing whole range of testing services that span the dev lifecycle (functional, security, load, localization and usability testing).

For our customers, this means they can find the testing expertise they need, no matter where they are in their SDLC. And for testers, it means provide more earning opportunities for those individuals with expertise in areas like security testing, performance engineering, or localization validation. Like I said, nothing major. </sarcasm>

In all seriousness, these are exciting times around the halls of uTest. We’ve spent the past 12 months trialing new types of testing services with select beta customers. And now, we’re ready to offer them to any and all companies, on demand. A quick introduction to uTest’s new suite of testing services:

Functional testing services to help ensure your applications function as intended. Our services related to functional testing include exploratory testing, test case execution, test case creation and writing automated test scripts.

Security testing services to help you avoid launching products with common security- and privacy-related vulnerabilities. Our services include tools-based static and dynamic security testing, as well as manual penetration from trusted, white hat security testers.

Load testing services to make sure your app is ready for peak traffic, and that performance won’t degrade under heavy load. Our services include live load, simulated load and a hybrid load offering that combines cloud-based load testing with live testers.

Localization testing to validate that your app is saying what you think it’s saying. Services include translation validation from native speakers who live in-market, as well as full L10N testing that covers content translations, currency, taxes, shipping options and more.

Usability testing to help you launch products that are intuitive, clean and achieve high conversions. Services include surveys-based testing with targeted focus groups (by age, gender, education, hobbies, location, etc) or usability audits from one of our UX experts.

Special thanks to our friends at Stein + Partners for all their help with our rebranding, as well as an epic month of late nights from the amazing uTest crew. And finally, a word of thanks to our testers for their help in this launch, and the dozens of customers who helped us learn so much about each of these new types of testing. If you’d like more info about any of these new services, drop us a note.

We’ve got more on the way in the coming months. We’re not going to rest until we’ve completely reinvented the way testing services are provided in this ever-evolving apps universe.

Have a comment? Want to tell us you hate/love the new look? Drop us a comment and let us have it!

Update: Mike Butcher over at TechCrunch just took this news prime time. Seems we’re not the only ones who recognize the need for better app security testing.

Working at a start-up is kind of like having kids. You get excited about every new milestone like a proud parent, even though you know it’s not the first time any of it has ever happened in history. This time it’s yours. Your child. Your company.

For parents, new milestones usually mean posting a select, few pictures to Facebook (give or take 50) of an an event like your child sitting up for the first time. Perched like a top-heavy teddy bear, staring wide-eyed at the camera and being photographed with adoration from many, many…many angles.

For uTest, it usually means hitting the local pub and celebrating together with a pint or two (give or take).

But this milestone was special. Our company, headquartered outside of Boston, opened up our first office in Silicon Valley. Exciting stuff for our founding members and newcomers alike! Granted, with so many recurring clients out here in the Bay Area, uTest has had a West Coast team for over a year, incubating in the Sunnyvale Plug & Play Tech Center.

But to keep up with demand, we’ve expanded pretty fast and definitely outgrew the nest. So now we have a space of our own in bustling downtown San Mateo, one block from the Cal Train and smack dab in the middle of a ton of great restaurants and shops.

What better way to celebrate the milestone and break in our new digs than throw a party! With our CEO Doron Reuveni and the executive team in town, that’s exactly what we did.

Living by the quote of our CMO Matt Johnston– “We take our work seriously, but not ourselves”– we spent a warm summer night celebrating with about 75 of our investors, customers, partners, new neighbors (the good folks at Attributor), new and old friends in the industry, the Chamber of Commerce…and even Mayor Jack Matthews.  Sushi, drinks, and music in the company of some of the Valley’s smartest, funniest, and most fascinating people. It doesn’t get better than this.

In fact, we’re already thinking about throwing another party in a few months for the many who couldn’t make it to this one.  Who knows…we may have a whole new milestone to celebrate.  And cleared space on our camera.

A little more than a year ago, we launched version 3.0 of the uTest platform. It was a bold rewrite that brought a better, faster, more powerful uTest experience. Last night, we launched a new version that takes that same experience and goes even further.

Besides an improved user interface, we’re launching three big new features: Bug Fix Verification, Test Case Support, and Tiered Bug Approval. Let’s take a look at them.

Bug Fix Verification
More and more of our customers are asking us to not only find bugs in their apps, but also to check to make sure the bugs were fixed as well. It makes sense – our testers are finding “real world” bugs, often times involving unusual combinations of devices, browsers, geographies, and more. For our customers, it may be completely impossible for a developer to replicate a particular bug or verify a fix in-house.

With this new service, customers can have a tester try to recreate a bug to determine whether or not a fix successfully eliminated a defect. The process is incredibly easy – uTest already knows the details about the bug, including the steps necessary to reproduce it and the environment on which it was discovered. For our customers, validating a bug fix is as simple as clicking a button and waiting for the uTest community to do its work.

Bug Fix Verification closes the loop between our community of testers and our customers’ developers. It is an additional service for our functional testing customers, and the pricing varies depending on the needs of each customer. If you are interested in learning more, talk to your project manager or contact us for more details.

Test Case Support
Many of our customers use test cases as part of their testing process. Customers write a step-by-step set of instructions for testing their application, making sure to cover all the components that must be validated. The uTest community follows these instructions using a wide variety of operating systems, browsers, locations, and more – providing comprehensive testing coverage. Customers can quickly learn if their app is ready for launch or if it has problems.

Up until now, handling test cases on the uTest platform was cumbersome and involved a lot of extra work. With this release, we are introducing a new feature that makes it easy to create and organize test cases within the uTest platform. Customers will be able to include test cases with their projects, and then see at a glance (and in real-time) the status of their testing coverage and where problems exist.

For testers, having built-in test case support makes it easy to follow the flow and report bugs about specific steps in the process. Bug reports will be tied to the exact step of the test case, making it much simpler to explain what went wrong and how.

Tiered Bug Approval
As the uTest community grows, we are always looking for ways to better align our testers and our services with the needs of our customers. With this new version, we’re launching a new feature called Tiered Bug Approval. Going forward, when customers approve bugs in the uTest platform, we’ll be asking them to rate the value of each bug.

For testers, bugs that are more valuable for our customers will yield higher payouts while bugs that are less valuable will yield lower payouts. Bug payouts will no longer be tied to bug type or severity, but will now be based on the customer’s assessment of approved bugs. This new feature is designed to align tester payouts much more closely with the real value that each tester creates for our customers.

Wrap-Up
Besides these new features, 4.0 includes dozens of other changes including UI enhancements, bug fixes, performance improvements, and more. Our product and engineering teams have been working hard on this update, and many of their best ideas came from our customers and our testers.

Have a great idea for our future product releases?  Testers should join our testing forums and check out our Platform Feedback section. Customers can contact their project manager or drop us a line.

If you’re following us on Facebook–and we hope you are!–the 1st contest in our “Under Hopper’s Magnifying Glass” competition has ended…and we have a winner! 

We asked people to guess what was under the magnifying glass and over 230 people chimed in.  We got some pretty funnny and creative guesses, ranging from an iPad cover (lobbying for the prize?  haha) to a mole to an Oreo.  One uTester even thought it was it was a meteorite!  But the correct answer was…drum roll, please… a dog’s nose! 

The first person to submit the correct answer was super sleuth Becky Presgraves, who won a brand new iPad 2. Congrats, Becky!!   

Check Out This Week’s Contest and Get Involved!

If you missed the 1st competition, you have three more chances to win an iPad 2.  We’re sponsoring one per week, so log-on and ante in! 

This week, we’re asking you to tell us what YOU want to see under Hopper’s magnifying glass!  If your idea receives the most ‘likes’ at the end of the week, you’ll will win an iPad 2.  So make sure to lobby your friends and get them to vote for your entry!  

For a chance to win, check out our specially designed Under Hopper’s Magnifying Glass app on Facebook here.  Good luck, and may (Hopper’s) force be with you-