Software Testing Blog

The OnDemand event in Palo Alto provided the stage where the top Internet companies disrupting the enterprise converged with the incumbent pioneers in cloud computing and SaaS. Kicking off the two-day event with keynote Marc Benioff, Salesforce.com CEO who ignited the on-demand revolution, sure set the bar high!

These days were packed with investor roundtables, CEO showcases, and panel discussions on the trends and opportunities in SaaS and cloud computing services.

Lucky for those of us who couldn’t make the show, all the presentations (including Marc’s!) were recorded and can be viewed on the video archives site. Check out uTest’s below! For a list of a few other great sessions, continue reading after the bump.

Read more…

Good usability is like pornography — it’s difficult to define, but “we know it when we see it” (it’s ok, this link is safe for work, we swear). Unlike pornography, however, an intuitive, seamless user experience is difficult to find in web & mobile apps.

And for something that’s supposed to make life easier, usability testing has been notoriously time-consuming, expensive and difficult to navigate.  Well, that ends today.

Now product managers, marketers and graphic designers can build a virtual focus group that matches their user base (by location, language, age, gender, education level, industries or hobbies) and get unbiased, qualified feedback before they launch. And the whole thing can be done in a matter of days, not weeks or months.  And at a fraction of the cost of lab-based usability testing.

uTest now offers a  couple of quick and cost-effective options to help companies refine the usability of their apps:

• We have a basic survey-based usability testing service that enables a virtual focus group to be hand-picked from our community of 25,000+ software-savvy professionals.  They will use the customers’ web, desktop or mobile apps and then complete a survey that the customer has created.
• We also offer a full-service usability testing option that includes consultation with a user experience expert.  This UX expert will provide a complete usability audit, create the use case(s), write the survey(s), and perform detailed analysis of the results.

And if you’re a seasoned usability expert — with formal training and practical experience in interactive design, information architecture and running usability testing projects — then drop us a line.  We’re on the lookout for more experts to help us keep up with growing demand for our usability testing services.

Boston Business Journal just announced their list of Best Places To Work. And with nearly 450 nominees and only 20 companies making the cut in the Small Company category (20-100 employees), it’s a tough list to get on. But we’ve never shied away from a tough competition, so we threw our hat in the ring. And lo and behold, we’re proud to report that uTest has joined this prestigious list of  the best places to work in Massachusetts!

We’re stoked to be on a list alongside global heavyweights like Google, Microsoft, Ritz-Carlton, Comcast, as well as local startup legends like Carbonite and HubSpot. So, how did we do it? Well, from our open bullpen layout to our always-stocked kitchen to tweeting from the slopes,  to a company outing that consists of climbing Mount Monadnock, uTest is not your typical company… even by startup standards.

Read more…

Software testers pay a lot of attention to software problems, but bugs and design flaws affect numerous industries.  So as we wrap-up the week, let’s stop to consider a few product failures that we really wish had never happened.

Blowout Preventers Fail
Last week, the Deepwater Horizon drilling rig exploded, tragically killing 11 people and injuring dozens more.  Now the well is leaking oil into the Gulf of Mexico at a rate of anywhere between 1,000 and 25,000 barrels of oil per day (nobody is quite sure).  One barrel of oil contains 42 gallons or about 159 liters, meaning a whole lot of oil is flowing into the Gulf.

Normally, oil wells are capped with a device called a blowout preventer (or BOP) that stops the flow of oil in an emergency.  A BOP features numerous valves designed to close a well quickly, including a last-resort valve called a shear ram that should work when all else fails.  It operates like a giant pair of scissors – cutting the drilling pipe and closing off the well.  Now BusinessWeek reports that a study performed by the U.S. Minerals Management Service found that 50% of shear rams failed at cutting the drilling pipe and closing properly.  Current evidence suggests that the BOP for the Deepwater Horizon failed to close as well.

(The picture above was taken by a robotic submarine trying to manually activate the shear ram on the Deepwater Horizon’s BOP.)

Read more…

It’s been a busy few weeks around the hallowed halls of uTest (ok, they’re really more of a beige).  Somewhere between delivering the keynote at the QUEST conference, interviewing testing guru Scott Barber, and launching v3.0 of our platform, we found time to sign a partnership with SOASTA.

For those who don’t know them, SOASTA leverages the cloud to offer on-demand load & performance testing, which enables companies to test the real-world performance of their web apps.  In fact, the Wall Street Journal recently named them one of the top 50 VC-backed companies in all the land.  So what’s a crowd company doing partnering with a cloud company?  A lot, actually.

For starters, this partnership expands upon uTest’s recently announced load testing services.  As our customers and their apps become more sophisticated, they’re turning to uTest to provide them with real-world load testing.  And this partnership with SOASTA enables us to offer a unique package of on-demand load testing and advanced analytics to help our customers identify bottlenecks and points of failure between their apps and their users.

Read more…

In the third and final part of our Q&A with testing guru Scott Barber, we discuss what qualities to look for when hiring testers; a world without testing teams; when to start blogging, and much more. If you missed our earlier interviews, here’s Part I and Part II.

uTest: It looks like you’ve hired your fair share of testers – what skills, experience and characteristics do you look for when making these critical hiring decisions?

SB: The most important things I look for in an individual are aptitude to learn whatever technologies, business information, processes, etc. that they will be expected to test – that and investigative curiosity.  Beyond that, I’m always looking to build a well-rounded and diverse testing team. I find that a team full of testers “like me” leads to us all missing the same kinds of bugs, so I’m always looking for a breadth of skills and experiences across the team. I’ve hired aircraft mechanics, medical billing specialists, call-center representatives, technical sales representatives, subject matter experts, developers… oh yeah, and even a few testers.

I know how to teach people technologies, the basics of a new programming language, how to report bugs, how to document process, and how to generate test ideas from whatever information we have.  What I don’t know how to teach people is how to be curious, the desire to investigate their curiosity, or to be excited by the thought of investigating things that strike them as odd.

So I guess I’m always on the lookout for someone with a tester’s attitude who fills a gap in perspective of the current team whose own knowledge or skill gaps fall in areas I know how to teach them or get them trained in.

Read more…

In part II of our interview with Scott Barber, we tackle the so-called “consensus” among the Context-Driven School; his start as a software tester; why software problems are mostly people problems; reporting out-scope-bugs and much more.

We will post the final installment tomorrow. By the way, did you miss Part I?

uTest: There’s a lot of consensus among the Context-Driven School of Software Testing, but are there any fundamental differences? You can’t agree with guys like Michael Bolton and James Bach on everything, can you? Speak freely, your secrets are safe with us!

SB: I don’t know that there *is* a lot of consensus. I know that I am very much Context-Driven. In fact, my license plate says “CONTEXT”. That said, the buzz and counter-buzz surrounding the Context-Driven School of Software Testing is still causing as much harm as good.

I think the underlying problem is that the creation of software is widely treated as a software development activity, which may or may not benefit from related activities like testing. I find this thought process just plain silly. Software development, in my experience, is done for one of four reasons; altruism (minimal), education (some), research and development (slightly more than some), and/or financial/business advantage reasons (most). I think the fact that software is created for different reasons and purposes makes software development at least context-dependent by definition. The difference between being context-dependent and being context-driven is fundamentally the difference between being reactive vs. being proactive.

I think there never would have been a need to call out or name the Context-Driven School of Software Testing if testing hadn’t been so undervalued, misunderstood, and too often ignored entirely for so long that some entrepreneurial folks decided to step up and start talking about “the right way” to test software. While I’ll be one of the first to admit that many of these folks published “right ways” that were far superior to the state of the practice of the time, none of these “right ways” were right for everyone or every situation.

Read more…

Our Testing the Limits guest this month is testing guru Scott Barber, the Chief Technologist of PerfTestPlus. A speaker, writer, teacher and entrepreneur, Scott has one of the most impressive resumes in the business, particularly in the realm of customized testing methodologies, embedded systems testing, personal security systems and other topics – all of which are discussed on his blog.

In Part I of our 3-part interview, Scott discusses the Manifesto for Agile Development (almost ten years after it was created); the expectations of today’s testing managers; the notion of testers as an “unfortunate necessity”, the 1983 War Games movie and more.

uTest: As a signatory on the Manifesto for Agile Development, can you comment on the progress being made by software companies in upholding these principles? Have they exceeded your expectations, or is there still a long way to go?

SB: Honestly, I think the buzz around the “Agile movement” has, in many cases, taken the industry in an unfortunate direction. I meet far too many people and companies who are completely unfamiliar with the Agile Manifesto and think of Agile as a collection of practices, processes, and tools. The reality is that Agile is a far more a mindset and a culture than it is a collection of practices, processes and tools. Agile isn’t the best fit for every situation, or for every person.

I believe that the trend to “go Agile” is misguided. If a company is developing good software, the people involved in developing that software are happy working there, the software development is sustainable, and the business is being adequately served by that software, there’s really no need for them to try to be more or less Agile. Agile has challenges like any other culture, but the single biggest challenge I find is companies trying to solve development, process, management, and/or schedule problems by “going Agile.” Teams who have grown up in a culture that is fundamentally different than Agile simply will not find it easy to “go Agile.”

Read more…

After months of hard work, we’re excited to announce version 3.0 of our testing platform.  This is much more than a simple refresh or a minor upgrade — this is a full-blown rewrite of our testing platform, from the UI design, some cool new features, back through the code and a new, open architecture.  And it’s all based upon feedback from customers, testers and all we’ve learned from 1,000+ test cycles in the past 18 months.

Our goal was to produce a faster, more usable, more powerful experience for uTest customers and testers alike.

So what’s new?

• User Interface: Our new UI was designed to provide a simpler, richer, more interactive experience with more intuitive navigation from bug-to-bug or test-cycle-to-test-cycle
• Improved Infrastructure: Our new architecture enables better scalability and, more importantly to you, is designed to provide improved performance (with faster page load times) around the world
• API and Integration: Our new and open APIs will enable us to rapidly expand our offering and better integrate with our customers, partners and third-party app developers

What’s in it for you?  If you’re a customer, keep reading.  To check out what’s new for testers, you can skip down to the tester features.

Customer Features
Once our customers get past our sleek new UI, we hope they notice the great new features and changes we’ve included.  Two features are really important.

First, we’ve built a new test cycle wizard.  Creating a clear, concise test cycle is the most important thing a customer can do to ensure success, but there are a lot of details required to make a test cycle perfect.  Our new wizard makes this process easy, making sure each part of the test cycle is well-documented.

Second, we’ve greatly enhanced our tester rating system.  Our platform will evaluate the past performance of each tester based upon activity levels (# of test cycles participated in, # of bugs reported, # of test cases completed) and the quality of that activity (bug approval %) and, once we have enough data points, we’ll assign them a rating.  Testers who are in the top 20% of all rated testers will receive a gold, silver, or bronze rating.

Some of the other new features we’ve added in v3.0:

• Resizable Columns – On any table or list, all columns can be re-sized and re-arranged by customers using a simple drag-and-drop.
• Tester Messenger - Enhanced tester messaging tools help you communicate with testers to get what you need from each bug report
• Testing Coverage Reports – Be confident that you’ve covered every corner of your testing matrix with our new coverage reports. You’ll know what’s been tested and what still needs to be tested in a single informative report
• Easier Filtering – All test cycles and bugs can be easily filtered to find exactly the information you need
• Smart CSV Exporting – Export a CSV file with just the information you need about your bugs or your test cycles
• Multiple Users – Add multiple members of your testing organization easily with our new account management tools

Tester Features
Besides the new UI, we’ve added a lot of cool new features for our uTester community.  However two features are really huge for testers.  The first is our new test cycle wizard that helps testers review test cycles more easily and completely, as well as telling uTest whether or not they intend to participate in that test cycle.  Now when a tester checks out a test cycle, they will be able to indicate whether or not they want to join.

Our second big feature is a new & improved tester ratings system.  In the past, testers were rated with a star system based upon subjective customer feedback.  Starting today, testers who are active will earn a rating and a “Rated” badge.  This shows that these testers are actively participating in test cycles contributing their expertise to the uTest marketplace.  Additionally, those testers who are in the top 20% of our rated testers will receive a gold, silver, or bronze badge.  The remaining testers will receive a “Rated” badge.  Testers who are not active or for whom there is not enough data will not receive a badge.  New testers will receive a “New” badge.

A tester’s rating will depend on many factors including total activity and recent activity (# of cycles participated in, # of bugs reported, # of test cases completed), as well as the quality of that participation (bug approval %, test case approval %, and accuracy in specifying bug types and severity).

Other great new features include:

• Resizable Columns – For any table or list, all columns can be re-sized and re-arranged using a simple drag-and-drop.  For example, a tester can move the test cycle name to be in the first column with a width that displays the entire test cycle name
• Tester Messenger – Enhanced communication tools between testers, uTest project management and customers
• Easier Filtering – All test cycles and bugs are easily filtered.  For example, testers may filter bugs by “My Bugs Only,” status, severity, and by test cycle; even for closed test cycles
• Clearer Test Cycles – All test cycles are formatted in a more digestible way to help testers more easily and completely review each test cycle
• Improved Attachments – Bug attachment size limit has been raised from 5MB to 10MB (there is a limit of three attachments per bug).  Also, more file types are permitted for upload.
• Smarter Emails - Cleaner, more informative notification emails from the uTest platform

Wrap-up
Our engineers and product managers have worked countless hours putting together version 3.0, but we also want to thank our customers and testers for their countless ideas and insights.  As we continue to improve and refine our platform, we are always open to thoughts and ideas about how to make our product better.

Have a great idea for our future product releases?  Testers should join our testing forums and check out our Platform Feedback section. Customers can contact their project manager or drop us a line.

A heads-up to our testers and customers — this weekend, we’ll be launching v3.0 of the uTest platform.  This will be a major, major (did I mention major?) upgrade of our product.  It also means that we’ll be taking the platform offline for a period of time beginning on Saturday morning.  We’re working closely with customers to minimize the number of test cycles that will be live during this period, but we do apologize for any inconvenience that may result from this planned downtime.

The changes in this new version will be obvious for all to see.  But we’ll publish a complete write-up of all the changes for customers and testers, including artwork to highlight what’s new & improved.  Stay tuned to the uTest blog or our uTester forums for updates on how the launch is going on Saturday.

UPDATE: As we approach noon EST, things are progressing well.  We’re still running down our long list of launch tasks, but things are looking good at this point.  More details coming soon.