Software Testing Blog

What better way to end a great year of Testing the Limits interviews than to pick the brain of Matt Evans, QA Director of Mozilla.His 20+ years of software testing experience include stints at Palm, where he managed the quality program for the WebOS Applications and Services of the Palm Pre smartphone, as well as Agitar Software, where he helped pioneer automated test generation from Java source code. Today, Matt is recognized as one of the foremost experts on open-source development and crowdsourced testing.

In Part I of our must-read interview, we get his thoughts on the diversity of the testing profession; the importance of developer-written unit tests; the evolution of Mozilla’s testing community; the biggest myths of crowdsourcing; the unique challenges of mobile testing and more. Be sure to check back in tomorrow morning for Part II of the interview.

********

uTest: You’ve been all over the tech spectrum throughout your career: You’ve worked at web companies and mobile companies, startups and enterprises, open source and closed source. But during that whole time, you’ve always been involved in testing and QA. What’s kept you in the testing space for the last 20 years?

ME: There are several reasons. First of all, software testing is a huge challenge and it takes a lot of different intellectual skills to do it right. It really boils down to asking the right questions about the application under test and continuing to do so. Testing an application well requires you to look at functionality from a lot of different perspectives: What are the different types of users? How will they go about using your product? In what different environments and conditions will users expect your application or product to work in? Drilling down on these questions and ultimately coming up with the test cases and test data to ensure you are adequately covering these conditions has always been very stimulating and rewarding to me throughout my career in the testing field.

Secondly, the exposure you get in the testing field is incredible. You can typically explore various technologies incorporated in an application and get well-versed with each. In fact, to do the job of a tester well, you are required to get a solid understanding of the technologies used in creating the application and the influences of the environment where the application is intended to operate. The more you understand these technical aspects of the application under test, the more you will know what are functional dependencies and environmental conditions that you must test the application under. In addition, you also must interact with the many players and stakeholders of a project. Obviously, at the top of the list are the users and customers of the application. You will need to understand their expectations and usage of the product, and translate those into testable use cases. Your relationship with development is also key. Providing the developers timely, contextual, and actionable feedback on the health of the application is critical to any software release. The exposure to technology and the various players on software projects have been key to my continued passion with software testing.

Lastly, there have always seemed to be good opportunities in the software testing area, whether it be traditional black box testing, test automation, or testing tools development. In my experience, the need for good qualified test professionals at every level has always been pretty consistent in good or bad economies.

uTest: For a mainstream web app in 2010, what’s the appropriate mix/interplay between automated functional testing and manual testing (both test case execution and/or exploratory testing)?

ME: It really depends on where the state of the software project is at. Hopefully, testing and test development are done at an early stage of the product life cycle and you have the time to develop test cases and write them as automated tests. With respect to automated tests, in my experience most of the new bugs are found at the point of developing the test cases and the initial runs of the automated scripts. Once these tests are running correctly, their future value is directly related to how often they are run on the updated code base. Ideally, they are run on the developer’s desktop before check-in as well as part of continuous integration. Actually, these days I think you are at a great competitive disadvantage if you don’t have a robust practice of developer-written unit tests and functional automated tests, all under the control of a continuous integration system. If you don’t have that in place, you need to invest in that now.

Having said that, I do also feel on any software project that there is and always should be some level of manual testing. The ultimate users of the program are not robots or automated test frameworks. They are live human beings with lots of different skill sets, backgrounds, geographic location, and expectations. You need to experience the product with your own fingers and eyes on the keyboard and screen. Most automated tests are pretty superficial and have rather myopic validation methods for determining a pass or fail. Because of that, good manual validation–especially with a fully configured and integrated product–is a must in my book. So, what’s the appropriate mix? I would target 70% automated and 30% manual.

uTest: You were recently one of the keynote speakers at the Google Test and Automation Conference (GTAC) in India, where you presented on crowdsourced testing. How much easier has it gotten to speak on this topic in the last few years? Is it starting to register with most people that crowdsourcing (particularly crowdsourced testing) is a viable method for helping to get important work done?

ME: I am relatively new to the concept of crowdsourcing and crowdsourced testing. It was a topic I started to get very interested in a couple of years ago. When I was at Agitar, we started a free service for JUnit test generation called JUnit Factory. It was a pretty successful open project that attracted more than ten thousand users. It was a win-win in that users received pretty good unit tests for their Java projects, and we got crowdsourced testing on our latest code base. Since then, I’ve kept an eye on crowdsourcing progress and I am pleased that it has made it to the software testing field. I do believe it is the right model for a lot of testing tasks, especially in the web and mobile space. So, I’m glad that uTest is making a go of it. I think the explosion of social networking over the past years has really made it apparent that the virtual crowd is a very powerful thing. It can solve lots of interesting problems we know about now, and I’m sure we will be discovering areas where crowdsourcing can be very effective in the future. So, I think crowdsourcing is a practice that is here to stay and will be with us in some form for as long as the web stays running.

uTest: The Mozilla community is unmatched in terms of sheer size, at nearly a million members. How can you possibly manage that much feedback on a regular basis? How do you separate the valuable data from the junk (aka: signal-to-noise ratio)? In short, how do you make something that sounds so chaotic seem so orderly?

ME: It certainly is a challenge and is a constant battle to stay on top of it. And the honest answer is that we do the best we can. As you have surmised there is a mountain of feedback that is received by Mozilla on a daily basis. Primarily the feedback we receive on Firefox is in the form of crash reports, Bugzilla bug reports, beta feedback and twitter. That list is in priority order of what we continuously monitor every day. We are constantly searching for any trends where a particular crash or reported bug starts to become and issue within the Firefox user population. Even a bug that affects a relatively small percentage of users say 0.5%, we are still talking about two million users. So there’s a constant vigilance to detect of there are any emerging issues within the code base.

In the latest beta releases of Firefox 4, there is the feedback feature at the top right of the browser. This allows users to quickly and with limited interaction give a thumbs up or down whether they are happy with the current behavior of the browser. This along with an optional comment is sent to a collection server for categorization and display: http://input.mozilla.com/en-US/. This has been a pretty successful method to get instantaneous feedback, although it does suffer from  a lot of noise contain within the data. But given the sorting and filtering tools we have certainly have received some key insights where we are weak and need more attention with our testing. We are exploring ways to allow users to participate more directly in the test effort through these types of feedback tool features.

uTest: People often (wrongly) view crowdsourcing as a passing fad, or something that only involves unskilled labor. In your experience, what’s been the biggest misconception when it comes to crowdsourcing or community-driven projects?

ME: I think the biggest misconception is that it is all lumped under one label and that crowdsourcing is done according to some crowdsourcing manual or standard.  The reality is that crowdsourcing and community based projects are as diverse as the people that are involved.  The interaction and participation of the project members can vary drastically from project to project. The mechanisms for interaction can foster high communication and collaboration among the members or complete isolation and independent input.  There is such a big spectrum of crowd-sourced project implementations and it isn’t correct to have crowd-sourced and community-driven projects under the same umbrella. Strict crowd-sourced projects tend to be targeted towards tasks that are discrete and distributed in nature and is better suited for projects that require a number of paralleled tasks. In strict crowd-sourced projects members tend not to collaborate together to get a consensus results and may even be in a competitive situation to get the best result.

Community based approaches tend to be longer lived projects and are typically for public benefit. Membership is considered a privilege and betterment of the community as whole is usually at the top of the list of shared goals amongst the members of the community project. In addition, you find the rise of leaders within community projects that drive the project forward. Mozilla certainly is an example of of a community-based project. Many people would be surprised at our mission statement: “to promote openness, innovation and opportunity on the web”. It just so happens that building a world-class competitive browser helps us achieve the mission. Community citizenship and passion for the project has been the key success factors for Mozilla success in my opinion.

uTest: You were the QA Director for Palm when they launched the Palm Pre Smartphone, as well as the WebOS apps and services. What’s been the biggest difference (if any) between launching a mobile product and a web product?

ME: The biggest difference between a web product and mobile device is the amount of testing and certification that must precede the launch of a mobile product. A smartphone such as the Pre is an incredibly complex and highly integrated piece of technology–much more so than a typical web application. First off, a smartphone contains a fully-functional OS, usually based on some variant of Linux running on very constrained hardware. It must perform all of its concurrent services utilizing limited memory and limited CPU horsepower. The smartphone must also respond correctly to the multitude of many current events, from those generated from the environment–like switching from wifi to a WAN internet connection–to handling data input from the user, as well as handling events from the onboard applications.

Launching a mobile product requires exhaustive certification of individual hardware components such as the CPU, modems, codecs, and displays. Even then, the finished product is really launched by the carrier and must go through their exhaustive certification tests as well. Testing an onboard mobile application is also a much harder testing task. There are so many conditions and constraints that are involved in testing a mobile application. A typical mobile application is nearly functionally equivalent to any counterpart desktop client-side web application. Take, for example, a mobile email application. It must behave and interact with the server-side application in nearly the same way as a desktop web client. The established protocols were designed for a stable communications environment, but this is just not the case in a mobile environment. The internet connection may be lost and reconnected very rapidly. The connection may even be lost for long periods of time. The application may, at any moment, be swapped out of memory. The system may be shut down abruptly. Lots of system conditions happen in a smartphone that would rarely or never happen in the context of a desktop web client application. However, a mobile application must perform its main functional operations of retrieving and sending messages flawlessly with no loss of data and full operational integrity. Testing mobile applications under these environmental scenarios is a huge challenge. In short, testing a web application is no easy task, but mobile applications and products represent a much tougher and larger testing challenge.

uTest: There’s been a lot of bold predictions about the future of native apps, the mobile web, tablets, and just about anything that relates to mobile. Having lived (and worked) through the dotcom boom and bust of the last decade, do you see any similarities with regard to mobile?

ME: Not really. In the dotcom boom and bust era, the problem was such rampant speculation about how we as consumers would utilize the web for just about every type of activity and commerce transaction. It seemed at that time that anyone with a slick PowerPoint presentation and an outrageous business plan could get startup funding. The main problem was that the technology of the day just couldn’t deliver the user experience that all the web startup companies promised. Here we are, nearly a decade after the bust, and I believe we are just beginning to fulfill some of the wild web usage predictions and capitalization by users and consumers. That’s quite a gap between the vision that was promised and when it was actually delivered. No wonder there was such startup company roadkill during the bust.

Mobile computing, like any other surge in adoption of technology, will have winners and losers. However, there seems to be no end in sight for all things mobile. There is such innovation happening in this space. Take, for instance, the iPad. It is a game-changer, in my opinion. It has defined a whole new computing platform between what we consider a mobile handset and a laptop, and surely is adding to Apple’s vast bank vault. Smartphones continue to wow us with their capabilities and the ability to combine consumer-based technologies into such a convenient package, and we are quickly seeing how these devices are changing how we interact and communicate with each other. Mozilla encourages people to think big thoughts about the future of mobile computing. Just check out this video about the concept mobile phone called seabird and you will see what I mean. I sure would like the phone for Christmas.

So here’s the big difference I see between the dotcom boom/bust period a decade ago: In this mobile era we are leading with innovation and seeing how the marketplace utilizes the technologies. Where as, in the dotcom/bust era it was a lot of empty promises about technology and investor riches that ultimately were never delivered. The current stage in mobile and web computing in general is set for an explosive boom possibly much bigger and broader than than the dotcom period.  But this time I think it will be sustainable. Mobile computing and platforms are here to stay and will be the magnet for innovation for the foreseeable future. The current biggest hurdles are battery life and communication bandwidth. The latter is being addressed as we speak. 4G networks will have broadband connectivity. Hopefully battery life will also be improved shortly. A bright future ahead is my prediction.

Editor’s Note: Read Part II of the interview.