So, for the love of our software testers around the globe, this brief post aims to debunk a few major misconceptions the outside world has with respect to the craft of software testing.

First, a major misconception about software testers is that they’re merely “bug hunters” – they dig for bugs that developers must then fix. While this is true at face value, good software testers go beyond the identification of problems and focus on the more important prevention of problems. In other words, testers are in the business of helping developers become better at developing software with fewer bugs. As the saying goes, you cannot “test” quality into a product – you can only build it in. Good software testers focus on the source of the quality spigot. So instead of “build-fix-build-fix-build,” most testers aim to “prevent-build-fix-build-prevent.”

Another misconception about  testers is that they’re purely reactive to the inevitable surfacing of defects, which is absolutely false. In addition to aiding developer as mentioned above, good software testers also champion the culture of quality into the team, group, and organization. They do so by recommending processes that help streamline the development of high-quality software. They also work with various stakeholders, such as product managers and end users, to have a more holistic view of the software’s features and its intricate dependencies.

This is not an easy task as you often find conflicting views from various stakeholders, but testers begin with the end in mind and work their way backwards. This approach is very much proactive and, if implemented correctly, ultimately save the organization precious resources.

Finally, software testers are not one-trick ponies, but rather Jacks and Jills of all trades. They often wear multiple hats in an organization, including but not limited to the following: process modeler, investigator, fact gatherer, teacher, project manager, production system trouble-shooter, corporate efficiency consultant, lab tech, technology researcher, intra-team mediator, end-user liaison, and ultimately problem solver (inputs from John Kotzian – Gold uTester). That’s a lot to ask of a tester or even a group of testers. So the next time you bump into a software tester, be sure to express your appreciation for all they do.

For more information on what software testers do to affect change in the quality system, be sure to view our pre-recorded webinar with Google’s Director of Test James Whittaker (specifically around the 9 minute mark). And if you have additional misconceptions that you’ve heard of, feel free to drop a comment below.

uTest: For readers who aren’t familiar with your body of work, why don’t you start by telling us a little bit about your background before T-Mobile?
MC: I have been leading Quality Assurance (QA) and Testing teams since 1997. I started at the ground floor, as a testing intern with a small company, and I worked my way up to leading an award-winning multi-national team with several hundred team members. Prior to joining T-Mobile USA, I was the Senior Director of Quality Assurance at Fair Isaac (FICO) and held Director-level positions in Quality Assurance at Inovis and Equifax.

Through the years, I have been active in the QA and Testing communities; from 2005-2007, I served as the President of the Atlanta QA Association (AQAA). I have also had the opportunity to present at numerous conferences and lead several QA and Testing workshops.

uTest: And what’s your role today?
MC: Currently, I am responsible for all aspects of QA and Testing in the T-Mobile USA Enterprise IT organization. T-Mobile is a national provider of wireless voice, messaging, and data services with America’s largest 4G network. Reporting directly to the CIO of T-Mobile USA, I lead a team of more than 500 QA, Testing and Infrastructure professionals. Together, my team and I ensure the quality of the software, systems, devices and services, which support more than 32 million customers, 24 care centers, and 1900 retail stores. Some of the systems we test include: Web, eCommerce, Customer Care, Retail, Billing, Back Office, ERP, Supply Chain, Business Intelligence, Wholesale, Handset Application Testing, and much more.

uTest: What are your day-to-day responsibilities?
MC: Every day is different; however, my first priority is ensuring the team is consistently delivering high-quality software releases. I also focus on making sure the entire team is working well together to deliver on our Quality vision and direction by setting clear expectations, and providing them the tools and environment to be successful. We’ve created a real-time dashboard and reports, which allows my management team and me to stay on top of day-to-day and long term progress.

Each and every work day, there is a 30-minute conference call with QA and Test Environment Planning teams to review project intake, progress, and any other issues. During major releases, I personally send encouraging e-mails to the onshore and offshore teams, every day. I often serve as the Quality Advocate and Evangelist with business and technical teams, which include Project Steering Committees and GO/NO GO meetings. I am also responsible for budgeting, SOWs, approving invoices and other requests, such as, new hires, travel, and promotions. Some of my business travels have included exciting places like Seattle, Texas, and India, where my onshore and offshore teams are located.

uTest: What projects are you currently working on?
MC: T-Mobile IT delivers more than 400 projects annually, not including product launches or the daily web content and scheduled price plan changes that we test. In addition to delivering business projects, we are currently working on a significant IT Modernization program, application and platform upgrades, as well as, the migrating of two state-of-the-art data centers. This migration is a very interesting testing opportunity because we were able to test the latest technology and benefit from our test automation efforts. Our team prevented many functional, performance, high availability and connectivity issues from ever getting to production. At the same time, we are re-tooling our retail and customer care systems with a user-friendly, high-performance interface. Another recent project, which I am extremely proud of, is Wal-Mart Family Mobile powered by T-Mobile. I led the testing initiative for this project and had the opportunity of working directly with the Wal-Mart testing leadership in Bentonville, to ensure the project was a success.

uTest: You made the shift to the mobile world when it was entering its infancy. How have you managed to adapt to this new environment without the benefit of learning from other’s mistakes? In other words, what’s it been like working in such an immature field?
MC: Regardless of the industry, software testing is software testing; however, the pace of the telecom industry is moving extremely fast, and the speed to market is a key success factor. I enjoy getting knee-deep into projects to really understand the complexity mobile application testing, as well as the new tools and manual and automation techniques.

uTest: You’re a mobile guru… so what kind of phone (device maker, model and OS) are you packing?
MC: My primarily devices are both Androids – the HTC G2 4G phone and a Samsung Galaxy Tablet. T-Mobile USA was the first carrier to roll-out the Android operating system with Google and it was real exciting to be a part of this pioneering project. It’s amazing to think that just a few years later, there are now more Android devices then there is BlackBerry or iPhones.

uTest: It should be clear to all by now that mobile apps present truly unique testing challenges. But what advice do you have for companies who are new to the mobile apps game?
MC: Test on the device itself, as soon as possible; you can miss defects if you only test on an emulator. Automate as much a possible so that you can efficiently test with many devices, platforms, and configurations. Be sure to pay attention to the feedback of you customers.

uTest: What’s the best way for them to overcome the unique challenges of mobile app testing?
MC: I have found that mobile app testing requires a more technical type of testing, and automation is key. We are also looking into crowdsource testing to augment our current testing handset application testing efforts.

uTest: What are the biggest pitfalls they should try to avoid?
MC: Don’t forget about performance and security testing when working with a partner, or a vendor of mobile testing.

uTest: How do you measure the effectiveness and efficiency of your organization’s testing efforts? Are there particular KPI that you find useful (or a load of bunk)?
MC: Yes, metrics are very important in my organization. We have created a homegrown real-time dashboard that allows us to also manage in real-time. I find planned versus actual metrics extremely useful. It initially took almost a year for the team to become accurate at setting planned test execution dates. By tracking and using historical data, we are able to create predictive models that we use for estimations and release health measurements.

Continue reading Part II of the interview >>>

In the second and final installment of our Testing the Limits interview with James Whittaker, we get his thoughts on some recent changes to Google’s test philosophy; why certain principles cannot span all types of testing teams; mobile testing challenges; the value of software testing; subject matter for his upcoming book, How Google Tests Software; and much more.

If you missed part I of the interview, you can find it here. Also, be sure to scroll down to the end of the interview for links to more material from James Whittaker. Enjoy!

***********

uTest: You’ve been at Google now for over two years. Looking back, what’s been the biggest hurdle you’ve had to overcome during this time? And how much have the company’s testing procedures changed over this period?

James: My boss Pat Copeland took me aside a few weeks into my starting at Google and said something like “I know you’ve accomplished a bunch of things outside Google, but that’s all in the past. You’ve got to accomplish something inside Google. If you don’t no one will listen to you.” It was good advice. The message was that my past got me into Google but would get me no further. I took leadership and responsibility for testing of Chrome and Chrome OS, hard problems, important problems that required things I am good at and things I am not good at. That was the hardest part, being pushed outside my comfort zone. I never liked the execution part before, schedules and plans and meetings and disasters. It’s a lot easier as a consultant where you can happily imagine those things without experiencing them firsthand. I can’t believe that I used to advise people on these sorts of things before. I was surprised at how much I was learning and how much I was able to contribute. Now I take every opportunity to work outside my comfort zone. That’s where growth occurs.

As to the second part of this question, Google’s testing procedures have changed a lot. I think the Test Engineer role has been completely reinvented in the past two years.

uTest: We would imagine that a lot of testers and managers at smaller companies that will view your book as interesting, but not necessarily relevant to their daily testing lives. Explain why this is not the case and talk a little about the challenges of writing for an audience that includes teams of all sizes.

James: But it is the case! You can say the same things about my other books too. My books are meant to make you think differently about testing. It’s up to the reader to make it relevant by putting it into practice. There is no way I can write a book relevant to any specific style of testing or the practices of any specific company (except Google of course). All I can do is offer information and ideas and deliver them in, hopefully, an entertaining way. The problem is that too many people work for companies who just want them to keep doing the same old stuff. Change is too hard for them. Even if they wanted to test the way my books suggest, they can’t. I feel sorry for those people. In a better economy I would tell them to get a new employer. In this economy, well, it’s tougher. But there are also a lot of people who do own their destiny and can make changes in the way their company does testing and treats testers.

And partly, I don’t care about the effect on daily work. My books are a celebration of testing and I want testers to read them to learn and to enjoy and to appreciate that many others are involved in the same activity they are. I hope my books tie testers together and that some of the enthusiasm I have for testing will rub off and there will be more passionate people involved in this discipline.

uTest: Being at Google must have given you great insight into the unique testing challenges of the mobile world. How has your view of mobile testing changed over the course of the last two years?

James: I didn’t do any mobile testing before so I can’t say. But, dude, a device is a device. The mobile labs are cooler. The tools aren’t as highly advanced. There is a little more emphasis on manual testing (I like that). But software is software at the beginning, middle and end of the day.

uTest: It’s common for certain departments of a company to view software testing and QA as a cost-center. This doesn’t seem to be the case at Google. How can testers and test teams do a better job at showing their value to a company, like your team does at Google?

James: Don’t bother trying. Value isn’t something you can make up. It is binary. You are either valuable or you are not. Be the former and you will get recognized.

uTest: We know your book is still a work-in-progress, but can you give our readers a sneak-peak at some of the products that will be mentioned? Will you be discussing specifics for things like Chrome, Google Voice and others? Or will it be more general and philosophic in nature?

James: Actually it is “our” book as I have two coauthors Jason Arbon and Jeff Carollo. But the answer to your question is all of the above. Right now the plan is to feature all the majors like Android, Gmail, Chrome and so forth and also talk about the roles of the SWE, SET and TE plus the tools we use. Lots of specifics. Probably the most detailed and specialized book I have ever written.

Rapid Fire:

• Favorite (non-testing) blog: siliconslacker.blogspot.com. Careful, not PG 13 and very funny. But then again I am a big fan of satire.
• Last show you DVR’d: HBO’s A Game of Thrones
• Which celebrity plays James Whittaker when the book (inevitably) becomes a movie? The comedian Ron White. Southern accent, always has a drink in his hand.
• More likely audio book narrator: Morgan Freeman or Casey Kasem? Freeman, love that guy.
• You wouldn’t be caught dead where? Every answer I could give to this would offend someone. Pass.
• Favorite mobile app: Glympse: “share your where”

Editor’s note: We hope you enjoyed our latest interview with Google’s Test Director James Whittaker. We’ll be giving away some free copies of his upcoming book when it’s published. Be sure to check back in regularly to find out how to win yourself a free, signed copy!

In the meantime, here are a few links “from the vault” of James Whittaker:

Webinars:

• More Bang For Your Testing Buck
• Exploratory Software Testing
• 5 Ways To Revolutionize Your QA
• The Future of Software Testing
  

Interviews:

• Testing the Limits – Part I (2009)
• Testing the Limits – Part II (2009)
• Testing the Limits – Part I (2010)
• Testing the Limits – Part II (2010)
  

Blog Posts

• How Google Tests Software – Part one of six
• The SET Career Path
• This Code is Crap

Books

• Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design
• How to Break Software: A Practical Guide to Testing W/CD
• How to Break Web Software: Functional and Security Testing of Web Applications and Web Services. Book & CD
• How to Break Software Security

Two years ago, we got this crazy idea to start interviewing the giants of software testing, for a series now known as Testing the Limits. To kick things off, we shot some questions back and forth with the distinguished testing author/teacher/speaker/consultant…(deep breath)…. James Whittaker!

A frequent guest of the uTest blog – as well as the author and host of several eBooks and webinars – James is known throughout the industry as the Test Director for a little company called Google. If you’ve stopped by the Google Testing Blog at all in the last year or two, chances are you’ve read his posts.

Anyway, we’re extremely excited to have James back for his third exclusive interview – and we’ve got a lot of ground to cover. In the first part of our Q&A, we discuss some recent changes at Google; a possible book deal; the future of cloud computing; testing Chrome OS; the problem with test automation; the upcoming GTAC event and why testing is (believe it or not) getting easier. For the second half of the interview, be sure to check back tomorrow.

**********

uTest: Good to have you back, James. Tell us what you’ve been up to? Anything different in your life at Google?

James: Funny you should lead with that! Yes, my role at Google is changing. I’ve passed the leadership of Chrome and Chrome OS testing to a colleague and taken over Cloud Computing. Lots of amazingly complicated back end/data center testing issues which are new and exciting. Google keeps pushing me outside my comfort zone in the most endearing way. Retirement in place hasn’t really been an option. But hey, one can dream.

uTest: We’ve noticed you’re blogging a lot more of late and you’ve started a Twitter account! How do you manage to find time for these things with the new challenges you’re faced with?

James: Believe it or not, the Cloud stuff is actually smaller than the Chrome/Chrome OS work in terms of the amount of time out of my day. The level of automation is very high and there is a lot we’ve been able to do put some very hard sub-problems on autopilot. My team here is seriously Jedi quality folk. Definitely good fodder for some future talk. But the Google-wide testing efforts in terms of security testing, testing tools, development tools and general evangelism is also officially part of my day job. I own the Google Testing blog and am also running GTAC this year. And yes, I finally succumbed to Twitter. It’s actually a lot of fun so far, 140 characters is so do-able.

uTest: Speaking of the blog, I noticed Pat Copeland mentioned the popularity of the current “How Google Tests Software” series. Any plans on turning that into a book?

James: More than plans, I am under contract with Addison Wesley and am working diligently on it. I am finding that as I dig deeper into Google internal testing processes I have to be more careful about what I publish. I have to avoid talking about confidential technology and innovations created by other Googlers who aren’t ready to discuss their work externally. So there is a backlog of stuff that requires review and once approval occurs, it makes sense to publish it all at once. The book format is ideal for that as opposed to trickling it out in blog form. Besides I love books, they are so cuddly and blogs are so not cuddly. And I also have two co-authors Jeff Carollo and Jason Arbon who I am mentoring through the writing process and that is bringing me a lot of satisfaction.

uTest: You made a comment in one of your blog posts about “automating to within the last inch of the human mind.” Can you tell us what you mean by that.

James: Hey, you’re stealing blog material from me! I was planning on describing this on the Google Testing blog as it did get a number of comments from curious readers. The central idea is that I think that test automation is often built to solve too big a problem. This broad scope makes automation brittle and flaky because it’s trying to do too much. There are certain things that automation is good at and certain things humans are good at and it seems to me a hybrid approach is better. What I want is automation that makes my job as a human easier. Automation is good at analyzing data and noticing patterns. It is not good at determining relevance and making judgment calls. Fortunately humans excel at judgment. So when I am doing exploratory testing, I want automation following me and helping. I can watch the UI (which it can’t) and it can watch the instructions hit the CPU (which I can’t). I can identify usability issues and the automation can trap exceptions and notice system issues. It’s cooperative and symbiotic. We have tools for this and are actively designing more tools where the automation gathers what it can and then presents it to the human for a decision…to within the last inch of the human mind. Get it?

uTest: Absolutely. I am sure the uTest crowd will like the message that the human mind is the most important part of the equation and that automation is here to help!

James: You bet. The human mind is key and uTest provides minds that scale. I may be trying to drive a lot of hard testing problems to extinction, but intelligence at scale is not one of them.

uTest: The theme of last year’s GTAC was “Test to Testability” – where you highlighted several methodologies and tools that could be used to build testability into products. What is the theme of this year’s GTAC conference and what are the implications for testers and testing managers?

James: “Cloudy with a Chance of Tests” It’s Cloud Computing with a twist. At Google we believe the cloud fundamentally changes not only the way testing tools and techniques are applied but also the very nature of testing itself. If this is too cloudy of an answer, stay tuned!

uTest: The world of testing moves incredibly fast at times. That said, is it possible to “future-proof” the theories and methods you write about in your books? Talk about some of the challenges of trying to stay current on the latest testing trends.

James: That’s easy. You stay current by inventing the latest trends not following them. Invention is less about being able to read the future and more about a willingness to experiment and be wrong. You should see our drawing room floor at Google, it’s a graveyard of failed experiments. We purposefully try to do things differently to make sure we’re not missing some great new idea or technique. We’ll try almost anything once and we push hard to fail fast. Staying open minded is the key. If you are willing to experiment and expect everything you try to fail, when something actually works, well, you know its something special.

uTest: Big picture question: With the emergence of the cloud, automated tools and other technologies….is software testing getting easier? Or is it a case of the “more we learn, the less we know”?

James: It is getting easier, way easier. There are so many options now. Computing power that makes testing millions of configurations a matter of a couple hours instead of completely impossible. Crowdsourcing that alleviates the need for expensive hardware acquisition and maintenance and hard to manage off-shoring. Developers who get quality are in larger number now and they write fewer bugs. Newer technology is making old and stupid test ideas and tools obsolete. Gotta love that. The web as a single platform increases test reuse. Dev to test ratios are coming in line with reality and sensibility. Anyone who says testing is getting harder is doing it wrong.

Editor’s note: Check back tomorrow for the second half of our interview with James Whittaker.

Despite the perception that Google’s testing is highly complex and indecipherable to us mere mortals, we find the opposite to be true. As Whittaker explains, testing scope is determined by “emphasizing  scope over form.” Plainly stated, testing comes in three sizes at Google: small, medium and large. Here’s his explanation:

Small Tests are mostly (but not always) automated and exercise the code within a single function or module. They are most likely written by a SWE or an SET and may require mocks and faked environments to run but TEs often pick these tests up when they are trying to diagnose a particular failure. For small tests the focus is on typical functional issues such as data corruption, error conditions and off by one errors. The question a small test attempts to answer is does this code do what it is supposed to do?

Medium Tests can be automated or manual and involve two or more features and specifically cover the interaction between those features. I’ve heard any number of SETs describe this as “testing a function and its nearest neighbors.” SETs drive the development of these tests early in the product cycle as individual features are completed and SWEs are heavily involved in writing, debugging and maintaining the actual tests. If a test fails or breaks, the developer takes care of it autonomously. Later in the development cycle TEs may perform medium tests either manually (in the event the test is difficult or prohibitively expensive to automate) or with automation. The question a medium test attempts to answer is does a set of near neighbor functions interoperate with each other the way they are supposed to?

Large Tests cover three or more (usually more) features and represent real user scenarios to the extent possible. There is some concern with overall integration of the features but large tests tend to be more results driven, i.e., did the software do what the user expects? All three roles are involved in writing large tests and everything from automation to exploratory testing can be the vehicle to accomplish accomplish it. The question a large test attempts to answer is does the product operate the way a user would expect?

Simple enough, yes? So if you’re in charge of testing at a growing company – and wish to follow in the footsteps of giants – you would be wise to read the entire series, starting here.

As a supplement to this series, here’s Whittaker’s last uTest appearance – a webinar titled “More Bang For Your Testing Buck” (after the jump). Enjoy!

While there are technically no wrong answers to these questions, some hold more weight than others. For instance, you’d be much better off listening to advice from a company like Google, as opposed to Uncle Joe’s Discount Software Shack.

Lucky for you, Google Test Director James Whittaker tackles this subject in part II of his “How Google Tests Software” series. If you’re new  to the uTest Blog, James Whittaker is the distinguished author of several uTest blog posts, eBooks and webinars. You would be wise to listen to his advice.

Anyway, here are a few short excerpts from his latest article, addressing the various roles of those in the development process:

The SWE or Software Engineer is the traditional developer role. SWEs write functional code that ships to users. They create design documentation, design data structures and overall architecture and spend the vast majority of their time writing and reviewing code. SWEs write a lot of test code including test driven design, unit tests and, as we explain in future posts, participate in the construction of small, medium and large tests. SWEs own quality for everything they touch whether they wrote it, fixed it or modified it.

The SET or Software Engineer in Test is also a developer role except their focus is on testability. They review designs and look closely at code quality and risk. They refactor code to make it more testable. SETs write unit testing frameworks and automation. They are a partner in the SWE code base but are more concerned with increasing quality and test coverage than adding new features or increasing performance.

The TE or Test Engineer is the exact reverse of the SET. It is a a role that puts testing first and development second. Many Google TEs spend a good deal of their time writing code in the form of automation scripts and code that drives usage scenarios and even mimics a user. They also organize the testing work of SWEs and SETs, interpret test results and drive test execution, particular in the late stages of a project as the push toward release intensifies. TEs are product experts, quality advisers and analyzers of risk.

What role do YOU think these positions should play in the testing process? Alternative ideas are encouraged.

Not that we’re encouraging you to leave the testing profession, but if you absolutely had to, here are a few options for you to consider:

1. Software developer / engineer: Aside from werewolf hunter, this is probably the most obvious career alternative, as great testers will eventually acquire the skills and understanding needed to succeed as a developer. But as a blog reader recently brought to my attention, this works both ways. He said that at his former place of employment, developers aspire to be testers, NOT the other way around. He writes, “ the Tester/QA path is the destination/pinnacle of the career path in SW development. You start out as a Jr. Programmer then… Sr. Programmer then… eventually Architect/System Designer…then…you eventually make it to Testing. Their thinking was that you can’t adequately test until you have proper understanding of the development process. In other words, you are truly considered an expert by the time you get to that level.”

2. Detective: Much like a detective, a tester’s bug-hunting prowess will depend largely on intuition – i.e. knowing the right questions to ask and having a sixth sense for odd irregularities. Testers are already possess the other traits found in successful detectives, including sound logic, analytical skills and patience. The only thing they are missing is an assistant named Watson and a trench coat. Both are available on Craigslist.

3. Journalist: There’s a very thin line between a tester and an investigative reporter. Like their journalistic counterparts, QA professionals must ask tough questions, dive deep into complex issues and report them to the layman in a clear, concise and objective manner. The hours stink and the pay is terrible, but there is some downside to the job however.

4. Actuary: As an actuary, it’s your responsibility to prepare and protect your client from risk in the financial world. Replace “financial” with “software” and you’ve got yourself a tester. Like QA pros, actuaries evaluate and quantify possible outcomes in order to minimize losses associated with uncertain undesirable events (like showstopper bugs). In this profession, you’ll need analytical skills, an understanding of human behavior and of the vagaries of information systems. Does that sound like it’ll be a problem?

5. Demolitionist: Sometimes, software testing really is about “breaking things.” So if testers can thrive while breaking web, desktop and mobile apps, it’s not much of a stretch to see them enjoy wrecking houses, office buildings and stadiums. And like real demolionists, testers must wield the virtual wrecking ball carefully, so as to not damage the surrounding structures.

6. Professor: As evidenced by the thousands of blog posts, forums chats, webinars and other mediums, testers have shown an amazing willingness to pass on what they have learned. A few notable testers who have already served in this role of professor (or course instructor) include James Whittaker, Cem Kaner, James Bach, Michael Bolton, Gerry Weinberg and dozens of others.

7. Architect: The same traits that would enable testers to thrive in demolishing buildings would also help in constructing them – that being knowledge of safety standards, usability and design, functionality, etc. As they say, “If you can build it, you can break it.”

8. Psychologist: Testers are often masters at understanding the mindset of their users. Not only do they know how the typical user will navigate a website; perform certain actions;  respond to error screens, etc. – they know why they are likely to do it. So if you ever become bored with the mind of the everyday software user, perhaps you could try your hand at the mid-life crisis guy or those suffering from Apeirophobia (fear of infinity). For this job, you’re going to need a more comfortable couch.

Have you recently made the jump from testing to another profession, or vice-versa? Please share your experience in the comment section.

Last week, more than three hundred software professionals from around the world tuned in to “More Bang For Your Testing Buck” – an exclusive webinar with best-selling author and Google Test Director James Whittaker. For those in attendance, we’re happy to post the following reprise of the Q&A session, where James addresses ACC requirements; the levels where testers are needed most; test tours; unit testing and more. See his answers below.

For those that missed the event, we have posted the webinar in its entirety (after the jump). If you’re interested in similar presentations, be sure to check out our webinar archives.

When is the right time to start with ACC in a development project? You just told us that ACC are not requirements. So ACC cannot be defined before requirements are clear? May ACC can even be defined just with the requirements?

Whittaker: Early in a project when there is a lot of flux, features added and cut/unreliable code being written, there isn’t much use in putting a lot of work into any sort of artifact and the ACC is no different. Once the features are locked, this is the time to begin in earnest. Even for large complicated products a good ACC can be built in hours, not days, so it’s unlikely that you’ll get too far behind. What we’ve found helpful is having a single test engineer maintain the ACC early in a product cycle and then when features are no longer in so much flux, that’s when additional testers can be added. Too many testers too early is just not productive. Developers know their code is buggy and it’s pretty unhelpful to keep reminding them of it.

The ACC and the requirements are very similar. In fact, I think the former could be a viable substitute.

It sounds like you think testers are only useful at the level of the UI?  Or did I misunderstand?  I currently work as a tester on an architectural team and code tests to test services and APIs.  Are you saying that is not a useful testing task, or that it should be owned by devs?

Whittaker: I want to make the distinction been early cycle testing, which are things like unit testing and test infrastructure, and late cycle testing which is more end-to-end. I think all late cycle testing should be done by testers and all early cycle testing should be owned by devs. Even within early cycle testing there are devs who specialize in writing such test code. At Google they are called Software Engineers in Test. They are engineers first and testers mostly in the sense that they write test code and not shipping features.

Now late cycle testing or end-to-end testing (if you like that phrase better) can be either UI or API level. At the UI we are stringing together inputs that make up user scenarios. At the API level we are writing code that utilizes the APIs with similar purpose: to test that the APIs can be used in real user scenarios.

The bottom line for me is that if you are thinking about the user, you are a tester. If you are thinking about the code, you are a developer.

Is there added risk that the useful highlighting of problem areas through the  “bad neighborhood tours” prevents someone looking for other problems with the highlighted area?

Whittaker: Only if you don’t also use other tours. The tours are meant to cover functionality as a whole and balance each other out. Each capability will be involved in any number of actual tours and if the right collection of tours is chosen, you should get good coverage. In fact, that’s kind of the whole point: tours are generalizations of test cases. They are a way to package up tests that are related in form, purpose or substance so they can be taken as a whole. So you might make a test pass that, hypothetically, looks like this:

• Bad Neighborhood Tour: 18 tests total
• Landmark Tour: 45 tests total
• Rained-Out Tour: 3 tests total
• FedEx Tour: 5 tests total
• Money Tour: 31 tests total

And so forth.

Should we be writing the unit test or just advising on their development?

Whittaker: What advice are you going to offer? This is a good question and I am being serious here. Unless you know the code base and how it was constructed can you really help here? Unit tests are code that test low level boundary conditions and test that specific control structures are right. If you aren’t familiar with the code, you’ll not be much help. That’s the devs domain and his/her job to get it right. Being a crutch for developers weakens their resolve to do proper unit testing, don’t give them the ability to blame their poor unit tests on you.

If a dev is consistently shipping you code that falls down easily then it is clear that they are not doing good unit testing. And you are wasting your time if you are testing such rubbish. Now perhaps you can advise them: write better unit tests so your code is harder for me to break. Unit testing should remove most easy bugs. If you aren’t working hard to find more subtle problems, the devs are not doing their jobs.

At a company that is entrenched in the notion of test plans and doesn’t have the tools that you demonstrated for seamless bug reporting, what are some strategies for making the most of our testing hours?

Whittaker: Be more organized and thoughtful. In fact, it was the result of being more organized and thoughtful that led us to ACC and it’s accompanying toolset. We purposefully questioned every testing activity we were doing and played devils advocate: Are our test plans really being used? What actual value are they providing? Which parts are simply not worth doing? How long are we spending reporting and triaging bugs vs actual testing? Why are our test cases so hard to reuse? Why are tests hard to find? Where do new testers often get confused? And so on. Ultimately we started throwing out things like test plan as documents and focusing more on test planning as an activity. The value was in the activity not in the artifact.

What we ended up with is a process that maximizes the time we spend doing productive work and minimized the time we spent creating throw-away docs. You can do this too. You may end up with something that looks like our process (in which case you can adopt our tools) or you may end up with something that fits your team and its culture. Either way, you end up working smarter.

Editor’s Note: And as promised, here is the full-length version of the webinar.

In part II of our Testing the Limits interview with Mozilla QA Director Matt Evans, we get his thoughts on mobile immaturity; the worst bug ever submitted by a Mozilla community member; the so-called “skills shortage” in Silicon Valley; skepticism for all things open-source; the next great browser innovation and more.

If you missed Part I, do yourself a favor and catch up here.

*******

uTest: In many ways, mobile is still playing catch up to the web. Is there one area in particular where you see the most room for improvement? If so, where?

ME: Well, there are some obvious platform deficiencies around inconsistent UI and whether Flash is going to be fully supported across mobile devices or not. But this is a testing blog, so let’s talk about that. As I mention elsewhere in this interview, mobile is just a really tough testing challenge. The big problem is that there is very little support for cross-platform mobile device test automation. I suspect most of mobile device and application testing is done 100% manually. If any environment needed more test automation, it is mobile. At Palm, we rolled our own test harness that ran on the Pre. This became extremely important for endurance testing and finding memory leaks in the Pre applications.

Mobile software companies have an uphill battle since developing automated system tests for every platform is very costly, both in time and resources. However, reliance on mostly manual testing has lots of quality risks. If the quality of mobile devices and software is to rise about what it is now, we need automated test tool support that works well across all device platforms.

uTest: What’s the best (and by that, we mean the worst) bug ever submitted by one of your community members?

ME: Recently, Alex Miller, a Mozilla community member, discovered a very critical security bug and was awarded $3000 for finding and reporting the bug. He’s been hard at work finding and discovering other security flaws in Firefox, too, and was even given clearance access to all Mozilla security-related bugs reported in Bugzilla. Very few people have this access.  Oh, I forgot to add a little fact about Alex: he’s only 12 years old. That’s an awesome accomplishment by a really smart kid. This exemplifies the opportunity Mozilla provides to the community: an incredible technology playground where anyone that spends the time to learn can participate at any level no matter who you are or what your background is. The more you prove what you can do, the more you will be encouraged and acknowledged for that effort. Finding bugs is a good place to start for anyone who wants to participate. Certainly, not everyone is going to develop the expertise to discover deep level security bugs, but believe me there is plenty of testing folks can really help us out with. If you are so inclined, we will welcome you with open arms. Please visit us here.

uTest: We keep reading about the skills shortage in Silicon Valley. Are you seeing this at all, particularly when it comes to software testers? If so, what do you suspect is the reason?  And how do you overcome this dearth of top-shelf talent?

ME: It’s always been hard to find good software testers. Good or bad economy, a seasoned and talented software tester is rarely unemployed. Typically, many good testers wind up in development because it often appears to be the growth path for a software technology career. Software testing as a career path is still viewed as something lesser than development. However, there has been a lot of gained respect for the software testing career path in recent years. The industry and practice has definitely matured since I started as a QA engineer when breaking software meant dumping your card deck all over the floor. So, I think the pool of quality testing folks is better now than ever, but it is still very hard and time consuming to finally hire someone. How do you fix this? We are always in search mode. We never stop looking for good QA engineers. Also having a team of great recruiters that can poach well is very key as well.

uTest: Which blogs/sites/magazines/conferences do you frequent to stay on top of the latest trends and tools in testing?

ME: I haven’t been to many conferences in the last ten years, except of course the recent GTAC conference in Hyderabad. Mostly, this is because I’ve been involved in startups which didn’t have the budget for sending folks to conferences, and I couldn’t afford the time as well. I have followed James Bach’s blog over the years; he’s as much an entertainer as he is a software tester. I also read a lot from Michael Bulton, mostly because I think he is the E. F. Hutton of the testing world. He is definitely worth listening to. Last year, I took some time off between jobs. It was a great time to have the luxury to really do a lot of web-surfing and getting caught up on things. I became rather fascinated with Yvette Francino. She’s a wonderful writer/blogger, and she chronicled her story of getting laid off as software test manager. It was quite the feel-good story to see how her use of blogging and online networking landed her a job as editor of SearchQuality.com, a site I still frequent today. Lastly, it was great to finally meet James Whittaker in person at GTAC this year. He’s quite a speaker, and I’ve learned a lot from his posts and books. I can’t wait to see how he will avenge being called the “octomom of testing books” at the GTAC conference .

uTest: What could software companies (eg: Palm/HP) learn from the open source movement about launching great products?  And vice versa?

ME: Traditional software companies are skeptical of open source tools and projects. They generally think that they have lesser quality that commercial counterparts. That may have been true some years ago, but I don’t think it is true today. Open source projects have one great advantage over commercially developed ones. The rate of updates and fixes is typically is much higher than commercial projects. I’m mostly talking about the bigger projects such as the popular linux distributions, apache tools and even Mozilla. We are unparalleled in our transparency and response to reported security flaws for instance. In fact being more transparent across the development cycle is a competitive advantage to open source. It encourages user feedback and therefore course corrections can be done incrementally and continuously and ultimately deliver a product a user is expecting. I think that software companies should be more transparent and allow their customers to participate more in the development process. What can open source projects learn from commercial companies? If there is anything it would probably be discipline around scope of the project. Too often, I see that open source projects get feature bloat or accumulate features that are cool to build but are not really needed by the user. Since, most open source developers are working on the projects for the pure enjoyment of it, they are often attracted to working on new stuff and are less interested in improving or fixing regression bugs.

uTest: Put on your prognosticator’s hat – what’s the next big “wow” feature or innovation in the world of browsers?

ME: I think Aza Raskin of Mozilla Labs fame has it right. Identity and browser identity awareness are the next big wow browser features in my opinion. Let me try to explain, although I’m sure Aza does a much better job of it. Your browser knows or could know all about you: your browsing habits, where you browse, what you buy, where you login to, who you communicate with, etc. etc. Currently, much if not all of this data is being captured on the web in many, many places. I won’t go into any of the arguments about whether your captured browsing habits and history are truly anonymous data or whether this data is being used only for commercial benevolence. But one thing is clear: you don’t own this data, and you can’t access or alter it either. And for many, that is a real problem. However, this data can be very useful to you. It is mostly used for a better experience on the web–or at least a merchant’s version of what it thinks will be a better experience for you. Well, your browser could do a much better job of it. It could capture this data directly and it would only be about you. This data could be stored locally, or even in the cloud using services such as Firefox Sync. It could do many cool things like real targeted pre-fetching of your favorite blogs, and then scan them for keywords that are important to you and arrange them in priority order. It could sense what tabs are used more often and arrange for a better workflow experience. I think there is a lot of assistance a browser could do either in the context of the browser interaction, e.g., automate tasks or perform operations based on the web content. Well, there is one huge obstacle, and that is security. How do we ensure that this personal data that absolutely defines you is kept safe and secure? And how do we prove it to you so that you feel safe in allowing a browser to continue capturing this data and making operational decisions based on this data? That may take awhile, but little by little we are making inroads towards this goal such as the Firefox Sync feature that securely stores your profile data in the cloud across instances of Firefox running on desktops and mobile devices. We are getting there, but I suspect it will be a while before you let your browser do your online banking for you.

uTest: What’s the greatest challenge that tech execs will face around app quality in 2011?

ME: The biggest challenge I see is around compatibility testing and maintaining a consistent user experience across the landscape of platforms. To be competitive in today’s web-enabled software market, you need to support a variety of platform interfaces. It is now expected that any web-accessible service will include feature-rich clients for iPhones, iPads, tablets, smartphones, desktops, and kiosks. The compatibility matrix starts to explode when you add in device models, OS versions, and other configuration-dependent attributes. In my experience, you do need to test on real hardware and configurations. Emulation capabilities do help, but at the end of the day you can not be sure of software behavior unless you test it on live hardware. Its a real problem that doesn’t seem to get better with the plethora of new platforms available to users.

Rapid Fire with Matt Evans:

• Favorite band or musician: Jason Mraz
• Last movie seen in theaters: Social Network
• Favorite sport and/or team: The Oregon Ducks football team
• What keeps you alert during those late night launches (eg: coffee, Red Bull, Mountain Dew, et al): Diet Monster
• Top restaurant within driving distance of Mozilla’s Mountain View offices: Cascals (walking distance from Mozilla Offices) We never drive to lunch!
• Fact about you that would surprise our testing community: I’m an avid NASCAR fan. Kevin Harvick is my favorite driver.
• Lucky number (or, failing that, favorite flavor of gum): 29
• If they had opposable thumbs, would cats or dogs make better testers: Dogs. They like to break things more the cats do.

In our latest uTest webinar, More Bang For Your Testing Buck, best-selling author James Whittaker will take a critical look at how to maximize your investments in testing, including an examination of the tools, best practices and methodologies that can help make testing a “happier place.”

The webinar is schedule for Friday, December 10 at 11:00 AM (EST).

Register now at Go-To Meeting >>>>

As the former Test Architect at Microsoft and the current Director of Testing at Google, James is one of the most accomplished professionals in the testing business. Long-time readers of our blog will know that he is also no stranger to the uTest community. Here are a few of the uTest projects he’s participated in:

Webinars
The Future of Software Testing

5 Ways To Revolutionize Your QA

Exploratory Software Testing

eBooks
5 Ways To Revolutionize Your QA

Interviews
Testing the Limits – Part I (2009)

Testing the Limits – Part II (2009)

Testing the Limits – Part I (2010)

Testing the Limits – Part II (2010)

Hope to see you at the next week’s webinar!