Software Testing Blog

No one likes to be the bearer of bad news. Yet this is often what software testers must do; their job depends on the inevitable surfacing of defects. So it’s no wonder that more often than not they are under-appreciated – blamed when critical bugs are found post-launch and taken for granted when they are not.

So, for the love of our software testers around the globe, this brief post aims to debunk a few major misconceptions the outside world has with respect to the craft of software testing.

First, a major misconception about software testers is that they’re merely “bug hunters” – they dig for bugs that developers must then fix. While this is true at face value, good software testers go beyond the identification of problems and focus on the more important prevention of problems. In other words, testers are in the business of helping developers become better at developing software with fewer bugs. As the saying goes, you cannot “test” quality into a product – you can only build it in. Good software testers focus on the source of the quality spigot. So instead of “build-fix-build-fix-build,” most testers aim to “prevent-build-fix-build-prevent.”

Another misconception about  testers is that they’re purely reactive to the inevitable surfacing of defects, which is absolutely false. In addition to aiding developer as mentioned above, good software testers also champion the culture of quality into the team, group, and organization. They do so by recommending processes that help streamline the development of high-quality software. They also work with various stakeholders, such as product managers and end users, to have a more holistic view of the software’s features and its intricate dependencies.

Read more…

For this month’s Testing the Limits, we shoot some questions back and forth with Michael Cooper, the Senior Director of Enterprise QA for T-Mobile. Subjects include T-Mobile’s latest testing projects; the unique challenges of mobile app testing; the importance of metrics and more. Enjoy!

uTest: For readers who aren’t familiar with your body of work, why don’t you start by telling us a little bit about your background before T-Mobile?
MC: I have been leading Quality Assurance (QA) and Testing teams since 1997. I started at the ground floor, as a testing intern with a small company, and I worked my way up to leading an award-winning multi-national team with several hundred team members. Prior to joining T-Mobile USA, I was the Senior Director of Quality Assurance at Fair Isaac (FICO) and held Director-level positions in Quality Assurance at Inovis and Equifax.

Through the years, I have been active in the QA and Testing communities; from 2005-2007, I served as the President of the Atlanta QA Association (AQAA). I have also had the opportunity to present at numerous conferences and lead several QA and Testing workshops.

uTest: And what’s your role today?
MC: Currently, I am responsible for all aspects of QA and Testing in the T-Mobile USA Enterprise IT organization. T-Mobile is a national provider of wireless voice, messaging, and data services with America’s largest 4G network. Reporting directly to the CIO of T-Mobile USA, I lead a team of more than 500 QA, Testing and Infrastructure professionals. Together, my team and I ensure the quality of the software, systems, devices and services, which support more than 32 million customers, 24 care centers, and 1900 retail stores. Some of the systems we test include: Web, eCommerce, Customer Care, Retail, Billing, Back Office, ERP, Supply Chain, Business Intelligence, Wholesale, Handset Application Testing, and much more.

uTest: What are your day-to-day responsibilities?
MC: Every day is different; however, my first priority is ensuring the team is consistently delivering high-quality software releases. I also focus on making sure the entire team is working well together to deliver on our Quality vision and direction by setting clear expectations, and providing them the tools and environment to be successful. We’ve created a real-time dashboard and reports, which allows my management team and me to stay on top of day-to-day and long term progress.

Each and every work day, there is a 30-minute conference call with QA and Test Environment Planning teams to review project intake, progress, and any other issues. During major releases, I personally send encouraging e-mails to the onshore and offshore teams, every day. I often serve as the Quality Advocate and Evangelist with business and technical teams, which include Project Steering Committees and GO/NO GO meetings. I am also responsible for budgeting, SOWs, approving invoices and other requests, such as, new hires, travel, and promotions. Some of my business travels have included exciting places like Seattle, Texas, and India, where my onshore and offshore teams are located.

uTest: What projects are you currently working on?
MC: T-Mobile IT delivers more than 400 projects annually, not including product launches or the daily web content and scheduled price plan changes that we test. In addition to delivering business projects, we are currently working on a significant IT Modernization program, application and platform upgrades, as well as, the migrating of two state-of-the-art data centers. This migration is a very interesting testing opportunity because we were able to test the latest technology and benefit from our test automation efforts. Our team prevented many functional, performance, high availability and connectivity issues from ever getting to production. At the same time, we are re-tooling our retail and customer care systems with a user-friendly, high-performance interface. Another recent project, which I am extremely proud of, is Wal-Mart Family Mobile powered by T-Mobile. I led the testing initiative for this project and had the opportunity of working directly with the Wal-Mart testing leadership in Bentonville, to ensure the project was a success.

Read more…

In the second and final installment of our Testing the Limits interview with James Whittaker, we get his thoughts on some recent changes to Google’s test philosophy; why certain principles cannot span all types of testing teams; mobile testing challenges; the value of software testing; subject matter for his upcoming book, How Google Tests Software; and much more.

If you missed part I of the interview, you can find it here. Also, be sure to scroll down to the end of the interview for links to more material from James Whittaker. Enjoy!

***********

uTest: You’ve been at Google now for over two years. Looking back, what’s been the biggest hurdle you’ve had to overcome during this time? And how much have the company’s testing procedures changed over this period?

James: My boss Pat Copeland took me aside a few weeks into my starting at Google and said something like “I know you’ve accomplished a bunch of things outside Google, but that’s all in the past. You’ve got to accomplish something inside Google. If you don’t no one will listen to you.” It was good advice. The message was that my past got me into Google but would get me no further. I took leadership and responsibility for testing of Chrome and Chrome OS, hard problems, important problems that required things I am good at and things I am not good at. That was the hardest part, being pushed outside my comfort zone. I never liked the execution part before, schedules and plans and meetings and disasters. It’s a lot easier as a consultant where you can happily imagine those things without experiencing them firsthand. I can’t believe that I used to advise people on these sorts of things before. I was surprised at how much I was learning and how much I was able to contribute. Now I take every opportunity to work outside my comfort zone. That’s where growth occurs.

As to the second part of this question, Google’s testing procedures have changed a lot. I think the Test Engineer role has been completely reinvented in the past two years.

uTest: We would imagine that a lot of testers and managers at smaller companies that will view your book as interesting, but not necessarily relevant to their daily testing lives. Explain why this is not the case and talk a little about the challenges of writing for an audience that includes teams of all sizes.

James: But it is the case! You can say the same things about my other books too. My books are meant to make you think differently about testing. It’s up to the reader to make it relevant by putting it into practice. There is no way I can write a book relevant to any specific style of testing or the practices of any specific company (except Google of course). All I can do is offer information and ideas and deliver them in, hopefully, an entertaining way. The problem is that too many people work for companies who just want them to keep doing the same old stuff. Change is too hard for them. Even if they wanted to test the way my books suggest, they can’t. I feel sorry for those people. In a better economy I would tell them to get a new employer. In this economy, well, it’s tougher. But there are also a lot of people who do own their destiny and can make changes in the way their company does testing and treats testers.

Read more…

Two years ago, we got this crazy idea to start interviewing the giants of software testing, for a series now known as Testing the Limits. To kick things off, we shot some questions back and forth with the distinguished testing author/teacher/speaker/consultant…(deep breath)…. James Whittaker!

A frequent guest of the uTest blog – as well as the author and host of several eBooks and webinars – James is known throughout the industry as the Test Director for a little company called Google. If you’ve stopped by the Google Testing Blog at all in the last year or two, chances are you’ve read his posts.

Anyway, we’re extremely excited to have James back for his third exclusive interview – and we’ve got a lot of ground to cover. In the first part of our Q&A, we discuss some recent changes at Google; a possible book deal; the future of cloud computing; testing Chrome OS; the problem with test automation; the upcoming GTAC event and why testing is (believe it or not) getting easier. For the second half of the interview, be sure to check back tomorrow.

**********

uTest: Good to have you back, James. Tell us what you’ve been up to? Anything different in your life at Google?

James: Funny you should lead with that! Yes, my role at Google is changing. I’ve passed the leadership of Chrome and Chrome OS testing to a colleague and taken over Cloud Computing. Lots of amazingly complicated back end/data center testing issues which are new and exciting. Google keeps pushing me outside my comfort zone in the most endearing way. Retirement in place hasn’t really been an option. But hey, one can dream.

uTest: We’ve noticed you’re blogging a lot more of late and you’ve started a Twitter account! How do you manage to find time for these things with the new challenges you’re faced with?

James: Believe it or not, the Cloud stuff is actually smaller than the Chrome/Chrome OS work in terms of the amount of time out of my day. The level of automation is very high and there is a lot we’ve been able to do put some very hard sub-problems on autopilot. My team here is seriously Jedi quality folk. Definitely good fodder for some future talk. But the Google-wide testing efforts in terms of security testing, testing tools, development tools and general evangelism is also officially part of my day job. I own the Google Testing blog and am also running GTAC this year. And yes, I finally succumbed to Twitter. It’s actually a lot of fun so far, 140 characters is so do-able.

uTest: Speaking of the blog, I noticed Pat Copeland mentioned the popularity of the current “How Google Tests Software” series. Any plans on turning that into a book?

James: More than plans, I am under contract with Addison Wesley and am working diligently on it. I am finding that as I dig deeper into Google internal testing processes I have to be more careful about what I publish. I have to avoid talking about confidential technology and innovations created by other Googlers who aren’t ready to discuss their work externally. So there is a backlog of stuff that requires review and once approval occurs, it makes sense to publish it all at once. The book format is ideal for that as opposed to trickling it out in blog form. Besides I love books, they are so cuddly and blogs are so not cuddly. And I also have two co-authors Jeff Carollo and Jason Arbon who I am mentoring through the writing process and that is bringing me a lot of satisfaction.

Read more…

Google Test Director James Whittaker recently concluded his fantastic “How Google Tests Software” series. We covered Part I a few weeks back, but I wanted to re-direct your attention to his latest post, which deals with the size and scope of their various projects.

Despite the perception that Google’s testing is highly complex and indecipherable to us mere mortals, we find the opposite to be true. As Whittaker explains, testing scope is determined by “emphasizing  scope over form.” Plainly stated, testing comes in three sizes at Google: small, medium and large. Here’s his explanation:

Small Tests are mostly (but not always) automated and exercise the code within a single function or module. They are most likely written by a SWE or an SET and may require mocks and faked environments to run but TEs often pick these tests up when they are trying to diagnose a particular failure. For small tests the focus is on typical functional issues such as data corruption, error conditions and off by one errors. The question a small test attempts to answer is does this code do what it is supposed to do?

Medium Tests can be automated or manual and involve two or more features and specifically cover the interaction between those features. I’ve heard any number of SETs describe this as “testing a function and its nearest neighbors.” SETs drive the development of these tests early in the product cycle as individual features are completed and SWEs are heavily involved in writing, debugging and maintaining the actual tests. If a test fails or breaks, the developer takes care of it autonomously. Later in the development cycle TEs may perform medium tests either manually (in the event the test is difficult or prohibitively expensive to automate) or with automation. The question a medium test attempts to answer is does a set of near neighbor functions interoperate with each other the way they are supposed to?

Large Tests cover three or more (usually more) features and represent real user scenarios to the extent possible. There is some concern with overall integration of the features but large tests tend to be more results driven, i.e., did the software do what the user expects? All three roles are involved in writing large tests and everything from automation to exploratory testing can be the vehicle to accomplish accomplish it. The question a large test attempts to answer is does the product operate the way a user would expect?

Simple enough, yes? So if you’re in charge of testing at a growing company – and wish to follow in the footsteps of giants – you would be wise to read the entire series, starting here.

As a supplement to this series, here’s Whittaker’s last uTest appearance – a webinar titled “More Bang For Your Testing Buck” (after the jump). Enjoy!

Read more…

One of the many “great debates” in the software industry is the role of testers in the development process. How soon should they be involved? What tasks should they be completing during the initial, middle and latter phases? Who should be responsible for designing data structures, reviewing code and writing test cases? The list goes on…

While there are technically no wrong answers to these questions, some hold more weight than others. For instance, you’d be much better off listening to advice from a company like Google, as opposed to Uncle Joe’s Discount Software Shack.

Lucky for you, Google Test Director James Whittaker tackles this subject in part II of his “How Google Tests Software” series. If you’re new  to the uTest Blog, James Whittaker is the distinguished author of several uTest blog posts, eBooks and webinars. You would be wise to listen to his advice.

Anyway, here are a few short excerpts from his latest article, addressing the various roles of those in the development process:

The SWE or Software Engineer is the traditional developer role. SWEs write functional code that ships to users. They create design documentation, design data structures and overall architecture and spend the vast majority of their time writing and reviewing code. SWEs write a lot of test code including test driven design, unit tests and, as we explain in future posts, participate in the construction of small, medium and large tests. SWEs own quality for everything they touch whether they wrote it, fixed it or modified it.

Read more…

We often ask our Testing the Limits guests what they would do in a world with no need for software testers. So far, answers have included mandolin player, pilot, stand-up comedian, sports announcer, werewolf hunter and other typical trades. This got us to thinking, “what other careers would software testers be good at?”

Not that we’re encouraging you to leave the testing profession, but if you absolutely had to, here are a few options for you to consider:

1. Software developer / engineer: Aside from werewolf hunter, this is probably the most obvious career alternative, as great testers will eventually acquire the skills and understanding needed to succeed as a developer. But as a blog reader recently brought to my attention, this works both ways. He said that at his former place of employment, developers aspire to be testers, NOT the other way around. He writes, “ the Tester/QA path is the destination/pinnacle of the career path in SW development. You start out as a Jr. Programmer then… Sr. Programmer then… eventually Architect/System Designer…then…you eventually make it to Testing. Their thinking was that you can’t adequately test until you have proper understanding of the development process. In other words, you are truly considered an expert by the time you get to that level.”

2. Detective: Much like a detective, a tester’s bug-hunting prowess will depend largely on intuition – i.e. knowing the right questions to ask and having a sixth sense for odd irregularities. Testers are already possess the other traits found in successful detectives, including sound logic, analytical skills and patience. The only thing they are missing is an assistant named Watson and a trench coat. Both are available on Craigslist.

3. Journalist: There’s a very thin line between a tester and an investigative reporter. Like their journalistic counterparts, QA professionals must ask tough questions, dive deep into complex issues and report them to the layman in a clear, concise and objective manner. The hours stink and the pay is terrible, but there is some downside to the job however.

Read more…

Last week, more than three hundred software professionals from around the world tuned in to “More Bang For Your Testing Buck” – an exclusive webinar with best-selling author and Google Test Director James Whittaker. For those in attendance, we’re happy to post the following reprise of the Q&A session, where James addresses ACC requirements; the levels where testers are needed most; test tours; unit testing and more. See his answers below.

For those that missed the event, we have posted the webinar in its entirety (after the jump). If you’re interested in similar presentations, be sure to check out our webinar archives.

When is the right time to start with ACC in a development project? You just told us that ACC are not requirements. So ACC cannot be defined before requirements are clear? May ACC can even be defined just with the requirements?

Whittaker: Early in a project when there is a lot of flux, features added and cut/unreliable code being written, there isn’t much use in putting a lot of work into any sort of artifact and the ACC is no different. Once the features are locked, this is the time to begin in earnest. Even for large complicated products a good ACC can be built in hours, not days, so it’s unlikely that you’ll get too far behind. What we’ve found helpful is having a single test engineer maintain the ACC early in a product cycle and then when features are no longer in so much flux, that’s when additional testers can be added. Too many testers too early is just not productive. Developers know their code is buggy and it’s pretty unhelpful to keep reminding them of it.

The ACC and the requirements are very similar. In fact, I think the former could be a viable substitute.

It sounds like you think testers are only useful at the level of the UI?  Or did I misunderstand?  I currently work as a tester on an architectural team and code tests to test services and APIs.  Are you saying that is not a useful testing task, or that it should be owned by devs?

Whittaker: I want to make the distinction been early cycle testing, which are things like unit testing and test infrastructure, and late cycle testing which is more end-to-end. I think all late cycle testing should be done by testers and all early cycle testing should be owned by devs. Even within early cycle testing there are devs who specialize in writing such test code. At Google they are called Software Engineers in Test. They are engineers first and testers mostly in the sense that they write test code and not shipping features.

Now late cycle testing or end-to-end testing (if you like that phrase better) can be either UI or API level. At the UI we are stringing together inputs that make up user scenarios. At the API level we are writing code that utilizes the APIs with similar purpose: to test that the APIs can be used in real user scenarios.

The bottom line for me is that if you are thinking about the user, you are a tester. If you are thinking about the code, you are a developer.

Read more…

In part II of our Testing the Limits interview with Mozilla QA Director Matt Evans, we get his thoughts on mobile immaturity; the worst bug ever submitted by a Mozilla community member; the so-called “skills shortage” in Silicon Valley; skepticism for all things open-source; the next great browser innovation and more.

If you missed Part I, do yourself a favor and catch up here.

*******

uTest: In many ways, mobile is still playing catch up to the web. Is there one area in particular where you see the most room for improvement? If so, where?

ME: Well, there are some obvious platform deficiencies around inconsistent UI and whether Flash is going to be fully supported across mobile devices or not. But this is a testing blog, so let’s talk about that. As I mention elsewhere in this interview, mobile is just a really tough testing challenge. The big problem is that there is very little support for cross-platform mobile device test automation. I suspect most of mobile device and application testing is done 100% manually. If any environment needed more test automation, it is mobile. At Palm, we rolled our own test harness that ran on the Pre. This became extremely important for endurance testing and finding memory leaks in the Pre applications.

Mobile software companies have an uphill battle since developing automated system tests for every platform is very costly, both in time and resources. However, reliance on mostly manual testing has lots of quality risks. If the quality of mobile devices and software is to rise about what it is now, we need automated test tool support that works well across all device platforms.

uTest: What’s the best (and by that, we mean the worst) bug ever submitted by one of your community members?

ME: Recently, Alex Miller, a Mozilla community member, discovered a very critical security bug and was awarded $3000 for finding and reporting the bug. He’s been hard at work finding and discovering other security flaws in Firefox, too, and was even given clearance access to all Mozilla security-related bugs reported in Bugzilla. Very few people have this access.  Oh, I forgot to add a little fact about Alex: he’s only 12 years old. That’s an awesome accomplishment by a really smart kid. This exemplifies the opportunity Mozilla provides to the community: an incredible technology playground where anyone that spends the time to learn can participate at any level no matter who you are or what your background is. The more you prove what you can do, the more you will be encouraged and acknowledged for that effort. Finding bugs is a good place to start for anyone who wants to participate. Certainly, not everyone is going to develop the expertise to discover deep level security bugs, but believe me there is plenty of testing folks can really help us out with. If you are so inclined, we will welcome you with open arms. Please visit us here.

uTest: We keep reading about the skills shortage in Silicon Valley. Are you seeing this at all, particularly when it comes to software testers? If so, what do you suspect is the reason?  And how do you overcome this dearth of top-shelf talent?

Read more…

If you were going to invest more money in testing, where would you place those bets? Testing early in the cycle? Automation? Manual testing? Better requirements and planning? Better documentation? Torture devices for your devs?

In our latest uTest webinar, More Bang For Your Testing Buck, best-selling author James Whittaker will take a critical look at how to maximize your investments in testing, including an examination of the tools, best practices and methodologies that can help make testing a “happier place.”

The webinar is schedule for Friday, December 10 at 11:00 AM (EST).

Register now at Go-To Meeting >>>>

As the former Test Architect at Microsoft and the current Director of Testing at Google, James is one of the most accomplished professionals in the testing business. Long-time readers of our blog will know that he is also no stranger to the uTest community. Here are a few of the uTest projects he’s participated in:

Read more…