How Google Tests Software: Small, Medium and Large

Google Test Director James Whittaker recently concluded his fantastic “How Google Tests Software” series. We covered Part I a few weeks back, but I wanted to re-direct your attention to his latest post, which deals with the size and scope of their various projects.

Despite the perception that Google’s testing is highly complex and indecipherable to us mere mortals, we find the opposite to be true. As Whittaker explains, testing scope is determined by “emphasizing  scope over form.” Plainly stated, testing comes in three sizes at Google: small, medium and large. Here’s his explanation:

Small Tests are mostly (but not always) automated and exercise the code within a single function or module. They are most likely written by a SWE or an SET and may require mocks and faked environments to run but TEs often pick these tests up when they are trying to diagnose a particular failure. For small tests the focus is on typical functional issues such as data corruption, error conditions and off by one errors. The question a small test attempts to answer is does this code do what it is supposed to do?

Medium Tests can be automated or manual and involve two or more features and specifically cover the interaction between those features. I’ve heard any number of SETs describe this as “testing a function and its nearest neighbors.” SETs drive the development of these tests early in the product cycle as individual features are completed and SWEs are heavily involved in writing, debugging and maintaining the actual tests. If a test fails or breaks, the developer takes care of it autonomously. Later in the development cycle TEs may perform medium tests either manually (in the event the test is difficult or prohibitively expensive to automate) or with automation. The question a medium test attempts to answer is does a set of near neighbor functions interoperate with each other the way they are supposed to?

Large Tests cover three or more (usually more) features and represent real user scenarios to the extent possible. There is some concern with overall integration of the features but large tests tend to be more results driven, i.e., did the software do what the user expects? All three roles are involved in writing large tests and everything from automation to exploratory testing can be the vehicle to accomplish accomplish it. The question a large test attempts to answer is does the product operate the way a user would expect?

Simple enough, yes? So if you’re in charge of testing at a growing company – and wish to follow in the footsteps of giants – you would be wise to read the entire series, starting here.

As a supplement to this series, here’s Whittaker’s last uTest appearance – a webinar titled “More Bang For Your Testing Buck” (after the jump). Enjoy!

Continue Reading

Essential Guide to Mobile App Testing

Know Your Role: Software Testing Lessons From Google

One of the many “great debates” in the software industry is the role of testers in the development process. How soon should they be involved? What tasks should they be completing during the initial, middle and latter phases? Who should be responsible for designing data structures, reviewing code and writing test cases? The list goes on…

While there are technically no wrong answers to these questions, some hold more weight than others. For instance, you’d be much better off listening to advice from a company like Google, as opposed to Uncle Joe’s Discount Software Shack.

Lucky for you, Google Test Director James Whittaker tackles this subject in part II of his “How Google Tests Software” series. If you’re new  to the uTest Blog, James Whittaker is the distinguished author of several uTest blog posts, eBooks and webinars. You would be wise to listen to his advice.

Anyway, here are a few short excerpts from his latest article, addressing the various roles of those in the development process:

The SWE or Software Engineer is the traditional developer role. SWEs write functional code that ships to users. They create design documentation, design data structures and overall architecture and spend the vast majority of their time writing and reviewing code. SWEs write a lot of test code including test driven design, unit tests and, as we explain in future posts, participate in the construction of small, medium and large tests. SWEs own quality for everything they touch whether they wrote it, fixed it or modified it.

Continue Reading

Essential Guide to Mobile App Testing

Trading Places: 8 Alternate Careers For Software Testers

We often ask our Testing the Limits guests what they would do in a world with no need for software testers. So far, answers have included mandolin player, pilot, stand-up comedian, sports announcer, werewolf hunter and other typical trades. This got us to thinking, “what other careers would software testers be good at?

Not that we’re encouraging you to leave the testing profession, but if you absolutely had to, here are a few options for you to consider:

1. Software developer / engineer: Aside from werewolf hunter, this is probably the most obvious career alternative, as great testers will eventually acquire the skills and understanding needed to succeed as a developer. But as a blog reader recently brought to my attention, this works both ways. He said that at his former place of employment, developers aspire to be testers, NOT the other way around. He writes, “ the Tester/QA path is the destination/pinnacle of the career path in SW development. You start out as a Jr. Programmer then… Sr. Programmer then… eventually Architect/System Designer…then…you eventually make it to Testing. Their thinking was that you can’t adequately test until you have proper understanding of the development process. In other words, you are truly considered an expert by the time you get to that level.”

2. Detective: Much like a detective, a tester’s bug-hunting prowess will depend largely on intuition – i.e. knowing the right questions to ask and having a sixth sense for odd irregularities. Testers are already possess the other traits found in successful detectives, including sound logic, analytical skills and patience. The only thing they are missing is an assistant named Watson and a trench coat. Both are available on Craigslist.

3. Journalist: There’s a very thin line between a tester and an investigative reporter. Like their journalistic counterparts, QA professionals must ask tough questions, dive deep into complex issues and report them to the layman in a clear, concise and objective manner. The hours stink and the pay is terrible, but there is some downside to the job however.

Continue Reading

Essential Guide to Mobile App Testing

More Bang For Your Testing Buck – Follow Up Q&A With James Whittaker

Last week, more than three hundred software professionals from around the world tuned in to “More Bang For Your Testing Buck” – an exclusive webinar with best-selling author and Google Test Director James Whittaker. For those in attendance, we’re happy to post the following reprise of the Q&A session, where James addresses ACC requirements; the levels where testers are needed most; test tours; unit testing and more. See his answers below.

For those that missed the event, we have posted the webinar in its entirety (after the jump). If you’re interested in similar presentations, be sure to check out our webinar archives.

When is the right time to start with ACC in a development project? You just told us that ACC are not requirements. So ACC cannot be defined before requirements are clear? May ACC can even be defined just with the requirements?

Whittaker: Early in a project when there is a lot of flux, features added and cut/unreliable code being written, there isn’t much use in putting a lot of work into any sort of artifact and the ACC is no different. Once the features are locked, this is the time to begin in earnest. Even for large complicated products a good ACC can be built in hours, not days, so it’s unlikely that you’ll get too far behind. What we’ve found helpful is having a single test engineer maintain the ACC early in a product cycle and then when features are no longer in so much flux, that’s when additional testers can be added. Too many testers too early is just not productive. Developers know their code is buggy and it’s pretty unhelpful to keep reminding them of it.

The ACC and the requirements are very similar. In fact, I think the former could be a viable substitute.

It sounds like you think testers are only useful at the level of the UI?  Or did I misunderstand?  I currently work as a tester on an architectural team and code tests to test services and APIs.  Are you saying that is not a useful testing task, or that it should be owned by devs?

Whittaker: I want to make the distinction been early cycle testing, which are things like unit testing and test infrastructure, and late cycle testing which is more end-to-end. I think all late cycle testing should be done by testers and all early cycle testing should be owned by devs. Even within early cycle testing there are devs who specialize in writing such test code. At Google they are called Software Engineers in Test. They are engineers first and testers mostly in the sense that they write test code and not shipping features.

Now late cycle testing or end-to-end testing (if you like that phrase better) can be either UI or API level. At the UI we are stringing together inputs that make up user scenarios. At the API level we are writing code that utilizes the APIs with similar purpose: to test that the APIs can be used in real user scenarios.

The bottom line for me is that if you are thinking about the user, you are a tester. If you are thinking about the code, you are a developer.

Continue Reading

Essential Guide to Mobile App Testing

Testing the Limits With Matt Evans from @Mozilla – Part II

In part II of our Testing the Limits interview with Mozilla QA Director Matt Evans, we get his thoughts on mobile immaturity; the worst bug ever submitted by a Mozilla community member; the so-called “skills shortage” in Silicon Valley; skepticism for all things open-source; the next great browser innovation and more.

If you missed Part I, do yourself a favor and catch up here.


uTest: In many ways, mobile is still playing catch up to the web. Is there one area in particular where you see the most room for improvement? If so, where?

ME: Well, there are some obvious platform deficiencies around inconsistent UI and whether Flash is going to be fully supported across mobile devices or not. But this is a testing blog, so let’s talk about that. As I mention elsewhere in this interview, mobile is just a really tough testing challenge. The big problem is that there is very little support for cross-platform mobile device test automation. I suspect most of mobile device and application testing is done 100% manually. If any environment needed more test automation, it is mobile. At Palm, we rolled our own test harness that ran on the Pre. This became extremely important for endurance testing and finding memory leaks in the Pre applications.

Mobile software companies have an uphill battle since developing automated system tests for every platform is very costly, both in time and resources. However, reliance on mostly manual testing has lots of quality risks. If the quality of mobile devices and software is to rise about what it is now, we need automated test tool support that works well across all device platforms.

uTest: What’s the best (and by that, we mean the worst) bug ever submitted by one of your community members?

ME: Recently, Alex Miller, a Mozilla community member, discovered a very critical security bug and was awarded $3000 for finding and reporting the bug. He’s been hard at work finding and discovering other security flaws in Firefox, too, and was even given clearance access to all Mozilla security-related bugs reported in Bugzilla. Very few people have this access.  Oh, I forgot to add a little fact about Alex: he’s only 12 years old. That’s an awesome accomplishment by a really smart kid. This exemplifies the opportunity Mozilla provides to the community: an incredible technology playground where anyone that spends the time to learn can participate at any level no matter who you are or what your background is. The more you prove what you can do, the more you will be encouraged and acknowledged for that effort. Finding bugs is a good place to start for anyone who wants to participate. Certainly, not everyone is going to develop the expertise to discover deep level security bugs, but believe me there is plenty of testing folks can really help us out with. If you are so inclined, we will welcome you with open arms. Please visit us here.

uTest: We keep reading about the skills shortage in Silicon Valley. Are you seeing this at all, particularly when it comes to software testers? If so, what do you suspect is the reason?  And how do you overcome this dearth of top-shelf talent?

Continue Reading

Essential Guide to Mobile App Testing