In the second part of our Testing the Limits with Anne-Marie Charrett, we get her thoughts on the meaning of exploratory testing, the challenge of agile adoption, how to grow as a tester and more. Enjoy!

uTest: Certain industries appear to be ahead of the curve when it comes to testing practices, while others remain in the proverbial stone age. Is this an accurate statement? Or have testing practices evolved at similar pace across all industries? As someone who has spent time in many sectors, we’re interested to hear your thoughts on this.

AMC: I think companies that demand value from their testing are generally more receptive to new ideas and change in testing. I don’t think it’s fair to silo this into industries.

Take for example the finance industry, yes many large insurance and bank corporations are risk averse and resist change but not all. For example Barclays Bank are using coaching & Rapid Software Testing.

I’ve worked with small companies in R&D who you would associate with flexibility and being pro-active, yet they want very traditional, heavily documented testing processes. Often this is because someone did testing ‘once’ and this is what they did.

I’ve seen testing practices change within sectors too. For example, the telco sector in the mid 1990‘s were typically heavily documentation orientated. Often testing went on for years before a product was released. By the late 90’s and early 2000’s testing practices had to evolve as smaller companies with lighter and more flexible delivery approaches challenged this paradigm.

uTest: There’s a good debate right now on the true meaning of exploratory testing, with people like James Bach and Michael Bolton chiming in with their opinions. What is your definition of exploratory testing? And in your view, what is the most misunderstood term in testing today?

AMC: So many questions!! The beauty of Exploratory Testing is that it can mean different things to different people. Thats why there are so many different perspectives on it.

There are some core values to Exploratory Testing, namely that it’s an approach (not a technique), it’s simultaneous learning, design and execution and that it’s tester centric.

The latter ideal is something that I cherish and hold dear.  I think it’s essential that we take responsibility for the testing we do. This means each tester decides on their testing approach, what they test and when they’re done. Owning these decisions is what matures a tester, helping them become skilled, confident and motivated to excel in their testing.

uTest: On a similar note, software testing is sometimes seen as a career with a definite ceiling. What advice do you have for testers who feel like they have peaked in terms of title, salary and responsibility?

AMC: This is a really difficult question to answer, but it’s so dependent on the tester what they want and the other factors in their lives.

Personally, I’ve experienced the frustration caused by lack of recognition, especially early on in my career as a tester. I’ve also quickly become disillusioned by roles I’ve taken to the point where I’ve asked myself, is this all there is to testing?

A couple of things radically changed how I viewed myself, my career and testing.

The first was to re-discover the joy or learning new things. There are always new things to learn, new things to discover in testing but have you sufficient curiosity to find them?

I’ve discovered how owning and taking responsibility for your learning liberates you. You own the learning. You’re not doing it because your job demands it, or your boss expects you to do it. You are doing it to satisfy your own curiosity, it’s yours. I find a sense of freedom from that. It makes me really happy.

But this is my own personal experience, it’s up to each individual tester to discover what floats their boat.

uTest: What’s Anne-Marie Charrett doing when she’s not making testing a better place?

AMC: I spend time with my two boys Nikolai and Alex and I enjoy running. One of my biggest pleasures is an early morning run, and then sitting by the sea watching the sun rise.

Rapid Fire:

Better wine: Australia or California?

You make wine in California?

Favorite testing blog….other than uTest :

My personal favourite has to be James Bach’s blog but other bloggers I hugely admire are Michael Bolton and Catherine Powell. Closer to home are Trish Khoo and Ben Kelly.

Twitter or Facebook:

Twitter for work, Facebook is strictly for family and friends.

Next vacation:

Up the coast to Hawks Nest for a week by the beach with my very supportive husband, two boys and our new addition to the family, our pet bear.

Ever walked out of a movie?

What a question! Funnily enough, I don’t think so. I keep waiting to see if the movie improves, or if it’s because I haven’t quite understood something.

Favorite waste of time:

My boys Nikolai and Alex.

Editor’s Note: We hope you’ve enjoyed our latest Testing the Limits interview. Until next time, happy testing!

To kick off another amzing year of Testing the Limits we reached out to Anne-Marie Charrett, an independent tester who has worked for the likes of Mercury Interactive, IBM (twice) and Nortel – just to name a few. She also arranges for speakers to visit Ireland as part of Softtest Ireland and blogs about her testing experience and offers coaching at mavericktester.com. 

In part I of this month’s interview, we learn what motivates Anne-Marie to coach via Skype, what’s caught her interest lately, how her book with James Bach is coming and what the biggest mis-conception about testing is. Come back tomorrow for part II.

uTest: In terms of writing, speaking and researching, you are one of the most active testers in the business. So we’ll start by asking you this: What hot topics within testing have captured your interest recently?

AMC: 2012 has kicked off with a flurry of activity. Key topics appear to be, How we learn, Rapid Test Management and more recently James Bach has been looking Exploratory Test Documentation.

It goes like this. Typically we write tests and charters as artifacts for other people as evidence of work performed. But writing is a lot more powerful than that, it has the ability to assist in design (think brainstorming in mind maps). Exploratory Test Documentation is about changing the purpose of writing from an end product to a by product.

I also like the way new conferences and peer workshops are happening at a grass roots level, for example Lets Test in Stockholm. These are not necessarily big conferences, but ones that offer value to testers and that encourage participation. I hope that this will be the conference circuit of the future!

uTest: You’ve made quite a name for yourself as a testing coach; offering advice to testers free of charge via Skype. In your experience, what areas require the most coaching on your part? In other words, what does a typical tester coaching session cover?

AMC: Often testers come looking for coaching in a particular skill (e.g Test Automation), but many fail to understand basic testing concepts such as: “What is testing?” and “How do you determine bugs?”

Understanding testing is key to improving your testing skill.  After all, if you don’t understand something, how can you improve it?

Software delivery typically doesn’t allow for this type of introspection. Our jobs demand we focus on delivery, often to the detriment of how well we are doing our testing.

Coaching is the breathing space that all testers need to learn and grow.

In coaching I encourage testers to work through tasks to acquire skill. I’m there to guide and help them, but they need to work out the answers. That way, their learning experience is deeper and more meaningful and empowering.

uTest: James Bach, a fellow Skype coach, has said that one of the reasons he coaches testers for free is that he is “motivated by people who entertain [him] with their passion for learning.” What is your motivation for coaching testers?

AMC: I like to think there is a little testing genius in all of us. I get a sense of satisfaction if I can tap into that and liberate it.  If a tester leaves my coaching sessions feeling motivated and confident about their testing skill, I feel motivated too.

uTest: James also mentioned that he was working with you on a book about coaching testers. First off, how is the book coming along? And secondly, what is the most significant thing you’ve learned from your coaching sessions?

AMC: The content is at the point where there is sufficient content to hold a couple of workshops on coaching. In fact, I’m holding one on the 4th May in London, and one with James Bach in Sydney on the 29th May.

I think this is a significant point in the maturity of the coaching model.

Coaching has taught me to be humble and respect people. I have a tremendous admiration for testers who ask to be coached. I think this takes courage especially for testers who have been in the industry for a while.

It also helps me gauge the amount of knowledge and skill I have as a tester. If you want to fully understand a topic, try teaching it! This has given me a tremendous confidence boost. I’m reaching goals I’d never even imagined possible a year ago.

uTest: Speaking of publications, you were an author of the popular book How to Reduce the Cost of Software Testing. Your chapter dealt with the cost of setting up a test team. For those who haven’t read it, what is the biggest misconception managers and executives have with regard to the cost of setting up a test team? And how can they avoid making these mistakes?

AMC: There’s a misconception that creating a visible testing structure is the equivalent of testing. This is not true. What makes a house a home? Not the roof or the external walls, but the people inside the house.

It’s the same for testing. Testing is about the testers, their skill and discipline and how they interact and behave with others.

That’s what companies need to focus on when setting up a team.

uTest: In terms of money, time and effort, what is the biggest waste of resources you see in testing today? Certifications? Test reports? Automated tools? You tell us.

AMC: Behind this waste lies the concept that there is a simple solution to a complex problem. The above solutions are wasteful when they aim to easily resolve a complex problem. If we could stop looking for the silver bullet, then perhaps these wasteful solutions wouldn’t look so tempting.

uTest: True or false: Startups are less concerned with quality and testing than larger, established companies.

AMC: Its not true or false.

There is never an easy solution to quality, both startups and larger companies struggle equally with it.

Many startups create quality products because if an idea is sufficiently new and valuable, people will consider it good quality.

Peoples perceptions on quality change over time, they start expecting more from a product. Often what was acceptable when a product is launched quickly becomes unacceptable to a user. It’s important that a startup understands and manages the transient nature of quality. One possible solution is a skilled tester, but it’s not the only solution.

Many larger companies equally struggle with quality. As companies grow and mature they become more concerned about demonstrating due diligence than being diligent about quality and become fixated on process and demonstrable results.

These are not easy problems to solve, but it helps to understand what quality is and how much it means to you.

uTest: We noticed that you spent a few years as a Y2k test specialist with IBM. First off, thanks for keeping civilization together. Seriously though, tell us a little bit about your responsibilities in that role. Was the run up to 2000 as hectic and scary as it’s often portrayed to be?

AMC: Glad to be of service!

On hindsight, Y2K was more about compliance and limiting liability than testing. Perhaps there was a valid cause for concern, but I never experienced any of it where I consulted.

In part II of our Testing the Limits interview with security expert Richard Stiennon, we get his thoughts on where companies are least prepared for a security breach; the explosion of mobile related threats; the future of cyber security and much more. If you missed the first installment, read Part I now.

uTest: What’s the one piece of security advice you would have for companies that are in their infancy?

RS: Plan ahead. As you grow so does your “target area.” When you launch your product or service you may not be in the cross hairs of a bad actor. But you will be. Sometimes security builds in “friction” that can slow down customer acquisition. So you may not want to require 12 character passwords and SMS authentication at the beginning but know that eventually you will have to deal with account theft, breaches, and spammers. Have a plan so you can implement defenses quickly; and ideally before a damaging attack or breach. You don’t want to be in the position of the Sony Play Station Network that implemented protections *after* a series of attacks that cost $170 million to recover form.

uTest: You’ve said before that mobile will not require its own anti-virus systems. That said, it seems that mobile threats are multiplying by the hour. In your view, what’s the biggest security challenge in terms of mobile?

RS: Apps, apps, apps. VPNs, firewalls, and carrier filtering are going to impede network based attacks. Containing and vetting applications is the biggest security challenge for the platform vendors.

uTest: Looking back to the 1990s, what’s surprised you the most about the evolution of cyber security? What’s been your biggest disappointment?

RS: For me the biggest surprise was the confluence of existing criminal organizations and cyber crime, especially arising out of the demise of the Soviet Union. In retrospect is seems obvious, but at the time it was a wake-up call for me that guys with guns and baseball bats were going high-tech. My biggest disappointment is that security has never become important enough to spawn secure networks or secure computers. Not a single ISP or carrier has gone to market with a secure network with complete content inspection. Not a single computer manufacturer outside the military has sold a computer whose primary feature is security. They are all happy to sell you security add-ons but not willing to step up and address the underlying vulnerability of their products.

uTest: What happens in the next decade of security is anyone’s guess, but your predictions carry a bit more weight. Care to make any bold predictions on the future of cyber security? The bolder, the better.

RS: As a rule I stay away from “the sky is falling” scenarios, but since you asked.
By 2015 the overall threatspace will be ten times worse than it is today. Think about that. There will be TEN breaches as critical as the RSA attack. There will be TEN Google Aurora’s. There will be TEN Stuxnets. There will be 300 thousand new malware variants a day.

Electronic warfare of the type contemplated by the Obama administration in Libya will occur.
Microsoft will no longer dominate either personal computing or enterprise IT.
There will be no consolidation of the IT security vendor space. There will still be 1,500 vendors.
There will be an international cyber police force.

uTest: Aside from Surviving Cyber War, what other books would be on your required reading list for someone who wants a better understanding of security?

RS: Start with America the Vulnerable by Joel Brenner for an overview of the problem. To get into practical aspects of security read the Hacking Exposed series. If you want everything, especially from the perspective of software assurance read Cybersecurity: The Essential Body Of Knowledge by Dan Shoemaker and Wm. Arthur Conklin.

uTest: You’re an extremely active speaker and lecturer – care to give our readers a heads up on where and when you’ll be presenting next?

RS: Follow me on Twitter (@stiennon) to get alerts on my speaking engagements and web events. There are several big conferences in the works for 2012 but no confirmations yet. I will be participating in a Wilton Park conference on cyber security June 27-29 in the UK.

Rapid Fire

• Who taught you how to bake bread (http://www.youtube.com/watch?v=JTY4WJRSzY8)? My father and older brothers were avid bread bakers and introduced me to the art.
• Favorite sci-fi hero? Lazarus Long.
• Any prizes planned for your 25,000th Twitter follower? I will #FF my 25,000th follower.
• More annoying: Pop-ups or Captcha? Pop-ups, especially the floating ones that seem to know where your eye is focused. I love reading hidden meanings into Captchas.
• Secret Talent: Juggling. Still working on my five ball cascade.

Editor’s note: We hope you’ve enjoyed another year of Testing the Limits interviews. We have big plans for 2012, so stay tuned. In the meantime, happy holidays from all of us here at uTest.

In part I of our interview, we get his thoughts on the difference between security and other types of testing; what the world would look like under full blown cyber war; the biggest threats to the typical web user; the motives of hackers and more. Tune in tomorrow for part II.

uTest: You started out in the field of aerospace as an engineer and wound up as one of the world’s top security experts (so typical). Kidding aside, what attracted you to the field of cyber security? And what’s kept you there for the better part of two decades?

Richard Stiennon: My transition from structural engineer to networking came when I started a dial-up ISP in Michigan, but I did not get the security bug until joining Netrex, which was an integrator of security products and services. Through Netrex I worked with a lot of the early security products and the founders of ISS, and Check Point Software. By the time Netrex was acquired by ISS (later to be acquired by IBM) I had moved on to PwC where I got exposed to large enterprise and performed audits on their security postures. From there it was on to Gartner and after that I was firmly entrenched in the IT security world. I have a low threshold for boredom. The security industry moves so fast you get left behind if you allow your eyes to glaze over for a second.

uTest: Is it fair to say that security testing requires a much different mindset/persona than other types of testing? If so, what specific qualities and characteristics are needed in a security tester? If you were assembling a team of security testers, what traits would you look for?

RS: Security testing of software throughout its development cycle is indeed different than quality and functionality testing. Instead of testing against end user use cases you have to have a mind set of an attacker, a completely different use case. In addition to meticulous use of security testing tools (HP-Fortify, Veracode, etc) a security tester must understand the application and how an attacker would leverage built-in functionality to subvert a system. A security tester must be diligent and detail oriented as well as imaginative and wily – a rare combination.

uTest: In 2010, you published Surviving Cyber War, which gave the world an inside look into the onset of state-sponsored cyber war. Since then, there’s been no shortage of similar incidents (Stuxnet, Anonymous, to name a few). Are you surprised at the speed in which cyber warfare is evolving?

RS: Every day, although I should not be surprised, because threats follow an ever rising and easily predictable ramp. As governments finally go through the learning curve they are recognizing the threat to their own systems as well as to critical infrastructure. So far there is a lot more talk than action but that will change.

uTest: In terms of nuclear war, most of us know the worst case scenario. Not so with cyber war. Care to give our readers a short description of what the world looks like under full-blown cyber war?

RS: It is hard to imagine a full blown cyberwar because I believe cyber attacks are going to be a component of all future wars between developed countries – not a stand alone conflict. That said, a full blown cyber attack is easier to contemplate. The most likely is an Internet outage. We have seen this before regionally with the worst event being the result of the SQL Slammer outbreak in 2003. The Internet was unusable for about eight hours. Imagine not being able to use Facebook for most of the day! Both Estonia and Georgia are familiar with the impact of such cyber attacks. Luckily the Internet is resilient and the extremely smart people who take care of the plumbing will bring it back on line in days. Anyone who has lived through a power outage from hurricanes or snow storms knows what an attack against critical infrastructure will look like. Of course even the most severe cyber attack cannot equal the risk from a major solar flare or electromagnetic pulse (EMP).

uTest: True or false: Governments are a greater security threat than private entities.

RS: True, but keep in mind that hostile governments and criminal elements are often inseparable. The combination of states hungry for industrial, diplomatic, and military intelligence with organized crime, drug cartels, and rogue actors (Anonymous, terrorists, people with a grudge) creates a threatscape that is hard to pin down.

uTest: What’s the greatest threat to day-to-day app security for the typical user:

A. Black hat hackers
B. Internet Explorer
C. Authentication via social credentials (FB, Twitter, etc.)
D. Unsecured wi-fi
E. Other (please elaborate)

RS: E. Cyber criminals. While they may employ the expertise of Black Hats they are the ones with the profit motive and will expend time, energy and dollars to break your app.

uTest: You’ve done your fair share of traveling over the last few years. In your experience, is security dealt with differently in different parts of the world? If so, any specific examples?

RS: Yes, the early adaptors of the Internet (US, Canada, UK) tend to be more exposed while Germany, LATAM, and parts of Asia thought about security when they got online and took precautions. At this point Estonia has the best formulated security strategy, thanks to their experience in 2007.

uTest: It seems to be getting more and more difficult to understand and predict the motives of hackers. Typically, it was for financial gain. Now, we see motives that go beyond money and into the realm of chaos and disorder. In your view, how important are motives in the context of security?

RS: Motives generate threats. Although the chaos-generating motive is not new at all; it is just seeing a new generation that is repeating the cycle that used to be called the hacker underground. Motives are a critical component in what I call threat-based security planning. Instead of so-called risk based security that starts with identifying and protecting critical assets I promote understanding threats and basing security on those.

uTest: What’s the one area where today’s companies are least prepared to deal with a serious security breach?

RS: Cyber espionage. In 1995 when I was at Netrex I would talk to companies that would say “Why do I need a firewall? Who would hack us? We have nothing to steal.” That sentiment is still prevalent but is expressed “Why would anyone attack us? We are not a bank, we are just a furniture manufacturer, or oil and gas exploration company.” They do not realize that they compete in a global market and even their business processes are valuable to someone.

Editor’s note: Be sure to check back tomorrow for Part II of our interview with Richard Stiennon.

In part II of our Testing the Limits interview with Noah Sussman, we ask him some questions on Etsy’s testing challenges; assembling a testing team; localization testing and more. Did you miss part I? You can catch up here.

uTest: Total job interview question: What’s been the biggest testing challenge at Etsy and how have you been able to overcome it?

NS: Total job interview answer: concurrency has been a big challenge for us as we scaled our CI cluster. Most xUnit test runners aren’t designed to deliver massive concurrency out of the box. Eg if you want to run all your PHPUnit tests concurrently, you have to build a special branch of the test runner.

And even then, most tests are not written with concurrency in mind either. So a lot of design, debugging and rewriting of legacy code have been required in order to get to a point where we could run our tests in as concurrent a fashion as the available hardware would allow.

uTest: A previous interview guest once wrote a lengthy post on the “victims of fake agile.” In your view, is there a danger of a half-assed move into the agile methodology? If so, what are some of the major consequences?

NS: I can only speak from personal experience and I don’t know how typical my experiences are. I agree with the author of that post in that adopting Agile won’t fix a dysfunctional team, and it won’t help an organization to learn to accept its limitations and work within them.

uTest: What types of traits/qualities/skills does Etsy look for when hiring testers?

NS: On my team there are two roles for which I hire. These aren’t formal roles and we switch them up sometimes.

I hire software engineers who value quality and who are interested in how complex systems fail. These are the people who customize our mocks and fixtures, manage our coding standards, build static analysis tools, develop Jenkins plugins, design the CI system in general other things of that nature.

Then there is another equally important set of people whose skillset is generally described as “hardcore QA chops and a deep connection to the Etsy community.” The people I’ve hired here have many years of formal QA experience in high-risk industries like finance. These are people who are willing to use what they know about formal process, to help Etsy avoid the need for formal processes.

Here I look for talented QA analysts who are longtime Etsy users and deeply involved in the Etsy community. These are the people who develop functional testing tools, manage Selenium integration with CI and work with others in the organization to formulate test plans and resolve bugs. They also design and improve the process by which we triage bugs found in production.

uTest: How does Etsy – which has a global user base – deal with testing issues related to localization?

NS: For the recent launch of Etsy in German, there was quite a bit of time spent doing user testing in our Berlin office. We have a growing international team and community, so that’s a convenient place to go when you suddenly need a testing resource in some geographically distant location.

Another example: when we started supporting non-US currencies, that was a massive developer testing effort, followed by a lot of manual testing performed by everyone across the organization. All-hands participation in testing is the norm for any major product launch.

uTest: Does the introduction of mobile, social media or location-based functionality make testing more difficult for a company like Etsy? What are some of the specific challenges that come with these variables?

NS: Automated test tools for the mobile Web are less mature and so working with them incurs a high research and development cost. Of course that’s part of the fun, because we get to help invent the future by figuring how to leverage new tools like GorillaPhone and Kif.

Social media integration just means hitting a bunch of external APIs that you don’t control. It can be frustrating when a test fails and you can’t immediately fix it, but that’s a lot better than not knowing that an external service has gone down. Nagios is a great tool for these kinds of automated checks.

uTest: We see that you’re a pretty active Quora user. What’s the dumbest question you’ve ever come across? On a more serious note, how have social sites like Quora enhanced your understanding of testing and technology in general?

NS: I was largely self-taught as a developer and for me online forums were always very important tools in terms of solving problems for clients. I learned quickly that access to information is critical to the success of a Web developer. I can’t tell you how many times I’ve re-read “how to ask questions the smart way” and “why we won’t help you.” Sites like Quora and StackOverflow are for me just a natural way to continue doing the research I always have done in order to remain competitive in a fast-changing industry.

I don’t recall any truly dumb questions on Quora, but there are some funny / sad / informative threads where people air their ideas on how to measure developer productivity. Surprisingly few people say things like: just make sure your engineers like what they’re doing and they respect your goals.

uTest: You seem to be a big fan of test tools. Any specific ones that have caught your eye in recent months? And on a broader note, how do you and your team go about purchasing, selecting (or developing) tools to make testing run more smoothly?

NS: GUI driven regression testing on mobile devices is a fascinating area of development. New and interesting things are appearing all the time. For example, Kif uses iOS accessibility tags, a strategy which has the potential to be much cleaner than addressing elements with XPath or CSS.

Most engineers at Etsy are constantly prototyping and trying out new tools. Experimentation is highly encouraged and if we find tools we like we can often adopt them wholesale in a matter of weeks. Behind the firewall, there’s no formal change management or tool adoption process. Wrote a new IRC bot? Let it loose. New shell script? Installed for everyone. The dev environment is chaotic and extremely fun to use.

That said, I believe simplicity is a core value of automated tests and of CI systems. In order for us to adopt a new tool into CI, it would have to meet a clear and previously identified need. Otherwise it wouldn’t be worth the added complexity of adopting and maintaining the new functionality. Right now we are pretty much only looking at adopting new mobile testing tools, and tools that would help us to make the CI cluster run our tests faster.

uTest: What’s Noah Sussman doing when he’s not improving the Etsy user experience?

NS: My wife and I spend a lot of time looking at art, fashion and nature in various ways. I also do ink drawings and I read a lot.

Rapid Fire:

• Secret talent: Forgetting unimportant details.
• Favorite mobile app: Instapaper.
• Dream car: Honestly if I could take an NYC taxi everywhere, I would do that.
• Programming language of choice: Currently a mixture of JavaScript, Ruby, PHP, Perl and bash.
• Better fictional gift: Hoverboard or Lightsaber? Lightsaber because I’m comfortable with power tools. I would fall off the hoverboard.

Editor’s note: That does it for this month. We have some great guests lined for the year end, so be sure to check back regularly for more Testing the Limits wisdom.

Our Testing the Limits guest this month is Noah Sussman, test architect at Etsy. If you recall, we chatted with Noah last month as part of our STPCon video interview series, which you can watch here. Noah’s professional background includes a stint at Barnes & Noble, where he was the Tools and Automation Specialist, in addition to being a technical adviser to several startups and organizations. To learn more about Noah, you can follow him on Twitter, or check out his posts on github or Quora.

In part one of our interview with Noah, we get his thoughts on how Etsy tests software; why engineers should take more responsibility for testing their code; testing at startups and much more.

uTest: When it comes to testing, there’s certainly no shortage of collective knowledge to draw from. Who are some of your software testing role models – in terms of both companies and individuals – that you strive to emulate?

NS: As far as people, I work with a great team at Etsy. I feel really lucky to work with this many talented and knowledgeable people. Check out our engineering blog, http://codeascraft.etsy.com, as well as http://github.com/Etsy. You’ll find a large number of people with a wide range of ideas, all of which I believe are interesting. Once you’ve read their blog posts on Code As Craft, you should check out their personal blogs too, because those are fascinating as well.

Sebastian Bergmann and Jason Huggins have also been very helpful to me as I tried to figure out the best way to make CI work with Etsy’s unique deployment process.

As to companies, Blizzard, Amazon.com and Facebook all are using the Internet to build new kinds of communities and businesses. All three are essentially collections of Web services. But they’re able to manage their enormous communities and are constantly adding new features that their users pretty much love.

uTest: You’re an established figure in the testing space now, but you’ve spent much of your career on the development side of things. What (if any) were some of the biggest challenges you faced transitioning from dev to testing? And conversely, what advantages has your prior dev experience given you in your testing career?

NS: When I began making Web sites in 1999, I manually tested each site I built. Sometimes I worked with a QA team and sometimes not, but I have always checked my own work before delivering it, regardless.

As the Web grew, my projects got more complex. Tasks like making sure there were no broken links on a site quickly became infeasible to do manually. Soon even manually viewing every page on a site became unworkable as well.

So I learned some Perl and suddenly I could do static analysis. Then I learned to parse and alert on the output from tools like HTMLTidy, the W3 CSS Validator and later JSLint. After that, the number of bugs in my code dropped dramatically.

At first most people thought I was a little nuts for taking this approach. In 2001 almost no one cared if a site was robust. Web sites were cheap, throwaway things — “brochureware.” But a few years later along came Google Maps and GMail and now Web sites were first-class applications, expected to perform well and last a long time.

After that I just found myself more and more involved in helping teams produce quality Web sites. I think once everyone realized testing Web sites was important, I just naturally got drawn into that because I was interested in the domain. There aren’t a whole lot of Web developers who are really deeply passionate about testing. Or if there are, I haven’t met most of them yet!

uTest: What’s the biggest difference between testing at a startup versus testing at a larger company like Etsy?

NS: That’s a funny question to me because I still think of Etsy as a startup. Most of the companies I’ve worked for were much larger and more bureaucratic.

Anyway, I think that for a very early stage startup without a (good) product in production, time to market is the only thing that matters.

That said, it is worthwhile to take the 20 minutes to set up Jenkins and have it run your tests and some static analysis every time anyone checks in code. Beyond that I would not spend any time or money on infrastructure until the first (successful) version of the product had shipped.

For a startup with a successful product, paying down technical debt becomes paramount. Especially the first release tends to be done with very little testing and a lot of last-minute hacked-together solutions. Emergent problem solving is good, really good, when all that matters is getting to market. Without expedient solutions you might never get to market at all. But that’s only the first phase in a Web site’s life cycle, and it’s important not to hold on to those initial strategies once their usefulness has been outgrown. Improving test coverage and refactoring for comprehensibility are tightly coupled activities, and both are very important for the long-term survivability of a complex piece of software.

uTest: We find that a lot of people in the software business – particularly those in testing – are reluctant to adopt the agile methodology. Was this the case for yourself and your peers at Etsy? And what advice do you have for testing teams that might be resisting the move to agile?

NS: I like to move fast and deliver quality and I have always tried to work for companies that share that philosophy. At Etsy they hired a team who believe that the right thing is to experiment a lot in the name of resiliency and adaptability. I think that hiring the right people was key.

I think when people are happy and respect each other, they’re likely to want to do what’s best for the team. So assuming you have a culture like that and that you’ve identified rapid iterations as the right workflow for your team, it should be just a matter of trial and error and hard work to get that kind of process in place.

I do think all those elements have to be present for success: the right team, the right values and a willingness to repeatedly fail in the interest of progress. If an organization doesn’t have all those attributes, that’s a problem that needs to be addressed before it will be possible to move forward with any kind of process improvement.

uTest: A big part of your STPCon presentation dealt with the idea of asking engineers to assume more responsibility for testing their code. First off, explain why this is a good idea. And second, which group would be more reluctant to accept this idea: testers or engineers?

NS: Tests are no more and no less than executable documentation. When writing new code, it makes sense that it should be documented by the person closest to it: the author. That’s my rationale for developer testing.

Comment “doc blocks” and wiki pages, are always found to be inaccurate to some extent. By contrast automated tests fail noisily whenever they go out of date. So with regard to documentation, automated tests are uniquely advantageous in that (given a CI server) you at least have the option of keeping your documentation up to date whenever it starts to drift away from reality

Etsy was a comparatively small company when the decision to embrace developer testing was made. So by the time I got there, they had a great product and their code was looking pretty clean. Given this great environment, they had no problem recruiting more engineers who also felt strongly that testing was a good idea.

Editor’s note: Continue reading Part II.

In Part II of our Testing the Limits interview with Michael Bolton, we get his opinions on the corporate fear of Rapid Software Testing; the challenges of coaching testers; the similarities between testing and sex; why programmers should learn to test; writing about testing; negotiating with the other Michael Bolton and more. Did you miss Part I? Then you can find it here.

uTest: It seems that your primary audience is fairly open the principles of RST. But for those who aren’t, what are their main objections? Have the skeptics made you rethink or refine any aspect of RST?

MB: The primary objection appears to be that people are reluctant to give up their safety blankets. They’re frightened of reducing bureaucracy and paperwork, because those things appear to make testing work more legible.  But that’s an illusion:  bureaucracy and paperwork make illusions about testing work more legible. (Have a look at James C. Scott’s Seeing Like a State for a wonderful description of legibility and how the neo-Platonists and the high modernists get it wrong for everyone except—sometimes–for themselves.)  People seem frightened to invest in skill, because even though skillful work can be very well documented, it doesn’t necessarily leave a familiar kind of paper trail.

People seem frightened of telling a different kind of testing story in a different way, because managers have become used to a certain kind of test reporting.  I think most companies are frightened of finding out what’s really going on in a product or a project, because the truth would be pretty horrifying in a lot of cases.  People are frightened of acknowledging the role of skill and tacit knowledge in technical work, because they’re frightened of having to replace people who leave. The assumption there is that the new hire only learns through documentation, but that’s clearly false; we learn through social interactions, by interaction with our products, and by doing meaningful work for which we’re held responsible.

One more thing:  many people whose jobs are titled “quality assurance” seem unwilling to give up their perception of their own authority. People want to be “influential”, to “own quality”, to “be the gatekeeper”, to “speak for the customer”.  I don’t have authority over the project unless I’m the project manager.  As a tester, I don’t have that authority, nor can I claim to speak for the customer more credibly than anyone else. I’ve met testers who believe that it’s their prerogative to tell programmers what to do or how to do it.  I recommend that such testers reflect on how they feel when they’re told what to do by people who’ve never done testing work.

As for what we’ve done to rethink or refine things, that’s a continuous process. James and I are currently working on a few important threads. I learned a great lesson from Jerry Weinberg in his Problem Solving Leadership workshop a few years back.  Someone accounted for a less-than-successful attempt at problem solving by saying “The complexity of the problem screwed us up.”  Jerry peered over the top of his glasses and replied, “Your reaction to the complexity of the problem screwed you up.”  So in the delivery of the class, we’re trying to focus on helping testers to see the simplicity behind complex situations, so the testers can be more confident in their ability to deal with them. On the other hand, we’re also focusing on helping testers to see the complexity behind apparently simple situations, so the testers have the wariness that they need to avoid being fooled too easily.  We’re also working on showing how to use Rapid Testing approaches in highly formalized or highly regulated environments, pointing to our experiences in financial and medical contexts. Excellent format testing (which is often confirmatory or demonstrative) begins with excellent informal testing. Consistent with that, we’re trying to make people aware that the bulk of their work is exploratory in nature, even though they might not have noticed it.  “Formal testing”, which often takes the form of dog-and-pony shows for regulators or auditors, is one thing. Testing to make sure that people don’t die or lose a fortune or pile on some horrible risk is another thing. Excellent testing of the former kind of testing depends on excellent testing of the latter kind.

uTest: If the Context-Driven School of Testing were a traditional school, there’d be a lot of students on the waiting list. In other words, the community is growing rapidly every day. What’s surprised you the most about the emergence of the Context-Driven community thus far?

MB: How slow it’s been, and how small it remains; how quick it’s been, and how big it’s getting. What’s surprised me mostly is that context-driven principles still seems non-obvious to other people. But I suppose they’re context-oblivious, or context-specific, or context-imperial as James has pointed out here.

uTest: We were interested to learn that you – along with your friend and colleague James Bach – spend a lot of time coaching testers over IM, free of charge. What sort of questions do these testers come to you with? Are they generally practical or abstract in nature? And what’s been the most significant thing you’ve learned from this experience?

MB: The questions are usually pretty specific.  They tend to be focused on how to relate to the rest of the project team, especially managers and programmers.  The general form of the answer is this:  learn how frame your testing; tell the product story; tell the testing story; and tell the story about the quality of your testing.  And practice doing that.  Then most of those problems go away.  The general form of answering the question is to give the testers model problems to solve, and then to help them work through those problems. I often get questions about how to deal when there’s no (or bad, or incomplete, or inconsistent) documentation; we help people understand that documents and artifacts are only one way (often a very poor way) to learn about something. I often get questions about how to estimate testing effort.  I’ve answered that here and here, but people learn better when they experience the dialogue.

uTest: A noted testing figure was recently quoted as saying “Testing is like sex. If it’s not fun, then you’re doing it wrong.” First, can you guess who that certain someone might be? Second, do you agree? And third, what are some simple steps testers can take to make testing more like sex (i.e. fun)?

MB: I’ve said this myself for years, but pretty circumspectly. I’m not claiming I was the first to say it; to me, the joke is so obvious that claiming credit for it would be embarrassing even for the first person who said it. But yes, absolutely I agree.

As for the third question… oy vey, it’s going to be tough to answer that without getting into lots of trouble. Mischievous fellow, aren’t you?  Okay, let me give you a few things that you can say about both sex and testing:

a) Don’t ever, ever use it to hurt or embarrass people.

b) We shouldn’t be scared to learn about it, because if we are, many people will be inclined do it irresponsibly and unsafely, which leads to it being not fun at all.

c) It’s how the human species has survived this long, although not without some cost.

d) Don’t let your eyes make your decisions for you.  The good-looking ones aren’t always the best, or the best for you.

e) Whatever your thinking about human sexuality might be, ideas weren’t meant to be monogamous, and they weren’t meant to live alone. Let ideas be frisky, and let them play with each other and make new ideas together. That goes double for ideas about testing.

There is one other thing that I have said before about the relationship between testing and sex.  Testing certification the way it’s currently practiced goes mostly like this.  You go to a room, typically in a hotel. You greet someone you’ve never met. You spend a few hours together, answer a few awkward questions.  Then you get up and leave a goodly sum of money on the table, they say “Thanks, you were great,” and then you never see each other again.  Does that remind you of anything?

uTest: You recently wrote a blog saying that testers should learn to program. Conversely, should programmers learn to test? Why or why not?

MB: Programmers should learn to test. They do learn to test. Most programmers attempt to run the code they’ve written and look at it critically to some degree. Great programmers do that testing in a really diversified and yet focused way. What I’d like to get rid of is the idea “I program, I test, and therefore I know all I need to know about testing. I’m good.” To test really well requires study of many ways that things can go wrong, and many ways in which we can be fooled, and many approaches for defending ourselves against them.  The best programmers I’ve ever worked with were themselves excellent testers, but they were also aware of their own limitations. They were really respectful of good testers, and, of course, good testers were respectful of them. It was a virtuous cycle.

It’s hard to figure where this goofy idea “programmers can’t test” came from.  It’s ridiculous. It’s probably a misstatement of the idea that programmers have blind spots with respect to their own work, but that’s just like everyone else in every discipline.  If you think a programmer can’t test, give him or her someone else’s code or product to look at; they’ll usually do really well at finding problems and inconsistencies.  I invite organizations to send programmers to the Rapid Testing classes, and they always do fine. Not surprisingly, they often find stuff that testers tend not to find. For example, during testing exercises, the programmers often go looking for source code, and then they find certain problems there that other testers don’t find.  But like everyone else, programmers get trapped by something in the exercises, and therefore like everyone else (including me), they learn something new about testing.

To me, the real issue, the real reason why we need testing specialists is this quite gross generalization:  in my experience, most programmers more passionate about creating programs than about exploring them and all the stuff around them.  They’d probably be fine at it, but it’s not where their centre of gravity is.  That’s okay; that’s what we testers are here for.

uTest: You’ve built quite an impressive readership on your blog dating back to 2004. We always urge testers in our community to start a blog themselves, so we’re wondering if you could offer them some quick advice on how to get started, how to keep it fresh and how to encourage discussion and debate among their fellow testers.

MB: Write from experience.  Write about the things that interest you, both inside and outside of testing.  Write from your head and your heart.  Read and watch and observe and synthesize like crazy to bring new stuff into the craft from whatever you do or whatever you see.  And once you’re written about it, let people in your community know you’ve written about it.  Use Twitter and LinkedIn and Facebook for that.

uTest: Similarly, you’re also constantly being introduced to new personalities in the testing space through your blog, your courses and presentations. Any rising stars of software testing that we should keep an eye on?

MB: The Weekend Testing community worldwide.  Pete Houghton. Markus Gaertner. Goranka Bjedov. Ben Yaroch. Griffin Jones. Anders Dinsen. The Dutch Exploratory Workshops on Testing community. Lynn McKee and Nancy Kelln in Calgary. Just this month, Eric Jacobson did a killer job at his first conference presentation, at STAR West. I’m really proud that Eric and many of these folks are Rapid Testing alumni. Those are just a few of the stars that I see rising. Google and Twitter will help you find them.

uTest: Obligatory Michael Bolton question: It’s looking like you’ve made quite a name for yourself on Twitter and in the press for not relinquishing your handle to the other Michael Bolton. Hypothetically, what would it take to wrestle that handle away from you? We’d be more than happy to act as a broker.

MB: Enough money for me to retire and to pay you the commission that you would so richly deserve for swinging the deal. Not that I want to retire; I’d just like to have enough money that I could. Oh, plus an apology for adopting my name.  His isn’t really Bolton, you know.  It’s Bolotin.  Russian, I believe.  He changed it because he was sure that radio DJs would never pronounce it correctly.

uTest: Where can we expect to see Michael Bolton next and what will you speaking/presenting about?

MB: I’ll be at EuroSTAR in Manchester this year, talking about measurement in a new one-day workshop, and talking about dashboards in a track session.  I’ll be doing public Rapid Testing classes in Nieuwegein, London, and Oslo before the end of the year, and there’s a public class penciled in for Finland in January.

Rapid Fire…..

Better beer: Canadian or German?

Duh. Guinness.

Favorite Frank Zappa album?

One Size Fits All.  Sofa, the instrumental, is a heartbreakingly beautiful piece of music.  But I also like the vocal arrangements on the mid/later period stuff—You Are What You Is, and Joe’s Garage.

Exotic pet of choice?

I used to be a dog person; then I was a cat person.  Now that I have a daughter who’s obsessed with horses, I’m seriously into low-maintenance mode.  Can I choose something stuffed?

Favorite inventor?

I’ve met Ward Cunningham.  He invented FIT—the requirements tool that’s at the conceptual core of FitNesse—and the Wiki. That’s a pretty good track record, plus he’s a great soul. Gotta consider what Les Paul did for us, though.

Mac or PC?

I’ve been around PCs for a long time.  What I liked about them at first was the idea that I could comprehend them, that I could control them.  That’s totally out the window (Windows?) now; both platforms are absurdly complicated and neither of them give me any sense of control.  Give me TTY, the anti-GUI!  That said, if only they made a Mac with a TrackPoint mouse, Apple would get all of my money.  Every time I use one of those touchpads, I feel like a kid pushing a toy car across the floor… vroom, vroom!  Give me something where can type AND point without taking my fingers off the home row!

MB: One more thing:  these have been great questions, and I very much appreciate the opportunity to chat with you.  I’ve really enjoyed reading other interviews in the series.  So thanks to you and to uTest!

Editor’s Note: Thanks for reading – and thanks to Michael Bolton for taking time out of his busy schedule. Until next time…happy testing!

Our Testing the Limits “reunion tour” rolls on this month with Michael Bolton, back for another lively session of Q&A. Michael is best known as the founder of DevelopSense, his Toronto-based testing consulting firm, and as a leading figure in Rapid Testing and the Context-Driven school of testing. In short, he’s one of the industry’s most highly regarded writers, speakers and teachers – and it’s a real pleasure to have him back. For more on Michael, be sure to check out his website, blog or follow him on Twitter.

In part I of our healthy two-part interview, we get his thoughts on test cases not being related to testing; the sub-par debate skills of testers; the quality chain of command; objections to Rapid Testing and much more. Be sure to check back tomorrow for Part II. Enjoy!

uTest: It’s been almost two years since our last interview. Where does the time go? We’ve followed you pretty closely during that time (on Twitter, don’t worry), but for those who haven’t, what have they missed? New publications? New courses? New ideas on testing? What’s new with Michael Bolton?

MB: I’ve been traveling like crazy this year, and I’m booked pretty heavily through the end of the year.  I’m beginning to set up my schedule for next year—so if people would like to schedule an in-house class, now is a great time to ask.  For new publications How to Reduce the Cost of Testing, a new book edited by Matt Heusser and Govind Kulkarni, has just been released. I’m pleased to say that I’ve got a chapter in there, with a number of other members of our community.

I don’t specialize in new ideas in testing so much, but rather in refining and reframing ideas we’ve had for years in more specific and, I hope, more useful ways. The other thing that I love to do is to bring ideas from elsewhere into testing.  Currently I’m fascinated by the work of Harry Collins, who studies the sociology of science and the ways in which people develop knowledge and skill. Tacit and Explicit Knowledge is his most recent book; The Shape of Actions is older.  I’m most interested in the idea of repair, which is Collins’ notion for the ways in which people fix up information as they prepare to send it, or as they receive and interpret it.

As an example, I’m 5’ 8” tall.  If I ask you how tall I am in centimeters (and provide you with the ratio of 2.54 centimeters to the inch), you’ll probably do a little math in your head to translate 5’ 8” into 68 inches.  If you do that, it’s because you have tacit knowledge that a foot is 12 inches, and it’s quicker to do five times 12 in your head and add eight than to work it out on the calculator.  Then you’ll report that I’m 173 centimeters (or 172), rather than what the calculator tells you:  172.72.  If you round the answer up or down to a whole centimeter, it’s because you have tacit knowledge that the extra precision is useless when my height changes more than that with every breath. The calculator doesn’t know that, but people often fix up the interaction with the tool, applying that kind of tacit knowledge without noticing that they’re doing it.  Collins argues that we give calculators and computers and machines more credit than they deserve when we ascribe intelligence or knowledge to them, even when we do it casually or informally.

My latest hobby horse is definitely not new, but I’d like to have a go at it anyway.  I’d like to skewer the idea of the test case having any serious relationship to testing.  Test cases are typically examples of what the product should do. That’s important; we often need examples to help explicate requirements and desires. But examples are not tests, so I’d like to call those artifacts example cases or examples rather than test cases. They’re confirmatory, not exploratory; checks, not tests. Brian Marick has written a lot about examples; Matt Heusser has too; so has Gojko Adzic. James Bach has been railing about test cases for a long time.  Often test cases are overly elaborate, expensive to prepare and maintain.  They’d be even more expensive if testers didn’t repair them on the fly, inserting subtle variations making observations that the test case doesn’t specify.  Just as Collins suggests about machines, test cases get more credit than they deserve.  As Pradeep Soundarajan would say, the test case doesn’t find the bug.  The tester finds the bug, and the test case has a role in that.  Now: the development of checks and the interpretation of checks—those things require all kinds of sapience and skill.

A test, to me, is an investigation, not a bit of input and output for a function.  Yet people tend to think of testing in terms of test cases.  Even worse, people count test cases; and even worse than that, they count passing and failing test cases to measure the completeness of project work or testing work.  It’s like evaluating the quality of a newspaper by counting the number of stories in it without reference to the content, the quality of the writing, the quality of the investigation, the relevance of the report, whether a given article contains one story or a dozen, and so forth.  Counting stories would be a ludicrous way of measuring either the quality of the newspaper or the state of the world. Yet, it seems to me, many development and testing organizations try to observe and evaluate testing in this completely shallow and ridiculous way. They do that because seem to think about things in terms of units of production. Learning, discoveries, threats to value, management responses… none of these things are widgets. They not things, either, for that matter.

uTest: In a recent blog post, you wrote about the inability of some testers to properly frame tests, mainly because they haven’t been asked to. Generally speaking, what other qualities or skills do you find testers to be lacking in?

MB: Oh, dear, it’s so sad because there are so many gaps. James Bach, in a recent chat with you, identified rhetoric—how to speak and write clearly, articulately, and precisely—as something that many testers are missing.  Many testers aren’t so good at developing an argument (in the sense of a line of reasoning rather than a fight).  Many testers see obstacles to their work as problems for testing, when in fact they’re pretty much always problems for the project.  Testing helps to reveal those problems if you have an appropriate mind set.  (I wrote about that recently here.)   It seems that many testers, like many programmers, often lapse into the binary fallacy—something is either one thing or another, yes or no, true or false, pass or fail.

Tom Waits put it beautifully in an interview a couple of years back, when someone asked him about how he finds the truth of his character. “Truths,” he said.  “Truth isn’t a word that should be used in the singular. It should always be used in the plural.”  Rob Sabourin has been observing gaps in testers’ command of math, and how to apply it in testing.  As a craft, we seem to be aiding and abetting management in bad measurement; we need more people to study how to do measurement well. At the Pacific Northwest Software Quality Conference, I was delighted to see Kristina Sitarski—who studied anthropology in university—deliver an excellent analysis of interactions between herself and a tester she was pairing with, focusing on learning and perceptual styles.  If we want to be well regarded as a craft, we need more reports like this, based on studies of what we actually do.  We don’t need any more of the inept works of fiction and fantasy that you see in neo-Platonist process manuals.

uTest: We see that you were disappointed in the New Yorker iPad app, tweeting something to the effect of “it might have been tested, but it wasn’t fixed.” In your view, is quality lacking in the mobile space? If so, why? Is mobile inherently more of a testing challenge than its web and desktop cousins?

MB: When I tweeted that, the New Yorker app was crashing for me within a few moments of starting up.  I had to wait a few weeks while that got sorted out. The magazine as delivered to the iPad is typically on the order of 150MB in size, where (comparable product heuristic!) The Economist is on the order of 3MB.  The product would frequently crash during a download, and wouldn’t pick up where the download had left off.  Even if downloads were successful, downloading the New Yorker every week would wipe out the base amount of data consumption on my mobile billing plan.  It would be cheaper to buy the dead tree version of the magazine.

Certainly there’s a great deal of extra complexity to be dealt with in the mobile space, when we look at the number of different systems and functions through which a given bit of data passes, or the enormous number of platforms on which people want to run apps.  Before Windows came along to abstract the hardware, there were drivers for each video card, each printer, each mouse, each network card times each operating system.  But then each application program came with special drivers to talk to each kind of hardware.  Developing and supporting all that stuff was completely nuts. Since there’s a perception of lots of opportunity and lots of money in the mobile space, there’s a gold rush and lots of people are heading for the Klondike. Now there are competing mobile OSs, times all those versions of those OSs, times all those handsets and tablets and mobile browser versions and interconnecting apps and services.  So in a way, we’re back to the late 80s and early 1990s, back in the DOS days, when I first got involved with programming and support and testing.  Hey you kids, get out of my yard!

uTest: You’ve said before that “decisions about quality are inherently subjective” and that “testers are not responsible for making decisions about quality, but rather for informing decisions about quality.” So our subjective question for you is this: Should testers be responsible for making decisions about quality?

MB: Like everyone else, testers should be responsible for making decisions about the quality of their own work. But we already have lots roles and titles for people who make decisions about the work of other people:  we call them “product manager”, “program manager”, “project manager”, “product owner”, “director of development”, “vice president”, “CEO”.  I urge testers: You want to manage a project? Become a project manager.  I urge quality assurance people: You want to assure quality?  Make sure you have real, final authority over the product and the people who produce it. That is, become a manager. You’re not a gatekeeper of quality; you’re a speed bump on the road to quality.  (Speed bumps are also known as “sleeping policemen”.  That’s apt.)

uTest: You’ve been traveling the world the last few years teaching courses in Rapid Software Testing. In your experience, are certain regions of the world more open to this testing mind-set than others? If so, why do you think this is the case?

MB: It seems that Rapid Software Testing has a lot of traction in certain circles in northwestern Europe, especially Sweden, places where there seems to be a deal of room for intellectual rigor and independence at the same time.  Rikard Edgren had an interesting explanation for the success of rapid and context-driven testing there.  He said that the Swedes in particular believe really strongly in the social contract, that people are interdependent, that society should take care of everyone, that things like education and health care are rights, not privileges, and that people should reasonably expect get them at a high level of quality. And to pay taxes for them.  Although, he said, the Swedes aren’t crazy; no one likes paying taxes, but it’s part of the deal if you want all this other good stuff.  So there’s this sense of mutual support and strong government, and the Swedes (broadly speaking, of course) believe in that… but they don’t like other people telling them what to do.  When Rikard say that, I thought it fit really well with what we espouse:  We’re all in this together.  We give ourselves and each other freedom to do the right thing and to screw up.  We also take responsibility for our actions, and we take responsibility for taking care of each other.  We work collaboratively, but we recognize that few people like being under someone else’s thumb.  That kind of freedom combined with responsibility allows people to blossom, I think.  I’d argue the Baltic countries have been ahead of North America for a while in terms of politics and social issues. Rapid Testing is popular in New Zealand and Australia too, to some degree. It’s that spirit of independent interdependence, if you will.  James is excited about Estonia, too, but I haven’t been there.  Yet.  The UK has this emerging group too.  So we’re seeing shoots coming up through the snow.

uTest: In our last interview, you mentioned specifically that New Zealand and Scandinavia were producing some excellent testers with fresh insight and new ideas. Have your travels uncovered any other areas of testing innovation?

MB: When I do, they’re almost always local geographical pockets, or some skunkworks when they’re inside larger organizations.  Steve Green runs this cool little testing services company in England that focuses on skilled testers and very fast turnaround. Paul Holland has been doing rapid testing for years at Alcatel-Lucent, with excellent results. Pradeep Soundarajan is running a testing services company in India that specializes in rapid testing approaches. That company is profitable in its first year.  Darren McMillan has done some really great work in explaining the ways in which he’s been using mind mapping. Those are only some of the prominent people. Alas, NDAs, company confidentiality policies, modesty, and fear restrain people from saying too much about what is and isn’t working.

In addition, everyone does rapid and exploratory work to some degree, but I’ve never seen process enthusiasts who have actually observed processes closely enough to notice that.  If you want to observe process, you have to observe people.  Most process enthusiasts I’ve seen observe artifacts, rather than real work in action.  The Social Life of Information talks about how quickly we could find blatant errors in our process models if only we took a more diversified anthropological approach.  Don’t get me wrong; I’m at best a dilettante in that stuff myself.  But I think we have to start getting serious about it.

Editor’s note: That’s it for now. Be sure to check back for Part II tomorrow!

In part II of our latest Testing the Limits interview with James Bach, we discuss what it would take to get him back on the client side of testing; free testing tools that he’s currently using; required reading for new testers; his upcoming speaking/book tour, sniper rifling in Middle Earth and more. Enjoy!

If you missed part I, you can find it here.

uTest: Your brother recently made the leap to the client side at eBay….will we see James Bach cross back over as well?

JB: Well, maybe a super-villain out there buys out my company and hires me full time to be chief tester for their front corporation. I would wonder at first why I was being paid three million dollars a year to test a variation of Angry Birds, but the money would blunt my natural inquisitiveness (I have my wife’s medical bills to pay, after all).

Only after a British secret agent drops in (literally) at my office to confront me with the horrible truth (that the program I was testing was actually a doomsday device) would I and Cortana (an AI construct rendered as a perky young female that switched sides after spontaneously evolving a sense of ethics) manage to skillfully mis-report critical bugs prior to sign-off. This would lead to the destruction of the doomsday machine seconds after our daring escape through the test data exhaust port.

But I doubt that will happen.

Basically, it would take an unreasonable amount of money to make me give up my independence.

uTest: You wrote a great blog post recently on the subject of tool vendors – specifically, how they can avoid your wrath. We’ve also heard you recommend free tools in the past (“because they can be freely abandoned”). What tools (either paid or free) have you discovered recently and how have hey helped your testing efforts?

JB: I’ve been playing with “R”. It’s a free statistical analysis system.

There are several books on it. I love tools that help me work with complicated data.

The tool that has helped me most recently is my Canon solid state video recorder combined with my micro tripod. I think I will never do professional testing again without a camera rolling. It’s so helpful to be able to roll back the film and watch what I just did in order to reproduce a difficult problem.

uTest: You’ve been following the testing of medical devices rather closely, recently highlighting that the FDA had come to recognize the value of exploratory testing. Are there other particular industries that need to make this same realization? If so, which ones?

JB: Banking, insurance, and aerospace come to mind. I’m working with a major world bank as well as a major insurance company right now, so that feels good. They are both “standardizing” on Rapid Testing. Which means that thinking for one’s self is becoming their standard.

uTest: We know you’re not much of a proponent of testing textbooks, but we know you’re a big fan of books and literature in general. Do you have a “required reading” list for new testers? If so, what are some of titles we could expect to find on it?

JB: The main books are by Jerry Weinberg:

• Perfect Software
• Introduction to General Systems Thinking
• Quality Software Management (Vols. 1 and 2 especially)
• Secrets of Consulting
• Exploring Requirements: Quality Before Design

And by other authors:

• Tools For Critical Thinking
• The Duck that Won the Lottery
• The Black Swan

uTest: You recently held a pretty unique testing competition at CAST last month. We’ll let readers check out the specific details on your blog, but we wanted to ask if you planned to hold more in the future. If so, what are some of your ideas for new testing competitions? We promise not to steal them.

JB: I don’t run competitions very often. I’d like to, though. It’s an interesting laboratory for testing practices. I think our industry needs more of them. We need a few high stakes competitions– something with big prizes.

I’d like to see a competition that focuses on test reporting (including test strategy and logistics), rather than just bug reporting. Everything produced should be put into the public domain, so that we gain more and better examples of testing practices and artifacts. And there should be a specification analysis competition, too.

It would be fun to do head-to-head testing competitions with live play-by-play commentary.

uTest: What scares you most about the testing industry today?

JB: The fact that the fake testing industry seems bigger than the real testing industry; and the fact that incompetence in testers is not only tolerated but vigorously encouraged in many organizations; and the new ISO testing standard– which is a tool that will be used by incompetent people to justify fake testing.

Software products are more complicated than ever. People will die because testing as a field won’t grow up and put away childish chew toys such as T-Map.

uTest: The last time we spoke, you had just published Secrets of a Buccaneer Scholar. Are there any other book projects in the works? If so, please give us a sneak preview, along with an estimated date of arrival.

JB: I’m working with Anne-Marie Charrett on a book about online test coaching. We don’t have a date, but I suspect late next year.

uTest: We were disappointed to learn that you wouldn’t be speaking at STPCon down in Dallas. Where and when is your next speaking appearance and what will you be discussing?

JB: I’ll be at the Star West conference, talking about critical thinking, exploratory testing, and formal testing.

Rapid Fire:

• Medieval weapon of choice: Barrett M107 Sniper rifle. Did they have those? Okay..Longbow, then.
• Secret Talent: Resigning.
• Programming language of choice: Perl.
• Star Wars or Lord of the Rings? Well, I served with the forces of Gondor in the battle before the gates of Mordor. Does that answer the question? I was the one with the sniper rifle.
• You wouldn’t be caught dead where? Working for The Man.

 Editor’s note: We hope enjoyed our latest Q&A with James Bach. If you have suggestions for future guests, send them to marketing@utest.com. Until next time, happy testing!

Back for another session on the Testing the Limits hot seat is James Bach – one of our most popular guests and one of the most widely recognized figures in testing today. A prolific author, speaker, coach and consultant, James has been disrupting the testing industry since 1987. For more on James’ background, his body of work and his testing philosophy in general, you can check out his blog, website or follow him on Twitter.

In part one of our latest interview, we get his thoughts on his previous life as a developer; how testers should interact with engineers; the biggest waste of time in testing today; the skills that most testers lack and more.

uTest: In a recent lecture you said that you hated being a developer, saying it was like completing 25 crossword puzzles every day (i.e. boring and repetitive). What’s the one piece of advice you would offer to those are who are thinking of making the transition from dev to testing?

JB: I advise: make that transition– but please, study testing as you do.

Testing is not just another programming problem. Yet, too often, a programmer mentality sees testing as just the process of manipulating software. Sometimes, programmers dream of robot armies to do their “testing.” The quest for the clever tool then eclipses the mission of excellent testing. This is a little like trying to invent an android that can talk to your wife so that you don’t have to.

You can use all your tech skills, programmers. It’s fine to write software. But what testing is really about is rapid, deep learning. To do that, you have to get your face, and all the rest of you, right up close to that product. You must wrestle with the product– yes– by hand. Or as we like to say where I come from: you need to do your testing sapiently.

uTest: In that same presentation you also talked about the importance of making friends with developers. Why is the tester-developer relationship so adversarial at times?

JB: I don’t know, why does my wife not like it when I report defects in her? People are strange that way…

Seriously, it’s easy to see the dynamic. Anyone who creates a piece of work and submits it for judgment is going to feel judged. That’s not a pleasant feeling. And the problem is compounded by testers who glibly declare that this or that little nit or nat is a “defect,” as if anything they personally don’t like is a quality problem for everybody. It is further compounded when programmers and testers are separated by large distances or other communication barriers, not to mention the process barriers.

Even though technical people are renowned and sought after for their social skills (I understand United Nations programmers were just about to solve the Arab-Israeli situation a few months ago, had it not been for the need to calm tensions between the Koreas.) testers and developers are people who see the world very differently. It’s a special challenge to relax and smile when a bug report seems to have been ignored.

Personally, I make a set of explicit commitments to any developer I work with on a project. I write them down and deliver them. This seems to help.

uTest: You’ve said many times that belief is a sin for testers – something we assume you’ve learned from experience. Was there anything in particular that led you to this truism? Or was it just years of experience?

JB: It’s not a matter of experience, but reason. Our job is vigilance. To lose vigilance is to abdicate our responsibility. Vigilance, in testing, means being a good skeptic. We must reject certainty in any form. We’re the Knights of May Be. To believe is to cease questioning; to fall asleep at our posts.

I use lots of information. I work hard not to believe it. This is my discipline.

uTest: What’s the biggest waste of time in testing today?

JB: Useless test documentation is a candidate for that– frequently compounded by wasting huge effort trying to automate the simplistic documented checks and keeping that automation running. Test metrics are also a waste, most of the time.

Of course you know I think that certification is a waste– a tape worm parasite, living off our flesh– but probably not the biggest waste.

Since testing maturity models bring all this together into a grand festival of waste, that has to be closer to the biggest.

That new ISO testing standard is a magnificent waste, too. It will not help anyone do anything worthwhile, I predict.

Okay, here is the biggest waste as I see it: All those things I mentioned conspire to keep our craft in a childlike state of fantasy and ignorance. Managers are coddled by cynical and/or craven consultants, like self-indulgent perennially drunk rich kids, instead of being doused in cold water and told to sober up. And this has led to a world where technology is routinely poorly tested, medical devices routinely recalled, security breaches routinely routine, and terrible sums of money are wasted cleaning up messes that didn’t need to happen in the first place.

That’s a lot of waste.

uTest: We were interested to learn that you spend a fair amount of time coaching testers over Skype…for free. What sort of questions do these testers come to you with? Are they generally practical or abstract in nature? And what’s been the most significant thing you’ve learned from this experience?

JB: Yeah, I like Skype coaching. I charge for some of it, but most I do for free. When I do it for free I tend not to schedule it in advance (people just catch me online) and I require that the student allows me to publish the transcript of our sessions. In this way, I’m gathering material for my test coaching book.

Sometimes the coaching is about answering questions. Mostly it’s about testers getting taking on challenges from me and working through them.

One of my goals is to create a change in the testers minds so that they can begin to perceive and process ideas that have been obscure to them, up to that point.

Sometimes it’s quite concrete. Other times it’s conceptual. This exchange happened yesterday:

—————–

James Bach: Whom do you serve?

Sue: The business.

James Bach: No, that’s not a person. You serve people, not concepts.

If you do a bad job concepts will not fire you, people will. I mean what people in your organization depend upon you to do your work? Who will be upset if you do it badly?

Sue: hhhmmmm you are making me think about this…if wrong insurance rates are produced; legal would be involved. But, I never considered them my client.

James Bach: You don’t have a manager? You don’t work in a group that has a leader? You don’t have co-workers?

Sue: Yes, I have a manager and co-workers.

James Bach: Is there some reason why you don’t consider your manager to be your client?

Sue: I never really did.

————–

Here, I’m working with Sue, who is a very experienced tester, to help her get a whole lot clearer about what she actually does for a living.

I asked her to write me a list of what she’s good at, and what she’s not so good at. I noticed her list was remarkably vague. This suggested to me that she wasn’t used to thinking about testing skills and issues in any pointed way, so I decided to use a common coaching pattern: drilling down.

I asked her to list skills of testing. I picked one item on her list (“mission setting”) and asked her to get specific about what that meant. This is when the exchange, above, happened. It was like walking into a very cluttered closet that turned out to be a kitchen connected to a parking garage. Later in the conversation she gets clearer about her job, and we begin to sort things out. Like a lot of testers, Sue knows what she’s doing in a visceral way, but has trouble putting it into words. That’s partly because no one has pushed her before to learn a useful rhetoric of testing.

When I encounter conceptual confusion I sometimes roll up my sleeves and begin to offer some operational distinctions that I hope will make communication and thinking clearer. I try to lead the tester there by framing questions that make them aware of the confusion they are laboring under. My questions can be pretty aggressive, so I generally coach people who are highly motivated.

Other times, I will suggest an exercise to develop a shared experience that will help cut through the confusion.

Also, sometimes I just come out and say “Here’s a good way to think, why not try that?”

That’s how the sessions go, more or less.

uTest: On a similar note, is there a particular skill or trait that you find most testers to be lacking? And what can they do to improve this?

JB: There are several that I notice in particular. One is practical mathematics. Another is writing. Another is, um, reading. (My theory is that people who were forced by their parents to go to university often decide to swear off all educational development once they get out of school. But I don’t really know why more testers don’t seem to study their craft all that much.)

Maybe more vital than those: rhetorical skill. This includes verbal self-defense as well as story-telling skill. This is a skill that requires practice, and apparently testers aren’t getting it, because many of them look like frightened bunnies to me when I challenge them to give a professional test report. Rhetorical skill improves when you practice giving presentations to your colleagues. This is why I try to start peer conferences wherever I go. I just did one in Estonia, and we’re planning one in Romania, next year.

If there’s a single skill that testers HAVE, that I wish they had LESS of, it’s the skill of faking their work. All over the world, fake software testing (what they call “Possum Testing” in New Zealand) is an accepted, normal practice. It’s disgusting. But, of course, testers are often afraid that they will be fired if they stop doing it.

Editor’s note: Continue reading Part II.