Testing the Limits With James Bach: Part II

In part II of our latest Testing the Limits interview with James Bach, we tried something a bit different this time, crowdsourcing some ojamesbachf the questions from our uTest Community members. Additionally, James shows us his lighter side and which of his picks won the World Cup — of his heart. 

Be sure to check out Part I of our interview, if you already haven’t.

What is the biggest hurdle to testing you see testers struggle with? (Jeff S.)

JB: The hurdles that come with having no credibility. Gain credibility, and every external hurdle gets a lot smaller. If you ever find yourself saying, “I want to do good work, but my manager insists that I test in a stupid way, instead,” then probably the issue is that your manager thinks you are incompetent. Fix that. Then when you politely tell your manager to mind his own business, he will let you get on with your work in the way you see fit.

Do you see the tide changing for development teams modernizing their testing philosophy? Or is entrenched thought winning the day? (Jeff S.)

JB: I don’t know, really. I don’t do polls or anything. I can say that business is good for me and my colleagues, right at the moment.

Which area or skill is best to focus on first as a tester to build a solid foundation or understanding of testing? (Frank B.)

JB: I would say: general systems thinking (GST). See the book Introduction to General Systems Thinking by Jerry Weinberg. Within the realm of GST, I suggest: modeling. It’s vital to gain control over your mental models of products. Models are a prison from within which you test.

Continue Reading

Testing the Limits With James Bach – Part I

JamesBach150James Bach is synonymous with testing, and has been disrupting the industry and influencing and mentoring testers since he got his start in testing over 25 years ago at Apple. Always a great interview, James is one of our most popular guests and we’re happy to have him back for his first Testing the Limits since 2011. For more on James’ background, his body of work and his testing philosophy, you can check out his blog, website or follow him on Twitter.

In Part One of our latest talk with James, he talks about a future that involves a ‘leaner’ testing world, the state of context-driven testing outside of the United States, and why you’re “dopey” if you’re a manager using certain criteria in hiring your testers.

uTest: We know you don’t enjoy certifications when it comes to testers. In fact, in a recent blog, you mentioned that ‘The ISTQB and similar programs require your stupidity and your fear in order to survive.’ Do you feel like certifications are picking up steam when it comes to hiring and if they’re becoming even more of a pervasive issue?

JB: I don’t have any statistics to cite, but my impression from my travels is that certifications have no more steam today than they did 10 years ago. Dopey, frightened, lazy people will continue to use them in hiring, just as they have for years.

uTest: Speaking of pervasive problems, what in your opinion has changed the most – for better or for worse – in the testing industry as a whole since we talked with you last almost 3 years ago?

JB: For the better: the rise of the Let’s Test conference. That makes two solidly Context-Driven conference franchises in the world. This is related to the general rise of a spirited European Context-Driven testing community.

Nothing much else big seems to have changed in the industry, from my perspective. I and my colleagues continue to evolve our work, of course.

uTest: In a recent interview, you mentioned that you see the future of testing, in 2020 for instance, as being made up just of a small group of testing “masters” that jump into testing projects and oversee the testing getting done…by people that aren’t necessarily “testers.” Do you see QA departments going completely by the wayside in this new reality of a leaner testing world? Wouldn’t this be a threat to the industry in general?

JB: I’m not sure whether you mean QA groups, per se, or testing groups (which are often called QA). I don’t see testing groups completely going away across all the sectors of the industry, but for some sectors, maybe. For instance, it wouldn’t surprise me if Google got rid of all its “testers” and absorbed that activity into its development groups, who would then pursue it with the ruthless efficiency of bored teenagers mopping floors at McDonald’s (a company as powerful as Google can do a lot of silly things for a very long time without really suffering. Look at how stupidly HP has been managed for the last 20 years, and they are still, amazingly, in business).

Continue Reading

Testing the Limits With Microsoft’s Seth Eliot

Our Testing the Limits guest this month is Seth Eliot, the Senior Knowledge Engineer of Test Excellence at Microsoft. In this role, he focuses on driving best practices for development testing across the entire company. Prior to Microsoft, Seth had a successful stint at Amazon in addition to several startups. Apart from his professional background, Seth is one of the industry’s very best bloggers, writers and presenters. For proof, check out his blog or follow him on Twitter.

In this must-read interview, we ask him about testing challenges at Microsoft, including those of Bing and the new Surface tablet; the notion of testing in production (TiP); the most rewarding testing project he’s ever worked on; big data and more. Enjoy!

********

uTest: You’ve spent the bulk of your testing career with two of the most successful companies of all-time: Amazon and Microsoft. Unfortunately, most testers spend their careers with companies that – how shall we put this – aren’t so successful. In your opinion, is testing easier or more rewarding when the company is doing well? And what advice do you have for testers who might be working at a dysfunctional company?

Seth: The most satisfying testing job I ever had was a small startup in Pittsburgh called CoManage. It ultimately fizzled, but at the time we thought we were all going to be millionaires and I was consistently surprised to walk out of the test lab to see it was dark outside and I didn’t even know where the day had gone. If your company is dysfunctional, ask yourself if there is something you can do to turn it around and turn it into one of those dream successes. Learn new strategies and approaches for software engineering, change the direction, and bring new life to the company. At best you will be the hero, at worst you will have learned some valuable skills and lessons for finding that next job.

uTest: Prior to your current role at Microsoft, you were the Senior Test Manager for the team working on Bing, where you were primarily tasked with exabyte storage and data processing challenges. What were some of the specific testing challenges here and how were you and your team able to overcome them?

Seth: Yes, this is an internal system called Cosmos – a massively scalable, distributed data processing system. The technical challenge to put it simply is how do you test something so big and complex? I was fortunate to have a really talented team of testers who built out tools and monitors that enabled us to evaluate end-to-end test cases leveraging actual jobs being run in the production system. This led to us to finding the bugs that really mattered – those that affect real users. We were even able to prioritize our test scenarios based on the revenue impact of the user workflows and on the current pain points experienced by those users. This is an advantage of having an internal customer, but with good monitoring you can also approach this level of insight with external customers too.

uTest: We’d be remiss if we didn’t ask you about the recent launch of Microsoft Surface. First off, did you know about the project beforehand? If so, good job keeping it a secret. Secondly, what are some of the big testing challenges you’d expect to be associated with this project? The hardware? The Windows 8 OS? The touch functionality? We’d love to hear your thoughts on this HUGE Microsoft endeavor.

Continue Reading

Testing the Limits with Zynga’s Galina Kramer

Our Testing the Limits guest this month is Galina Kramer, the Senior QA Manager of Localization for Zynga. In this role, she is responsible for L10N testing of the company’s web and mobile games, including hits like CastleVille, Mafia Wars Hidden Chronicles and Poker. Galina has over 13 years of experience in quality assurance, with stints at Bill.com, Wells Fargo and others. For more on Galina’s background, check out her LinkedIn profile.

In this interview, we asked Galina about the challenges of testing high-profile applications in more than 16 different languages; what testing is like at Zynga; her criteria for hiring testers; switching industries and other topics. Enjoy!

***********

uTest: You’ve spent much of your QA career in the healthcare and financial sectors. Now, you’re focusing on a different aspect of QA (localization) in a totally different industry (social gaming). What’s been the most difficult adjustment you’ve had to make during this transition? Similarly, what advice do you have for QA professionals switching industries?

GK: The most challenging aspect of working at Zynga overall is the release cadence. I was used to weekly releases, but not daily. Having such a short release cycle brings its own challenges to functional QA, which I managed at Zynga for a year. Now that I am working in Localization QA, it is even more complicated since everything has to get translated first.

My advice to anyone switching industries – GO FOR IT! Try new things and have some fun!! I love working in the gaming industry and never thought work can bring so much enjoyment.

uTest: At Zynga, you oversee L10N for all of the company’s web and mobile games, which are currently supported in 16 different languages (with more to come we assume). What languages or markets have been the most challenging thus far and why?

GK: When we first started localizing our games, we had a lot of issues with staffing for certain languages and that was a real challenge. Now that we have established a robust process, we are smooth sailing. It helps having great partnerships with both internal and external teams in order to make things happen as fast as we need them to.

uTest: Not only are you supporting 16 languages, you’re also supporting countless versions of operating systems, mobile devices, wireless carriers and other factors. How is your team able to manage such a complex testing matrix?

GK: It’s all about prioritization and risk assessment. We shuffle resources on a daily basis depending on a priority and work together. Teamwork is the key, truly.

uTest: With so many users who are willing to report issues, it’s often assumed that in-depth testing isn’t needed in certain industries, such as social media and gaming. This is obviously not the case at Zynga. What is the company’s philosophy when it comes to testing and QA? And has it changed as they continue to expand their global footprint?

Continue Reading

Testing the Limits with Gerald Weinberg

Gerald M. WeinbergIn the latest installment of Testing The Limits we speak with Gerald Weinberg. Jerry has been practicing, teaching, lecturing, consulting, coaching and writing about software programming and testing since the 1950s. With decades of experience and accumulated knowledge he’s written more than 80 books and has dedicated his life to helping others be the best testers they can be – despite ever changing testing trends. In today’s Testing The Limits interview we’ll find out the biggest lessons Jerry’s learned over the years, how his books remain top sellers 20 years after their release and what the biggest issues facing testers today are. To keep up with Jerry visit his website or follow him on Twitter.

uTest: You’ve spent nearly 50 years working with computers and software in one form or another. Obviously, much has changed during this time, but surely some things have stayed the same. In your experience, what is the most notable “constant” in the world of software?

GW: The infatuation with the latest fad which is supposed to “increase productivity” by some arbitrary amount, with no clue as to how that “productivity” is measured. For the most part, these fads are usually a new set of names for old practices. Yet at the same time a segment of the developer population goes ga-ga over the fad, a much larger segment doesn’t even attempt to learn what is good about the fad. The majority of developers are working the same primitive way their predecessors did 50 years ago, just using more machine power to do it.

uTest: In a recent “testing roundtable” discussion, panelists were asked what they considered to be the biggest weakness in the way companies test software. James Bach and Cem Kaner both cited a lack of testing skills and a lack of means to acquire such skills. Do you agree? As a long time consultant, what do you think is the biggest weakness in the way companies test software?

GW: To me, the biggest weakness is not considering software testing anything but a (barely) necessary evil. Testing is seen as something that could be done by a troop of monkeys, so serious testers are treated like third-class individuals. The lack of means of acquiring testing skills arises from this attitude, as do most of the other poor practices in the testing business. You treat people as if they are stupid, then they will wind up acting stupid.

uTest: A good debate has recently sprung up on the subject of whether testers should be able to code. As an expert in programming (and testing) where do you stand on this matter? Are coding skills (or at least a base knowledge of coding) a requirement for good software testing? Are they a nice-to-have? Or are they totally superfluous?

GW: You can be a great tester if you have programming skills. You can also be a great tester if you have no programming skills at all. And, you can be a lousy tester with or without programming skills. A great tester will learn what skills she needs to continue to be great, in her own style.

Continue Reading

Testing the Limits With Anne-Marie Charrett – Part II

In the second part of our Testing the Limits with Anne-Marie Charrett, we get her thoughts on the meaning of exploratory testing, the challenge of agile adoption, how to grow as a tester and more. Enjoy!

uTest: Certain industries appear to be ahead of the curve when it comes to testing practices, while others remain in the proverbial stone age. Is this an accurate statement? Or have testing practices evolved at similar pace across all industries? As someone who has spent time in many sectors, we’re interested to hear your thoughts on this.

AMC: I think companies that demand value from their testing are generally more receptive to new ideas and change in testing. I don’t think it’s fair to silo this into industries.

Take for example the finance industry, yes many large insurance and bank corporations are risk averse and resist change but not all. For example Barclays Bank are using coaching & Rapid Software Testing.

I’ve worked with small companies in R&D who you would associate with flexibility and being pro-active, yet they want very traditional, heavily documented testing processes. Often this is because someone did testing ‘once’ and this is what they did.

I’ve seen testing practices change within sectors too. For example, the telco sector in the mid 1990‘s were typically heavily documentation orientated. Often testing went on for years before a product was released. By the late 90’s and early 2000’s testing practices had to evolve as smaller companies with lighter and more flexible delivery approaches challenged this paradigm.

uTest: There’s a good debate right now on the true meaning of exploratory testing, with people like James Bach and Michael Bolton chiming in with their opinions. What is your definition of exploratory testing? And in your view, what is the most misunderstood term in testing today?

AMC: So many questions!! The beauty of Exploratory Testing is that it can mean different things to different people. Thats why there are so many different perspectives on it.

There are some core values to Exploratory Testing, namely that it’s an approach (not a technique), it’s simultaneous learning, design and execution and that it’s tester centric.

The latter ideal is something that I cherish and hold dear.  I think it’s essential that we take responsibility for the testing we do. This means each tester decides on their testing approach, what they test and when they’re done. Owning these decisions is what matures a tester, helping them become skilled, confident and motivated to excel in their testing.

Continue Reading

Testing the Limits With Anne-Marie Charrett – Part I

Testing the Limits with Anne-Marie CharrettTo kick off another amzing year of Testing the Limits we reached out to Anne-Marie Charrett, an independent tester who has worked for the likes of Mercury Interactive, IBM (twice) and Nortel – just to name a few. She also arranges for speakers to visit Ireland as part of Softtest Ireland and blogs about her testing experience and offers coaching at mavericktester.com

In part I of this month’s interview, we learn what motivates Anne-Marie to coach via Skype, what’s caught her interest lately, how her book with James Bach is coming and what the biggest mis-conception about testing is. Come back tomorrow for part II.

uTest: In terms of writing, speaking and researching, you are one of the most active testers in the business. So we’ll start by asking you this: What hot topics within testing have captured your interest recently?

AMC: 2012 has kicked off with a flurry of activity. Key topics appear to be, How we learn, Rapid Test Management and more recently James Bach has been looking Exploratory Test Documentation.

It goes like this. Typically we write tests and charters as artifacts for other people as evidence of work performed. But writing is a lot more powerful than that, it has the ability to assist in design (think brainstorming in mind maps). Exploratory Test Documentation is about changing the purpose of writing from an end product to a by product.

I also like the way new conferences and peer workshops are happening at a grass roots level, for example Lets Test in Stockholm. These are not necessarily big conferences, but ones that offer value to testers and that encourage participation. I hope that this will be the conference circuit of the future!

uTest: You’ve made quite a name for yourself as a testing coach; offering advice to testers free of charge via Skype. In your experience, what areas require the most coaching on your part? In other words, what does a typical tester coaching session cover?

AMC: Often testers come looking for coaching in a particular skill (e.g Test Automation), but many fail to understand basic testing concepts such as: “What is testing?” and “How do you determine bugs?”

Understanding testing is key to improving your testing skill.  After all, if you don’t understand something, how can you improve it?

Software delivery typically doesn’t allow for this type of introspection. Our jobs demand we focus on delivery, often to the detriment of how well we are doing our testing.

Coaching is the breathing space that all testers need to learn and grow.

In coaching I encourage testers to work through tasks to acquire skill. I’m there to guide and help them, but they need to work out the answers. That way, their learning experience is deeper and more meaningful and empowering.

Continue Reading

Testing the Limits With Richard Stiennon – Part II

In part II of our Testing the Limits interview with security expert Richard Stiennon, we get his thoughts on where companies are least prepared for a security breach; the explosion of mobile related threats; the future of cyber security and much more. If you missed the first installment, read Part I now.

uTest: What’s the one piece of security advice you would have for companies that are in their infancy?

RS: Plan ahead. As you grow so does your “target area.” When you launch your product or service you may not be in the cross hairs of a bad actor. But you will be. Sometimes security builds in “friction” that can slow down customer acquisition. So you may not want to require 12 character passwords and SMS authentication at the beginning but know that eventually you will have to deal with account theft, breaches, and spammers. Have a plan so you can implement defenses quickly; and ideally before a damaging attack or breach. You don’t want to be in the position of the Sony Play Station Network that implemented protections *after* a series of attacks that cost $170 million to recover form.

uTest: You’ve said before that mobile will not require its own anti-virus systems. That said, it seems that mobile threats are multiplying by the hour. In your view, what’s the biggest security challenge in terms of mobile?

RS: Apps, apps, apps. VPNs, firewalls, and carrier filtering are going to impede network based attacks. Containing and vetting applications is the biggest security challenge for the platform vendors.

uTest: Looking back to the 1990s, what’s surprised you the most about the evolution of cyber security? What’s been your biggest disappointment?

RS: For me the biggest surprise was the confluence of existing criminal organizations and cyber crime, especially arising out of the demise of the Soviet Union. In retrospect is seems obvious, but at the time it was a wake-up call for me that guys with guns and baseball bats were going high-tech. My biggest disappointment is that security has never become important enough to spawn secure networks or secure computers. Not a single ISP or carrier has gone to market with a secure network with complete content inspection. Not a single computer manufacturer outside the military has sold a computer whose primary feature is security. They are all happy to sell you security add-ons but not willing to step up and address the underlying vulnerability of their products.

uTest: What happens in the next decade of security is anyone’s guess, but your predictions carry a bit more weight. Care to make any bold predictions on the future of cyber security? The bolder, the better.

Continue Reading

Testing the Limits With Richard Stiennon – Part I

What an honor it is to have security expert Richard Stiennon cap off another great year of Testing the Limits. Aside from being the most followed IT security analyst on Twitter, Richard is an accomplished writer, having authored Surviving Cyberwar and the soon to be published Cyber Defense: Countering Targeted Attacks. Richard is currently Chief Research Analyst of IT-Harvest, a security analyst firm with a wide range of high-profile clients. For more on his background, click here.

In part I of our interview, we get his thoughts on the difference between security and other types of testing; what the world would look like under full blown cyber war; the biggest threats to the typical web user; the motives of hackers and more. Tune in tomorrow for part II.

uTest: You started out in the field of aerospace as an engineer and wound up as one of the world’s top security experts (so typical). Kidding aside, what attracted you to the field of cyber security? And what’s kept you there for the better part of two decades?

Richard Stiennon: My transition from structural engineer to networking came when I started a dial-up ISP in Michigan, but I did not get the security bug until joining Netrex, which was an integrator of security products and services. Through Netrex I worked with a lot of the early security products and the founders of ISS, and Check Point Software. By the time Netrex was acquired by ISS (later to be acquired by IBM) I had moved on to PwC where I got exposed to large enterprise and performed audits on their security postures. From there it was on to Gartner and after that I was firmly entrenched in the IT security world. I have a low threshold for boredom. The security industry moves so fast you get left behind if you allow your eyes to glaze over for a second.

uTest: Is it fair to say that security testing requires a much different mindset/persona than other types of testing? If so, what specific qualities and characteristics are needed in a security tester? If you were assembling a team of security testers, what traits would you look for?

RS: Security testing of software throughout its development cycle is indeed different than quality and functionality testing. Instead of testing against end user use cases you have to have a mind set of an attacker, a completely different use case. In addition to meticulous use of security testing tools (HP-Fortify, Veracode, etc) a security tester must understand the application and how an attacker would leverage built-in functionality to subvert a system. A security tester must be diligent and detail oriented as well as imaginative and wily – a rare combination.

uTest: In 2010, you published Surviving Cyber War, which gave the world an inside look into the onset of state-sponsored cyber war. Since then, there’s been no shortage of similar incidents (Stuxnet, Anonymous, to name a few). Are you surprised at the speed in which cyber warfare is evolving?

Continue Reading

Testing the Limits With Noah Sussman – Part II

In part II of our Testing the Limits interview with Noah Sussman, we ask him some questions on Etsy’s testing challenges; assembling a testing team; localization testing and more. Did you miss part I? You can catch up here.

uTest: Total job interview question: What’s been the biggest testing challenge at Etsy and how have you been able to overcome it?

NS: Total job interview answer: concurrency has been a big challenge for us as we scaled our CI cluster. Most xUnit test runners aren’t designed to deliver massive concurrency out of the box. Eg if you want to run all your PHPUnit tests concurrently, you have to build a special branch of the test runner.

And even then, most tests are not written with concurrency in mind either. So a lot of design, debugging and rewriting of legacy code have been required in order to get to a point where we could run our tests in as concurrent a fashion as the available hardware would allow.

uTest: A previous interview guest once wrote a lengthy post on the “victims of fake agile.” In your view, is there a danger of a half-assed move into the agile methodology? If so, what are some of the major consequences?

NS: I can only speak from personal experience and I don’t know how typical my experiences are. I agree with the author of that post in that adopting Agile won’t fix a dysfunctional team, and it won’t help an organization to learn to accept its limitations and work within them.

uTest: What types of traits/qualities/skills does Etsy look for when hiring testers?

NS: On my team there are two roles for which I hire. These aren’t formal roles and we switch them up sometimes.

I hire software engineers who value quality and who are interested in how complex systems fail. These are the people who customize our mocks and fixtures, manage our coding standards, build static analysis tools, develop Jenkins plugins, design the CI system in general other things of that nature.

Then there is another equally important set of people whose skillset is generally described as “hardcore QA chops and a deep connection to the Etsy community.” The people I’ve hired here have many years of formal QA experience in high-risk industries like finance. These are people who are willing to use what they know about formal process, to help Etsy avoid the need for formal processes.

Here I look for talented QA analysts who are longtime Etsy users and deeply involved in the Etsy community. These are the people who develop functional testing tools, manage Selenium integration with CI and work with others in the organization to formulate test plans and resolve bugs. They also design and improve the process by which we triage bugs found in production.

uTest: How does Etsy – which has a global user base – deal with testing issues related to localization?

Continue Reading