Your Overconfidence Is Your Weakness: Lessons from a “Crash Specialist”
We’re very excited to have Pradeep Soundararajan as our guest blogger this month. Aside from being an active 
uTester (and recent Bug Battle winner), Pradeep is the author of TesterTested – one of the web’s most popular testing blogs. A resident of Bangalore, India, Pradeep is an independent test consultant who tests, coaches, speaks and writes on software testing and problem solving. His experience (as he describes it) is “6,500,000 mistakes and counting.” In this post, he takes a look at testing coverage, and the value of exploratory testing.
Finding a lot of bugs can be both a good and a bad thing for a tester. While the good thing might seem obvious, the bad thing is not. In fact, it’s rarely even talked about. Finding a lot of bugs can be bad in the context of making a tester feel overconfident about their skills.
Here is my own story of overconfidence…and what I learned from it.
About six months after I started testing software, I was given the title of “crash specialist.” Had you seen my stiff collar back then, I bet you wouldn’t have wanted to talk to me.
All humans are fallible. The problem with overconfidence is not about facing failure, but rather being unable to digest failures. I had a lot of indigestion problems as a result of being overconfident. As a “crash specialist” it was hard for me to accept the fact that end users found crashes that I had not. Determined to not let go of my reputation, I ended up pushing myself too hard – that is, 18 hours of work every day, including Saturday and Sunday. No excuses.
It didn’t take long for me to figure out that I needed to run a lot of different kinds of tests if I were to find a lot of problems. Unfortunately, the document test cases were horrible. And who, might you ask, drafted the documentation? Me!
It was the fear of losing my reputation that prompted me to do more exploratory testing. I desperately wanted to find all of the bugs in a release before it reached the end users. I had gone crazy and continued testing a specific build after its release to customers – to rebuild my confidence and assure myself that I had found all of the bugs.
I wanted to do more tests, but the developers kept approaching me for my support on verifying fixes. It obviously slowed down my testing. I understood that supporting developers was part of my work, but it would actually stop me from finding more bugs. So what was I to do? As it turns out, this was my first lesson on the opportunity cost in testing.
A few days after the release – ahhhh! – customers had began reporting bugs! It was around that time that I had decided it had been too long since I visited the pub. As I sipped my Kingfisher, I kept asking myself, “What’s wrong with me? Why am I unable to achieve my goal of finding all bugs?”
What was wrong with my testing approach? I can sum it up in one word: Coverage.
I had no idea about coverage. I thought of myself as a one-man army. I thought I was extremely talented (please laugh), and that someday, I’d grow up to become a great tester who finds all bugs in just one cycle of testing. As I look back at that moment, I can easily say that was the peak of my dumbness as a tester.
“Coverage is the extent to which we have modeled and tested the system.” – Michael Bolton (no, not the singer).
Let me explain this in the context of the product I was testing.
Regional coverage: The product I was testing was supposed to be released to in Japan. I had neither traveled there nor had an opportunity to interact with any customers. When the feedback from customers started flowing, I wondered why I didn’t have a clue of any of those problems.
Finger size coverage: Sound strange? No, I am not trying to coin a new term and add to the existing confusion. The product I was testing had lots of variations of buttons and key press that were comfortable to me and my team (maybe because of the bias we had developed) but probably didn’t go well with a couple of people from other countries, as they had a fatter thumb (or a thinner one) than us that made it difficult for them to operate our product without making a mistake.
When I participated in the uTest Bug Battle, I was constantly watching out for coverage the products under test were getting by various testers slogging bugs and pushing the bug count to 600+ in just a week: 100+ different platforms, 5 different OS, 50+ browser types, 100+ variation of internet connection speeds, 1000+ testers.
I also noticed a pattern of tester’s approach. In trying to compete with each other and to not end up reporting a duplicate, most testers used an exploratory testing approach. When you are in a battle, you want to use the best weapons. What most managers and testers don’t realize is this: Everyday they are in a battle!
I see uTest acting as a catalyst for lessons on coverage and exploratory testing to newbies, novice testers and to those who claim to be experienced (just because the industry thinks they have been here long enough).
If you still haven’t learned the lessons on coverage and the value of exploratory testing, you should participate in Bug Battles, experience the coverage, explore exploratory testing and lower your overconfidence. You’ll be a better tester because of it.
Outstanding post, Pradeep! Insightful and entertaining. Looking forward to working together more closely in the future.
@MattJ,
Outstanding post, Pradeep! Insightful and entertaining. Looking forward to working together more closely in the future.
My pleasure Matt. I am glad you liked this post.
Hi Pradeep,
Fantastic Topic! This post is so relevant to uTesters at uTest. We more often get trapped in finding each bug(and earning an extra dollar) that we forget the value that our bug is going to add to the product. This post speaks about the pitfalls that result in mindless bug hunting and emphasizes on using the Freedom of Exploratory Testing in an effective way.
Happy Testing,
Parimala Shankaraiah,
http://curioustester.blogspot.com
Hi Pradeep,
Brilliant as always..
Through each post you brings/teaches a new idea, which helps the thinking process.. 
Regarding the coverage, I think most of the testers will go about testing an application as per requirements. Even if they are performing exploratory testing, they might be bound by the limits of this so called SRS. I feel that is why testers miss something like the “Finger size coverage”
Also, I completely agree with Pari for this matter.. With the dollar in mind, we tend to forget the value we add to the product..
(Thanks for that eye opener Pari 
)
@Nandagopal,
Regarding the coverage, I think most of the testers will go about testing an application as per requirements. Even if they are performing exploratory testing, they might be bound by the limits of this so called SRS.
We are in the dinosaur age of testing. In 5 million years from now all testers will realize that what can be captured in the requirement document is just probably less than 5% ( a made up number ) of what our users might actually need. Added to that if it is frozen – then the need of the user has changed but we haven’t.
I feel that is why testers miss something like the “Finger size coverage”
The finger size coverage was an interesting lesson to have hit me right at the start of my career.
A nice article.
I too have experienced problems that might not be problems had someone considered finger size coverage. I have a mobile handset that has buttons that are quite close together. That’s not so much of a problem in itself, but when the delete button is right next to the play button, when the confirm button is right next to the delete button, and when the cancel button is right next to the confirm button, then data loss is just a slip away.
Alas, it appears that the dog ate a little of Pradeep’s posting. I know that he’s working pushing a correction through, but for now, it would be more correct to say that coverage is the extent to which we’ve modeled and tested the system. Coverage is never isolated; it’s always with respect to some model.
For those who are interested, there’s more about test coverage (which is not just code coverage) in some articles that I wrote for Better Software:
Got You Covered
Cover or Discover
A Map By Any Other Name
—Michael B.
Hey Michael,
Hope all is well. If you ever want to writea guest blog post for our community of ~20,000, I’d welcome it. If you’re interested, just drop me a line at mattj[at]utest.com.
- Matt J.
Bingo!
Some day ago I started to think about my weakness in testing. Pradeep totally won my attention with this ‘overconfidence’ for this day
@MichaelB,
Thanks for your comment and liking this piece.
@Alexi,
You made me curious. What else did you identify about your weakness in testing?
When I heard and hear Code Coverage for a model, I asked “is there anything to cover apart from the written code to give an usable model?”
‘Finger size coverage’ remembers me the ‘No Fingers Delay Coverage’ in the charter, which made model under test to crash within 8 seconds on using it.
This post uncovered most of my little learning to learn again.
@Pradeep
I will write about that apart.
hi sir…congratulations for winning bug battle….ur article rocks………A complex topic (such as exploratory testing) could be best understood through examples and experience……the article is best of its kind…truely an explorer of exploratory testing…..
Thanks Pardeep for this post..
This is a very good post and i will try to learn from it.
A single conversation with a wise man is better than ten years of study. ~Chinese Proverb
Thanks Again .. you got a new fan !!! that’s me
@ Babu,
Thanks for your comment. I just have a thing to say here – exploratory testing isn’t a complex topic – it is what we do most of the time without know it. People had been struggling to communicate it properly and other people abused the idea by trying to communicate it in their own words without any research on it.
It also appears to be complex when people close their ears when we talk and open their ears after we are done and also simultaneously open their mouth to say, “that doesnt work”. I dont understand why they dont understand they are foolish.
You may ask why do we still talk – because we care about them and we dont want them to be fools forever. We have seen a lot of change and we will see more change.
Kaner, James, Michael, Heusser, Elizabeth ( and you may consider the Indian tester – Pradeep, too ) have been struggling to help people understand about it.
@Atul,
You won my appreciation as well for quoting the source of this one A single conversation with a wise man is better than ten years of study. ~Chinese Proverb
Not many testers are matured enough to quote the source of the ideas they borrowing and you did it.
I learn a lot about testing when I meet wise people and talk to them.
hi sir…thanks for ur comments….. the reason i mentioned exploratory testing as a “COMPLEX TOPIC” is because, i thought basically as the name exploratory indicates that i’m going to explore the product which is not an independent module……its like an artificially produced fruit of an artificial tree capable of recycling the fruit to its requirements(n my requirements are also not precise) for a particular set of consumers…. I have been given this fruit to test it before it is out to the customer…..and my job as a tester would be to help fill in all the holes in the structure,and also the holes that would be formed by filling these holes and so on and so forth…… in order to test this fruit i have to explore the entire tree n this tree is connected to another large tree….so my boundaries are not confined,they are scattered in every possible direction….its like infinity because i’m going in the reverse direction….. it would be easy to derive an equation given a precise out put……but the reverse process of deriving the equation is like deriving a new derivation for every line i proceed backwards…..this is such a laborious process….and after having been through the entire process several number of times i would still encounter bugs because my boundaries are not confined….and this is about only a single module,there will be many such modules that put together would produce the complete product to be delivered to the customer……there is also the time constraint…. and in any case i’ll not be given the whole product to test it…so thinking about all these things,i felt exploratory testing would be a complex job ,only a dedicated team work would produce the best results……..in the context i think every thing in this universe is defected (except god and his creations…) by some way or the other,only the intensity of the defect varies….he he………… . i dont know whether my thought process is right or wrong….please do comment on my defects……..sorry for replying in such a big paragraph,could not compress it……have a nice day sir……
@Babu,
Here are some suggestions for you:
1. Stop addressing as “sir”. “Pradeep” is fine.
2. You might be watching TV channels like Nat Geo, Discovery and might have observed that animals by virtue have been explorers with mission of finding food, better place, new lands, possible new water locations… and humans have been more successful explorers than other animals. The reason for the industry treating exploratory testing as a tertiary approach to test is for business purposes. This might be of interest and high relevance: Traditional Software Testing Process – The cash cow demystified
3. Exploratory testing focuses on freedom for a tester and hence there is much less stress when performing it as compared to scripted testing.
4. Exploratory testing depends on “you” ( tester ) and not on robotic scripts and hence it is more fun than scripted approach.
5. Practice more testing
6. Use spell and grammar check in MS Word. Not that I am perfect, I at least write in a way that spell and grammar mistakes do not pop out of the article to say, “hey, I am here”.
7. You should start blogging.
thank you sir…
Thnx!
Today i realized that a fool with a tool is still a fool
@Bibek,
The quote “A fool with a tool is still a fool” is from Marshall McLuhan
You might want to consider quoting him if you might want to use that quote elsewhere and hey, its a good practice.
[...] Your Overconfidence is Your Weakness: Lessons from a “Crash Specialist” – by Pradeep Soundararajan: In our most-popular guest post to date, noted blogger Pradeep Soundararajan explains why finding lots and lots of bugs isn’t necessarily a good thing. Reliving his days as a “crash specialist” Pradeep examines how a tester’s ego can get in the way of their objective. His advice is as funny as it is useful. Simply put: a must read. [...]
[...] (Ajay plus Sharath Byregowda, Manoj Nair, and Parimala Shankaraiah) are students of our colleague Pradeep Soundararajan. They gather online every week to spend an hour or so testing some freeware or some Web service, [...]
Hi Pradeep,
I really enjoyed reading this post (just like most of the other posts). This one caught my attention due to the work overconfidence. Very often developers give you some title (I have never understood whether there is any ulterior motive it that) its a feel good thing for us.
That is where problem starts, you start acting that image of yours. You just start putting undue pressure on yourself just to retain that image. It can be really tiring. I have suffered it.
Keep posting..
@ Novesh,
Very often developers give you some title (I have never understood whether there is any ulterior motive it that) its a feel good thing for us.
Ah, nice. I also think we testers get so easily happy with just one bug and one appreciation for that. It can be a trap unless you have trained your brain to be skeptical and stay mission focused.
I have suffered it.
If you had told me you had 10 years of experience, my response would have been, “Well, that doesn’t tell me anything about you” but what you said above tells me about your experience. Now, that is great that you can make more sense in talking about your experience with just a few words instead of numbers.
Besides learning technical stuff… Its very important to have attitude as a tester…
Thanks for the post…
Sharing your experience helps others not to do the same and struggle
@ Sowjanya,
Thanks! I hope every tester shares their experiences for the benefit of the community.
Amazing article.. As ususal
… very fascinating to understand the fact that just a simple bug like finger coverage.. can be of utmost important to the customers. Japan incident has happened with me too, but in this case I found a bug which I thought might be bittie (very small) but it stopped the shipping of the product. Great quote that I would like to quote “It took me a long time not to judge myself through someone else’s eyes. ” ~Sally Field
@Neha,
Thanks for dropping by.
Japan incident has happened with me too, but in this case I found a bug which I thought might be bittie (very small) but it stopped the shipping of the product
That’s an important learning to have. I just hope those testers who have this lesson also realize that they need to think about the reverse of it – What they think as a show stopper could not be one such and there is nothing to be disappointed about it.