6 Testing Mistakes to Avoid Like the Plague

mistakes“It’s okay to make mistakes,” they’ll tell you. “As long as you learn from your mistakes.” Not the worst advice you could get, but a wiser person would tell you that it’s far better to learn from someone else’s mistakes. Wouldn’t you agree?

If so, you’re in luck, because organizations all over the world continue to make the same software testing mistakes over and over again – all so that you don’t have to. While there are far too many to list here, we wanted to share a few of the more common gaffes. Here they are, in no particular order:

Mistake #1: Testing too late

How do you know if you’ve waited too long to test your apps? One sign would be if users are publically complaining via social media about defects they found on their own. When it comes to quality, the sooner you can involve the test team, the better off you are going to be. As you’ll see throughout this post, many of the great testing blunders occur for this very reason. Teams that see testing as the “last line of defense” not only misunderstand the role of QA, but they put themselves and their business in dangerous territory. Brian Marick explains the advantages of testing early:

Tests designed before coding begins can improve quality. They inform the developer of the kinds of tests that will be run, including the special cases that will be checked. The developer can use that information while thinking about the design, during design inspections, and in his own developer testing. Early test design can do more than prevent coding bugs.

Mistake #2: Testing with amateurs

In an ideal world, every testing project would have two types of experts: a testing expert and a domain expert. The reasons for having the testing expert on hand should be obvious (to readers of this blog anyway). The great testers understand, regardless of the product, where flaws and vulnerabilities are likely to be found. They are, as James Bach has said, professional skeptics. Domain experts on the other hand might not know much about testing per se, but they will immediately pick up on inconsistencies and shortcomings in the product from a feature/functionality perspective. Unfortunately, many companies are making the mistake – the BIG mistake – of having neither a testing expert nor a domain expert on their team. Don’t be one of them.

Mistake #3: Testing without a scope

“Just look for bugs and tell me what you find.” Famous last words! While this type of “improv” testing can often yield quality results, it should never be standard operating procedure. Unfortunately it is inside many companies. A note to testers: If this is the direction you’re asked to go in, here is some good advice from Jon Bach:

There are *always* requirements, but they aren’t always written. They are both implicit and explicit.  Some are hidden, some may be obvious.  Some take time to emerge, others hit you right away.  You find them and they find you. You’ll know you found one when you sense a user has an unmet need.

Techniques for testing without requirements:

  • Ask stupid questions
  • Read the FAQ
  • Look at competing products
  • Read out-of-date documents
  • Invite a developer to the whiteboard, then diagram your assumptions of the architecture or some states

Testing is an active search, not just for bugs, but for requirements.  Approach your requirements-gathering the same way you approach testing.  You have to go find the bugs – they often won’t reveal themselves to you on their own.

Mistake #4: Testing “one and done”

A lot of time and energy goes into a software launch. So it’s only natural to relax a bit when the application is released to users. It may seem as though all is said and done from a testing perspective, but it ain’t. As we’ve seen time and again, many of the more serious bugs and issues crop up well after a product launches, often times through no fault of the engineering team. Maybe the application isn’t synching with a recently updated third party app. Maybe a browser setting changed. The list is endless. Whatever the circumstance, companies would do well to avoid the mistake of failing to run regression tests.

Mistake #5: Testing in a controlled environment

These days, users are likely to consume your application under all sorts of various conditions; an endless combination of devices, operating systems, browsers, carriers and locations. Why then do so many companies only test their applications within the confines of a highly controlled test lab? Why are they not moving a portion of their testing into the real world? There was a time when companies could get away with 100% on-site testing – and many did. Unfortunately, many continue this practice (old habits die hard, after all) resulting in production bugs that could have easily been found by testers in-the-wild.

Mistake #6: Testing too fast/slow

The Agile testers can’t seem to catch their breath, while the waterfall testers look like they’re about to die of boredom. Our last testing mistake has to do with the pace of testing – specifically, why many organizations can’t seem to keep it consistent. Much of the problem (again) boils down to the misperception of QA. When testing is seen as the last step in a long process (usually with a looming deadline) it’s natural for testers to feel rushed. When the role of QA is properly defined and agreed upon, testing will never be rushed and it will never slow to a crawl. Instead, it will become an integral part of the organization, not merely a department being waited on.

Of course these mistakes are all matters of opinion, and so we’d love to hear yours. What are the biggest testing mistakes in your view? Please share your thoughts in the comments section below.

Note: This post was inspired by the timeless Classic Testing Mistakes by Brian Marick. Give it a read if you haven’t already.

Essential Guide to Mobile App Testing

Comments

  1. says

    Great blog here! Also your site loads up fast!
    What host are you using? Can I get your affiliate link to your host?
    I wish my site loaded up as fast as yours lol

  2. Nazia says

    I would like to add another point over here. That is, “Testing with a split concentration”. In my experience I have found some testers start testing with a lot of thing in their mind. As a result a slow and incomplete test is done, which ultimately produce eleventh hour’s bugs and even production bugs.

  3. says

    Hey! I could have sworn I’ve been to this blog before but after reading through some of the post I realized it’s new to me.
    Nonetheless, I’m definitely happy I found it and I’ll be book-marking and checking back often!

  4. Zhou_fin says

    Yousaid: The great testers understand, regardless of the product, where flaws and vulnerabilities are likely to be found. They are, as James Bach has said, professional skeptics.

    Would you please specify more how to find flaws and vulnerablties?

  5. Diana says

    Great information! I think there is also another mistake that should be listed, if not included in one/many of the other mistakes. Mistake: Not having a business analyst engaged from the beginning of the project all the way through test and implementation. The BA, may or may not be the domain expert, but has insight and understanding of the requirements and business processes/goals of the software. It’s our job to get the requirements right from the business perspective, development perspective, and QA/test perspective.

  6. Dayle Fish says

    A very good article. However I cannot develope a test till I know what you are going to build.
    I like to sum testing up as have a plan if possible in today’s world it can be two pages but have a plan. Testing and QA are two different birds. One actually tests, the other compiles as many pieces of paper about the test as possible (metrics?). Have a plan. (said that) Tell the team what you are building, how is it supposed to work, that is how all the pieces are supposed to work preferably at a planning meeting. As developer you need to do your unit testing. Then you need to conduct your Integration testing of all the pieces. Do not give it to test if any of the above doesn’t quite work, you are wasting testings time. (don’t as you are heading out the door at the end of the day open a door and throw your app at testing and go home). This means have a schedule. (plan). Talk about, it write it down, make certain you keep testing informed from the start . A status meeting (30 minutes) or two won’t hurt, MAKE it a TEAM effort. While you are developing, we are creating tests. before new delivery’s, test the App, test new deliveries fixes, then regression test the app again.

  7. Champ says

    Nice article ,I agree and these are some of the most common mistakes . In my view one of the biggest testing mistakes is not having a well defined testing structure . Due to the agile mantra (mostly misinterpreted and tweaked to suit deliver dates), I am increasinly seeing organizations wanting to save money by not having a structure and expecting ‘Testers’ to play multiple roles (test analyst test lead , test manager etc) , this in my opinion is compromising the quality of the testers work in turn giving rise to a rushed feeling which only snowballs over time affecting the the quality of the products . cannot agree with you more as you rightly point out in mistake # 6
    ‘When the role of QA is properly defined and agreed upon, testing will never be rushed and it will never slow to a crawl. Instead, it will become an integral part of the organization, not merely a department being waited on.’

    Do you have any tips on how to impress the importance and need for a defined test structure ?

  8. Michael Broido says

    Re Mistakes 1-4:
    Testers need to assert themselves on the team, the sooner the better. In particular, design of the test cases, including set-up/preconditions, should be subject to review by the developers as much as the testers review the developers’ work products.

    The flow of test products in the waterfall/military models is one phase behind, but the testers should not be operating on their own. In particular, the developers should be reviewing the test cases to assure that if the product is tested in the proposed way, will that establish the given requirement fully and completely? Does the proposed test approach demonstrate that the original requirement has been understood by the test team?

  9. Panicos M. says

    The article consist of very useful points written, i am sure, from person/s with experience of effective software development. Here we are implementing a Banking System using methodologies ( Agile ). There are 6 Agile teams each consisting of a Domain expert ,who has turned into Business Analyst, and an experienced Analyst/Programmer located next to each other. For each domain the planing, scope definition, analysis, analysis, design, development, testing (including UAT), documentation is done only by these two persons based on Agile principles and following most of the points mentioned in the article. Two people with the assistance, when needed, from specialists and myself is the ideal number towards effectiveness and quality even for difficult areas like loans and commissions. The system is demonstrated throughout its building cycle to active business users, representing different departments ( Quality Committee). Bugs and Functional improvements are identified throughout the process either by the Domain expert or by the programmer/analyst or by QA. These are welcome to apply based on the 80-20 rule. Result? Effectiveness, quality and low project cost.

  10. Janakiram says

    My suggestions :
    1. Testers should have the attitude and aptitude towards testing. They should not be i-did not-get-a-programmer’s job-hence-doing-testing kind of people. They should be always in the test-to-break kind of attitude. If they have some programming experience, excellent !
    2. Testers should be treated on-par with Developers. Right importance should be given to them.

  11. says

    Views on Mistake #1: Testing too late
    For me testing is situation and context based. Decision on whether to test too late or too early depends on situation and context like business case of QA, availability of test basis, testing approach, skills and competency of professionals, etc
    ——————-
    Views on Mistake #2: Testing with amateurs
    Testing is a role and testing shall be collaborative. Amateurs also add value. Its always good to perform testing by involving test experts, domain experts, functional consultants, naive users, and so on.
    ——————-
    Views on Mistake #3: Testing without a scope
    Testing shall not be scope bound but objective bound. Testing with scope uses just analytical thinking. Testing with objectives involves design thinking (analytical + intuitive thinking). So final objective of testing shall be: to provide information on Fitness to Use, Fitness to Standards, Fitness of Cost, and Fitness to requirements and latent requirements.
    ——————-
    Views on Mistake #4: Testing “one and done”: System testing is not just confined to testing before release. Now system testing is continuing even after release since some coding is required towards addressing requirements towards integration with other systems, configuration, installation etc. So system testing goes beyond release of system and making it operational and involves stabilization as well.
    ——————–
    Mistake #5: Testing in a controlled environment: Its impossible and also, unrealstic to create deployment environment with ever increasing complexities because of content-communication-computation-connectivity aspects of software based systems. Hence testing is a controlled environment is not a good proposition
    ———————
    Mistake #6: Testing too fast/slow: Testing depends on availability of work products for testing and also, on many dependent factors like test environment, test data, other resources etc. So testing shall be aligned to objectives, business and context.

  12. Ro says

    According with my experience, the biggest testing mistake is unprofessional stuff. Especially in the management. For me better don’t have a QA manager then have a manager who doesn’t know what does ‘testing’ mean.

  13. Dave Maschek says

    From recent experience, here are five best tester practices that I’ve found can help overcome five of the six testing mistakes. I don’t really have a response to mistake #5.
    1. Have a tester on your agile team from the moment the project is kicked off.
    2. The tester needs to be a domain expert or learn fast. Good communication with the product manager and developers is a big help.
    3. The tester should develop a set of acceptance tests for each user story at the start of the iteration and ask if the developers can add any more.
    4. Each user story must be tested while it is developed and a test case created so that the user story can be tested again before release (aka regression testing).
    6. The tester should be regarded as an equal member of the agile development team. Tester best practices 1 through 4 flow from this.

Trackbacks

Leave a Reply

Your email address will not be published. Required fields are marked *