The Client is Always Right – Especially When They’re Wrong
How should testers respond when their bugs are rejected by clients?
Here to address this topic is Bill Ricardi of Northern Ireland, this month’s guest blogger. An active member of the uTest community (especially in the uTest forums), Bill describes himself as a “huge nature fiend, with professional ties to advertising, real estate, gambling, and writing.” For more on Bill – including the upcoming publication of his first book – check out his Google Knol page.
In the field of contract software testing, you won’t always see eye to eye with the client. What you consider a critical bug, they might see as a non-issue (or worse, a ‘feature’). What you call a major security flaw, they might consider such a remote possibility that it doesn’t even deserve a mention.
You might ask how you bridge such a gap between your level of testing and the client’s level of acceptance and understanding of product integrity and the testing process in general. The answer is simple:
You don’t.
It isn’t your job to convert the client to your way of thinking. Yes, you can contest a bug that they reject out of hand if you were technically correct to report it. Sometimes they’ll accept it as valuable feedback, but most of the time they’ll just ignore any contested bugs. This is something that you have to live with.
You cannot impose your standards upon a client. No matter how passionate you are about the quality of your work and your understanding of their product, no matter how much you study the test parameters and the client’s requirements… only they know what they really want out of the test cycle. They WILL usually refuse bugs that they don’t consider ‘real’ bugs, because they’ve probably had meetings about what their programming philosophy should be, and they probably have a preconceived notion of what they have time and budget to fix.
So you’ll get hit with the dreaded duo of rejection reasons: ‘not a bug’ or ‘out of scope’.
My advice: Once you’ve made your case, and made it again if you contested their bug rejection, MOVE ON! As an independent tester, you only have 2 options; you can continue to look for bugs on their project, or you can move on to a new project. Don’t attempt to control the client, they won’t appreciate it. Instead, focus on your self control and keeping your personal standards high.
Ya ..its true..
I think ii should follow what is said ..but because i am a not having that much experience “Self control” is a tough..
Good if I can achieve that as early as possible
But again, if there is something that you believe will really clash the system, and believe that the client does not want it to be exposed to his higher management, i think that you need to stand your ground!!!
Thinking about this case again, if there is something that you believe will really clash the system, and believe that the client does not want it to be exposed to his higher management, i think that you need to stand your ground!!!
Great article. If you really feel its genuine bug and counterpart is not accepting. Just escalate it through higher management and lay back on your seat.
They will fight to do the needful
Good discussion.
Bill, I think you should write a follow up post addressing these concerns.
A possible title could be “I Told You So”
This surfaces a significant ethical issue in cloud-sourcing. If the contractor is paid by the bug both the client and the engineer have strong incentives that clash here. Money is at stake.
The value of test lies not in just discovered bugs but in test coverage and quality assurance. Which simply takes time and effort.
So a “paid by the bug” paradigm with no regard to time or effort, that also enables the client to arbitrarily choose what results are contractually valid, is rather deplorable.
If I’m the tester, I held up my end of the bargain. I tested, I reported results. The coverage, the results, all holds value to the client no matter what recourse they take. I delivered.
No party has control over what defects will exist or what their ultimate value will be. For payment to be contingent on such a factor is deplorable! But if I agree to such exploitative terms and *then* the client is so enabled to arbitrarily determine whether the bug is valid, it’s just insult to injury.
Jacob,
A few important points of clarification to your premises and your conclusion:
* I assume by “cloud-sourcing”, you mean crowdsourcing. I’m not nitpicking, but I want to clarify that we’re talking about the same thing (crowdsourcing meaning working with a community of live testers; cloud-sourcing being provisioning automated testing from the cloud).
* Testers in the uTest community are paid for all forms of approved work — not just bugs. Many uTest test cycles contain guaranteed payouts for completing test cases, usability surveys or load testing participation.
* Customers don’t pay per approved bug. They pay per test cycle. So they have no financial incentive to be overly strict or stringent with their bug approval/rejection decisions.
* If a bug report is rejected by a customer, they must provide a specific reason code (eg: “out of scope”, “works as designed”). The tester can dispute that rejection and uTest will review the bug, the scope and the decision to reject.
* uTest’s project management team works extremely closely with customers and the participating testers to ensure that customers define a detailed scope of work for each test cycle, and that testers get any questions answered about what’s in scope vs. out of scope. In fact, top testers have 90+% of their bugs approved.
For these reasons, clients don’t “arbitrarily” make approve / reject decisions. These are QA managers (or directors of engineering or CTOs) who understand the value of testing — and testers. They also know that, in an online community, reputation matters. And if they want to continue to have the top testers in the community work on their projects, they have to build a reputation for fairness and consistency.
Also, our testers get to accept or ignore any project and work only on those test cycles that interest them — and that fit within their schedules. And they get to build their tester reputations and skills while they get paid for their work. That’s why we’re attracting 700+ additional testers each month.
I’m not going to try to change your opinion about crowdsourcing, as you seem to have made up your mind. But I do want to paint a more detailed, accurate picture of how WE treat our community here at uTest — including their payouts, our reputation system, as well as offering training & learning opportunities, and tester networking.
Beyond that, we take the concepts of fairness and consistency extremely seriously — as thousands of our active testers and hundreds of our customers would attest.
Regards,
Matt Johnston
VP of Community & Marketing
uTest
good article!
Sometimes we as testers think the issue is a showstopper but its not a hindrance for customers in their business.
Farooq
Thanks for the respectful reply, Matt.
First, yes, my mistake. Crowd-sourcing. Thanks for correcting.
* You’re right. There are other compensated work items. I didn’t mean to say a bug was the only compensated work item, but it’s the one that is germane to the context of this blog post, which is essentially bug triage. Sorry, didn’t communicate that well.
* Didn’t realize that the clients pay per test cycle. So again, my mistake, and thanks for clearing that up. However I don’t think that fully alleviates the concern I mentioned.
They pay by the cycle, but uTest pays contracted engineers by the approved bug. Ultimately the source of funds to compensate the engineers for the bugs is what the client paid for the test cycle, right? If only a few bugs are found and approved, does that leave funds available for further testing and bug finding?
if that’s the case — which is my assumption, although I assumed incorrectly earlier — then there is still a net benefit — and thus incentive — for the client if a bug is not approved. It enables further testing within the test cycle without spending extra money.
But I do concede, now that I know they pay by the cycle, that does reduce the ethical conflict. Just doesn’t eliminate it.
* I definitely understand tangible justification is constituted in triaging the reported bugs, as accepted or rejected. Certainly. There is still some latitude in rejecting bugs, though. Arbitrary rejection can still be argued and supported. Ultimately if clients are able to specify the scope they have some control over this.
But mostly, the potential for clients to reject bugs that they oughtn’t is not my concern. I even find it unreasonable to expect uTest to be able to fully mitigate such an element to the model, so I want to make clear that that is not my criticism. To the contrary I commend the structure uTest has in place in that regard.
So I’m glad you affirmed that clients can’t unreasonably dismiss bugs out of hand with no justification. But affirming that runs contrary to the message in this blog post. which was why I spoke up. And it supports my disagreement with Mike Brown, that the client should not be so enabled, nor the engineer’s recourse undercut by the coercion of blogging peers.
But again, no, my concern is not really whether or not bugs are justly rejected. My concern is the overall pay per bug compensation model. It ought to be coverage-based compensation.
The quality, diligence, efficiency, depth, creativity and ultimate value of an engineer’s test coverage does not determine how many bugs will be found. The quality of engineering of the product determines that.
I’d be insulting myself as a tester if I suggested there is no skill in discovering defects. There is. A better skillset will maximize your discovery proficiency.
But ultimately bugs will be found, that are inherent in the development. So testers have , at best, marginal control over how many bugs they will find. I don’t think it’s fair compensation to pay them for approved bugs found rather than test coverage performed because of it.
Further, as I said before, test coverage has an intrinsic value to the client. They receive this benefit for no charge, except where bugs are found. In crowd-sourcing projects, while much of it may be redundant they receive a massive amount of test coverage. Work is performed. Testers are doing their jobs. But payment only comes if bugs are found.
The bug count is not accurately reflective of that quality, diligence, efficiency, depth, creativity and ultimate value of an engineer’s test coverage that I described before.
So I just think it’s unjust. I think testers should also get compensated for coverage performed.
Now, I also recognize how much more easily the contract engineers could exploit that kind of pay model, enough to so make it not feasible. Or not easily feasible. And I have no solution to that. I just don’t know. But I haven’t brainstormed about it. And that not being feasible doesn’t justify ignoring the ethical concern within this paradigm. Let’s not make the better the enemy of the best!
I’ve paid some attention to uTest for the better part of a year now, and I sincerely believe you that you take concepts of fairness very seriously, and so I sincerely thank you for that. That kind of commitment is fantastic. And that’s also why I felt compelled to raise this issue. I actually have hope that it will be heard, and considered, and hopefully acted upon. So again I really appreciate the response. I’m not a hater of crowd-sourcing, and I recognize the unique values it has. Just implore that this concern be considered further. Thanks, Matt!
A very well thought out reply Jacob. Thanks for the comment. Two quick points of clarification:
I assume the disagreement you referenced with me was actually a disagreement with the author of the post, Bill Ricardi. Yes?
If not, then I think you misunderstood my comment. I was simply urging Bill (or any tester, for that matter) to write on the subject of being correct about a bug when their clients (or bosses) didn’t take them seriously. Seems like there are probably a million of these stories out there, and I would be interested to hear them. I’m sure other testers would as well.
Best,
Mike Brown
Ah, another mistake on my part! Sorry, Mike, sincerely. Bill Ricardi was the author of that story, then.
There are hospitals, where nurses do the operations of the patients and medical doctors care about the financial accouting and the accoutants do the marketing and the medical students wipe the blood off the floor.
You don’t believe it?
But there are enterprises, in which IT is engineered by…. project management done by … and marketing determined by … and the testing is applied by …
Fill in the dotted gaps with the real professionals that you’ll find at the site at hand and you’ll know why it looks like in the virtual hospital described above.
the problem though with an independent testing service like uTest is that the tester has no idea as to what the client’s requirements are. unless of course you are following a test case that the client gave you.
does the author have an example of a ‘security bug’ that he raised, but the client rejected? I am guessing not. remember, there are always 2 sides to a coin.
[...] and move on. A member of the uTest community, Bill gave his advice on this matter as part of our Guest Blogger series, [...]
[...] A LOT of embarrassment. When it comes to the more technical issues, you would be wise to read “The Client Is Always Right – Especially When They’re Wrong” by uTester Bill [...]
[...] full length book: Living Cheaply in the UK. You may recall his previous entry on the uTest Blog, The Client Is Always Right – Especially When They’re Wrong. On a similar note, Bill offers testers valuable advice on the best ways to manage their downtime. [...]