Doing a full run of software testing in house is difficult. It takes a lot of man power, countless hours (on a tight deadline none-the-less) and covering the entire testing matrix in-house is incredibly cost prohibitive. In the face of these challenges it makes sense that many companies look for outside help. Outsourcing has been the option of choice for years now, but it’s far from perfect. Austin Engery dealt with some frustrating outsourcing issues and eventually chronicled their dumpy journey for Information Week. This story perfectly highlights some of the biggest shortcomings of outsourcing.
Dealing with workers who are physically apart from the rest of your employees can be difficult, but more often than not outsourcing complicates the matter even more. The outsource office can literally be time zones away, making getting in touch with someone complicated. Turnover at outsourcing companies can be high, so you may not even be dealing with the same staff for the duration of your project. Difficult or lack of communication can drag projects out and frustrate both parties. Here’s a few examples of Austin Energy’s communication issues:
“[Austin Energy CIO Alan Claypool emailed an IBM executive after] months of correspondence between himself and his staff and IBM project managers on multiple problems.”
“My concern is the lack of progress in identifying the root cause and, frankly, the pace and quality of response from IBM.”
“They have been responsive. I would characterize it as, in their words, ‘incremental response.’ We would like to see a faster response.”
Trouble with Accountability
In an outsourcing situation, you are not the only project on the company’s plate. Your project manager could be simultaneously juggling multiple projects which means A. Their attention is divided and B. More than one company is trying to get a hold of that project manager in the few common working hours – which further delays matters if something goes wrong. Austin Energy had trouble with the quality of work, but had few options for recourse.
“We continue to be gravely disappointed at the delays and seemingly ad hoc methods toward managing this project.”
“When we contracted with IBM for implementation services, we did not plan for, staff, or anticipate using our system and folks as the Alpha testers for the IBM code deliveries. Prior to delivery, we expect the code has been reviewed, validated, and will perform without software defects.”
“IBM has failed to properly set up and fully test their payment solutions for the Online Customer Care Portal.”
“We had an unscheduled outage while teams were preparing for our major testing this morning.”
“The second [error] was by an employee who was not available for executing a key task at the previously scheduled time.”
“At one point, Austin Energy’s general manager asked then IBM CEO Sam Palmisano to personally intervene, but Palmisano passed the request to a subordinate.”
Outsourcing requires a contract before work can begin. These contracts cover exactly what will be done during testing. Being so locked into a contract leaves little room for change or adjustments if you are unhappy with the work or if requirements change. Having to define the entire project before work begins can also leave holes you didn’t even know existed until it’s too late. If the work isn’t completed to your standards, or missed vital aspects you didn’t foresee, you’ll end up paying more money to have more testing done, essentially paying double for what should have been one project.
[When IBM finally addressed Austin Energy's concerns] “Kern outlined five steps IBM would take. They included ‘improving communications on business impacts caused by known defects,’ ensuring that all outstanding issues were documented and assigned to the right people, implementing ‘best practice processes to ensure repeatable success,’ working more closely with third-party vendors such as Oracle, and ‘identifying any remaining gaps that are outside the project scope and providing recommendations.’”
“In the future, the utility’s outsourcing contracts will include much more stringent penalties for non-performance. The penalties around specific areas like system availability need to be much stronger.”
The problems and delays often encountered with outsourcing can lead to additional, unplanned costs both in terms of fixing the problems and launch delays. Delayed launches often equate to lost revenue or lost savings.
“Austin Energy officials voiced concerns that some of IBM’s proposed fixes, such as adding more server capacity, would mean that the utility would have to pay more than it originally planned on the project. ‘IBM implies there will likely be additional cost to the City as a result of IBM’s decision – as part of the recovery effort – to move to more powerful servers,’ wrote Austin Energy’s Kerry Overton to IBM’s Chris Lilley on Nov. 23. ‘We do not support additional cost for the replacement architecture under the fixed price contract.’
“Little had changed since Kern proposed his remediation program: ‘The City of Austin continues to sustain substantial financial and operational losses resulting from two multi-million dollar procurements in which IBM has failed to provide the fully functional systems required out of three major system integration contracts we have with you.’”
“More than a year later, the City is still suffering from serious, continuing issues in getting the system running and using it to conduct routine City of Austin business.”
Some companies are so fed up with outsourcing that they’re abandoning the practice. GM recently moved away from outsourcing so its teams can be more nimble and creative. From Information Week:
[CIO Randy] Mott’s philosophy on outsourcing at GM, as it was a HP, Dell, and Wal-Mart, is tha the company needs more creative, business-changing ideas from IT, and IT teams need to deliver those innovative projects faster. Mott doesn’t think GM can be creative or fast enough with outsourced IT. ‘When the business says ‘go,’ then that means we start working on a contract, we don’t start working on a project,” Mott says of the current outsourced model.
With the increasing dissatisfaction with outsourcing, companies will be looking for alternatives. Some companies will beef up their in-house QA efforts, some will restructure their entire development method, some will probably look into beta testing and some will begin to realize that crowdsourced testing companies offer many of the benefits of outsourced testing (and then some), while addressing many of the issues. All testing methods have their own advantages and disadvantages, the important part is to know the true cost of software testing for each of the methods and pick the one that fits your business and project the best.