Audit Report On The Hewlett-Packard System Integration Contract Expenditures Associated With The Emergency Communications Transformation Program

Audit Report In Brief

The Emergency Communications Transformation Program (ECTP) was initiated in 2004 to transform and consolidate the City’s 911 Emergency Dispatch System.  The objective of the ECTP is to centralize and integrate the call-taking and dispatch operations between the New York Police Department (NYPD) and Fire Department of New York (FDNY), including its Emergency Medical Dispatch (EMD) division, into two fully integrated Public Safety Answering Centers (PSAC 1 and PSAC 2) equipped with state-of-the-art hardware and software communications systems.

On April 1, 2005, the Department of Information Technology and Telecommunications (DoITT) contracted with Hewlett-Packard Company (HP) as system integrator for the ECTP.  The contract includes, but is not limited to, development for PSAC facilities, organizational transformation of the call-taking process, application integration for all Computer-Aided Dispatch (CAD) systems and the Automated Vehicle Location (AVL) system, and the communication infrastructure supporting the ECTP.

The ECTP system integration contract has a not-to-exceed amount of $380 million over a five-year term with two additional options to extend the contract for an additional year through June 30, 2012.¹   On January 6, 2012, the projected contract expenditures were $346 million and as of April 17, 2012, the City expended approximately $309 million of the $346 million.

The Comptroller’s Office issued the Audit Report on the Project Management for the Emergency Communications Transformation Program by the New York City Department of Information Technology and Telecommunications on March 20, 2012 (Audit #7A11-104). The current report addresses a more detailed audit of one aspect of that overall engagement – the HP ECTP system integration contract.

Audit Findings and Conclusions

We do not have reasonable assurance that the expenditures of the HP ECTP system integration contract were reasonable and justified and that the scope of services was met. DoITT’s questionable selection of HP as the system integrator on this project in 2005, compounded with HP’s poor performance and its inadequate oversight over consultants’ performance, resulted in the failure to complete all original component projects (i.e., unified CAD system and PSAC 2) of this contract.²   The preceding factors, combined with poor contract management by DoITT, resulted in significant cost overruns.  The estimated cost overrun of the ECTP system integration and overall project management could be as much as $362 million.

As of April 17, 2012, the City has paid HP $309 million of the $346 million currently projected for this contract.³   Effective January 2011, DoITT awarded a second system integration contract, not to exceed $286 million, to complete one of the original components identified in the project definition (PSAC 2).⁴   Consequently, the anticipated overall system integration cost increases to $632 million ($346 million current ECTP projected expenditures plus $286 million, the cost of the second system integration contract awarded to a different contractor).  Based on our analysis of the budget breakdown provided by DoITT, we estimate the cost overruns could be as much as $362 million ($632 million minus $270 million originally estimated for equipment and system integration services for both PSACs).

DoITT should not have awarded the system integration contract to HP in 2005 because 1) HP did not receive the minimum technical score DoITT required to be considered a viable contractor and 2) DoITT did not maintain documentation to justify awarding this contract to HP.⁵    According to DoITT, after the only other contractor withdrew its proposal, HP was awarded the contract even though it did not achieve the minimum technical score because it received outstanding recommendations from two government agencies (NYPD and the US Air Force). However, DoITT did not maintain any documentation to substantiate the alleged outstanding recommendations that supported its decision.  In fact, our audit found documentation to the contrary—that on November 4, 2004, HP was rated as a poor performer on another ECTP-related contract (with NYPD) prior to being awarded this contract and thus DoITT had no valid justification for selecting HP.

DoITT also did not exercise good judgment when overseeing the contract and ensuring HP provided qualified consultants to work on the project.  Specifically, DoITT did not reduce the maximum contract amount when it decided HP could not implement the unified CAD system or complete the work related to PSAC 2.  In addition, DoITT allowed HP to use very large mark-ups for the services provided by subcontractors, changed fixed price component projects to time and material basis, and allowed HP to charge a portion of its administrative costs related to this contract to the City.

After being awarded this contract, HP’s performance did not meet the City’s expectations.  In 2007, the then-DoITT Commissioner expressed that DoITT, NYPD, and FDNY were disappointed by HP’s performance.  A year later, DoITT rated HP’s performance as “Needs Improvement” and “Unsatisfactory” in the 2008 VENDEX Contractor Performance Evaluation.  HP’s deficiencies were also detailed in a May 2009 report prepared by Gartner Inc. (Gartner), DoITT’s independent quality assurance consultant, which reviewed HP’s performance under the contract for the period of April 2005 through April 2008.  However, HP was allowed to continue its work on this contract.  During this period, April 2005 to April 2008, HP was paid $113 million.

Even after 2007, HP continued to show signs of poor performance over the execution of the contract.  Our audit found instances when HP did not ensure that the consultants hired had the qualifications or experience necessary for the positions for which they were billed.  Moreover, the consultants’ timesheets were not approved in a timely manner.  The approval of timesheets was as long as 885 days after the performance of work.

Further, HP did not properly monitor its billings and had a significant billing error rate.  Our review identified instances when HP billed the City for consultants at a higher title than they were working at and billed for unsupported consultant hours, questionable hours, and unauthorized overtime.  In many instances, the timesheets reviewed and accepted by HP lacked a detailed description to determine the specific work the consultant was being paid for and, in other cases, described activities that added no benefit to the system integration project or activities that were reported before the date of performance.  Based on our review of the documents submitted by HP for payment under the time and material section of this contract, the City should recoup at least $2,509,451 from HP.  We, therefore, do not have reasonable assurance that HP properly billed for consultant work associated with the time and material portion of the ECTP project.  As a result, DoITT may be able to recoup as much as $106 million.

Audit Recommendations

We make 11 recommendations, including that DoITT should:

• Use the findings in this report as a starting point to determine how much of the expenditures over original budget can be recouped.  Specifically, DoITT should review the documentation supporting the invoices submitted by HP with a view toward recouping as much of the $113 million paid HP from April 2005 to April 2008 and where HP’s unsatisfactory performance deprived DoITT of receiving the full benefit of the services bargained for in the contract.  DoITT should also conduct an in-depth review of HP’s invoices and supporting documents to determine how much of the $106 million—in addition to the $2,509,451 that was specifically identified in this report—can be recouped because HP incorrectly billed for its time and material services.  (It should be noted that $56 million of the $106 million is included in the $113 million paid to HP for unsatisfactory performance for the period from April 2005 to April 2008.)

• Review its contract budget monitoring procedures to ensure that, in the future, if a contractor is no longer expected to perform one or more of the tasks that were originally expected, the maximum contract amount is appropriately reduced.

• Revise its procedures to ensure that the contract solicitation procedures are not ignored and ensure that, in the future, only qualified contractors are selected.

• Maintain in its bid documents written documentation of the reference checks conducted on contractors.

• Review all resumes to ensure that consultants are qualified for the positions they are working in.

• Properly review HP’s subsequent invoices, timesheets, and other documentation before approving payments to ensure that payments are appropriate and accurate.

• Limit the mark-up percentage that a contractor can add to the actual cost it pays to its vendors in future contracts.

• Insert a liquidated damage clause into all DoITT’s contracts that allows it to assess liquidated damages for each day that delivery is delayed. The contracts should also allow the City to seek remedy for poor performance.

City Response

The Office of Citywide Emergency Communications (OCEC) responded to our draft report rather than DoITT.  We, therefore, refer to “City” officials rather than OCEC or DoITT in our comments.  City officials disagreed almost entirely with the audit’s findings and conclusions and disagreed with seven of the 11 recommendations.  Furthermore, they maintain that the audit’s findings and conclusions are “premised on a fundamental misunderstanding of the scope of HP’s work as system integrator on PSAC 1.  This misunderstanding is also the basis of the audit’s unsupportable conclusion that the system integration work for ECTP ‘could be’ up to $362 million over budget.”

We strongly disagree with the City’s position.  In its response, the City presents several questionable arguments which attempt to refute the report’s findings, but each argument was disproved. In some cases, statements are made that are contradicted by the City’s own documents and appear to be factually inaccurate. This final report may have been less contentious if the City had fully and openly discussed its concerns with the audit team prior to submitting its written response, which is the customary practice.

The City’s disagreement with seven of our recommendations is largely based on its disagreement with the report’s findings.  Because we believe our findings to be correct, we re-iterate our recommendations, which we believe should be implemented.

The full text of the response received from City officials is included as Addendum to this report.  Our comments concerning the response are included as Appendix II, which precedes the Addendum.

---