A Review of the Management and Fiscal Controls Over the City’s ECTP Upgrade to its Emergency 911 System

Executive Summary

On May 19, 2014, Mayor Bill de Blasio announced a halt to major work on the Emergency Communications Transformation Program (ECTP) pending reviews of various aspects of the project by the Department of Information Technology & Telecommunications (DoITT), the Department of Investigation (DOI), and New York City Comptroller Scott M. Stringer. The administration asked the Comptroller to conduct a review of the history of the project, including its oversight structure and financial management. The following is the report of the review conducted by Comptroller’s Audit and Investigations Bureau in response to the Mayor’s request.

The Comptroller’s review found that the ECTP’s governance was inappropriately structured, overly reliant on consultants, and ineffectively monitored, which resulted in a failure to achieve full agency participation and coordination. Instead of strong City governance, the project outsourced critical responsibilities to consultants who were insufficiently accountable to the City. As a consequence of inadequate governance, the project has taken far longer than anticipated, stretching from the initial estimate of five years to the current estimate of fifteen years, with the end date now projected to be 2018.¹

We found that project delays and cost overruns mounted as technical and other difficulties emerged and the existing governance structure, the project consultants, and assigned personnel proved inadequate to address the project’s needs. These problems did not go unnoticed. As the ECTP failed to deliver critical products on time and on budget, various reports and analyses were produced with a myriad of recommendations for improvement. Just three years into what was supposed to be a five-year project, DoITT recommended rebidding the main contract for system integration services, which had been let to Hewlett-Packard (HP). However, DoITT’s recommendation and many of those provided in connection with prior audits, investigations and expert studies were not acted upon in a timely manner—if at all—leading to further delays and increased costs.²

The Comptroller’s review also found a lack of transparency around the City’s expenses on the ECTP and its prior related emergency system technology upgrades. As a result, the project’s total cost was not clearly disclosed to the public. In 2004, the ECTP’s estimated capital cost was projected to $1.345 billion. ³ By last month, as detailed in the City’s Fiscal Year 2015 Capital Commitment Plan, the estimated cost had soared by 73%, to $2.326 billion. Even then, the Comptroller’s review found that these capital cost estimates were understated by at least $39 million, which reflects two contracts that do not appear to have been included as part of prior ECTP cost estimates.

Moreover, none of these projected cost estimates take into account the expenses that were and will continue to be incurred by DoITT and the other City agencies involved with the ECTP development and operations. ECTP maintenance costs alone are projected to cost DoITT $50 million in Fiscal Year 2015.
Further, our review found that the ECTP cost estimates do not appear to have fully accounted for relevant City’s expenditures to upgrade the 911 system incurred by the City prior to the ECTP.⁴ Only by including those expenditures can the complete cost of the ECTP be known.

When all of the above factors are taken into consideration–previously unaccounted for capital contracts, expense budget items, and relevant prior technological enhancements–our preliminary review indicates that the total cost to the City for the ECTP may have been understated by in excess of $200 million. However, a full audit of the City’s ECTP expenses would be required to identify the total program costs, which is beyond the scope of this review.

The absence of effective project governance and complete information about the cost of the ECTP hindered the City’s ability to properly evaluate, plan for, and manage an extremely complex multi-billion dollar project. Decisions about project scope, technology, outsourcing, product choices, vendor selection, and change order approvals should have been supported by complete information and should have been promptly acted upon. Further, effective project governance and full transparency were needed to ensure that past operational and management failures informed project planning and vendor selection.

To help address the issues encountered with the ongoing development of the ECTP and other active or future IT projects we offer the following recommendations:

• A project governance structure should be imposed on the ECTP that ensures that the City proceeds with a single vision, direct control, the ability to make decisions quickly, and the authority to see that all objectives are carried out. Such authority should not be delegated or outsourced to consultants who are not answerable to the public. Neither should authority be vested in so many City stakeholders that they are unable or unwilling to act quickly and efficiently in a coordinated manner.
• The City should directly employ appropriate technical experts to closely monitor the project in order to advise the contract’s governance committee on the project’s status and to make timely decisions.
• The City should disclose and consider all costs associated with the project in order to make fully informed decisions about its scope and progress.
• Where contractors are paid on a time and materials basis, the City should ensure that agency personnel properly review the consultants’ timesheets and vendors’ invoices before authorizing payments to contractors.
• The City should implement the controls set forth in Comptroller’s Directive 31 on all future contracts for technology services. Moreover, in light of the historic difficulties with the ECTP, where feasible, the City should seek amendments of existing contracts for technology services to conform with the Directive.