A Compilation of Systems development Audits and an Assessment of Citywide Systems-Development Strategy

Audit Report In Brief

Given the amount of taxpayer money spent on computer systems, the Comptroller’s Office has dedicated a portion of the resources of the Audit Bureaus to conduct audits of computer system-development projects implemented by City agencies.

Audits conducted by the information technology (IT) division during the period of fiscal years 2005 through 2009, have documented instances of mismanagement of system-development projects.  These instances of mismanagement have included: excessive cost overruns; missed deadlines; systems not developed as planned; and, systems that simply did not meet agency needs and that were abandoned.

Report Findings and Conclusions

For this compilation report we revisited the lessons learned from these audit reports when viewed in total.  We focused on the system development process and the costs associated with these projects.  Based on our re-evaluation, we conclude that up to $190.7 million of the $299.6 million IT system-development projects examined may have been poorly spent, specifically: up to $125.3 million on cost overruns; $50 million on a system that did not meet its initial business and system requirements; and, up to $15.4 million on systems that due to issues of functionality are at risk of not accomplishing the intended tasks.  In general, based on the results of our audits of IT system development projects, we have determined that the City has not created a successful unified City-wide strategy for developing IT systems.  As a consequence, the resources invested in these projects are at risk.

However, we did conclude that there appears to be an improvement in the process of developing IT system projects.  Our earlier audit reports identified cost overruns or funds wasted, as well as reservations regarding whether the systems met their original business and systems requirements and overall goals.  Our more recent audit reports disclosed systems that are operational, although they identified instances of deficiencies or incomplete deliverables from which it may be concluded that some portion of the associated investment in the system may be at risk.

Report Recommendations

To address these issues we make seven recommendations for improvement:

1. Management must be realistic about the results they want from the new system and when the system will be fully operational. The use of performance indicators can help identify potential problems early in the development.
2. Requirement planning should include all users that are able to specify the requirements precisely as to what the finished system should include in order for it to be well-designed and effective.  These users should be involved in planned tests, adequately trained as testers, and they must be allowed sufficient time to achieve the testing objectives.
3. Project time-frames should be short, which means that large system development projects should be split into separate modules.
4. The consistent use of the System Development Life Cycle as defined by Department of Information Technology and Telecommunications’ (DoITT) Project Management Office by all City Agencies.
5. An independent Quality Assurance (QA) consultant must be employed at the outset of project development with specific instructions to objectively evaluate the progress of the development and evaluate the performance of the vendor as defined in VENDEX (Vendor Information Exchange System) to augment the evaluation performed by the specific City Agency.
6. An Oversight Committee comprised of City representatives with technical expertise should be established to review all project plans to see if they are realistic.  Participants of the Oversight Committee should be encouraged to challenge the development team as to the viability of the timely completion of the project.  Also, this Committee should be empowered to monitor the progress of each technology project undertaken throughout the City with a specific ‘go or no go’ process. This would thereby help to close the void that currently exists in the development of system projects.
7. A team consisting of agency management, an independent oversight committee, and the QA consultant should evaluate the impact that requested changes (either legal or user specified) will have on system requirements, costs, and it should consider the magnitude of project risks caused by these changes.