Audit Report on the Development and Implementation of the New York City Fire Department’s Fire Inspection Revenue and Enforcement System

Audit Impact

Summary of Findings

The audit found that the New York City Fire Department’s (FDNY) Fire Inspection Revenue and Enforcement System (FIRES) replaced its legacy system, and some ancillary systems and manual processes. FDNY staff can now conduct mobile inspections, and the public can access a portal with self-service capabilities related to inspection, enforcement, and revenue activities.

However, FIRES did not fully achieve the major goals of automating processes and making them more efficient. This is due in part to the fact that certain inspection and enforcement functionalities were never implemented and FDNY staff continue to rely on manual processes and external systems.

A survey of FDNY staff revealed dissatisfaction with FIRES’ usability, functionality, and connectivity with inspectors’ mobile devices. In addition, a survey of public users revealed dissatisfaction with the public portal self-service capabilities, including the ease with which users can submit requests, schedule inspections, and communicate with FDNY.

In addition, FDNY did not provide evidence that FIRES was tested to ensure it met all the functional and technical requirements before deployment, and FDNY allowed FIRES to operate in an unsupported environment for two years. This exposed the system to potential security vulnerabilities and performance issues.

Intended Benefits

The audit’s recommendations are intended to enhance FIRES functionality to make it more user friendly and to meet business needs, and ultimately, to allow FDNY personnel to operate more efficiently.

Introduction

Background

In addition to fighting and responding to fires and other hazardous conditions, FDNY is responsible for preventing and mitigating them. The agency fulfills this goal by reviewing and approving plans for fire alarm, suppression, and communication systems, and conducting tests and inspections of these systems.

FDNY conducts inspections to ensure that buildings, structures, and other facilities comply with the New York City Fire Code, Fire Department Rules, and other laws and regulations. The agency also issues permits, licenses, and certificates related to the manufacture, storage, handling, and transportation of hazardous and combustible materials, among other things.

FDNY Inspections Structure

Inspections are primarily conducted by the Bureau of Fire Prevention (BFP) and the Bureau of Fire Operations (Fire Ops). (Please see Appendix I for a list of BFP and Fire Ops units and a brief description of the types of inspections they conduct.) BFP and Fire Ops may issue a Violation Order, Notice of Violation (NOV), or summons to property and business owners if they fail to comply with relevant laws, rules, and regulations. BFP has approximately 400 fire inspectors and conducts more than 250,000 inspections each year.

The Bureau of Legal Affairs (Legal Affairs) is responsible for processing and prosecuting NOVs and summonses.^[1] NOVs and summonses are resolved through the New York City Office of Administrative Trials and Hearings (OATH) Environmental Control Board (ECB) and New York State Criminal Court, respectively. Inspectors may be required to appear at OATH, ECB, and NYS Criminal Court proceedings.

The Bureau of Revenue Management (RM) is responsible for billing and collecting fees for plan reviews, tests, inspections, permits, licenses, certificates, and penalties for violations.

The Need for a New Inspection, Revenue, and Enforcement System

FDNY previously relied heavily on manual processes to schedule, conduct, and track inspections; issue and process violations and summonses; schedule hearing appearances and update hearing dispositions; and process appeals. Those processes were largely paper based, coordinated via phone calls and emails, and tracked in Excel spreadsheets and ancillary systems. In addition, FDNY used a 30-year-old, unsupported legacy system for the billing and collection of fees and fines.

FDNY proposed developing a new system to replace its legacy system and automate processes to make them more efficient, save time, and enhance data collection and oversight. In 2014, FDNY issued an Invitation to Bid to solicit a vendor with experience implementing Accela software to support the design, configuration, scripting, integration, and implementation of a new system.^[2] In 2016, FDNY contracted with GCOM Software, Inc. (GCOM) to develop and implement the FIRES.

Under the terms of the contract, GCOM was responsible for designing, developing, and implementing FIRES based on FDNY’s functional and technical requirements. FDNY’s Bureau of Technology Development and Systems (BTDS) was responsible for overseeing and approving those activities.

The major goals of FIRES included:

• Replacing an existing mainframe legacy system (the Fire Prevention Information Management System [FPIMS]), ancillary support systems, and manual processes used to support inspection, enforcement, and revenue management activities;
• Increased efficiencies due to more optimized scheduling and routing;
• The ability to conduct mobile inspections, reducing time spent on data entry at FDNY Headquarters, and increasing available time in the field;
• Automated inspection processes and enhanced data availability in the field; and
• Implementing a portal to allow the public to access information and provide self-service capabilities related to plan reviews, inspections, permits, licenses, certificates, and violations.

FDNY paid GCOM approximately $18.4 million to implement FIRES. It paid another vendor, Gartner, Inc., $5.5 million for quality assurance services and project oversight. FIRES went online in September 2021 and was operated on-premise through September 2023. In October 2023, FIRES was migrated to a cloud environment.^[3]

In addition to implementation costs, FDNY must pay Accela annual fees for license subscriptions and support. While FIRES was operating on-premise, FDNY paid Accela $544,912 per year for 800 user licenses and support. After FIRES migrated to the cloud, FDNY paid Accela $1,054,764 per year for 879 licenses and support.

Due to the sensitivity of the information and the potential risk associated with the release of certain information, this audit resulted in two reports: one restricted report covering access and security controls related to FIRES, and this one covering the development and implementation of the system, which will be released to the public.

Objectives

The objectives of this audit were to determine whether FIRES meets its overall goals as stated in the system specifications and has adequate functions to ensure its information process is reliable and secure.

Discussion of Audit Results with FDNY

The matters covered in this report were discussed with FDNY officials during and at the conclusion of this audit. An Exit Conference Summary was sent to FDNY on August 23, 2024, and discussed with FDNY officials at an exit conference held on September 16, 2024. On December 6, 2024, we submitted a Draft Report to FDNY with a request for written comments. We received a written response from FDNY on December 20, 2024. In its response, FDNY generally agreed with four recommendations (#5, #6, #7, and #8), partially agreed with one (#4), and disagreed with the remaining three recommendations (#1, #2, #3). In addition, FDNY disagreed with the audit’s findings regarding system testing and updates.

FDNY’s written response has been fully considered and, where relevant, changes and comments have been added to the report.

The full text of the FDNY response is included as an addendum to this report.

Detailed Findings

The audit found that FIRES replaced FPIMS, FDNY’s legacy system, and some ancillary systems and manual processes. FDNY staff are now able to conduct mobile inspections, and the public can access a portal which has self-service capabilities related to inspection, enforcement, and revenue activities.

However, FIRES did not fully achieve the major goals of automating processes and making them more efficient. This is due in part to the fact that certain inspection and enforcement functionalities were never implemented in the system, and because FDNY staff continue to rely on manual processes and external systems, including Microsoft Outlook and Excel.

A survey of FDNY staff revealed that approximately 33.1% of respondents disagreed or strongly disagreed that FIRES allows them to perform tasks easily and efficiently and that processes and workflows are streamlined and not complicated or time-consuming. A quarter of the respondents disagreed or strongly disagreed that FIRES improved business processes and workflows, data collection, and tracking compared to FPIMS.

In addition, although FIRES provides the ability to conduct mobile inspections, FDNY staff reported that the mobile application was not user-friendly and that they experienced frequent connectivity issues when using iPads in the field, which the audit team also observed. These issues keep FIRES from achieving its goals of enhancing data availability in the field, reducing time spent on data entry, and increasing available time in the field.

A survey was also distributed to public users to gauge their experience using FIRES’ self-service functions. Many respondents reported that they were dissatisfied or very dissatisfied with public portal self-service capabilities, including the ease with which users can submit requests (28.6%), schedule inspections (26.8%), and communicate with FDNY (32.6%).

The audit also found that FDNY did not provide sufficient evidence that FIRES was tested to ensure that it met all the functional and technical requirements prior to deployment. Furthermore, FDNY allowed FIRES to operate in an unsupported environment for two years. This exposed the system to potential security vulnerabilities and performance issues.

These issues are discussed in the detailed findings below.

FDNY Did Not Ensure that GCOM Implemented Required Inspection and Enforcement Functions

The contract scope of work as detailed in the Request for Proposals (RFP) stated that FDNY requires the new system (FIRES) to support inspection, enforcement, and revenue management functions. The contract provided a high-level overview of future state functional requirements (i.e., a description of the features and functionality that should exist in the new system), and further specified that the “Contractor shall be responsible for all configuration and code development required to meet the functional and technical requirements” for FIRES.

However, FDNY did not enforce the contract requirements and ensure that GCOM implemented certain functions that were delineated in the contract—including key functions for plan application and inspection workflows, automating inspection scheduling and optimizing routing, enforcement activities, detailed design specifications, and system testing documentation as detailed below.

Over the course of the contract, FDNY completed seven performance evaluations for GCOM covering the period July 1, 2016 through September 30, 2022. Although GCOM did not develop a system that met the future state functional requirements, FDNY gave GCOM ratings of satisfactory, good, or excellent, and did not hold GCOM accountable for its performance.

Unique Plan Application and Inspection Type Workflows Were Not Implemented

FDNY officials provided the audit team with a FDNY FIRES Project Acceleration Option, dated April 10, 2020, which detailed changes to the contract scope of work related to plan application and inspection type workflows, among other things. According to this document, FDNY wanted to make changes to the contract’s scope of work due to the impact of COVID-19 and the “desire to finish the project at a quicker pace.” The changes included removing from the contract scope the ability to create multiple application types that solicit different information, documents, plans, and other data by application type; routing the appropriate application components to different departments for review, if applicable; and validating that the application meets all requirements for the selected service. The changes also entailed removing unique inspection workflows.

FDNY omitted or reduced these plan application and inspection requirements in favor of a common plan intake and review model to support all plan types and a single inspection model to support all inspectional units. The only difference among inspections were checklists for each inspection type provided by FDNY.

FDNY officials informed the audit team that FDNY exercised FDNY FIRES Project Acceleration Option through contract change orders and in particular, Contract Change Order #4, which became effective May 26, 2020. However, this change order did not include the FDNY FIRES Project Acceleration Option as an attachment or otherwise reflect contract changes, as required by the Rules for City of New York.[4] Furthermore, even though the changes resulted in omitted or reduced work, FDNY did not make changes to the contract amount due to the vendor.

FDNY’s decision not to implement unique plan application and inspection workflows as intended, resulted in user dissatisfaction. In response to a user survey, FDNY employees expressed concern that creating a single workflow to suit unrelated processes, rather than multiple workflows tailormade to reflect specific programmatic needs, resulted in users developing workarounds outside the system.

Automated Inspection Scheduling and Routing Functions Not Fully Implemented

According to the contract, GCOM was to develop a system which was capable of automatically scheduling inspections and optimizing routing based on pre-defined business rules. Inspections were to be scheduled based on (1) inspector availability as documented in an online calendar, and (2) automated scheduling and routing functions. Once inspections were scheduled, they were to be posted in an online calendar. Further, inspection assignments and routes were to be optimized using a proprietary software (Click) or an equivalent solution. However, GCOM did not develop a system that met these functional requirements.

While BTDS internally developed and worked with the vendor to implement an automated inspection routing and scheduling process (RAS) to some degree, the process is inefficient in that supervisors must review proposed inspection lists and modify them based on staff availability maintained outside of FIRES (e.g., in Outlook calendars or Excel spreadsheets) and based on their knowledge and experience of feasibility and logistics (e.g., inspection type, location, duration, complexity, and staffing).[5] Supervisors must then finalize inspection dates and forward inspection assignments to staff.

FDNY does not require all inspection units to use this functionality—instead, each inspection unit decides whether and to what extent they use RAS versus manual processes to schedule and route inspections.[6] Some inspections are still conducted using paper checklists in the field and staff manually enter information in FIRES when they return to the office.

Although GCOM was required to implement functionality to automatically schedule inspection assignments if supervisors fail to finalize them within a predetermined time, FDNY did not enforce this requirement or ensure this control was implemented. Worse, FIRES automatically cancels inspections if supervisors fail to finalize them within a specified period and does not automatically reschedule them. FDNY officials stated that FIRES “will automatically cancel the inspection if no action has been taken by [the] Supervisor after 45 days.”

Since FDNY did not implement automated inspections and optimized routing as intended, FDNY may not be using its resources efficiently and effectively and may not be scheduling and conducting inspections in a timely manner (i.e., once per year, or once every two or five years as required). This increases the risk that inspections aimed at preventing or mitigating fires and other hazardous conditions are not being scheduled or conducted.

FDNY District Offices are responsible for conducting the largest volume of inspections. Based on an Overdue Annual Inspections report provided in May 2024, 49,834 inspections are past due. FDNY officials stated that the inspection backlog is due, in part, to the COVID-19 pandemic and other reasons such as inability to access inspection sites.

FDNY officials approximated that they would need to increase the number of inspectors by 30% to complete all inspections on time and resolve the backlog. However, overdue inspections may also be caused, in part, by inefficient scheduling, routing, and cancelations of inspections. Timely inspections help identify and mitigate issues that may pose risks to public safety.

By allowing staff to manually schedule inspections, FDNY has created an environment where irregularities can occur, including prioritizing or expediting certain inspections. For example, in February 2024, it was reported that the Department of Investigation was investigating whether FDNY inspections were being expedited in exchange for payments.

At the exit conference, FDNY officials stated that the RFP scope of work contained high-level requirements, and that contract requirements changed as FIRES was implemented and were documented in change orders. However, based on the audit team’s review of contract change orders, FDNY did not authorize changes to the scope of work for automated inspection scheduling and routing. Therefore, FDNY should have ensured that GCOM implemented this functionality.

Enforcement Activities Are Not Fully Automated

The contract required GCOM to automate enforcement activities which had previously been conducted using manual processes, worksheets, and ancillary systems. The contract detailed the then-existing business environment and functionality that should exist in the new system to automate enforcement activities for issuing, transmitting, and processing NOVs and summonses, scheduling and tracking hearing dates and scheduling inspector appearances, updating hearing dispositions, and processing appeals, as detailed in Table I below.

Table I: Existing Business Environment and Expected Future Functional Requirements for Key Enforcement Activities

In its proposal, GCOM stated that the system can be configured to meet the functional requirements, some of which are to provide the “ability to automatically schedule hearing participants” and “the ability to automatically update the status of appeal.”

However, FDNY did not enforce the contract requirements and ensure that all enforcement instruments are issued and transmitted electronically. Summonses are still issued on paper, using pre-printed forms and hand-delivered to Legal Affairs. Administrative staff record summons information in FIRES and hand-deliver summonses to NYS Criminal Court.

In addition, FDNY did not implement functionality to track and schedule court dates and inspector appearances through an online calendar. FDNY also did not implement bidirectional interfaces for hearing dispositions, which would streamline the business process.[7] FDNY allocated $190,500 of the contract amount for implementing this optional bidirectional interface that was not implemented, and did not hold GCOM accountable. GCOM received full payment for this work, which was not performed. As a result, Legal Affairs staff continue to rely on manual processes and external systems including Microsoft Outlook and Excel.

Legal Affairs officials stated that during the development phase, they requested to implement the above functionalities regarding the summons, hearing, and appeal processes so staff can perform their jobs more effectively and efficiently.

FDNY officials stated that tracking and scheduling court dates through an online calendar was not implemented and was not a valid requirement. However, they did not provide an explanation or any documentation to support this assertion.

FDNY officials further stated that they are awaiting resources to develop the feature for batch summonses to be served but did not provide an estimated timeframe to implement this feature.

Regarding the requirement to update hearing dispositions via an automated electronic data interface, FDNY officials stated that BTDS receives batch files from OATH daily which include ECB hearing dispositions. BTDS generates a report of hearing dispositions for Legal Affairs and posts it in a shared drive. However, this process is inefficient because Legal Affairs staff must still manually input information in FIRES.

FDNY officials stated that the appeal functions were developed in FIRES but were not in-use. The audit team discussed this issue with Legal Affairs officials, and they stated that the unit experienced challenges with inadequate system features and complex processes that required staff to perform numerous steps to complete their tasks. Therefore, the unit prefers to manually process appeals outside of FIRES. FDNY officials stated that they will work with Legal Affairs to develop and enhance system functionalities to streamline the appeal process within FIRES.

Without automated functionality for enforcement activities, FDNY risks cases being dismissed and delays in correcting violations critical to public safety.

At the exit conference, FDNY officials acknowledged the importance of automating enforcement activities and stated that they are working with Legal Affairs to develop and enhance system functionalities within FIRES. FDNY officials stated that the most important item to implement is a direct interface with OATH but indicated that there have been resource challenges at both FDNY and OATH which impacted implementation.

FDNY Did Not Provide Evidence of System Testing and Rollout Approval

As defined in the contract scope of work, testing services are activities that should be performed to confirm that “the individual components work together properly and as a whole perform their specified functions.” The contract specified that the minimum acceptable testing criteria should include product, system, system integration, regression, performance and volume, user acceptance, and pilot/parallel testing. (Please see Table II below for a description of each of the required tests.)

Table II: Description of Minimum Required Testing Services

Before initiating each test phase, FDNY was required to approve code review and test results in writing to ensure the system met the requirements and specifications. However, FDNY did not provide final approved test results for any FIRES releases. The audit team repeatedly requested testing documentation (including Final UAT results for each release and final sign-off) to determine whether FIRES met all functional and technical requirements and was ready for deployment.

In response, FDNY provided only one preliminary UAT result which covered testing of business workflows related to the submission, review, and approval of building safety plans and building systems for compliance with the New York City Fire Code and FDNY Rules. Within FDNY, the Emergency Planning and Preparedness Group (EPPG) is responsible for reviewing these plans and applications, and the Tech Management Unit is responsible for technical review and approval.

This UAT report stated that due to the significant number of defects and unavailability of some functionalities in the production environment, UAT could not be completed. The first system release—Release 1, Deployment 1 (R1D1)—would include only EPPG functions, and that Tech Management functions would have to wait until a later release.

The report also stated that “due to the unacceptable amount of defects, lack of confidence of the stakeholders and untested functionalities, it was decided that there will be another round of SIT and UAT.” BTDS also recommended a full review of all requirements and adoption of a Requirement Management Process and Defect Management Process.^[8] FDNY did not provide documentation to show that another round of tests was performed, that defects and outstanding issues were resolved, or that test results for R1D1 were approved, as required by the contract. The agency provided only an internal email stating that “GCOM and FDNY Project teams coordinated to get everything resolved, retested, and secure” and “everything for the release of R1D1 […] is a ‘go’.”

After the exit conference, FDNY officials provided the audit team with a FDNY FIRES Project Acceleration Option dated April 10, 2020, which detailed changes to the contract scope of work related to system testing, among other things. According to this document, FIRES testing would be executed, monitored, and controlled using JIRA, a test management tool used to record test results and monitor and track remediation of defects and issues. Further, JIRA exports would be used to satisfy the contract deliverables related to test plans and protocols, test scripts, and test exit criteria documentation. In addition, FDNY officials provided the audit team with a JIRA file that recorded certain system development and testing activities.

However, as previously mentioned, the FDNY did not reflect changes to contract scope in change orders and did not make changes to the contract amount for omitted and reduced work. Furthermore, based on the information provided, the audit team could not determine whether FDNY addressed and resolved all defects prior to rollout.

FDNY Did Not Provide Evidence that It Conducted Price and Cost Analysis for Changes to Contract Cost Related to Minimum Viable Product

The Rules for the City of New York, Section 4-02(b)(1)(ii), state that changes to contract amounts due to authorized omitted work require appropriate price and cost analysis to determine reasonableness.

Contract Change Order #7 stated that the payment amounts for milestones 14–16 would be reduced by $506,928.44 due to Minimum Viable Production (MVP) Product for deliverables, including 90 Inspection Checklist items and 19 Standard Form Orders. The change order stated that “MVP is a development technique in which a new product is developed with sufficient features to satisfy initial needs.”

Rather than reducing the contract price, however, the $506,928.44 was reallocated to milestones 17–20, which pertained to Release 3 enhancements. Change Order #7 did not detail the scope of work for those enhancements.

The audit team requested that FDNY provide the required price and cost analysis to determine the reasonableness of changes to contract amounts. FDNY officials stated they did not have this information but were sure that Project Managers “at that time had a precise assessment of the level of effort required which is generally details by resources and hours.” Without this documentation, FDNY cannot be assured that the changes to contract costs for the reduced scope of work for inspection checklists and standard order forms, was a reasonable and appropriate exchange for the added system “enhancements”.

User Survey Respondents Indicate Dissatisfaction with FIRES

To determine whether FIRES is meeting system goals from the perspective of those who use it, the audit team distributed two surveys: one to 670 FDNY credentialed users, and another to 3,200 external public users.^[9] The survey asked respondents to rate their experience regarding ease of use, support, and functionality. Both surveys revealed that approximately 25% of respondents were overall dissatisfied or very dissatisfied with FIRES.^[10]

FDNY Users Indicate FIRES Is Difficult to Use, Time-Consuming, and Inefficient

As previously stated, the major goals of FIRES included replacing FPIMS, automating processes, providing FDNY with the ability to conduct mobile inspections, and increasing efficiencies and available time in the field. However, a third of FDNY users (33.1%) who responded to the survey reported that FIRES does not allow users to perform tasks easily and efficiently, and that the processes and workflows are time-consuming.

In addition, 25% of respondents disagreed or strongly disagreed that FIRES improved business processes and workflows, data collection, and tracking as compared to FPIMS, the legacy system FIRES replaced. Significant percentages of FDNY users were also dissatisfied with FIRES training, ease of use, functionality, connectivity in the field, and support, as detailed below.

Many respondents who indicated that they were satisfied with or neutral about the system also provided negative feedback in their comments.

Training

The survey revealed that approximately 25.3% of respondents were dissatisfied or very dissatisfied with the training provided. Some respondents stated that the training was informal, disorganized, and insufficient, which required them to learn the system on their own. Respondents commented that the training material was already outdated and that no updates were provided to reflect the changes in the system. Inadequate training may prevent staff from working effectively and efficiently and could negatively impact business operations and productivity.

Based on contract requirements, GCOM developed training materials and provided “train-the-trainer” sessions for FDNY. These FDNY trainers later provided hands-on training to their staff. FDNY officials stated that newly hired inspectors receive basic training for FIRES, and individual business units are responsible for providing on-the-job training for employees who need to use FIRES to perform their job duties. Further, FDNY officials stated that they have optional e-learning materials for end users, such as training videos, user manuals, and quick reference guides.

The audit team attended a training class at the FDNY Training Academy to gain an understanding of the business operations and basic FIRES instructions for new inspectors. The training documentation provided the team with a general understanding of the business process and FIRES usage. However, some documents provided were dated August 2021—before the FIRES release—and do not reflect the current business workflow.

Ease of Use

The survey revealed that 33.3% of respondents disagreed or strongly disagreed that the system layout and design is easy to use and navigate. The survey also revealed that 24.8% of respondents indicated that entering data into the system is not easy, and 30.6% reported that the system is not user-friendly. Specifically, the respondents’ comments included statements that the system requires multiple steps to complete a task, and that inspections take longer than expected.

Functionality

According to 33.1% of respondents, FIRES does not allow the user to perform tasks easily and efficiently, and the processes and workflows are time-consuming. The survey revealed that 14.3% of respondents disagreed or strongly disagreed that FIRES had adequate functionality to perform their jobs, and 15.1% of respondents indicated that the system does not generate the documents needed to perform their jobs.

Some respondents commented that FIRES is limited in its capabilities, processes are cumbersome and take too many steps to complete, and inspections take longer. In addition, some respondents stated that the system lacks features that would allow them to generate criminal summonses and validate addresses.

Mobile Devices/Connectivity

Although FIRES provides the ability to conduct mobile inspections, the survey results indicate that the mobile application is not user-friendly and that there are frequent connectivity issues which impede FDNY staff from conducting inspections in the field.

Specifically, 21.6% of FDNY respondents disagreed or strongly disagreed that the mobile application was easy to use and navigate, and 27.4% indicated that it was not user-friendly. In addition, FDNY respondents provided comments regarding iPad slowness, redundant checklists and numerous steps required to complete inspections, and issues generating and printing documents and uploading pictures.

FDNY respondents also reported frequent connectivity issues when using iPads to conduct inspections in the field. Specifically, 32.3% of respondents indicated that they experience connectivity issues daily or multiple times a week, and an additional 43.8% reported connectivity issues at least once a week.

The connectivity issue was discussed with FDNY officials and noted during the on-site field observation. The auditors observed inspectors having to move around to search for an internet signal. The inspectors stated that they often experience this issue, especially in basements or subbasements, and as a result cannot record inspection data or complete the checklist. Further, the auditors noted the loss of internet connection on several occasions during inspections. The inspectors also experienced lagging issues that slowed down the inspection progress. The mobile device connectivity issue hinders the field inspection process and prevents inspectors from performing their primary tasks effectively and efficiently.

Frequency of Use

In response to the survey, 93.2% of respondents reported that they regularly use FIRES daily (82.9%), multiple times per week (7.9%), or at least once per week (2.4%). However, based on the audit team’s review of City Payroll Management System (PMS) records and the FIRES user list, 94 of 682 accounts (13.8%) were either never used or inactive for more than 90 days—29 user accounts were never used after they were created, and 65 accounts were inactive for periods ranging from 96 days to over three-and-a-half years.[11]

Public Users Indicate Difficulties Navigating the Site, Using Self-Service Options, and Communicating with FDNY Customer Service

As previously stated, one of the major goals of FIRES was to implement a portal allowing the public to access information, providing self-service capabilities related to plan reviews, inspections, permits, licenses, certificates, and violations.

However, significant percentages of public users who responded to the satisfaction survey indicated that FIRES is not user-friendly and that it is difficult to communicate with FDNY via the portal. Further, many respondents indicated that when they reported issues to Customer Service, those issues were either not resolved or took more than a month to resolve.

Ease of Use and Functionality

More than a quarter of respondents (28.5%) indicated that the portal layout is not easy to navigate or user-friendly. Roughly 28% reported they were dissatisfied or very dissatisfied with the ease of submitting application, inspection, or permit requests through the public portal.

An additional 26.8% reported that it is not easy to schedule, cancel, and reschedule an inspection or test, describing the online request process as “overly complicated, redundant, and repetitive,” “difficult to use,” and “almost impossible to update data [when] mistakes are made.” One respondent noted “various issues scheduling appointments for written tests and uploading documentation needed for the appointment and for qualification review.”

FDNY Communication and Customer Service

Roughly one-third of respondents reported that it is hard to communicate with FDNY within the portal. Many reported having to send multiple emails and getting either delayed responses or no responses. Another respondent noted, “It would also be helpful for there to be a way through the portal to send questions/follow up requests and receive responses directly.”

In addition, 36.9% of respondents indicated that when they reported issues to FDNY’s Customer Service Center, those issues were either not resolved or took more than a month to resolve. One respondent noted that they had been “calling the business number […] for days and never an answer [sic]. Really frustrating.” Another explained that, when their case had not been uploaded, “it was hard to get a non automated response” and that they “could not reach a human to assist.” Some respondents stated that they never received important documents despite receiving email confirmations that the documents had been paid for, processed, and/or mailed.

Without promptly addressing and providing timely resolutions to users’ inquiries, delayed submissions of application, request for inspection, and fines potentially result.

At the exit conference, FDNY officials stated that they are reviewing survey results, working with BFP to provide refresher training and materials to the users, and working with Accela to improve user interfaces. FDNY officials did not address survey results regarding mobile device connectivity issues or customer service.

FIRES Operated in an Unsupported Version for Two Years

Software support is essential for maintaining a secure, reliable, and effective technology environment. This occurs through system upgrades and technical support by the vendor. According to Accela, there are two major software upgrade releases per year to improve system functionalities, business workflows, and security compliance. In addition, Accela releases monthly “service packs” and “hot fixes” when necessary. However, FDNY did not update FIRES for two years and operated in an unsupported version of Accela while it was located on-premise from September 2021 through September 2023.

FDNY officials stated that they did not upgrade FIRES while it was operating on-premise due to resources, staffing, and the cost of maintaining and supporting the system, and because FDNY was planning to migrate the system to the cloud in the future.

In September 2022, FDNY entered into an agreement with the vendor Accela to migrate the system to a cloud environment. Under the terms of the agreement, Accela is responsible for future upgrades, maintenance, and security compliance for FIRES. According to FDNY officials, this project was expected to be completed by June 2023. However, the migration was not completed until October 2023.

For two years, FIRES operated with legacy software that failed to maximize and improve the system workflow and impacted user performance (i.e., for mobile application, geographic information, and online customer portal). In addition, without the updates, FIRES was at a greater risk of security breach and exposure of sensitive information, including names, email addresses, addresses, Social Security numbers, and Taxpayer Identification Numbers.

At the exit conference, FDNY officials stated that they did not experience any security breaches when FIRES was hosted and managed on-premise. They also stated that “the lack of updates to an application housed within our fully updated and tested servers posed no additional security risk” and updates pertained only to system performance enhancements.

However, according to Accela’s update release descriptions, they included new capabilities to “provide higher degrees of protection for sensitive information…[which] makes it easier to ensure the safety, security of the information held in records.” The updates also added capabilities to customize and enforce stronger passwords and support multi-factor authentication for mobile devices both of which reduce the risk of unauthorized access.

Recommendations

To address the abovementioned findings, the auditors propose that FDNY:

1. Consult with legal counsel regarding available contract remedies for FIRES functionalities that were to be provided under the contract but were not developed and tested by GCOM or approved by FDNY.

FDNY Response: FDNY disagreed with this recommendation, stating that “FDNY did not pay for any item specified in the Request for Proposal (‘RFP’) that was not implemented and, thus, commencing legal action would be inappropriate….

The RFP was formulated prior to 2014, and while the vendor (i.e., GCOM) was contractually obligated to implement functionalities specified in the RFP, the COVID-19 pandemic and the constraints of a fixed-price contract necessitated good-faith negotiations between FDNY and GCOM for several amendments…

Change orders specified the milestones and associated payment amount. If an item agreed upon in the RFP was removed, it was replaced with another item of commensurate complexity, time, and resources.”

Auditor Comment: The recommendation does not suggest FDNY begin legal proceedings, but that it consult with counsel regarding potential remedies.  As detailed in the report, FDNY did not specify the scope of work that was omitted, reduced, or added in change orders or provide evidence that the department conducted price and cost analysis to determine the reasonableness of changes to contract amounts as required by the Rules for the City of New York. In the absence of this information, FDNY cannot determine whether “replacement work” for omitted and reduced scope of work was in fact commensurate.

2. Consult with BFP and Fire Ops on business needs and fully implement an automated inspection scheduling and routing solution within FIRES based on inspector availability and profiles and require all business units to use for scheduling inspections.

FDNY Response: FDNY disagreed with this recommendation, stating that, “FDNY built FIRES in cooperation with the units that would utilize the program, and it was built to BFP specifications. It would not be functional for the business needs of the FDNY to have a fully automated system. The scheduling of inspections requires the approval of routes by supervisors on a weekly basis, which prevents a fully automated solution. The FDNY continuously works on improving the productivity of inspectors, including modifying the Routing and Scheduling System (‘RAS’) outside its original requirement and updating checklist.”

Auditor Comment: The FDNY’s response is contrary to the intent of the contract and major goals of FIRES. FDNY contracted with GCOM to configure Accela Software as the core solution to support BFP inspection scheduling and tracking, among other things. One of the major goals of FIRES was to automate inspection processes and increase efficiencies via optimized scheduling and routing.

As detailed in the report, the FDNY’s automated inspection scheduling and routing process, RAS, is inefficient and some units still rely on manual processes outside of FIRES to schedule and route inspections. Therefore, we reiterate that FDNY should consult with BFP and Fire Ops on business needs and fully implement an automated inspection scheduling and routing solution within FIRES, based on inspector availability and profiles, and require all business units to use for scheduling inspections.

3. Implement the feature to automatically alert units that an inspection requires scheduling and disable the feature that cancels inspections that are not assigned within a specified time.

FDNY Response: FDNY disagreed with this recommendation stating that, “This recommendation is not functional for the needs of FDNY. It is essential for units to have the flexibility to schedule inspections manually, especially in situations where immediate action is required for public safety…If cyclical inspections are cancelled, they are reintroduced through RAS. A monthly scheduling batch identifies accounts due for an inspection and processes them through RAS, creating a new inspection based on specified business rules. FDNY will review this to determine if any adjustments are needed to refine the process.”

Auditor Comment: During the audit, FDNY officials did not inform the audit team that RAS reintroduces canceled cyclical inspections or demonstrate this functionality.

Furthermore, although RAS may restore canceled cyclical inspections, not all inspections are cyclical and not all units utilize RAS. As a result, some canceled inspections may not be rescheduled promptly and may potentially be missed. Therefore, the auditors reiterate that FDNY should implement this recommendation.

4. Consult with Legal Affairs on business needs and implement functionalities and bi-directional interfaces to automate summons issuance and transmission, track and schedule court dates and inspector appearances through an online calendar, update hearing dispositions, and process appeals.

FDNY Response: FDNY partially agreed with this recommendation. FDNY acknowledged that “the OATH outbound interface has not yet been fully developed and implemented, which is due to unforeseen circumstances such as the COVID-19 pandemic and subsequent reduction in FDNY’s workforce. In the foreseeable future, we aim to allocate and mobilize resources to facilitate the completion of this task.”

However, FDNY stated that the department “implemented both hearing scheduling and appeals functionality that it demonstrated to the auditors and created with the input of the FDNY Enforcement Unit (“EU”). In addition, FDNY stated that “There is a daily inbound interface that updates FIRES enforcement records based on OATH’s events…Once OATH makes a decision, information such as disposition, adjournment, and rescheduled dates are updated in FIRES automatically as intended.”

Auditor Comment: As detailed in the report, FDNY did not implement functionality to track and schedule court dates and inspector appearances at hearings, and the process for updating hearing dispositions is inefficient because Legal Affairs staff must still manually input information in FIRES.

In addition, while FDNY developed appeals functionality, Legal Affairs staff do not use it because it is inadequate and overly complex. As a result, the unit continues to manually process appeals outside of FIRES. Therefore, the auditors reiterate that FDNY should fully implement this recommendation.

5. Conduct refresher training for all FIRES users.

FDNY Response: FDNY agreed with this recommendation.

6. Consider survey results, solicit additional feedback from internal and external users, and promptly address concerns related to FIRES’ ease of use, functionality, connectivity, and customer service.

FDNY Response: FDNY agreed with this recommendation.

7. Ensure that future information system development and testing contract requirements are fully met and results are approved and documented prior to system deployment.

FDNY Response: FDNY agreed with the recommendation but disagreed with the finding and stated that, “Before each major release, test plans and scenarios were created and executed. Failed defects were prioritized, and the high or medium defects were resolved before going live. FDNY will continue to utilize a similar process to meet future development and testing contract requirements before rolling out system updates.”

Auditor Comment: As detailed in the report, FDNY provided only limited  documentation to demonstrate certain system development and testing activities. FDNY did not provide sufficient evidence to show that failed defects were addressed and fully resolved prior to deployment.

8. Ensure FDNY information systems continuously receive updates to improve system functionality and optimize business workflow.

FDNY Response: FDNY agreed with the recommendation but disagreed with the finding that there was a greater risk of security breach and exposure of sensitive information because FIRES was not updated. FDNY stated that “There has never been an observed or reported case of a risk or potential risk due to the FIRES application functioning on an outdated version. Although an upgrade was necessary to optimize the out-of-the-box added functions and out-of-the-box intelligent dashboards, none of those upgrades were required to address security.”

Auditor Comment: As detailed in the report, FIRES did not receive Accela updates which included new security capabilities to provide higher degrees of protection for sensitive information, customize and enforce stronger passwords, and support multi-factor authentication for mobile devices.

The auditors found that the system did not comply with certain OTI security requirements when FIRES was hosted on-premise and those issues were resolved after the migration when FIRES was updated.

Recommendations Follow-up

Follow-up will be conducted periodically to determine the implementation status of each recommendation contained in this report. Agency reported status updates are included in the Audit Recommendations Tracker available here: https://comptroller.nyc.gov/services/for-the-public/audit/audit-recommendations-tracker/

Scope and Methodology

We conducted this performance audit in accordance with Generally Accepted Government Auditing Standards (GAGAS). GAGAS requires that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions within the context of our audit objective(s). This audit was conducted in accordance with the audit responsibilities of the City Comptroller as set forth in Chapter 5, §93, of the New York City Charter.

The scope of this audit covered the period from system development and implementation of FIRES in June 2016 through October 2024. At the FDNY’s request, this audit was postponed from August 2022 to February 2023, due to transition of FIRES operational environment and FDNY staff shortage.

To understand FDNY management structure and operations, the auditors reviewed the agency- wide and bureau-wide organization charts, the FDNY’s Mayor’s Management Report FY 2023 and FY 2024, and the New York City Comptroller’s Directive #1 Calendar Year 2017 and 2018 Checklists.

To understand the project scope and contractors’ responsibilities, auditors reviewed the FIRES contracts and related change orders, functional matrix, and the amendments for system development. Additionally, the audit team reviewed FIRES data migration and test plans, and user acceptance testing documentation to determine whether FIRES had quality assurance controls in place as part of system development and implementation. The auditors analyzed FIRES Project Acceleration Option document to determine whether project changes were properly documented. Additionally, the auditors reviewed the user manuals, training documents, and attended training session to gain an understanding of the system’s usages and business workflows. To understand the FIRES cloud migration process and operations, the auditors reviewed migration documentation that includes vendor agreement.

In addition, the audit team conducted system walk-throughs with the BFP, FireOps, Legal Affairs, RM, and Bureau of Technology Development and Systems to gain an understanding of how business units and the personnel performed their tasks and operations. The audit team also observed field inspections to understand the inspections process through mobile device.

Furthermore, the team reviewed the FIRES user role matrix to understand the roles and permissions for multiple units including Plan Intake, Plan Examination, Range Hood and Auxiliary Radio Communication System (ARCS) Inspections, Revenue Management, and Legal Enforcement. The auditors also performed system functional tests in a non-production environment to determine whether the system performed as intended and meet the overall goals as stated in the system specifications.

To obtain system user feedback, the audit team surveyed 670 internal FIRES users regarding system training, usage, functionality, ease of use, and reporting capability. The audit team also distributed another survey to 3,200 external public users regarding the system process and inspection scheduling, satisfaction, and support service. These internal and external surveys also provided information on users’ feedback regarding FIRES’ usability, functionality, information accuracy, and effectiveness of troubleshooting.

Although the results of the above tests were not projectable to their respective populations, these results, together with the results of the audit procedures and tests, provided a reasonable basis for the audit team to evaluate and support the findings and conclusions within the context of the audit objective.

Appendix 1

Inspection Unit Description

Appendix 2

FIRES Internal User Survey Results

 

Appendix 3

FIRES External Public User Survey Results

 

 

 

 

Addendum

See attachment.

---

Endnotes

[1] Violation Orders are issued by inspectors and processed and prosecuted by Legal Enforcement Unit. If conditions are corrected and compliance achieved, no court action is involved.

[2] In 2013, FDNY performed an analysis and determined that Accela, a commercial off-the-shelf software, could meet most of the new system’s functional requirements either out-of-the-box or with configuration or scripting.

[3] On-premise software is installed and hosted locally on an entity’s IT infrastructure; cloud software is hosted and managed by a third-party vendor.

[4] The Rules for the City of New York, Section 4-02, states that changes to contracts must be authorized and reflected in change orders, and changes in contract amounts due to authorized omitted work require appropriate price and cost analysis to determine reasonableness.

[5] RAS generates lists of recommended scheduled inspections for recurring tests and inspections which are required every one, two, or five years, and certain new inspections.

[6] FDNY units that use RAS include District Offices, and Bulk Fuel and Lab. Further, the Fire Suppression, Public Assembly, and High-Rise units partially use RAS. The Fire Alarm, Hazardous Cargo, Explosive, and ARCS units do not use RAS.

[7] Bidirectional interfaces allow for two-way transfer of sending and receiving information/data between two systems/programs.

[8] A Requirement Management process validates the functions needed for the business process. The Defect Management process identifies, documents, prioritizes, tracks, and resolve defects that arise during the development lifecycle.

[9] The team received responses from 165 of the 670 (24.6%) FDNY internal users and 473 of the 3,200 (14.8%) public users.

[10] The percentage is calculated according to the number of responses to each individual question and have been rounded to the nearest tenth.

[11] The 682 accounts belonged to users who were currently employed and with active status in PMS.