Audit Report on the New York City Police Department’s Oversight of Its Agreement with ShotSpotter Inc. for the Gunshot Detection and Location System

Audit Impact

Summary of Findings

The audit found that the New York City Police Department (NYPD) ensures that the billing, invoices, and payments to ShotSpotter are accurate, and that ShotSpotter generally meets its currently specified contractual obligations. The auditors found that sensor coverage areas were initially set up in Brooklyn and the Bronx because the two boroughs had the highest number of confirmed shootings, and, within the two boroughs, they were generally placed in precincts with the highest number of confirmed shootings. The audit also found that ShotSpotter met its contractual performance targets, which focus on avoiding “missed incidents,” most of the time.

However, the audit found that the contractual performance standard does not measure whether alerts sent to NYPD result in confirmed shootings. When measured against the contractual performance standards set by NYPD, ShotSpotter met its 90% target for avoiding missed incidents in almost all boroughs except Manhattan, but when measured against the number of confirmed shootings, performance is far lower. During the sampled months of review in 2022 and 2023, ShotSpotter alerts only resulted in confirmed shootings between 8% and 20% of the time.

During the month of June 2023, for example, out of the 940 ShotSpotter alerts that NYPD responded to 771 could not be confirmed as shootings upon arrival at the scene (82%), 47 were determined to be unfounded (5%), and 122 were confirmed as shootings (13%). NYPD officers spent 426.9 hours investigating alerts that were not confirmed as shootings. If only one officer responded, this equates to almost 36 twelve-hour shifts; if two officers responded, this number doubles.[1]  NYPD does not currently track the amount of time ─ or the associated staff costs ─ spent responding to such instances.

NYPD does not agree that confirmed shootings should be used to measure ShotSpotter’s performance.  It asserts that ShotSpotter improves the response time to possible shots fired which in turn increases the ability to provide assistance to victims, increases officers’ safety, and provides a more accurate location of the possible shooting than a 911 call alone.

However, NYPD does not measure ShotSpotter alert response times in comparison to 911 call response times to shots fired outside, the metric most closely aligned to ShotSpotter alerts. The audit analyzed the two data sets, NYPD’s internal OCD ShotSpotter Tracking report and the Open Data NYPD Call for Service report, and found that during the month of June 2023 average response times to ShotSpotter alerts were 1 minute and 38 seconds faster than response times to reports to 911 of outdoor shots fired (3 minutes 50 seconds for ShotSpotter versus 5 minutes 28 seconds for 911), far less than the difference of 5 minutes claimed in publicly-available data.^[2]

The audit found that NYPD’s data collection should be improved, analyzed more critically, and published in the interest of transparency before ShotSpotter’s contract, which expires in December 2024, is renewed. The data currently collected and published by NYPD does not support a comprehensive assessment of the tool’s effectiveness or economy, does not fully inform the public or government officials interested in ShotSpotter’s performance, and therefore does not currently support renewal of the contract.

Intended Benefits

The audit increases transparency around the use of ShotSpotter and raises questions concerning NYPD’s assessment of its performance.  NYPD has already spent over $45 million on this product and is committed to spending a further $9 million before the end of the current contract term. The audit calls on NYPD to critically and comprehensively assess ShotSpotter’s performance and determine whether its continued use remains in the City’s best interests.

Introduction

Background

ShotSpotter System

The ShotSpotter System (ShotSpotter) is a Gunshot Detection and Location System (GDLS) that uses acoustic sensors to detect gunshot activity to assist the NYPD in pinpointing the location of suspected gunfire within areas covered by the system. There are currently more than 2,000 sensors installed in the five boroughs. The system was created and is managed by ShotSpotter, Inc.^[3]

When ShotSpotter sensors pick up a suspected gunshot, that activity is reported to ShotSpotter’s Incident Review Center, which is staffed by ShotSpotter employees. Each recorded alert is analyzed at ShotSpotter’s Incident Review Center. The review is intended to confirm or change the machine classification of the incident type and is designed to be completed in less than 60 seconds.

Most activities detected by ShotSpotter sensors are reviewed and dismissed as non-gunfire. These do not result in alerts to NYPD. If the reviewer classifies the incident as a gunshot, ShotSpotter sends an alert to NYPD with details, such as number of shots detected, whether it appears that multiple shooters are involved, and whether high-capacity or fully automatic weapons are believed to have been used.

Next, the NYPD’s Operations Division receives an alert through their Domain Awareness system. The potential gunfire incident is created in the system and routed to the proper NYPD precinct, where police officers receive alerts on their phones.^[4] They may then be dispatched to the incident location to investigate. Closed Circuit Television (CCTV) cameras accessed by NYPD may also be used in conjunction with the GDLS to view the location detected by the system.

Community Concerns/Debates

Though ShotSpotter is widely used in major cities throughout the United States, there are ongoing debates over the use of this technology.^[5] Supporters have credited ShotSpotter’s detection with faster police response to shooting incidents, while critics have expressed concerns about the accuracy of ShotSpotter alerts. Critics note that false positives—such as fireworks or other loud sounds—may lead to over-policing in minority neighborhoods since officers are dispatched when no crime has been committed.

Critics have also expressed concerns about ShotSpotter’s ability to surveil private conversations. In July 2019, at the request of ShotSpotter, The Policing Project at NYU Law School conducted a Privacy Audit and Assessment of ShotSpotter, Inc.’s Gunshot Detection Technology. NYU’s audit found that the ShotSpotter technology “present[s] relatively limited privacy risks,” but also noted a small possibility that the technology “could capture voices of individuals near the sensors, and conceivably could be used for deliberate voice surveillance.”

The Policing Project’s audit made several recommendations which ShotSpotter states it has adopted. These included reducing the audio stored on their sensors, improving controls and supervision over audio access, agreeing not to share sensor locations, and denying requests and subpoenas for audio data.^[6] During this audit, NYPD informed the team that ShotSpotter does not disclose the exact location of its sensors to NYPD.

NYPD Contract with ShotSpotter

In 2014, NYPD entered into an agreement with ShotSpotter to implement the Gunshot Detection and Location System. Phase I of the agreement required ShotSpotter to set up and maintain the system in five coverage areas, which ShotSpotter divided into zones within the boroughs, covering about 15 square miles within New York City for one year. According to the agreement, the coverage areas “encompass NYPD precincts with not only the highest volume of confirmed shootings, but also with a high number of reported (911) shots fired calls.”

The agreement was amended once in August 2015 to expand into three additional coverage areas (Phase II), for a total of eight coverage areas.

At the conclusion of the first agreement, NYPD entered into a second agreement  with ShotSpotter in 2016 (Phase III) to maintain the existing coverage areas and to significantly expand to 20 coverage areas, with coverage of about 60 square miles. The contract requires ShotSpotter to provide all services and equipment that are necessary for the maintenance and expansion of the system. The terms of the 2016 agreement included four years of subscription services for the eight coverage areas under the 2015 agreement and five years of subscription services for the expanded areas under this agreement.

NYPD subsequently amended the agreement four times (Phases IV–VI) to further increase the coverage area to a total of 90.33 square miles. To date, all five boroughs have ShotSpotter sensors installed in areas with the highest rates of confirmed shooting incidents.

In 2021, the NYPD exercised the first three-year renewal option to extend the contract to December 2024. The three-year renewal increased the contract amount by $22 million, bringing the total contract amount to $54.6 million. Between August 14, 2014, and June 30, 2023, NYPD paid ShotSpotter $45.4 million.

2021 NYPD Contract Renewal and Performance Standard

When the agreement was renewed in 2021, the NYPD included a service level agreement (SLA) as an added level of oversight, establishing a performance standard where “90% of unsuppressed, outdoor gunfire incidents using standard commercially available rounds greater than .25 caliber inside the coverage areas will be detected and located within 25 meters of the actual gunshot location.” If ShotSpotter misses or misclassifies a certain number of gunshot incidents and consequently fails to meet the 90% performance rate in at least two months of the quarter for any borough, an SLA credit equal to ShotSpotter’s fees for two weeks of service for the affected borough is applied. The credit can be taken against ShotSpotter’s invoice for the next annual subscription term, or towards a ShotSpotter expansion, or a refund to NYPD if they elect not to move forward with the next subscription renewal.

Performance and the 90% target are defined in the contract. This and other metrics for measuring the effectiveness of the tool are discussed further below.

Objectives

The objectives of this audit were to determine whether NYPD properly monitors and evaluates the Gunshot Detection and Location System, properly reviews and approves ShotSpotter invoices, and to confirm whether sensors were initially placed in locations throughout the City with the highest shooting rates.

Discussion of Audit Results with NYPD

The matters covered in this report were discussed with NYPD officials during and at the conclusion of this audit. An Exit Conference Summary was sent to NYPD and discussed with NYPD officials at an exit conference held on November 17, 2023. On November 27, 2023, we submitted a Draft Report to NYPD with a request for written comments. We received a written response from NYPD on December 22, 2023. Following an updated review of NYPD’s response time to ShotSpotter alerts compared to 911 calls to report outside shootings and an analysis of time spent by NYPD concluding preliminary investigations of ShotSpotter alerts, information in the audit was modified and a revised Draft Report was issued on May 9, 2024. We received a written response from NYPD on June 3, 2024.

In its response, NYPD agreed with two recommendations, disagreed with one recommendation, and did not specify whether it agreed or disagreed with two recommendations.

NYPD’s written response has been fully considered and, where relevant, changes and comments have been added to the report.

The full text of NYPD’s response is included as an addendum to this report.

Detailed Findings

While NYPD generally exercised appropriate oversight over the vendor, the audit found that the contractual performance standard NYPD uses to measure ShotSpotter performance does not measure whether alerts sent to NYPD result in confirmed shootings.

The Performance Standard Adopted by NYPD Results in Artificially High Ratings for ShotSpotter

The contract between NYPD and ShotSpotter measures performance based on a formula that results in very high ratings for the tool in most boroughs except Manhattan. Since March 2022, ShotSpotter has been required to meet a performance target of 90% “of detectable (outdoor, unsuppressed) community gunfire which occurs within a coverage area.” Whether or not this target is met is based on the following formula:

Published incidents / (published incidents + missed incidents + false negatives) ^[7]

“Published Incidents” are recorded alerts identified by analysts in the ShotSpotter Incident Review Center as probable shooting incidents detected by ShotSpotter devices and reported to NYPD. Analysts examine the incident audio along with the “visual characteristics of the detected pulses and the incident, such as the number of participating sensors, the wave form, pulse alignment, and the direction of sound.” These factors determine whether an alert is sent to NYPD.

“Missed Incidents” are defined as confirmed shooting incidents that occur outdoors, within ShotSpotter coverage areas, that are not detected by the system. If NYPD is alerted by 911 or patrol of a shooting incident and no alert was sent by ShotSpotter, NYPD notifies ShotSpotter immediately of a missed incident.

“False Negatives” are shooting incidents incorrectly classified as non-gunfire based on a review by ShotSpotter personnel, or when automatically dismissed by the system. In these instances, no alert was sent to NYPD, but NYPD informed ShotSpotter that a shooting incident did occur.

To determine ShotSpotter’s performance rate, the previously mentioned formula is applied, and the total number of published incidents is divided by the combined total of published incidents plus all missed incidents and false negatives, for each period under review.

Applying this standard, ShotSpotter met its performance targets in Calendar Year 2022 in all zones except those located in Manhattan, where ShotSpotter failed to reach the 90% performance target in 8 of 12 months, with performance rates ranging from 78% to 89%.

In the months during which ShotSpotter did not meet its target, there was a higher number of both missed incidents and false negatives, as shown below in Table I.

The number of confirmed shooting incidents that were missed by ShotSpotter in Manhattan peaked at 25 in February 2022. In 10 of 12 months in 2022, ShotSpotter missed 10 or more confirmed shooting incidents. The number of false negatives ranged between 2 and 11 during this time.

Table I: Performance Rate in Manhattan (2022)

The auditors’ review of available data from previous years showed that this was a consistent problem between August 2019 and December 2022. During that period, ShotSpotter failed to meet the 90% performance rate in Manhattan in 30 of 41 months.^[8]

According to ShotSpotter, sensors did not detect some gunshot activity for several reasons including the high level of noise, the nature of construction in Manhattan’s Harlem area, and the density of buildings. ShotSpotter stated that it has taken steps to improve its performance in Harlem by moving and replacing sensors and investigating coverage areas.

Monthly reports show that between January and June 2023, the contractual performance rate has risen above 90% in all five boroughs.

Very Low Rates of Confirmed Shots Detected

When assessed according to the formula agreed upon by NYPD and ShotSpotter, the tool performs very well. However, the performance standard in the contract does not consider false positives or otherwise directly assess the tool’s ability to identify confirmed shooting incidents.

To do so, the auditors reviewed the data in NYPD’s ShotSpotter Tracking report (OCD report) to determine the correlation between ShotSpotter alerts and confirmed shooting incidents during sampled months of Fiscal Year 2023. NYPD’s Operations Division maintains the OCD report to track ShotSpotter data, including the number of alerts and confirmed shootings.  ShotSpotter alerts are considered confirmed shooting incidents when NYPD recovers evidence such as firearms, ballistics, or video, or if there are eyewitnesses, victims shot, summary arrests, or 911 calls that report a shooting.

The auditors initially reviewed the months of July 2022 and September 2022 and found that only 20% and 17% of ShotSpotter alerts in these months were confirmed shootings. Due to the low percentage of confirmed shootings recorded during these months, the sample was expanded to include the period from January through June 2023. During this period, the percentage of confirmed shooting incidents was even lower, dropping to rates ranging from 8% to 13%.   The analysis supporting these numbers appear below in Charts I and II.

On September 28, 2023, the auditors met with NYPD to discuss the low percentage of confirmed shooting incidents credited to ShotSpotter alerts. NYPD contends that the number of confirmed shootings in the OCD report is not an appropriate measure of performance because it only captures confirmed shootings based on the investigation conducted at the scene. NYPD claims that the number of confirmed shootings may be higher. NYPD suggests that later events not currently tracked may later change an alert’s disposition to confirmed; however, this assertion cannot be tested precisely because NYPD does not collect the necessary data.

Chart I: Number of Confirmed Shooting Incidents vs. Total ShotSpotter Alerts

Chart II: Percentage of ShotSpotter Alerts that Led to Confirmed Shooting Incidents

426.9 Hours Spent Responding to Unfounded and Unconfirmed Shots in a Single Month

Each time a ShotSpotter alert is deemed worth investigating, police officers are deployed to the scene. If evidence of a shot is not found, the deployment and any associated time spent investigating the alert represents a potential waste of resources. NYPD does not directly collect or analyze this information, but the auditors were able to estimate the time spent responding to ShotSpotter alerts during the month of June 2023, using the time of deployment in response to an alert, the “closing time” recorded in the Open Data report, and the reported disposition of the investigation that immediately followed. [9]

During the month of June 2023, there were a total of 940 ShotSpotter alerts (duplicates deducted) received by the NYPD. As shown in Table II below, the majority of ShotSpotter alerts were either unconfirmed (82%) where NYPD could not confirm a shot was fired due to lack of evidence, or the investigation resulted in an alert being categorized as unfounded/false positive (5%). Only 122 of the alerts (13%) resulted in confirmed shootings. However, NYPD officers were dispatched in response to all 940 alerts and time was spent investigating them.

NYPD officers responded to 47 unfounded and 771 unconfirmed alerts. The records show that officers spent an average of 20 minutes investigating unfounded alerts and 32 minutes investigating unconfirmed shots. This represents a total of 940 minutes spent investigating unfounded alerts and almost 25,000 minutes investigating unconfirmed alerts in a single month.  The two categories combined represent 426.9 hours during a single month.  If a single officer responded, that equates to 36 twelve-hour shifts; if two officers responded, the number of wasted shifts doubles (see Table II below).

Spread across a year, this potentially represents significant waste of officer hours. This in turn has fiscal consequences which the City can ill-afford.[10]  At a meeting held on May 15, 2024, and in its audit response, the NYPD explained that officers are not called in to do overtime to respond to ShotSpotter alerts since officers already on patrol assigned to the precincts will respond. However, the potential exists that in responding to unconfirmed alerts, and conducting unnecessary investigations, the overall use of officers’ time may be impacted and this in turn may affect the overall need for overtime.

Table II: June 2023 Average Time for Responding NYPD Unit to Complete Preliminary Investigation

NYPD Does Not Analyze or Publish  Comprehensive ShotSpotter Performance Data

NYPD’s evaluation of ShotSpotter’s performance is inadequate to hold the vendor accountable, and the data captured, analyzed, and reported are not sufficient to demonstrate the effectiveness of the tool.

NYPD Should Report Confirmed Shootings

Although NYPD has the capacity to count the percentage of alerts that are confirmed as shootings, it does not calculate or publish this figure. NYPD disputes that this is a useful measure of ShotSpotter’s effectiveness, and moreover contends that the actual number of confirmed shootings may be higher than calculated by the auditors. This is because it only captures the initial disposition after investigation in the OCD report.

NYPD does not track if a shot that is unconfirmed in the immediate aftermath of an alert is later confirmed as a shooting, but this is not a good reason not to measure ShotSpotter based on the detection rate of confirmed shootings. NYPD should capture and evaluate all information necessary to accurately determine the detection rate of confirmed shootings.

Knowing whether ShotSpotter accurately detects shootings is an essential piece of information needed to determine whether it is an effective tool.

NYPD Should Count Waste When Assessing ShotSpotter

NYPD does not currently assess the potential waste of officer hours spent responding to alerts that are unfounded or unconfirmed. NYPD has the data available to perform such an analysis, but the way data is currently captured makes this a laborious process. NYPD tracks the time from receipt of a ShotSpotter alert to the arrival of the first patrol unit, and the immediate disposition, in the OCD report used to capture ShotSpotter data, but it does not capture the end of the initial investigation or departure time of the officers dispatched in response to an alert. This information is instead captured as a “closing time” of each incident in the NYPD Call for Services Open Data report. To calculate the time spent responding to unfounded or unconfirmed alerts requires a manual matching of events in the two data sets, and then a manual calculation of time spent.

NYPD should ensure that it can readily identify and report on time spent by officers responding to unfounded and unconfirmed reports of shots fired—as well as the associated PS (staffing) costs, on a regular basis.

Tracking Relative Response Times

NYPD contends that response time to ShotSpotter alerts saves lives, but it does not currently capture and compare the difference in response times to alerts versus 911 calls made to report outdoor shootings. The auditors were able to derive an average response time by comparing two data sets, NYPD’s internal OCD ShotSpotter Tracking report and the Open Data NYPD Call for Service report, and found that during the month of June 2023 average response times to ShotSpotter alerts were 1 minute and 38 seconds faster than response times to reports to 911 of outdoor shots fired (3 minutes 50 seconds for ShotSpotter versus 5 minutes 28 seconds for 911), far less than the difference of 5 minutes claimed in publicly-available data.  Deriving the average response time was labor intensive; NYPD should improve its capacity to generate and review this data so that it can assess the relative response times on an ongoing and consistent basis.

During a meeting on May 15, 2024, and in its audit response NYPD contends that the time difference, in fact, is greater since the data cannot capture the time it takes for an individual to call 911. NYPD explained that ShotSpotter alerts are sent to NYPD on average within 60 seconds of reviewing the sound, while a person calling 911 to report a shooting can hesitate and wait minutes before calling 911. As stated at the meeting, it is not feasible for NYPD or the auditors to know or to analyze the number of minutes it takes a person to call 911 after hearing possible gun shots.

Nevertheless, NYPD’s responsibility for monitoring and overseeing ShotSpotter extends to ensuring that data needed to fully assess the tool’s effectiveness and economy are captured, critically assessed, and publicly reported.

Placement of Sensors Generally Correlates to Areas with High Shooting Rates

Because of public concerns about the impact of the system in neighborhoods with mostly minority populations, the auditors questioned how NYPD determined coverage areas.^[11] NYPD and ShotSpotter stated that coverage decisions were based on the number of confirmed shootings investigated by NYPD and 911 reports of shots fired. According to NYPD officials, the Office of the Chief of Department and the Bureau of Crime Control Strategies used shooting information from previous years and historical crime data to decide the initial coverages areas, using the same methodology each time the program expanded. The auditors’ review of the confirmed shooting data for 2014 and 2022 generally supports this assertion, with some caveats.

To determine whether sensors were initially placed in the areas with the highest number of confirmed shootings, the auditors reviewed and compared the confirmed shooting data for 2014 reported by NYPD on NYC Open Data and matched it to the sensors’ coverage areas in the related police precincts.^[12] This was complicated by the fact that shooting data is recorded by precinct, while coverage zones do not correspond directly to precinct boundaries—zones cover overlapping areas that may include one or more precincts or parts of them.

In 2015, NYPD decided to establish five coverage zones covering 14 precincts in Brooklyn (nine precincts) and the Bronx (five precincts). This was determined based on the prior years’ shooting data, which showed that Brooklyn and the Bronx had the highest overall shooting statistics of the five boroughs. This continued to be true through 2022.

See Chart III below for a comparison of total gunshot incidents by borough for 2014 and 2022.

Chart III: Total Confirmed Gunshot Incidents by Borough (2014 & 2022)

As shown in Chart III, in 2014, the Bronx and Brooklyn had the highest number of confirmed shootings—446 in the Bronx and 614 in Brooklyn.

Auditors also looked closely at the initial placement of sensors by precinct. The analysis found that sensors were not always placed in precincts with the highest number of confirmed shootings. While all but one of the precincts with the highest number of shootings were located within the Bronx and Brooklyn, sensors were not always placed in the precincts with the highest number of shootings within these two boroughs.^[13]

Sensors were initially placed in five zones encompassing 14 precincts—nine precincts with some of the highest number of confirmed shootings (ranging from 39 to 86 per precinct) and five neighboring precincts that had lower (but still high) numbers of confirmed shootings (ranging from 21 to 37 per precinct). However, there were five other precinct areas with high shooting rates (ranging from 38 to 59 per precinct) that were inexplicably not included in the initial placement of sensors.

Sensors were initially set up in the following precincts: 40, 41, 42, 46, and 48 in the Bronx; and 67, 71, 73, 75, 77, 79, 81, 83, and 90 in Brooklyn. They were not placed in precincts 43, 44, 47, 69, and 113—all of which had high shooting rates. These precincts were subsequently added to the coverage zones in 2016, 2017, and 2018. Table III below lists the years when coverage was added.

Mapping Confirmed Shootings

To examine the placement of sensors throughout the City, the auditors mapped confirmed shootings by borough and precinct. These maps show a heavy concentration of shootings in coverage areas in 2014 and 2022. The dots on Map I and Map II below show confirmed shootings by precinct in 2014 and 2022, respectively. Map III shows ShotSpotter coverage areas as of 2022, overlaid with confirmed shootings.

Map I: Total Confirmed Gunshot Incidents by Precinct (2014)

Map II: Total Confirmed Gunshot Incidents by Precinct (2022)

Map III: ShotSpotter Coverage Areas (2022)

For Calendar Year 2022, the auditors determined that sensors were generally placed in areas with the highest numbers of confirmed shootings. This was done by overlaying the ShotSpotter coverage areas on a map showing the number of confirmed shootings in 2022 for all boroughs and precincts.

Again, this effort was complicated in some cases, because ShotSpotter coverage areas do not directly correlate with precinct boundaries and can encompass multiple precincts, which can include smaller sectors within a precinct. For example, the 41st Precinct in the Bronx may not have the most shootings overall, but the area with sensors in the 41st Precinct overlaps with the 40th and 42nd Precincts in a zone with a high number of confirmed shootings.

Precinct Level Data

NYPD asserts that it established ShotSpotter zones based on the highest number of shootings and historical crime data, and further states that it precisely determined ShotSpotter zones based on higher shooting rates that did not correlate to precincts as a whole. As a result, ShotSpotter zones may encompass areas of multiple precincts that have many shootings within overall coverage areas.

This was difficult for the auditors to verify since NYPD does not know the exact location of sensors.  However, the coverage areas are broadly consistent with precinct-level shootings data. Had NYPD placed sensors based purely on confirmed shooting data available for each precinct, ShotSpotter would have been deployed in order of highest to lowest, in the precincts shown below, in Table III. Had NYPD relied solely on this data, four precincts—47, 44, 43, and 69—would have been covered sooner. All but one of the precincts below falls within the Bronx and Brooklyn initial coverage zones.^[14]

Table III: Top 14 Precincts by Shootings (2014)

NYPD Does Not Pay Invoices in a Timely Manner

Section 4-06 of the Procurement Policy Board Rules (PPB Rules) requires prompt payments to firms and organizations that do business with the City. It states that the invoice payment process should take 30 days or less, after the invoice receipt or acceptance date (IRA date).^[15]

After examining ShotSpotter invoices and the NYPD invoice payments for FY2021 through FY2023, the auditors found that NYPD took 94 days on average to pay invoices. NYPD stated that this was because many employees were at home during the pandemic and did not have access to the physical files at the time.

However, the auditors found that NYPD still paid its invoices late in 2023. For example, an invoice that was submitted in December 2022 was not paid until April 2023. NYPD stated that ShotSpotter has not issued any penalties for paying invoices after the 30-day deadline following the IRA date.

It is important that NYPD pay its vendors on time, as the PPB Rules allows vendors to charge interest on late payments. It is also the policy of the City to process contract payments efficiently and expeditiously to assure payment in a timely manner to firms and organizations that do business with the City.

Recommendations

To address the abovementioned findings, the auditors propose that NYPD should:

1. Decline to renew the ShotSpotter contract when it expires in December 2024 without first conducting a more thorough performance evaluation, considering the very low rates of confirmed shooting incidents detected, and factoring in the extensive NYPD officer time spent responding to alerts not ultimately confirmed as shootings.NYPD Response: NYPD disagreed with this recommendation, stating that “…non-renewal of ShotSpotter services may endanger the public and not renewing the ShotSpotter contract until the Department conducts further analysis would be a premature measure. The Department will continue to have discussions with ShotSpotter in order to enhance the performance and evaluation of the technology; however, it is not feasible to conduct an evaluation prior to the contract renewal period….In summary, loss of the ShotSpotter program would result in a less safe working environment for Officers and an increased chance of violent encounters for all New Yorkers.”

   Auditor’s Response: The auditors reiterate the need for NYPD to reassess the performance of ShotSpotter, and its ability to detect shootings, before the contract is renewed; we note that almost 6 months of the current contract term remains, allowing ample time to reconsider the product’s overall performance.

2. Develop more meaningful performance standards of ShotSpotter’s accuracy that better factor in the very high rate of alerts not ultimately confirmed as shootings.NYPD Response: NYPD did not disagree or agree with this recommendation.  It stated, “NYPD is limited in what it can consider a “confirmed shooting” in conjunction with a ShotSpotter alert by the nature of police work and alerts which don’t result in the recovery of evidence (i.e. ballistics, property damage, shell casings/live ammunition, firearms, video, ear or eyewitnesses and/or victims). As discussed in previous meetings, the Department is open to any auditor recommendations for improved standards calculations but have not received a recommended standard.”

   Auditor’s Response: The auditors’ findings show a significant difference in assessing the tool based on the contractual performance standard and assessing the tool’s ability to detect actual shootings.  This discrepancy is meaningful and should be addressed, NYPD contents in its response that confirmed shootings are not viable as a means to measure its effectiveness, in part because it contends that “confirmed shootings” does not include confirmations that come after more extensive investigation and other activities occur.  NYPD should be able to assess the overall accuracy of the tool in a manner which accounts for confirmations that come later in time as more evidence becomes available.

3. To increase transparency, NYPD should collect and publish relevant data, including the number of published alerts, percentage of alerts which result in confirmed shootings, the number of false negatives and missed incidents, time and staff costs spent responding to alerts that are not ultimately confirmed as shootings, and the relative response times to ShotSpotter alerts versus 911 reports of shots fired outside.NYPD Response: NYPD did not agree or disagree with this recommendation, it stated that “Shooting numbers are ever changing based on investigations and other factors that follow the shootings and therefore, parties that are not familiar with the data could misinterpret the information. In addition, this would need to go through various channels to determine feasibility and legality of such information request.”

   Auditor’s Response: The auditors reiterate the need for greater transparency.  There is nothing inherently confidential in the various data points recommended for publication and given the level of public debate concerning the merits of this tool, there is strong public interest in making the data available for review.

4. Continue to follow up with ShotSpotter, Inc. on coverage areas where the 90% performance rate is not met and ensure that ShotSpotter, Inc. is upholding the agreed upon Service Level Agreement Performance Rate.NYPD Response: NYPD agreed with this recommendation.

5. Pay its invoices within 30 days after the invoice receipt or acceptance date per the Procurement Policy Board Rules § 4-06.NYPD Response: NYPD agreed with this recommendation.

Recommendations Follow-up

Follow-up will be conducted periodically to determine the implementation status of each recommendation contained in this report. Agency reported status updates are included in the Audit Recommendations Tracker available here: https://comptroller.nyc.gov/services/for-the-public/audit/audit-recommendations-tracker/

Scope and Methodology

We conducted this performance audit in accordance with Generally Accepted Government Auditing Standards (GAGAS). GAGAS requires that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions within the context of our audit objective(s). This audit was conducted in accordance with the audit responsibilities of the City Comptroller as set forth in Chapter 5, §93, of the New York City Charter.

The scope of this audit was July 1, 2020 through June 30, 2023.^[16]

To obtain an understanding of the contract between NYPD and ShotSpotter, Inc. for the Gunshot Detection and Location System, the auditors obtained and reviewed the following:

• Procurement Policy Board (PPB) Rules – Rules of the City of New York – Title 9
• Comptroller’s Directive #1 – “Principles of Internal Control”
• NYC Administrative Code – Title 6 – Contracts, Purchases and Franchises
• NYPD Contract Agreements, Amendments and Renewal Agreement with ShotSpotter, Inc.
• NYPD Administrative Guide – Procedure 325-43

To achieve the audit objectives and obtain an understanding of NYPD’s oversight of its contract with ShotSpotter and its payments process, the auditors conducted walkthroughs with the following NYPD officials:

• The Director and an Auditor of NYPD’s Fiscal Accountability Unit,
• The Director and Captain of Fiscal Affairs Division of the Information Technology Bureau (ITB),
• The Deputy Inspector of ITB,
• The Deputy Inspector and two Lieutenants of NYPD Operations Division,
• The Deputy Chief of Crime Control Strategies

To obtain an understanding of the Gunshot Detection and Location System, the auditors conducted walkthrough with the following ShotSpotter Inc. officials:

• The Senior Vice President of Customer Support & Professional Services,
• Technical Support Engineer

To examine the accuracy and timeliness of NYPD invoice payments to ShotSpotter, the auditors independently extracted the Payment Request History per Vendor data file from the City Financial Management System (FMS) for FY2021 to FY2023 and compared them against ShotSpotter invoices for FY2021 to FY2023.

To determine the reliability of NYPD invoice payment data, the auditors examined CheckBook NYC data for FY2021 to FY2023 to compare with FMS data for FY2021 to FY2023.

To determine the effectiveness of the Gunshot Detection and Location System, the auditors judgmentally selected the month with the lowest performance rate (July 2022) and the most current month of data at the time of initial testing (September 2022) and analyzed the Scorecard Report from ShotSpotter and compared it against the alert tracking reports from the NYPD Operations Division and the ShotSpotter Misses repot. Auditors then further examined additional months from January 2023 to June 2023.

To determine the accuracy of NYPD Shooting Data, using the same two months selected above (July 2022 and September 2022), the auditors examined the ITB Bloodshooting report generated by NYPD and compared it against the Shooting Incidents Report from the NYC Open Data website.^[17]

The auditors examined two months of ShotSpotter activations by judgmentally selecting for testing the month with the lowest performance rate in FY2023 (July 2022), and the most current month of data at the time of initial testing (September 2022). Auditors further examined additional months (January 2023 to June 2023) as well.

To confirm that locations of ShotSpotter sensors were selected by NYPD based on highest number of shootings, the auditors reviewed gunshot data from 2014 to 2022 obtained from the NYC Open Data website. The auditors used mapping software Tableau to map out shooting locations and compared them with the ShotSpotter coverage area map provided by NYPD.

To obtain information on demographics of the ShotSpotter coverage areas, the auditors used Census 2020 data from the NYC Population FactFinder interactive map on the NYC Department of City Planning’s website (https://popfactfinder.planning.nyc.govhttps://popfactfinder.planning.nyc.gov). Auditors overlaid NYPD precinct borders on the Population FactFinder interactive map, then selected the approximate areas for demographics analysis based on the ShotSpotter coverage area map provided by NYPD.

To determine if ShotSpotter improved NYPD’s response time, auditors reviewed NYPD’s alert tracking report for September 2022, which was the most current month of data at the time of initial testing, and reviewed the ShotSpotter alert receiving time to the NYPD’s arrival time and calculated the average time. Auditors then compared the ShotSpotter alert average response time to the NYC 911 Reporting End to End Response time for critical incidents for September 2022. To conduct an updated and specific analysis on average response time, the auditors reviewed the NYPD’s ShotSpotter tracking report for June 2023 (the last month in our scope), and calculated the averages from incident to first unit arrival.  The auditors also conducted an analysis of Open Data report NYPD Call for Services and extracted ShotSpotter alerts in order to review the Closing Timestamp (time of final disposition) which is only found in the Open Data report. The auditors then calculated the average time responding officers took to close the preliminary investigation of unfounded, unconfirmed, and confirmed alerts.  We also calculated the average response time of 911 calls involving shots fired outside in order to obtain more specific information on the 911 end to end response to this specific critical incident (excluding all other types of incident not involving shots fired outside).

To test the accuracy and validity of the computer data from the Gunshot Detection and Location System, the auditors judgmentally selected the month with the lowest performance rate (July 2022) and the most current month of data at the time of initial testing (September 2022) and examined the ShotSpotter Scorecard report generated from ShotSpotter, the OCD ShotSpotter Tracking report, and ITB Bloodshooting report generated by NYPD. Auditors further examined additional months from January 2023 to June 2023.

To determine whether ShotSpotter billed NYPD in accordance with the contract, the auditors compared the ShotSpotter contracts and amendments against the invoice payments for FY2021 to FY2023. The auditors recalculated the invoice amounts based on the subscription fee rates listed in the contracts and amendments and then compared them to the invoice payments.

To determine whether NYPD paid ShotSpotter, Inc. the correct amount, the auditors compared FMS data for FY2021 to FY2023 against the ShotSpotter invoices for FY2021 to FY2023.

To determine if ShotSpotter met the MWBE vendors contract requirement, the auditors examined the MWBE Utilization report from NYPD and Checkbook NYC data to determine whether NYPD achieved the minimum of employing MWBE vendors for 4% of the total dollar value of the contract.

While the results of the tests are not projectable, they do provide sufficient and appropriate evidence to support the findings and conclusions as to whether the NYPD properly monitors and evaluates the Gunshot Detection and Location System and properly reviews and approves invoices.

Appendix 1

Analysis of Response Time

An analysis of publicly available information for the month of September 2022 found that the average response time from NYPD’s receipt of ShotSpotter alerts to the arrival of officers is four minutes and the average 911 call response time for critical occurrences is almost nine minutes. However, the End-to-End Response Time data available on the City’s website (https://www.nyc.gov/site/911reporting/reports/response-time-trends.page) is for “critical occurrences” but this category includes not only shots fired, but also assist police officer, robbery, burglary, larceny from person, assault with knife and assault with weapon.

The audit isolated data from 911 calls during June 2023 to identify response times to shots fired outside to compare to the response time to alerts generated by ShotSpotter, which are also outdoors.  The auditors reviewed and analyzed data for the month of June 2023 and determined the average response time from receipt of a ShotSpotter alert to first arrival of police officers, the average response time from receipt of a 911 call for critical occurrences that involved outdoor shots to first arrival of police officers, and the average time from the arrival of police officers in response to a ShotSpotter alert to the final disposition time (closing of preliminary investigation).

The auditors found that the average response time from receipt of the alert to the arrival of the first unit of police officers for 943 ShotSpotter alerts that occurred in June 2023 was 3 minutes and 50 seconds. Also, for this same 943 occurrences, the auditors found that it took an average of 43 seconds for ShotSpotter to determine that the sound was a possible gunshot and alert the NYPD.

Average Response Time for 943 ShotSpotter Alerts in the Month of June 2023:

• Average time from Incident to Alert to NYPD 43 seconds
• Average time from Receipt of Alert to Dispatch of Officers 50 seconds
• Average time from Dispatch to First Unit Arrival 3 minutes

For 911 occurrences, the auditors downloaded the Open Data report NYPD Calls for Services and extracted all the 911 calls deemed “critical,” which include categories such as, shots fired, assist police officer, robbery, burglary, larceny from person, assault with knife, assault with weapon, unusual incident and found that 723 calls involved “outside shots.”[18]  The average time for outside shots from receipt of 911 calls to arrival of the first unit of police officers was 5 minutes and 28 seconds.  According to NYPD, its Central Dispatch operations dispatch police offers for all incidents whether it comes through ShotSpotter or 911 calls.

For the 122 confirmed shootings alerted by ShotSpotter, there were 89 separate calls to 911 for the same incident. There were 33 confirmed incidents where there were no 911 calls (27%).

Appendix 2

Bronx ShotSpotter Coverage Area Demographics

Brooklyn ShotSpotter Coverage Area Demographics

Manhattan ShotSpotter Coverage Area Demographics

Queens ShotSpotter Coverage Area Demographics

Staten Island Coverage Area Demographics

Appendix 3

---

Endnotes

[1] An article from May 4, 2023, in Police 1.com entitled NYPD tests 12, 10-hour shifts to improve morale, retain officers reported that the NYPD had begun a pilot program of 12-hour shifts for police officers.

[2] The End-to-End Response Time data available on the City’s website (https://www.nyc.gov/site/911reporting/reports/response-time-trends.page) is for “critical occurrences” but this category includes shots fired, assist police officer, robbery, burglary, larceny from person, assault with knife and assault with weapon. The audit isolated data from 911 calls during June 2023 to identify response times to shots fired outside to compare to the response time to alerts generated by ShotSpotter, which are also outdoors.  For additional information on this analysis, please refer to Appendix 1.

[3] On April 10, 2023, ShotSpotter changed its name to SoundThinking, Inc. Auditors use “ShotSpotter” throughout the report since it was the name used for most of the audit scope period.

[4] All NYPD personnel can access snippets of the ShotSpotter alerts through the NYPD’s Domain Awareness System via an application on their phones.

[5] Chicago, Denver, and Houston are some of the major cities that use ShotSpotter.

[6] It is possible that a court could order ShotSpotter to comply with subpoenas.

[7] The formula used to determine ShotSpotter’s performance rate is calculated using information from both ShotSpotter and NYPD. ShotSpotter has the numbers of published incidents (incidents alerted to NYPD) and NYPD provides ShotSpotter information on missed incidents or false negatives. The terms used in the contract differ from the terms used in ShotSpotter’s Scorecard; however, the definitions essentially are the same. For this audit, we used the terms used in ShotSpotter’s Scorecard.

[8] The performance target was added to the contract in March 2022 and allows NYPD to deduct a fee when the target is not met. The auditors verified that credits for quarters two and three of Calendar Year 2022 were deducted from invoices in accordance with the contract.

[9] NYPD stated that the closing time may not always be the time the police officers leave the location. The closing time recorded in the Open Data report is when Central Dispatch is informed of the final disposition which can happen at any point during the investigative process, whether the officers are still at the location or have moved on to another incident.

[10] NYPD’s use of overtime has grown dramatically in recent years, and much of it has been spent on uniformed services of the type needed to respond to ShotSpotter alerts.  See Overtime Overview: A Deep Dive into NYPD Uniformed Overtime Costs  https://comptroller.nyc.gov/reports/overtime-overview/.

[11] For specific information on the demographics of the neighborhoods, please refer to Appendix 2.

[12] NYC Open Data is a website containing datasets published by City agencies. It allows New Yorkers to review information that is produced and used by City government. The web address is https://opendata.cityofnewyork.us/

[13] The exception was the 113th Precinct, which is located in Queens. Sensors were not placed in this precinct until 2017 even though it had a high number of shootings.

[14] The 113th Precinct is located in the JFK area of Queens, and this borough was not part of the initial coverage area. See Appendix 3 for a complete list of precincts and neighborhoods.

[15] According to § 4-06 of the Procurement Policy Board Rules, the IRA date is “the date a proper invoice is actually received by the designated billing office if the agency annotates the invoice with the date if receipt at the time of receipt.”

[16] To confirm that locations of ShotSpotter sensors were selected by NYPD based on the highest number of shootings, auditors reviewed data from 2014 through 2022.

[17] The ITB Bloodshooting Report includes all incidents during which a person was shot.

[18] In this analysis the auditors excluded any ShotSpotter alerts that were misclassified under Outside Shots and any events with duplicate numbers which as explained by NYPD were multiple reports received of an incident. The focus was on independent 911 calls for outside shots.