Audit of the Department of Health and Mental Hygiene’s Emergency Contract with Executive Medical Services PC for COVID-19 Related Services

AUDIT IMPACT

Summary of Findings

The audit found that the Department of Health and Mental Hygiene (DOHMH) established testing and vaccination services in communities most affected by COVID-19, as required. The audit also found that sampled invoices submitted to DOHMH were generally well supported by documentation, but that certain improvements could be made in monitoring future contracts.

DOHMH did not implement a monitoring plan with reasonable, clear, and understandable standards for assessing contractor performance, and primarily relied on the vendor to set staffing levels that were later billed and paid for under the terms of the contract.

Based on a review of invoices totaling more than $200 million in payments for COVID-19 tests and vaccinations, the staffing levels set by EMS PC were not well monitored. As a consequence of this, the costs of administering tests and vaccinations were not controlled.

On average, the ratio of administered tests/vaccinations to staffing hour paid for by DOHMH was less than one to one.  An analysis of sampled invoices shows that only one test was administered approximately every 1.61 staff hours paid at the eight long-term/fixed testing sites, and at the 23 long-term/fixed vaccination sites, only one vaccination was administered approximately every two staff hours.

The auditors also found significant variations in cost by location. An analysis of sampled invoices shows an overall average cost of $405 per test, but that New York City paid between $202 to $937 per administered test, depending on location. Similarly, the overall average cost per vaccination at long-term/fixed sites in the sample was $334. However, the variance by location was even greater, with the average cost per vaccination per site ranging from $169 to $2,423.

In addition, the audit found that DOHMH did not complete the vendor performance evaluation for the EMS PC contract within the specified timeframe, as required by the New York City Procurement Policy Board (PPB).

Details concerning each of the findings follows below.

Intended Benefits

The audit identified areas where DOHMH’s oversight and monitoring of contracted vendors could be improved to more effectively control expenditures and to minimize waste.

INTRODUCTION

Background

DOHMH’s mission is to protect and promote the health and well-being of New Yorkers. DOHMH engages with communities to develop and implement public health programming and policy recommendations, enforces health regulations, responds to public health emergencies, and provides limited direct health services.

DOHMH is integral to New York City’s COVID-19 public health emergency response. According to the Fiscal Year 2021 Mayor’s Management Report, DOHMH’s top COVID-19 priorities in 2020–2021 were to rapidly address any resurgence in COVID-19 infections; to vaccinate New Yorkers for influenza and COVID-19; to partner with City communities to develop effective local strategies; and to address “parallel pandemics” related to COVID-19, including other infections (e.g., tuberculosis), chronic diseases (e.g., heart disease), mental illness, drug abuse, racism, and social and economic instability.

As part of its response to the COVID-19 pandemic and the emergency declaration, DOHMH entered into an emergency contract with Executive Medical Services PC (EMS PC) on behalf of the City on May 22, 2020.[1] The contract stipulated that EMS PC would provide COVID-19 shelter and at-home COVID-19 swab tests, with an initial term of May 22, 2020 through May 21, 2021. During this period, DOHMH would reimburse EMS PC for funds expended, with a Maximum Reimbursable Amount (MRA) of $500,000.

The contract was amended six times between November 2020 and September 2022. The scope of services to be provided, the term of the contract and the MRA were all impacted. Services were expanded to include non-COVID 19 services and services in residential as well as community-based settings. The term of the contract was ultimately extended through December 31, 2022, with a final MRA of $460 million. According to the City’s Financial Management System (FMS), DOHMH paid EMS PC approximately $389.8 million for services under this contract by March 15, 2023.

Community-based (non-adult care facility) testing and vaccination services provided under the EMS PC contract consisted of both long-term/fixed and pop-up sites throughout the City. Long-term/fixed sites were generally permanent locations, mostly in brick-and-mortar buildings, that provided services for several weeks to months at a time, whereas pop-up sites were temporary locations providing services for short time periods. The majority of the pop-up sites for testing and vaccination were one-day events, with some sites being operational for several days.

Most of the sites staffed by EMS PC and covered under the contract were pop-up sites. There were 9 long-term/fixed and 166 pop-up testing sites, and 36 long-term/fixed and 101 pop-up vaccination sites operating at different points in time during the contract period . (A complete list of these testing and vaccination locations is presented in Appendices III and IV, respectively.)

During the COVID-19 emergency, DOHMH was tasked with establishing testing and vaccination sites throughout New York City. For testing services, DOHMH stated that it used community-informed and data-driven approaches to increase the uptake of COVID-19 testing in priority neighborhoods, with a focus on neighborhoods with longstanding inequities and the highest burden of disease. For vaccination services, DOHMH stated that the City’s efforts emphasized equitable access to vaccines while focusing on high-risk populations, low coverage neighborhoods, and locations near public transportation for increased access.

According to DOHMH officials, decisions on sites to prioritize COVID-19 testing and vaccination services throughout New York City were a collaborative effort that involved DOHMH leadership, City Hall, the Department of Education (DOE), the Health and Hospitals Corporation (H+H), and other City agencies and key stakeholders. EMS PC was one of the providers that the City used in its COVID-19 response efforts. DOHMH officials further stated that many vaccination sites (whether operated by EMS PC, H+H, or others) were set up in neighborhoods identified by the NYC Taskforce on Racial Inclusion & Equity (TRIE).[2]In addition, officials stated that decisions about utilizing EMS PC to provide vaccination and testing services (e.g. the sites to set up) were determined in partnership with City Hall, Incident Command System (ICS), and other City agencies, and occurred at selected sites, including City Hall, parks, Rikers Island, DOE schools, and community- and faith-based organizations.

The scope of this audit was limited to community-based COVID-19 related services over the period from May 2020 to December 2022.

Objectives

The objectives of this audit were to determine whether DOHMH:

• ensured that payments to the vendor were allowable and fully supported;
• ensured that staffing levels set by the vendor were commensurate with demand; and
• equitably established testing and vaccination sites consistent with contract parameters.

 Discussion of Audit Results with DOHMH

The matters covered in this report were discussed with DOHMH officials during and at the conclusion of this audit. An Exit Conference Summary was sent to DOHMH on April 24, 2023 and discussed with DOHMH officials at an exit conference held on May 2, 2023. On May 11, 2023, we submitted a Draft Report to DOHMH with a request for written comments. We received a written response from DOHMH on May 30, 2023.

In its response, DOHMH agreed with three of the audit’s five recommendations (#1, #2, and #5), disagreed with one recommendation (#4), and believed one recommendation (#3) was not needed as the process was already in place.

DOHMH also disagreed with the audit’s finding that the agency did not adequately control and monitor EMS PC’s staffing levels, and stressed the context within which they were working. The auditors acknowledge upfront the many challenges faced by DOHMH during this time. While the audit recognizes that DOHMH was responding to an unprecedented pandemic situation without the benefit of official cost control guidance, effective cost-management was not incompatible with DOHMH’s overarching mission, and indeed, is an essential part of emergency preparedness.

As contracts of this nature proceed, it is important that New York City’s fiscal interests are safeguarded by prudent systems of oversight and monitoring. DOHMH’s written response has been fully considered and, where relevant, changes and comments have been added to the report.

The full text of the DOHMH response is included as an addendum to this report.

DETAILED FINDINGS

The auditors found that DOHMH was able to establish testing and vaccination services in communities most affected by COVID-19, as required. The audit also found that sampled invoices submitted to DOHMH were generally well supported by documentation, but that certain improvements could be made in future contracts.

In fiscal terms, based on the audit’s review of sampled invoices totaling more than $200 million for COVID-19 tests and vaccinations , the auditors found concerning indications that staffing levels were not well monitored or controlled  and that, as a consequence, the number of tests and vaccinations administered and paid for under the contract averaged fewer than one per staff hour. Additionally, the audit found that significant variations in cost occurred by location. This raises questions about whether staffing levels established by the vendor and paid for by New York City were truly necessary.

The auditors also found that DOHMH did not complete the vendor performance evaluation for the EMS PC contract within the specified timeframe, as required by the New York City Procurement Policy Board (PPB).

DOHMH Did Not Adequately Control or Monitor Staffing Levels

DOHMH did not implement a monitoring plan with reasonable, clear, and understandable standards for assessing contractor performance, and relied heavily on the vendor to set staffing levels that were later billed and paid for under the terms of the contract.

The contract provided for EMS PC to be reimbursed based on the number of staff on a per hour basis, plus a flat fee per test or vaccination. Costs were as follows:

• Testing sites: $150 per hour, plus $100 per test administered;[3] and
• Vaccination sites: $150 per hour, plus $25 per vaccine administered.

The hourly rate paid to EMS PC under the contract included both administrative and personnel costs.

According to Section 5(A) of the original EMS PC contract, DOHMH was responsible for developing and implementing a monitoring plan that specified “reasonable, clear and understandable standards” for assessing contractor performance and included a “time frame for all management and oversight functions.” Further, according to Section 5(B), the contractor’s performance was to be evaluated based on, among other things, the achievement of the level of services.

DOHMH did not establish or communicate optimal staffing level benchmarks or standard service levels for administering tests or vaccinations, or design any formal mechanisms for assessing EMS PC’s staffing levels on an ongoing basis. DOHMH’s monitoring plan did not set standards for assessing whether EMS PC appropriately staffed vaccination and testing sites based on demand. DOHMH did not capture the data necessary to determine how many EMS PC staff members were directly testing and vaccinating patients and how many were providing support. As a result, DOHMH was unable to effectively determine the number of tests and vaccines administered per hour by each EMS PC staff member assigned and DOHMH had limited means of effectively controlling expenditures or minimizing waste.

During the audit and in its response to the draft report, DOHMH disputed the accuracy of this finding. DOHMH officials indicated that for testing sites, the assignment of staff members was a collaborative effort between DOHMH and EMS PC. DOHMH directed EMS PC when and where to provide services for COVID-19 testing and handled general communications with EMS PC, but DOHMH conceded it was EMS PC’s responsibility to assess and determine the number of staff members needed for an event.

For vaccination events, DOHMH indicates that EMS PC was given guidance regarding staffing levels and states that staffing levels were adjusted each week based on anticipated demand and other factors. DOHMH officials asserts that they utilized appointment data, on-hand inventory of various types of vaccines, and floor plans to determine the staff level required at each vaccination site in a calendar grid and shared the grid with EMS PC weekly.

DOHMH officials stressed that in the middle of the COVID-19 pandemic emergency, the agency was focused on providing testing to as many people as possible to help control the spread of COVID-19. Officials said that EMS PC supplied sufficient staff to register patients, monitor patient flow, and administer COVID-19 tests, while managing patient influx, so people could be tested as frequently as necessary and minimize wait times.

According to DOHMH, its contracted testing and vaccination operations required more staffing at certain sites, in part due to the size and/or layouts of the facilities being used. DOHMH also explained that there may have been additional time needed to test and vaccinate patients because of several factors, including language barriers, a patient’s age (e.g., younger children), acquiring additional screening information if necessary, and social distancing measures. Officials also cited safety measures as a significant factor that was critical to the vaccine operation, as required by the Centers for Disease Control and Prevention (CDC), that played a large part in determining the staffing levels.

DOHMH also states that assessments were performed on a regular basis and that they involved a close review of event summaries, site operations data, and verbal conversations. DOHMH indicates in its response to the draft report that such communication was provided to auditors.

While the auditors were given copies of communication with EMS in relation to staffing at Riker’s Island, examples of emails between DOHMH and EMS PC regarding vaccination sites, and copies of weekly calendar grids showing DOHMH-approved staffing levels, DOHMH did not provide evidence of communication regarding staffing levels at any other testing sites and it provided no documentation illustrating how its staffing assessments were performed or how approved staffing levels were established.

DOHMH also disagreed with the audit’s assessment that the agency’s inability to effectively determine the number of tests and vaccines administered per hour by staff assigned to perform such duties played any role in cost control. DOHMH stated that the CDC required that both clinical and non-clinical staff fulfill key roles at testing and vaccination sites. However, this requirement does not negate the audit’s finding that DOHMH was unable to assess the actual service levels at the sites because it did not know how many EMS PC staff members were in clinical versus non-clinical roles.

DOHMH further stated in its response that it has strong internal controls to prevent and detect fraud, waste, and abuse and these controls were in place during the management of the EMS PC contract. Nevertheless, the auditors have concluded that the absence of a quantitative monitoring plan that included an assessment of staffing levels contributed to both (1) an average of less than one test or vaccination administered for every staff hour billed to the City, and (2) the wide variance among the sites in the average costs of the tests and vaccinations conducted, which are discussed in the following section.

As discussed further below, the results of the invoice review clearly show that costs were not effectively controlled and that DOHMH’s ability to ensure that only necessary staff were paid for by New York City was limited.

Invoice Analysis Shows on Average Fewer Than One Test or Vaccination Administered per Every Staff Hour Billed and Per Unit Costs of Services Varied Significantly

The auditors reviewed a judgmental sample of 49 invoices (and supporting documentation) totaling $317.7 million. Of this total, $239.1 million was paid to EMS PC for COVID-19 testing and vaccination services provided at 315 sites for either the entire duration of a testing/vaccination event or until the month of June 2022 (whichever was sooner).[4] Breakdowns of the types of events and the amounts paid for these 315 sites are shown in Tables I and II below.

An analysis of these invoices shows that at the eight long-term/fixed testing sites, an average of 0.62 tests were administered per staff hour paid to the vendor. In other words, invoices show that one test was administered approximately every 1.61 staff hours paid in the sample.

Similarly, a review of invoices for the 23 long-term/fixed vaccination sites found that an average of 0.49 vaccinations were administered in one staff hour; therefore, one vaccination was administered approximately every two staff hours . The analysis showing the average number of tests and vaccinations per staff hour for the sampled long-term/fixed sites is shown below in Tables III and IV.

The auditors also found that the per unit costs of services varied significantly among the sampled long-term/fixed sites. An analysis of sampled invoices shows an overall average cost of $405 per test. However, the amounts in the sample varied significantly among locations; New York City paid between $202 to $937 per administered test.

The overall average cost per vaccination at long-term/fixed sites in the sample was $334. However, the variance by location was even greater, with the average cost per vaccination per site ranging from $169 to $2,423.

A series of Tables and Charts appears below. Additional information regarding sites included in the sample also follows in the Appendices.

Table I: Number of Sampled Sites Analyzed

Table I, above, shows a summary of the total number of invoices and the total number of sampled long-term/fixed sites reviewed by the audit. There were 23 invoices for COVID-19 testing that included billing for eight long-term/fixed sites and 164 pop-up sites. In addition, there were 26 invoices for COVID-19 vaccinations that included billing for 23 long-term/fixed and 120 pop-up sites. In total, there were 49 invoices reviewed and 315 sites included in the audit’s analysis.

Table II, below, shows a summary analysis of the total payments made for these 315 sites, including the total number of staff hours and patients serviced, and the audit’s calculations of the average cost per patient for the long-term/fixed sites, pop-up sites, and overall average considering both, broken down by testing and vaccination services. In total, there were 1,434,746 billed staff hours to provide testing and vaccination services to a total of 706,817 patients, totaling $239.1 million.

Table II: Summary Analysis of Payments Made for the 315 Sampled Sites

$15 million of the approximately $17.6 million paid for testing services shown in Table II was for services provided at long-term/fixed sites ; this can be seen from Table III below, which shows an analysis of the payments made for the eight long-term/fixed testing sites selected.

Table III: Analysis of 8 Sampled Long-Term/Fixed Testing Sites

Table III shows that other than services administered at City Hall, which had an average of almost four tests administered per staff hour, the average number of tests administered did not exceed 0.64 tests per staff hour. Overall, on average, 0.62 tests were administered at the eight sampled long-term/fixed sites per staff hour.

Table IV, below, shows an analysis of the payments made for the 23 long-term/fixed vaccination sites selected. Of the $221.5 million paid for vaccination services, almost $218 million related to services provided at long-term/fixed sites.

Table IV: Analysis of 23 Long Term/Fixed Vaccination Sites

As shown in Table IV, except for services administered at Manhattan Worth Street and Manhattan Chelsea, which had an approximate average of one vaccination per staff hour, the average vaccination rate did not exceed 0.82 vaccinations per staff hour. Overall, on average, 0.49 vaccinations were administered at the 23 fixed/long-term sites per staff hour.

The significant variance in costs among the sampled long-term/fixed testing sites is further illustrated in Chart I below, in which the average cost per test at each site is compared to the average cost of $405 per test for the eight sites overall . The site that was open for the longest duration (16 months) was at Rikers Island Correctional Facility . This site was for the facility personnel and visitors. DOHMH stated that the duration of the Rikers Island program was set by the Mayor’s Office.

Chart I: Average Cost per COVID-19 Test for Sampled Long-Term/Fixed Sites

As shown in Chart I, the average cost per test at three long-term/fixed locations—Christopher Blenman NSC, Grant Square NSC, and Hugh Gilroy NSC—was more than twice the overall average cost among all sites. According to DOHMH, these locations are DFTA sites for which the primary clients were older adults, a population with high vulnerability. Consequently, it was important that these sites remained open. Nonetheless, there may have been opportunities to reduce the number of staff that EMS PC utilized at these sites.

The significant variance in costs among the sampled vaccination sites is further illustrated in Chart II below, in which the average cost per vaccination at each site is compared to the average cost of $334 per vaccination for the 23 sites overall. The vaccination services provided at Morrisania Emergency Clinic were more than seven times the average and almost $1,000 more than the second highest average cost ($2,422 versus $1,427). The average cost was high because the demand for vaccination was very low at Morrisania from March to June 2022 and staffing levels were not adjusted accordingly.

Chart II: Average Cost per COVID-19 Vaccine for Sampled Long-Term/Fixed Sites

According to DOHMH, the three locations (Empire Outlets, Morrisania Emergency Clinic, and Edenwald YMCA) with highest average cost per vaccine were in areas—St. George (in Staten Island), Morrisania (in the Bronx), and Edenwald (in the Bronx), respectively—where residents had high vaccine hesitancy. They also argued that these sites were located in at-risk neighborhoods identified by the TRIE taskforce suffering from a disproportionate impact of COVID-19. However, as suggested with the DFTA sites mentioned above, there may have been opportunities for EMS PC to reduce the number of staff utilized at these sites. Please refer to Appendices I and II for further analysis of testing and vaccination sites examined during the audit, including the pop-up sites. As noted above, because DOHMH does not collect data distinguishing between support staff and staff administering vaccinations and tests , staff hours recorded in invoices included both categories.

Appendix I shows that the average cost per test was $250 or more at 91 (55%) of the 164 pop-up sites. At 82 (50%) pop-up sites, an average of one test or less was administered per staff hour.

Appendix II shows that the average cost of administering vaccinations was $500 or more at two-thirds of the 120 pop-up sites the auditors reviewed. The overall average vaccination rate per staff hour was one or less in 114 (95%) pop-up sites.

DOHMH officials argued, in response to the finding, that the testing and vaccination sites were generally non-traditional, non-clinical sites that were being used for clinical purposes, and therefore needed more staff members, providing services in various roles, than would be needed at a typical clinical setting.

In its response, DOHMH disagreed that the lack of monitoring on its part played a role in this finding, stating that the response to COVID-19 services varied among communities, the pandemic was constantly evolving over the time period reviewed, and activities were affected by circumstances beyond its control or monitoring.

The auditors disagree, based on the existence of significant variances among locations in staffing and persons served over extended periods of time, and the absence of evidence that DOHMH’s monitoring led to awareness of such variances or that it took any action to address them.[7]

DOHMH also took exception to the use of judgmental samples in conducting the invoice review. However, the use of judgmental sampling is entirely consistent with the audit standards mandated for use in the City Charter, and moreover, the size of sample was significant, covering $239.1 million (61%) of the $389.8 million paid under this contract through March 15, 2023.

The auditors find no basis to amend this finding.

Significant Portion of Testing and Vaccination Sites Were Placed in Communities Having Significant Black and Hispanic Populations

During the audit DOHMH provided considerable information concerning the methodology it employed in establishing sites. The auditors requested DOHMH to provide the methodology and data used for selecting 30 sampled sites—16 testing sites and 14 vaccination sites. However, DOHMH did not provide the site-specific data that was directly used for the selection of the 30 sampled sites requested and the auditors were not able to assess the extent to which the protocols were followed for 16 sites (8 testing and 8 vaccination). This was largely due to the absence of documented decisions. The remaining 14 sites (8 testing and 6 vaccination) were in neighborhoods identified by the TRIE as suffering from a disproportionate impact of COVID-19.

Nonetheless, a review of the locations selected in connection with this contract confirm that a significant proportion of sites were placed in communities that had significant Black and Hispanic representation, aligning with DOHMH’s preliminary findings that Black and Hispanic people were disproportionately impacted by COVID-19.

An analysis of the 59 community districts by ethnicity showed that 29 districts had Black and Hispanic households comprising more than half of the population. Testing services were provided in 16 (55%) of those districts and vaccination services were provided in all 29 districts.

Based on the American Community Survey (ACS) 2014–2018 (as reported on the Department of City Planning website), Black and Hispanic households formed the majority of households in 29 (49%) of the City’s 59 community districts. Maps showing the proportion of Black and Hispanic households in each district and (1) the 175 testing locations and (2) the 136 vaccination locations operated by EMS PC are shown below in Figures 1 and 2, respectively. As shown, 73 (42%) of the testing sites and 81 (60%) of the vaccination sites were in districts in which Black and Hispanic households made up the majority . According to DOHMH officials, the testing locations designated to serve the surrounding communities were the pop-up sites. The long-term testing locations were generally for the purpose of testing City employees; they did not cover community/ households, omitting the long-term testing sites from the analysis, the percentage breakdowns remained relatively unchanged. Of the 166 pop-up testing locations, 69 (42%) of them were in districts in which Black and Hispanic households made up the majority.

Figure 1: DOHMH-Contracted Testing Sites Operated by EMS PC and Proportion of Black and Hispanic Households by Community District

Figure 2: DOHMH-Contracted Vaccination Sites Operated by EMS PC and Proportion of Black and Hispanic Households by Community District

Based on the above, auditors found that a significant percentage of districts with large Black and Hispanic populations were among those in which testing and vaccination sites were located in connection with this contract.

Protocols for Ensuring Appropriately Supported Payments Were Generally Adequate

The audit found that DOHMH generally had adequate controls over its payments to EMS PC under the emergency contract. However, one of the controls instituted by DOHMH (signoffs of summary reports) was not implemented until seven months into the contract term and was not consistently enforced.

Section 4.3 (“Control Activities”) of Comptroller’s Directive #1, “Principles of Internal Control,” states, “Internal control activities help ensure that management’s directives are carried out. They are, basically, the policies, procedures, techniques, and mechanisms used to enforce management’s direction.” In addition, Section 5.7 (“Segregation of Duties”) states that “[k]ey duties and responsibilities need to be divided or segregated among different staff members to reduce the risk of error or fraud.”

DOHMH instituted several controls to help ensure that staff were aware of their responsibilities, responsibilities were appropriately segregated and payments were reviewed. Those controls included:

• Written Procedures Governing the Invoice Review Process: DOHMH had three written procedures governing its planning, monitoring, and invoice review processes for COVID-19 testing services provided at congregate residential settings/adult care facilities and community events, and for vaccination services. These documents provided detailed descriptions of the invoice review process and descriptions of the required planning and monitoring of testing and vaccination services.
• Segregation of Duties over the Payment of Invoices: DOHMH’s program officials responsible for overseeing the testing and vaccination services conducted initial reviews and approvals of the invoice-supporting documents submitted by EMS PC. DOHMH’s COVID Contract Manager or Contract Analyst conducted a secondary review to double-check the accuracy of the rates applied and the calculation in the invoice before DOHMH’s Disease Control Division processed the payments.
• Periodic Invoice Reviews: DOHMH’s Fiscal Management Unit within its Finance Division conducted periodic post-payment reviews. As of May 12, 2022, the unit reviewed 18 invoices totaling approximately $145.7 million. The unit identified 21 instances of duplicate billing and other errors totaling $133,400 and recouped the funds from EMS PC.

DOHMH Did Not Consistently Sign Off on Summary Reports

In December 2020, DOHMH began requiring EMS PC to submit daily summary reports to provide additional details about the testing events that EMS PC held. (Vaccination events did not begin until January 2021.) Prior to that date, DOHMH officials did not know the number of EMS PC staff present at congregate or community testing events until DOHMH received the next monthly invoice.

The summary reports established in December 2020 were to include the number of EMS PC staff present and the number of patients tested at community events and adult care facilities. In addition to an EMS PC official, the adult care facility representative also signed the reports to attest to the numbers reported. However, for community events, only EMS PC signed the reports as DOHMH personnel were not always present at the testing sites. For vaccinations administered at fixed and pop-up sites, the report was signed both by EMS PC and DOHMH Team or Regional Leads. This instruction appeared in DOHMH AP Vaccination Site: Guidance Document.

This was a useful control measure that was designed to ensure that only services that were provided were reimbursed. Unfortunately, the summary reports for testing services were not established until seven months after the start of the contract, and the reports were not consistently signed off by agency staff to attest to the numbers reported. This raises concerns about whether they were reviewed.

The audit found that 49 (24%) of the 201 reports reviewed lacked the required signature of a DOHMH employee . These reports were used by DOHMH to support payments to EMS PC totaling $2.3 million . DOHMH stated that a possible reason for a missing signoff was that DOHMH staff and the AP [Affiliated Physicians] Program Manager may have missed each other at the end of a shift, leaving a summary report unsigned. However, DOHMH could not quantify the number of signoffs that were missed for this reason.

The effect of this was that DOHMH did not have independent confirmation of the number of EMS PC staff and hours worked at testing events, prior to making payment. DOHMH paid 12 invoices totaling $5.4 million for testing services prior to December 2020.

Supporting Documents Generally Justified DOHMH Payments

Apart from the above, the auditors found that DOHMH generally collected adequate information to verify the testing and vaccination services provided by EMS PC . DOHMH obtained the number of COVID-19 testing administered at adult care facilities and community events from independent sources, such as Maven and DOHMH’s Incident Command System (ICS) Integrated Data Team.[8] [9]

The auditors reviewed sampled invoices and found that the documents supporting these invoices and DOHMH’s review of them were generally adequate. The 10 sampled invoices totaled $48.9 million— seven for vaccination services ($46.4 million) and three for testing services (totaling $2.5 million).[10]

DOHMH Did Not Timely Complete the Required Vendor Performance Evaluation

The auditors found that DOHMH did not complete the vendor performance evaluation for the EMS PC contract within the specified timeframe, as required by the New York City Procurement Policy Board (PPB).

According to Section 4-01(b) of the PPB Rules, the contracting agency is required to conduct an annual performance evaluation, which should be recorded in the City’s centralized procurement platform (currently PASSPort).

Auditors reviewed PASSPort and found that although the system created a draft performance evaluation for the initial contract term of the EMS PC contract timely (May 22, 2021, one year after the contract start date), DOHMH did not prepare the evaluation until July 22, 2022, more than a year after the PPB Rules required that it be completed. According to the PASSPort evaluation document, the evaluation end date was June 21, 2021.

DOHMH officials stated that “[d]ue to COVID-19, the PE [performance evaluation] was not created and assigned to an evaluator until 2022.” Furthermore, DOHMH did not complete the 2021–2022 performance evaluation until February 28, 2023, more than eight months after the required completion date. The timely recording of performance evaluations in PASSPort assists other City agencies by providing information on vendors that other agencies are considering for the award of new contracts.

RECOMMENDATIONS

To address the above-mentioned findings, the auditors propose that for future contracts, DOHMH should:

1. Develop and implement an effective and standardized process for assessing and setting appropriate staffing levels that are commensurate with demand, and document the results of these efforts.

DOHMH Response: DOHMH agreed with this recommendation.

2. Conduct regular cost analyses after the initial stage of contract implementation, including—when feasible—comparisons to other providers of the same services to continuously monitor and adjust for efficiency.

DOHMH Response: DOHMH agreed with this recommendation.

3. Ensure that all verification documents (e.g., summary reports) include appropriate agency certification that the services claimed by the contractor were provided.

DOHMH Response: “The Health Department believes that this recommendation is not needed because a receiving report process is already in place.”

Auditor Comment:  DOHMH previously indicated that its receiving report process does not include a verification that the number of staff reported by EMS PC on its invoices was accurate. During the audit period, auditors found that almost a quarter (24%) of the summary reports—intended to independently confirm the number of staff and provided services as reported in EMS PC’s invoices—were not properly certified by designated personnel. The auditors urge DOHMH to reconsider its position and implement the recommendation.

4. Ensure that it formally documents key decisions regarding site selections, including reasons for opening a site, its duration, and the closing of a site.

DOHMH Response: “The Health Department disagrees with this recommendation as it is not needed. As stated above, the COVID-19 pandemic response was a collaborative effort. Key protocols and mechanisms were in place to deal with decisions surrounding testing and vaccination sites.”

Auditor Comment: Formally documenting key decisions provides a level of transparency to stakeholders and facilitates subsequent reviews of those decisions and their impact. The auditors continue to recommend that key decisions such as these on future contracts be formally documented.

5. Ensure that required vendor performance evaluations are completed in PASSPort at least once annually.

DOHMH Response: DOHMH agreed with this recommendation.

Recommendations Follow-up

Follow-up will be conducted periodically to determine the implementation status of each recommendation contained in this report. Agency reported status updates are included in the Audit Recommendations Tracker available here: https://comptroller.nyc.gov/services/for-the-public/audit/audit-recommendations-tracker/

SCOPE AND METHODOLOGY

We conducted this performance audit in accordance with Generally Accepted Government Auditing Standards (GAGAS). GAGAS requires that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions within the context of our audit objectives. This audit was conducted in accordance with the audit responsibilities of the City Comptroller as set forth in Chapter 5, §93, of the New York City Charter.

The primary audit scope was May 22, 2020 to December 31, 2022.

To obtain an understanding of the policies, procedures, and regulations governing the testing and vaccination services provided by EMS PC, the auditors reviewed and used as criteria:

• Original contract agreement between DOHMH and EMS PC and subsequent amendments;
• DOHMH Planning, Monitoring, and Invoice Review for Community Testing Services;
• Congregate Settings Investigation and Response Unit (CSIRU) Invoice Review and Approval Process;
• Disease Control Division Management & Systems Coordination (DMSC) Invoice Review and Approval Process;
• DOHMH planning, monitoring, and invoice review of Executive Medical Services (EMS), Specimen Collection & Diagnostics in Congregate Residential Settings;
• COVID-19 Vaccination Services: DOHMH Planning, Monitoring, and Invoice Review;
• DOHMH-AP Vaccination Site: Guidance Document;
• Centers for Disease Control (CDC) COVID-19 Vaccination Program Interim Operational Guidance Jurisdiction Operations;
• CDC COVID-19 Vaccination Program Interim Playbook for Jurisdictions Operations Annex; and
• DOHMH Interim COVID-19 Vaccination Plan Executive Summary.

The auditors reviewed the NYC Office of the Comptroller’s Internal Control and Accountability Directive #1, “Principles of Internal Control,” to understand key internal controls and processes integral to an agency’s achieving its mission and assuring full accountability for its resources, including processes used by agency management to provide reasonable assurance regarding the achievement of objectives, efficiency of operations, and compliance with applicable laws and regulations.

The auditors also reviewed samples of the following documents to obtain a further understanding of the information available to DOHMH personnel who monitor the provided testing and vaccination services:

• Summary Reports submitted from EMS PC to DOHMH after each day of a testing and vaccination event starting in December 2020;
• AP Test Tracking schedule;
• NYC Daily Track Report containing daily information on vaccination appointments scheduled and doses administered submitted daily by EMS PC and containing legacy data prior to implementation of Salesforce;
• Salesforce reports of daily COVID-19 vaccination doses administered starting in April 2021 and fully adopted in May 2021;
• NYC calendar grid of scheduled COVID-19 vaccination appointments;
• AP Maven Test Count Report ACF;
• CORP Taskforce Program Update: 11220;
• Invoice supporting schedule;
• AP Point Prevalence Survey (PPS) schedule, used to help plan the anticipated number of residents to be tested at an adult care facility and the necessary number of nurses for EMS PC to bring; and
• Vaccination clinic floor plans identifying clinic patient flow and staff distribution.

The auditors reviewed the following documents associated with DOHMH’s decision to select EMS PC as a vendor to provide COVID-19 testing and vaccination services:

• Request for Quote for mobile vaccination teams and related addenda;
• Request for Quote: Community vaccination contractors for the 2019 novel coronavirus (2019-nCoV) emergency response and the response form completed by EMS PC’s response;
• Community Vaccinator Vendor Contact List;
• Vaccination Response Form completed by 4 vendors; and
• Specimen collection Response Form for emergency purchase: In home specimen collection and transport, completed by EMS PC.

To obtain an understanding of the reimbursement process for federal emergency funds expended by the City, the auditors reviewed the Public Assistance Program and Policy Guide Version 4 (effective June 1, 2020) and samples of the following documents necessary to obtain a reimbursement of state and federal COVID-19 assistance funding:

• CDC ELC Enhancing Detection Monthly Financial Report;
• Health & Human Services (HHS) Annual Federal Financial Report;
• NYC ELC Cares Budget Template; and
• Supplemental IP19-1901 Enhanced Influenza Campaign and COVID 19 Vaccine Response Preparation Budget Narrative.

To obtain an understanding of DOHMH’s payment process for the COVID-19 testing and vaccination services provided by EMS PC, the auditors selected and reviewed one invoice package for testing services provided in August 2020 (totaling $1.2 million) and one package for vaccination services provided in April 2021 (totaling $4.8 million). The packages included the invoice , the supporting schedule , and a receiving report.

To obtain an understanding of the controls that were relevant to the audit and to help achieve the audit’s objectives, the auditors interviewed relevant DOHMH personnel to learn about the planning, monitoring, invoice review, and approval processes of testing and vaccination services provided by EMS PC under the emergency contract. The auditors also interviewed DOHMH officials involved with the oversight of testing and vaccination services and periodic assessments of staffing levels, establishing the emergency contract with EMS PC, and preparing and submitting requests for reimbursement of state and federal COVID-19 related funding. To understand DOHMH’s process of selecting and monitoring testing and vaccination sites, the auditors judgmentally selected three adult care facility testing sites, three community testing event sites, and three vaccination sites and discussed the selection and monitoring processes followed by DOHMH’s program officials. Auditors also interviewed the following DOHMH personnel:

• SARS-CoV-2 Testing Team Lead (for adult care facilities testing);
• COVID Project Manager;
• Congregate Settings SARS-CoV2 Testing Coordinator;
• COVID Contract Manager;
• Contract Analyst;
• CRS IV-A, DMSC (for CE testing);
• Assistant Commissioner, Disease Control Division Management and Systems Coordination (DMSC);
• Financial & Operations Lead (for vaccination);
• Chief Operating Officer/Executive Deputy Commissioner (for vaccination);
• COVID Data Analyst;
• Senior Director of Cross Bureau Collaboration, DMSC;
• Director of Adult and Clinical Services;
• Deputy Commissioner, Disease Control;
• Assistant Commissioner, Internal Control and Audit;
• Assistant Commissioner, Bureau of Vaccination;
• Director of Information Strategy;
• ACCO/Assistant Commissioner;
• Executive Director, Revenue; and
• Executive Director, Budget Administration.

The auditors received a list of 51 payments made to EMS PC by DOHMH as of January 13, 2022, generated from the City’s Financial Management System (FMS). The payments were for the services provided by EMS PC during the period May 2020 to November 2021 under emergency contract for the total amount of $227.6 million. To determine the completeness and accuracy of the DOHMH-provided payment list, the auditors generated an independent payment report for this contract from FMS and compared the two lists.

To determine whether DOHMH had adequate controls to ensure that payments to EMS PC were for allowable expenditures and adequately supported, and that personnel resources utilized by EMS PC were commensurate with demand, auditors selected a sample of 10 invoices totaling $48.9 million from the total population of 51 payments made as of January 13, 2022. In addition to the previously mentioned two invoices selected (August 2020 for testing and April 2021 for vaccination), auditors selected and requested all testing-related invoices for September 2020 and April 2021, and all vaccination-related invoices for April and May 2021. Eight additional invoices totaling $42.9 million were provided—$1.3 million for two testing invoices and $41.6 million for six vaccination invoices. The auditors judgmentally selected all invoices for testing services provided by EMS PC in August 2020, September 2020, and April 2021 because they believed DOHMH’s controls over testing should have been established by then, since the EMS PC contract went into effect in May 2020. There were three invoices associated with the testing conducted in the three sampled months totaling $2.5 million.[11] Similarly, the auditors judgmentally selected all invoices for vaccination services provided by EMS PC in April and May 2021 because they believed DOHMH’s controls over vaccination should have been established by then since EMS PC started administering COVID-19 vaccinations in January 2021. There were seven invoices for vaccination services for the two sampled months totaling $46.4 million.

DOHMH required EMS PC to submit a detailed supporting document (Excel file) with its monthly invoice submission. The Excel file had three tabs:

1. “Program Summary,” which included lists test/vaccination sites broken down by date, the number of staff assigned at each site (labeled RNs), site hours, and the number of patients tested/vaccinated per day at each site. Based on these, the total staffing cost and total specimen collection and lab processing cost ($165 per specimen collected) or vaccination administration cost ($25 per patient) is calculated in the worksheet;
2. “Pt Vaccination Detail,” which included a list of sites, the unique patient ID number that received testing or vaccination each day, and the type of test/vaccination administered; and
3. “Staff Detail,” which included a list of staff names assigned at each site, each day, and details of the hours worked by each staff member.

From December 8, 2020, DOHMH also established the practice that EMS PC submit summary reports daily to DOHMH documenting the number of EMS PC staff present at the site and the number of patients tested or vaccinated. For testing events at adult care facilities, EMS PC submitted a summary report and a NYC ACF Time and Event Summary, later of which was signed by the adult care facility contact person. EMS PC also submitted summary reports daily for community testing and vaccination events. Summary Reports for vaccination events were signed by DOHMH employees present at the events to verify the numbers recorded in the report. To determine whether the payments made for the sampled 10 invoices (three for testing services and seven for vaccination services) were for allowable expenditures and adequately supported, auditors reviewed the invoice support document and performed the following steps:

• Verified that all patient and staffing totals for testing and vaccination were correctly calculated and the correct rate applied on the Program Summary tab and matched all the totals to invoice totals.
• Verified the numbers of EMS PC staff reported on the Program Summary tabs by matching those numbers to the EMS PC staff listed in the Staff Detail tabs.
• Verified the number of patients tested/vaccinated reported on the Program Summary tabs by matching those numbers to the unique patient ID numbers recorded each day for each testing/vaccination site listed in the Patient Detail tabs.
• Verified the patient data in the Patient Detail tabs by applying a formula to determine that there were no duplicate patient ID number reported by EMS PC.
• Sorted the staff names provided in the Staff Detail tab by name and date to determine whether EMS PC billed for duplicate shifts, overlapping shifts, or same staff at multiple locations at the same timeslot.
• Verified the number of patients reported on the Program Summary tabs of community event invoice to the number of daily tests reported in DOHMH’s Task Force’s (CORP) program update.
• Verified the number of patients reported on the Program Summary tabs of community event invoice to the number of daily tests recorded in DOHMH’s internal tracker schedule.
• Reviewed all summary reports for community testing events and testing at adult care facilities associated with one invoice for services provided in April 2021 to verify the number of EMS PC staff and patients tested for COVID-19 billed. For testing at adult care facilities, we also reviewed NYC ACF Time and Event Summaries to verify the presence of required signature of the adult care facility contact person.
• Reviewed all summary reports associated with two invoices for vaccination services provided in April and May 2021 to verify the number of EMS PC staff and patients vaccinated for COVID-19 billed and to verify the presence of required signature of a DOHMH on the summaries.
• Compared the number of patients billed for vaccination to the NYC Daily Track Report and DoITT (Department of Information Technology and Telecommunications) Salesforce, which are programs and platforms to monitor vaccination services, to verify the accuracy of the numbers.

The auditors sent the list of discrepancies found during the audit’s review to DOHMH for its review and feedback.

To determine whether personnel resources utilized by EMS PC were commensurate with demand to protect against waste, the auditors calculated the average number of tests and vaccinations administered per staff hour billed by EMS PC and determined the average dollar amount spent for each COVID-19 test and vaccination administered each day as billed by EMS PC.  For adult care facilities, the auditors also compared the number of tests administered at each facility to the number of patients and staff that were anticipated to be tested according to the AP PPS schedule, to determine whether staffing levels were appropriately planned compared to anticipated demand.

To better assess whether EMS PC’s utilization of personnel resources was commensurate with demand over time, the auditors judgmentally selected six long-term testing sites and four fixed vaccination sites and requested all associated invoices and supporting documents for the sites for the complete duration of the event or until June 30, 2022, whichever was sooner. There were 20 additional invoices and supporting documents for the selected six testing sites and 19 additional invoices and supporting documents for the selected four vaccination sites provided. The auditors calculated the weekly average costs of testing and vaccination in these 10 sites to determine whether personnel resources were used efficiently.

To understand the composition of staffing at the four vaccination sites and to determine the productivity rates at those locations, auditors requested the floor plans associated with the sites; they did not request the floor plan for one of the vaccination sites since it was not under DOHMH’s control. The DOHMH floor plan is a schematic diagram of the lay-out of a vaccination site. It identified the numbers and placing of the registration and vaccination stations, observation area and the placing of floor monitors.

For the 49 invoices and supporting documents reviewed (10 from the initial sample and 39 additional invoices mentioned above), the auditors extracted the data on the total staff present, staff hours utilized, patients tested or vaccinated, and the total amount paid to EMS PC for those events that the complete billing information was available (up to June 30, 2022) and consisted of 53 testing events at adult care facilities , eight long-term/fixed community testing sites , 164 pop-up testing sites , 23 long-term/fixed vaccination sites , and 120 pop-up vaccination sites. To determine the efficiency and cost of administering tests and vaccinations at each event, the auditors calculated the overall average rate of administering COVID-19 test and vaccination per staff hour and the overall average cost of administering a COVID-19 test and vaccination.

The auditors reviewed DOHMH’s evaluation of EMS PC’s contract in PASSPort (a vendor portal for NYC government contractors) and the supporting documents for the evaluation to confirm that DOHMH submitted the required evaluation within established timeframes and to obtain an understanding of DOHMH’s vendor performance evaluation process.

To review how DOHMH’s community-based testing and vaccination sites were distributed throughout the five boroughs, the auditors mapped all of DOHMH’s community-based testing and vaccination sites operated by EMS PC in the borough maps using ArcGIS online mapping software. For testing, there were nine long-term sites (providing testing services from 2 to 16 months) and 166 short-term or pop-up events that lasted from one day to three weeks to provide COVID-19 testing. For vaccination, there were 36 fixed locations and 101 short-term or pop-up events to provide COVID-19 vaccination.[12] The auditors extracted the information on Race & Hispanic Origin from American Community Survey 2014–2018 data by community districts posted on the DCP website and determined whether COVID-19 testing and vaccination sites were established in community districts where the majority household makeup was Black and Hispanic, which is the city resident population that DOHMH’s preliminary COVID-19 surveillance data found were disproportionately affected by COVID-19.

To understand the decision-making process for establishing COVID-19 testing and vaccination sites, the auditors reviewed the COVID-19 Organizing Response Plan (CORP) Taskforce Planning Overview, NYC DOHMH Interim COVID-19 Vaccination Plan – Executive Summary , and DOHMH narrative. The auditors randomly selected 30 sites (16 testing and 14 vaccination) serviced by EMS PC and requested DOHMH to provide the basis to establish these sites and any supporting documentation used in selecting a location for an event.

The results of the above tests, while not projectable to their respective populations whenever a sample was used, provided a reasonable basis for the auditors to evaluate whether DOHMH had adequate controls to ensure that payments to the vendor were for allowable expenditures and adequately supported, and that personnel resources utilized by the vendor were commensurate with demand to protect against waste.

APPENDIX I

List of Sampled Long-Term/Fixed and Pop-up Testing Sites: Total Amount Paid and Auditor’s Calculated Average Cost per Test & Average Number of Tests Administered Per Staff Hour

APPENDIX II

List of Sampled Community Long-Term/Fixed and Pop-up Vaccination Sites: Total Amount Paid and Auditors’ Calculated Average Cost per Vaccination & Average Number of Vaccinations Administered per Staff Hour

APPENDIX III

Complete List of Long-term/Fixed and Pop-up Testing Sites

APPENDIX IV

Complete List of Long-Term/Fixed and Pop-up Vaccination Sites

ADDENDUM

May 26, 2023

Maura Hayes-Chaffe
Deputy Comptroller for Audit Office of the Comptroller
1 Centre Street Room 1100 New York, NY 10007-2341

Re: Response to Draft Audit Report NYC Department of Health and Mental Hygiene’s Health Emergency Contract with Executive Medical Services PC for COVID
Related Services MJ22-085A

Dear Maura Hayes-Chaffe:

The NYC Department of Health and Mental Hygiene (Health Department or DOHMH) reviewed the draft audit report on its Emergency Contract with Executive Medical Services (EMS PC) for COVID-19 Related Services. The stated objectives of the audit were to determine whether the Health Department ensured that payments to the vendor were allowable and fully supported, staffing levels were commensurate with demand and testing and vaccinations sites were equitably established.

The Health Department thanks the auditors for the opportunity to respond to the draft report and appreciates the courtesy and professionalism of your team during the audit process.

COVID-19 pandemic provides a stark reminder of systemic health inequities that existed pre-pandemic. In collaboration with City Hall, Health + Hospitals, other City agencies and partnership with communities, the City addressed disparities in COVID-19 through community engagement, public education, and strategic placement of vaccination and testing efforts. In the post pandemic era, it will be imperative to work with policy makers and with cross-sector partners to implement the lessons learned from COVID- 19 to address the compelling needs of the City’s diverse communities.

Attached is the Health Department’s response to the draft audit report. If you have any questions or need further information, please contact Sara Packman, Assistant Commissioner, Audit Services, at spackman@health.nyc.gov or at (347) 396-6679.

RESPONSE TO THE AUDITORS’ DRAFT AUDIT REPORT ON
NYC HEALTH DEPARTMENT’S EMERGENCY CONTRACT WITH EXECUTIVE MEDICAL SERVICES PC FOR COVID-19 RELATED SERVICES
MJ22-085A

The NYC Department of Health and Mental Hygiene (Health Department, DOHMH) reviewed the draft audit report on its Emergency Contract with Executive Medical Services (EMS PC) for COVID-19 Related Services. The stated objectives of the audit were to determine whether NYC Health Department ensured that payments to the vendor were allowable and fully supported, staffing levels were commensurate with demand and testing and vaccinations sites were equitably established.

The Health Department thanks the auditors for the opportunity to respond to the draft report and begins this response with contextualizing the COVID-19 pandemic and the EMS emergency contract.

The novel coronavirus disease (COVID-19) outbreak began in China in December 2019 and quickly spread around the world, leading to the declaration of a pandemic by the World Health Organization on March 11, 2020. The high contagiousness of COVID-19 became an overwhelming challenge to health care system around the world.

The first case of COVID-19 in New York City was confirmed on March 1, 2020. Within weeks the life of New Yorkers was forever changed as they had to adjust to a once in a century historic public health emergency. By March 31, 2020, over 1,000 New Yorkers had died from COVID-19. NYC issued guidance to avoid densely packed public spaces and to wear a mask. In addition, schools were closed and New Yorkers were introduced to remote work. As of May 18, 2023, the City had more than 3.2 million COVID-19 cases and 45,000 deaths.

By April 2020, it became apparent from the Health Department’s surveillance data that the COVID-19 pandemic was exacting a toll on communities of color. Black and Latino city residents were dying at twice the rate of white and Asian New Yorkers.

The City used a multifaceted approach to address the pandemic that was grounded in guidance from the Centers for Disease Control and Prevention (CDC). This approach entailed identifying specific at-risk geographic areas and populations, working with community leaders and trusted members of the community, identifying barriers to service provision, and addressing them, and nimbly making changes to address the changing face of the pandemic.

The City also announced the unprecedented Taskforce on Racial Inclusion & Equity (TRIE) in response to the disproportionate impact of COVID-19 on communities of color. To that end, 33 neighborhoods were prioritized based on the number of COVID-19 cases, COVID-19 deaths, and other health and socioeconomic indicators.

The City, and by extension the Health Department was forced by circumstances to make COVID- 19 related decisions in real time with constantly evolving situation as New York City became the epicenter in the country for surging COVID-19 cases and deaths.

The COVID-19 vaccination and testing response was a collaborative effort involving the Health Department, Health + Hospitals, City Hall, Vaccine Command Center, and other City agencies. Executive Medical Services PC (EMS) was one of the providers that the City used in its COVID- 19 response efforts. Decisions about the utilization of that vendor to provide vaccination and testing services — e.g., what sites to set up– were determined in partnership with City Hall, Vaccine Command Center, Health Department, and other City agencies.

The EMS PC vaccine operation, as directed by the Health Department was unique. No comparable vaccination model existed prior to the pandemic due to the scale, duration and scope of the crisis, the specific needs of diverse communities and systemic health inequities. Further, the operation evolved over time with changing eligibility criteria, the introduction of new types of vaccine, and other State and federal policy and programmatic changes. The Health Department had to set the standards for this type of operation in real time.

The vaccine and testing sites were in venues that were not designed for use as clinical settings. Repurposing these sites into safe clinical settings influenced staffing levels and added staffing time — to set up and break down equipment, to maintain line flow and separations, and to always ensure line of site to patient observation for patient safety.

In the following sections, the Health Department addresses the auditors’ conclusions and recommendations.

The auditors conclude that the Health Department established testing and vaccination services in communities most affected by COVID-19. Based on the auditors’ review of sampled invoices, the auditors conclude that the invoices were generally well supported and justified the Health Department payments. However, the auditors assess that the Health Department did not adequately control and monitor staffing levels and relied heavily on the vendor to set staffing levels.

The auditors make five recommendations related to assessing and setting staffing levels; conducting regular cost analysis and adjusting for efficiencies; and ensuring that verification documents include agency certification, that key decisions were documented, and that vendor performance evaluations were completed timely.

Issue: “DOHMH did not adequately control or monitor staffing levels”

The auditors assess that “DOHMH did not establish or communicate optimal staffing level benchmarks on the standard service levels for administering tests or vaccinations in the pandemic environment or design any formal mechanisms for assessing EMS PC’s staffing levels on an ongoing basis. DOHMH’s monitoring plan did not set standards for assessing whether EMS PC appropriately staffed vaccination and testing sites based on demand.”

The Health Department strongly disagrees with the auditors’ assessment. Contrary to the auditors’ assertion, as was communicated to the auditors the Health Department used CDC guidance that requires specific onsite staffing at testing and vaccination sites. A vaccination on-site staffing include: site managers; pre-entrance screeners; registration staff; medical screeners; patient educators; vaccine inventory managers; vaccine preparers; vaccinators; runners or flow monitors; data input/management staff; interpreters; and post-vaccination observers. Staffing numbers were pre- determined based on the layout of a particular venue, number of vaccine products offered and anticipated demand. There was always an expectation that there would be demand for the COVID- 19 services made available to New Yorkers. The Health Department launched multi-million dollar advertising campaigns and consistent public officials messaging to encourage testing and vaccination uptake. Actual foot traffic to the sites varied and was beyond the immediate control of the Health Department.

Contrary to the auditors’ assertion, the Health Department implemented multiple mechanisms to assess EMS PC staffing levels on an ongoing basis. Standing weekly meetings were held with EMS to discuss staffing levels and other operational matters. The Health Department’s approved staffing levels were recorded in weekly calendars and other operational documents. The Health Department’s staff was always present at vaccination sites during hours of operation and wait- times was one indicator that the staff monitored as part of staffing level determination.

Assessment of staffing levels was a continuous and iterative process, performed on a regular basis and involved a close review of event summaries, site operations data, and verbal conversations. Evidence of this communication and documented staffing levels were provided to the auditors.

The auditors’ claim that “. . . DOHMH did not capture the data necessary to determine how many EMS PC staff members were directly testing and vaccinating patients and how many were providing support. DOHMH was therefore unable to effectively determine the number of tests and vaccines administered per hour by each EMS PC staff member assigned to conduct the testing and administer the vaccinations. As such, DOHMH had limited means of effectively controlling expenditures or minimizing waste.”

The Health Department strongly disagrees with this assessment. As directed by the CDC and discussed above, both clinical and non-clinical staff fulfill key roles at a testing and vaccination site. Before a patient presents before a nurse to be tested or vaccinated, a site-staff must complete clinical safety and operational activities. These activities include checking the individual’s temperature, screening the individual to ensure that they meet eligibility criteria before electronically or manually registering the individual. These pre-vaccination and testing activities are integral to an efficient testing or vaccination event and should not be considered as separate expenditures that could have been reduced. The Health Department has strong internal controls to prevent and detect fraud, waste, and abuse and these controls were in place during the management of the EMS PC contract.

Issue: “Invoice Analysis Shows on Average Fewer Than One Test or Vaccination Administered per Every Staff Hour Billed and Per Unit Costs of Services Varied Significantly”

The auditors stated that they “. . . believe that the absence of a quantitative monitoring plan that included an assessment of staffing levels contributed to both (1) an average of less than one test or vaccination administered for every staff hour EMS billed and (2) the wide variance among the sites in the average costs of the tests administered and vaccination shots given”.

The Health Department notes that the auditors judgmentally selected invoices for testing and vaccination sites that were in a vast array of neighborhoods and were intended to serve varied populations. Likewise, community response to the availability of the COVID-19 services varied. In certain communities, longstanding distrust in the government contributed to lack of vaccine confidence. So that, while those neighborhoods were prioritized for services, the demand for those services was sometimes low. In other communities, where such obstacles did not exist, there was more demand for the services.

In addition, the sampled invoices covered time periods during which the pandemic was constantly evolving. For instance, demand for vaccine was high and supply limited in early 2021 when the vaccine initially became available. Large crowds were commonplace at vaccine sites during this time. Over time, as vaccine supply increased and more sites were opened, the long lines dissipated.

COVID-19 testing and vaccination operations were affected by the situations discussed above. It is therefore conceivable that the financial variations that the auditors noted will exist. However, the auditors’ assertion that the absence of a quantitative monitoring plan contributed to these variations is not a valid one. It does not accurately depict the City and Health Department’s efforts to address the COVID-19 pandemic. The activities covered in the invoices that the auditors reviewed were affected by circumstances that were beyond the control or monitoring of the Health Department.

The lives of New Yorkers were saved because of the City’s efforts to provide testing and vaccination services. It is not possible to place a monetary value on a life.

The auditors present charts of average cost per COVID-19 test at sampled testing-sites, and separately, average cost per vaccination. The auditors flagged three testing sites that served older adults and three vaccination sites in at-risk neighborhoods that had average costs that were notably higher than other sites. They further stated that, “there may have been opportunities for EMS PC to reduce the number of staff utilized at these sites”. We disagree with the auditors’ assessment.

As was previously explained and demonstrated in site-specific floor plans that were shared with the auditors, a COVID-19 testing or vaccination site requires certain key staff roles to operate. Staffing –both clinical and non-clinical– is key factor in a venue’s setup. To operate a vaccination clinic according to CDC safety standards, CDC guidance requires that on-site staff includes site managers; pre-entrance screeners; registration staff; medical screeners; patient educators; vaccine inventory managers; vaccine preparers; vaccinators; runners or flow monitors; data input/management staff; interpreters; and post-vaccination observers. Each role is integral to the operation. Similarly, there are staffing requirements for testing sites. To comply with CDC standards, if a site was open–even if demand for testing or vaccination service was low– it was essential that key staffing roles be filled.

Further, the auditors stated that two-thirds of the 120 vaccination pop-up sites that they reviewed had average cost of administering vaccinations of more than $500. It is noteworthy that approximately half of these sites are schools where the services were mandated to protect the students who chose to return to in-school learning. The remaining sites were primarily in public housing buildings, long-term care facilities, and City Agencies offices. In Fall 2021, then Mayor de Blasio mandated that public school and other City employees must be vaccinated. As such these sites were in place to eliminate any barriers for vaccination and meet the need of targeted populations.

The auditors acknowledge that:

“Significant portion of testing and vaccination sites were placed in communities having significant Black and Latino Populations”

While the auditors concede that a significant portion of testing and vaccination sites were placed in communities having significant Black and Latino populations, they state that the Health Department “did not provide site-specific data that was directly used for the selection of the 30 sampled sites . . .”

No such data exists. The 30 sampled sites were selected to provide COVID-19 testing and vaccination services as part of the City’s overarching approach to provide these services to identified populations.

• Fourteen of the 16 sampled testing sites were opened in schools in Fall 2020 to facilitate in-school testing of students. The other two sites are community centers in at-risk
• The 14 sampled vaccination sites include public housing buildings, a correctional facility, schools, and outdoor pop-up sites. These sites were put in place to make it easier for a certain populations to get For instance, vaccination services were offered at the sampled schools in Fall 2021 to vaccinate students 12 and older. Per the CDC, “Administering the COVID-19 vaccine to people in all communities across the United States is dependent on having multiple vaccination sites that are both convenient and accessible”.

The Health Department’s Response to Auditors’ Recommendations:

1. “Develop and implement an effective and standardized process for assessing and setting appropriate staffing levels that are commensurate with demand and document the results of these efforts.”

Response: The Health Department agrees with the auditors’ proposed recommendation for future contracts. Using the lessons learned from the City’s response to the COVID-19 pandemic, the Health Department will have baseline benchmark models for staffing levels that address the changing needs of NYC’s communities.

2. Conduct regular cost analyses after the initial stage of contract implementation, including—when feasible—comparisons to other providers of the same services to continuously monitor and adjust for efficiency.

Response: The Health Department agrees with the auditors’ recommendation. . The lessons learned from the City’s response to the COVID-19 pandemic will help to inform future decisions.

3. Ensure that all verification documents (e.g., summary reports) include appropriate agency certification that the services claimed by the contractor were provided.

Response: The Health Department believes that this recommendation is not needed because a receiving report process is already in place.

4. Ensure that it formally documents key decisions regarding site selections, including reasons for opening a site, its duration, and the closing of a site.

Response: The Health Department disagrees with this recommendation as it is not needed. As stated above, the COVID-19 pandemic response was a collaborative effort. Key protocols and mechanisms were in place to deal with decisions surrounding testing and vaccination sites.

5. Ensure that required vendor performance evaluations are completed in PASSPort at least once annually.

Response: The Health Department agrees with this recommendation and will ensure that vendor performance evaluations are completed timely.

---

Endnotes

[1] EMS PC is also known as Affiliated Physicians (AP).

[2] The NYC Taskforce on Racial Inclusion & Equity (TRIE) was established by New York City in April 2020 in response to the disproportionate impact of COVID-19 on communities of color. TRIE brought together City leaders from 67 agencies and offices to work in teams to monitor the COVID-19 response and identify solutions to deliver immediate and long-term relief to those communities that suffered a disproportionate impact of COVID-19. TRIE prioritized neighborhoods based on COVID cases, deaths, and other health and socioeconomic indicators.

[3] The $100 per test fee was modified in the first contract amendment (dated September 2020) requiring DOHMH to pay an additional $65 fee to EMS PC (for a total of $165) for each test processed at a private commercial laboratory.

[4] An event duration refers to a time period an event was in operation. An entire event duration refers to a site that was opened and closed (no longer operating) during the invoice review period.

[5] There were an additional 14 long-term/fixed sites (1 for testing and 13 for vaccination) that were included in the reviewed invoices, however, these sites were excluded from the audit’s analysis of the staffing and average cost per patient because the invoices did not include the complete testing and vaccination data for these locations.

[6] There were 13 event days for 64 pop-up vaccination sites that were included in the reviewed invoices; however, these events were excluded from the audit’s analysis of the staffing and average cost per patient because the patient data and the staff hours data were comingled with other events, and the auditors were unable to identify the complete testing data for these locations.

[7] One example of a site with a low number of vaccinations per staff hour with no evidence of DOHMH’s input regarding staffing was Empire Outlets, for which invoices covering the 2 ½ month period of April 14 – June 30, 2022 indicate that, on average, a minimum of 10 staff were daily on staff to vaccinate anywhere from 10-30 clients per day. For this period, an average of 0.11 vaccinations were administered per staff hour paid by DOHMH.

[8] Maven is DOHMH’s Electronic Disease Surveillance System that contains results reported by laboratories about communicable disease and case investigation information collected by DOHMH program staff throughout the agency.

[9] During the City’s COVID-19 emergency response, DOHMH’s Integrated Data Team assisted with the sharing of key COVID-19 data (e.g., hospitalizations, deaths, and test positivity rates) among ICS emergency response teams to help ensure that ICS/DOHMH leadership and external stakeholders had access to accurate and timely COVID-19 data.

[10] The auditors found only three minor errors, consisting of two overpayments and one overlap (totaling $4,125), that were not identified by DOHMH’s post-payment review process. DOHMH confirmed that these payments should not have been made and stated on May 8, 2023 that it is in the process recouping these funds from EMS PC and is conducting an independent review to ensure that there are no other remaining overpayments.

[11]  Included in this amount was $7,275 for training and providing results of the tests only.

[12] There were 36 fixed locations for vaccination. One location in the Bronx (at Morrisania Emergency Clinic) was used twice as a fixed clinic, 11 months apart. As a result, we counted 36 fixed locations for vaccination. However, only 35 fixed locations are visible on our maps.