Audit Report on Intensive Mobile Treatment Initiative Carried out by Department of Health and Mental Hygiene

To the Residents of the City of New York,

My office has audited the Department of Health and Mental Hygiene (DOHMH) to determine whether DOHMH has established measurable performance targets for the Intensive Mobile Treatment (IMT) initiative that align with its stated objectives, as well as the degree to which the initiative effectively serves its clients.

The audit found that the IMT program has mixed success in effectively servicing clients who have been poorly served through traditional mental health treatment models. The audit’s examination of sampled clients found that only 41% of clients were seen by a psychiatric care practitioner at least 3 out of every 4 months they remained in the program.

Regarding the program objective to reduce incarcerations, prior studies conducted by DOHMH found that incarceration rates decreased for clients after entering the program. However, this analysis is not performed regularly and the most recent one was completed over a year and a half ago.

The audit also found that DOHMH does not track individual progress of clients or report outcomes in the aggregate. Consequently, there is no effective measurement of the overall treatment rate of IMT clients and associated outcomes.

The audit makes nine recommendations intended to improve how the initiative serves its clients. DOHMH should: require providers to submit information related to key treatment services provided to IMT clients; develop a less labor-intensive mechanism to obtain incarceration data for clients and assess whether incarceration rates are reduced; develop reasonable targets for treatment provided to clients and establish protocols for when those targets are not met; and establish performance measures for the program.

The results of this audit have been discussed with DOHMH officials, and their comments have been considered in preparing this report. Their complete written response is attached to this report.

If you have any questions concerning this report, please e-mail my Audit Bureau at audit@comptroller.nyc.gov.

Sincerely,
Brad Lander
New York City Comptroller

Audit Impact

Summary of Findings

The audit found that the Department of Health and Mental Hygiene’s (DOHMH) Intensive Mobile Treatment (IMT) program has mixed success in effectively servicing clients who have been poorly served through traditional mental health treatment models.[1] The stated objectives of the program are to: (1) increase client retention in the program to facilitate treatment; (2) reduce incarcerations; and (3) provide assistance in securing stable housing. The audit assessed these impacts and found that:

• DOHMH’s client retention rates are good, but retention does not equate to continuous psychiatric treatment;
• Although previous studies by DOHMH show a positive impact on incarcerations, the most recent study was completed in February 2022 (based on data of clients enrolled in IMT as of December 2018). This analysis has not been updated since; and
• The percentage of clients who were able to acquire housing has decreased from 47% to 30% over a 27-month period, and the percentage of clients who were able to retain stable housing has gone from 44% to 37% over a 21-month period. DOHMH attributed these declines to an increase in the number of clients who need housing along with a shortage of available and affordable housing.

The audit also found that DOHMH does not track individual progress of clients or report outcomes in the aggregate. Consequently, there is currently no effective measurement of the overall treatment rate of IMT clients and associated outcomes.

Intended Benefits

This audit identified areas for improvement in DOHMH’s oversight of the IMT program, to assist the agency in assessing the degree of effectiveness in helping the program’s clients.

Introduction

Background

In 2016, the New York City Mayor’s Office, in partnership with the Department of Health and Mental Hygiene (DOHMH), created the Intensive Mobile Treatment (IMT) program to provide adults with access to more flexible mental health services, substance abuse treatment, guidance within the criminal justice system, and provision of homeless services. IMT clients include individuals who have had frequent contact with the mental health, criminal justice, and homeless services systems; have a history of engaging in conduct and behavior that is unsafe and escalating; and, generally, were poorly served by traditional treatment models.

DOHMH’s stated objectives for the IMT program are as follows: (1) increase the number of clients who remain in the program; (2) increase the number of clients placed in stable housing or assist those who are already housed to retain stable, long-term housing; and (3) decrease the number of client incarcerations. DOHMH believes that these objectives are crucial for improving the quality of life of program participants and ensuring the overall success of the program.

DOHMH contracts with providers who employ mobile treatment teams responsible for providing continuous and direct support and treatment to individuals within their communities. These teams are made up of a program director, three behavioral health specialists, a psychiatrist, a nurse, an administrative assistant, and two peer specialists.  Each team serves a maximum of 27 clients.

DOHMH receives referrals to the program from various sources, including but not limited to jails, hospitals, and community organizations. A DOHMH Client Care Coordinator (CCC) reviews referrals and enters them in Maven, a computer system used to enroll clients in programs and store relevant information regarding a diagnosis and future treatment. If a client qualifies for IMT, the CCC directs the referral to the IMT team in the client’s borough (when feasible). Since July 2020, each IMT team has been required to use Maven to provide DOHMH with monthly updates regarding enrolled clients.

These updates include the client’s status in terms of medication, mental and medical health treatments, housing, current location, hospitalization, and adverse events, among other things. Providers are required to submit these updates by the 15th of each month. Providers are also required to maintain detailed files for each client, in which they record assessments, treatment plans, history of treatment and services provided, provider’s interaction with the client, as well as overall notes of the client’s progress.[2]

DOHMH program specialists offer programmatic and technical support to each provider to assist with establishing, developing, and maintaining the operations of the IMT teams. In addition, they conduct comprehensive reviews of each team separately. This consists of reviewing three to five client case files per team; observing meetings with team members where they discuss client care and review provider’s policies and procedures; and assessing whether the physical space of the provider is safe and activities are conducive to providing IMT services to clients.[3] At the completion of a comprehensive review, program specialists issue a site visit letter summarizing DOHMH’s findings.

If DOHMH program specialists identify egregious or persistent problems during a comprehensive review, they request a corrective action plan (CAP) from the IMT team, which the provider must respond to within 30 days. Program specialists are responsible for confirming the implementation of all elements of the CAP, the status of which is reviewed by the program specialist during the following comprehensive review.

Since its inception, IMT has gone through several phases of expansion. During its pilot stage in January 2016, DOHMH contracted three providers for the services of three IMT teams. DOHMH subsequently added additional teams and currently has contracts with five providers, consisting of  31 teams in total. Providers’ services are covered by funds DOHMH receives from the Office of Management and Budget (OMB). IMT’s budget for Fiscal Year 2023 was $37.6 million and the preliminary budget for FY2024 is $42.1 million.

As of March 3, 2023, there were 827 clients enrolled in the program. The maximum capacity for enrollment in the program for both FY2023 and FY2024 was 837 clients (31 teams with 27 clients per team).

Objectives

The objectives of this audit were to determine whether DOHMH has established measurable performance targets for the IMT initiative that align with its stated objectives, as well as the degree to which the initiative effectively serves its clients.

Discussion of Audit Results with DOHMH

The matters covered in this report were discussed with DOHMH during and at the conclusion of this audit. An Exit Conference Summary was sent to officials on November 7, 2023, and discussed at an exit conference held on December 1, 2023. On December 7, 2023, we submitted a Draft Report with a request for written comments.

We received a written response from DOHMH on December 21, 2023. In its response, DOHMH generally agreed with seven recommendations (#1, #2, #3, #4, #5, #6, #9), partially agreed with one recommendation (#8) and disagreed with one recommendation (#7).

Detailed Findings

The audit found that the IMT program providers have mixed success in delivering services to clients. While DOHMH’s retention protocols result in few clients being released from the program, the audit’s examination of sampled clients found that retention did not necessarily translate to clients receiving continuous psychiatric treatment—only 41% of clients were seen by a psychiatric care practitioner at least 3 out of every 4 months they remained in the program.

One of the program objectives is to reduce incarcerations. To assess this objective, auditors reviewed analyses performed by DOHMH that indicated that incarceration rates decreased for clients after entering the program. However, DOHMH does not monitor this rate on a regular basis, and the most recent analysis was conducted in February 2022, over a year and a half ago.

Another objective relates to stable housing. The data reviewed by the auditors shows that program effectiveness has declined. The percentage of clients who were able to acquire housing has decreased from 47% to 30% over a 27-month period. In addition, the percentage of clients who were able to retain stable housing has gone from 44% to 37% over a 21-month period. DOHMH attributes this to greater numbers of clients entering the program in need of housing coupled with a shortage of available and affordable housing.

Regarding the overall performance of providers serving the mental health needs of clients, DOHMH reviews case files and monitors whether clients receive prescribed services during comprehensive site visit reviews. These reviews are performed on a client-by-client basis, but DOHMH does not track individual progress of clients and does not track or report outcomes in the aggregate.

Moreover, inconsistencies in review methodologies prevent the agency and auditors from aggregating the results. This means the overall treatment rate of IMT clients and associated outcomes are not measured effectively.

Providers’ Success in Delivering Services to Clients Is Mixed

DOHMH requires providers to submit a monthly update for each client, relating to the program’s primary objectives—retain and treat clients for mental health services, reduce incarcerations, and assist in securing stable housing. This update includes, but is not limited to, the following information: basic biographical information, housing situation, medical and mental health related information, and community risks, as well as any adverse events (e.g., arrests). The auditors’ review of these updates and DOHMH’s monitoring records indicates that the agency has had mixed results in these areas.

Significant Percentage of Sampled Mental Health Clients Did Not Continuously Receive Treatment or Take Prescribed Medication

DOHMH considers the retention of clients within the program to be synonymous with treatment. DOHMH explained that traditional psychiatric care is not suitable for IMT clients, who are often ill and living on the streets and are not capable of managing their own mental health needs. In the IMT program, it is the responsibility of the IMT team to reach out to the clients—it does not rely on clients seeking treatment. Based on this structure, DOHMH’s reasoning is that if clients are enrolled and retained in IMT, they will inevitably receive the treatment they need. However, merely remaining in the program sets a very low bar.

The audit’s review of IMT enrollment data for 602 clients admitted to the program from its inception in 2016 through March 3, 2023, found that the retention rate was 99.5% at the three-month interval after enrollment in the program, 98% at the six-month interval, 96% at the nine-month interval, and 93% at the 12-month interval.

Treatment for a client may encompass several services, including but not limited to substance abuse intervention, peer support, vocational services, and behavioral therapy. However, these areas are reported on an inconsistent basis (if at all) in the monthly data that DOHMH receives from providers. The only area that is consistently included in the monthly data pertains to psychiatric treatment. A review of that monthly data revealed that, despite the high retention rate, a significant percentage of sampled clients did not meet with a psychiatric care practitioner on a continuous basis. The auditors reviewed 52 sampled clients with a diagnosis of mental illness to determine whether the clients were visited by psychiatric care practitioners at least once during a month.^[4] The review found that:

• 6 (12%) clients continuously met with an IMT psychiatric care practitioner each month of their enrollment.
• 15 (29%) clients met with an IMT psychiatric care practitioner 75–99% of the months enrolled in IMT.
• 10 (19%) clients met with an IMT psychiatric care practitioner 50–74% of the months enrolled in IMT.
• 20 (38%) of the clients met with an IMT psychiatric care practitioner less than half the time that they were part of the program. It should be noted that 13 of these clients did not see a psychiatric care practitioner at any point between 10 to 27 months while considered enrolled in the program.
• One (2%) enrolled client never met with an IMT psychiatric care practitioner.

It is noteworthy that all 52 clients were enrolled and retained in the program; however, not all were seen by a psychiatric care practitioner throughout the enrollment period.

While enrollment may potentially improve the chances that a client receives treatment, equating retention with treatment is based on a faulty premise. The two examples below further demonstrate how retention in IMT does not guarantee that a client will receive treatment, and that the two should not be considered synonymous.

• Client A was enrolled in IMT for 13 months and was reported missing for eight of them. They were seen by the psychiatric care practitioner only once during the period of enrollment.
• Client B was enrolled in IMT for 14 months and was reported missing for nine of them. They were seen by the psychiatric care practitioner only twice during the period of enrollment.

Both clients were counted as enrolled and retained, but neither received regular treatment from a psychiatric care practitioner. DOHMH does not define enrollment by criteria; clients are considered enrolled until they are dismissed from the program.  At the exit conference, DOHMH attributed the findings stated above to the fact that this is a difficult population, for which clients are referred to the program rather than coming in on their own. Officials claimed that the majority of clients in the program are resistant to regularly meeting with psychiatric care practitioners.

Only 29% of Sampled Clients Reportedly Took Prescribed Medication on a Regular Basis

Of the 52 sampled clients, 44 were prescribed oral or injectable medication to help them manage their mental health conditions. The auditors’ review of medication adherence data recorded in Maven showed that:

• 5 (11%) reportedly took their prescribed medications at all times.
• 8 (18%) reportedly took their prescribed medications most of the time.
• 16 (37%) reportedly took their prescribed medications only sometimes.
• 1 (2%) reportedly took their prescribed medications rarely.
• 14 (32%) had no information regarding how often they took their prescribed medication.

It should be noted that the terms to describe the frequency with which clients take their medications—”always,” “most of the time,” “sometimes,” and “rarely”—are preset terms that providers select when entering the information in the report. Without additional context, it is difficult to assess the meaning of those terms or the frequency with which medication was taken.

DOHMH officials stated that they lack the staff required to analyze the monthly data for treatment and medication. DOHMH also stated that reviews of the program are conducted during the comprehensive reviews and consist of a review of 15 to 24 sampled clients per provider.[5] However, following the review, program specialists do not report the degree to which services were or were not provided (see further information below), and even if they did, there are other issues that prevent a meaningful analysis of outcomes based on the reviews conducted.

These include the following factors:

• Providers are not required to report to DOHMH the number of times clients should meet with psychiatric care practitioners, so DOHMH cannot determine through the periodic reports whether the clients were seen by such practitioners the number of times that they were scheduled to be seen.[6]
• DOHMH has not set a standard for a target number or percentage of contacts that IMT team members (e.g., psychiatrist, psychiatric nurse, behavioral health specialist, peer specialist) should have with each client.

At the exit conference, DOHMH officials stated that they do not place a heavy reliance on any of the monthly data received from the providers, pointing to the medicine component as an example, saying that it is generally self-reported by clients and largely unverifiable. When asked why the agency requires the providers to report data, which is voluminous, officials indicated that they are in the process of reassessing the data that they collect and creating new performance goals and measures so that the information collected from providers is more useful and meaningful.[7]

The FY2023 budget of the IMT program is $37.6 million—amounting to $45,000 per client per year. Despite this significant investment, DOHMH is hindered in assessing the efficacy of the program because it has not set expectations or targets for the levels of treatment clients should expect to receive and it does not set goals around medication adherence. DOHMH does not consistently review monthly data containing all significant information about client treatment. As a result, DOHMH cannot determine whether the program is actually helping clients make progress with their treatment.

Incarceration Rates Have Not Been Tracked for More Than a Year and a Half

One of the goals of the IMT program is “to measure and reduce […] criminogenic and violence risk to decrease future incidents of violence and/or criminal justice involvement.” According to DOHMH, this is measured by the rate of incarceration for clients before and after entering the program.

Since the inception of the program, DOHMH has performed two studies related to this topic.  These found that clients were less likely to be incarcerated one year after their enrollment in IMT compared to one year prior to entering the program. However, this indicator is not tracked, and the last study was performed in February 2022.

The first study, completed in 2020, was based on data from January 2016 through March 2018 and showed a 32% drop in the number of incarcerations for clients enrolled in the IMT program when compared to the year before their enrollment. The second study, completed in February 2022, was based on clients enrolled in IMT as of December 2018 and showed a 22% drop in the number of incarcerations.

The audit’s review of the monthly updates that providers submitted to DOHMH for 53 randomly selected clients who entered the program between September 2020 through August 2022 showed that two clients entered the program while they were in prison and nine were imprisoned (from one to six months) after enrolling into the program. However, auditors did not have access to the clients’ histories prior to enrolling in the program; therefore, an assessment of whether overall incarcerations for these clients decreased could not be completed. For this same reason, the auditors were not able to corroborate the findings of the two studies conducted by DOHMH.

According to DOHMH, it currently does not track or analyze this performance measure because it is difficult to coordinate and obtain the criminal activity data from the Department of Correction. DOHMH identified COVID as another reason for not tracking the information, stating that it was difficult to obtain a baseline for determining whether clients were released from prison due to good behavior or because of COVID. At the exit conference, officials also stated that this analysis, which is performed manually, is both labor and time intensive, and staff shortages restrict the agency’s ability to conduct this analysis on a regular basis.

Unless this indicator is tracked, DOHMH is unable to monitor trends or measure success against this objective. In June 2023, DOHMH stated that it is in the process of selecting additional key performance indicators/measures that will be used for internal reporting and evaluation purposes; however, DOHMH has not to date identified what the proposed indicators will be. Once these are solidified, the agency stated it will begin using a reporting system that will allow for, at minimum, a quarterly review of selected key performance measures.

Percentage of Clients Who Have Secured or Retained Stable Housing Has Declined

According to the terms of the contract, the goals of the program are to improve the quality of life of participating clients. One of the methods used by IMT teams to achieve this goal is connecting homeless individuals with housing and assisting those who are housed to retain their housing for six months or longer. However, the audit found that the number of clients directly benefiting from these services has gone down over time.

The auditors’ review of the aggregate data reported for the period July 1, 2020 through September 30, 2022, showed that the percentage of clients who secured stable housing fell from 47% to 30%.[8] In addition, between January 1, 2021 and September 30, 2022, housing retention declined from 44% to 37%.[9]

Auditors reviewed the housing status for 52 randomly selected clients who entered the program between September 2020 and August 2022 and found that 37 were either living on the street, living in shelters/mental institutions, or were incarcerated. As of March 3, 2023, only 8 (22%) of the 37 clients were able to obtain stable housing, and two (5%) had been approved for housing and were awaiting placement. Fifteen of the 52 clients were housed at the time they entered the program; 14 of them remained in permanent housing at the time of the auditors’ review.

DOHMH stated that the falling percentages are due to an increase in the number of clients who need housing either prior to or after joining the program, as well as the shortage of available and affordable housing.

DOHMH’s Assessment of Program Effectiveness Is Limited

DOHMH views program retention and treatment as a key measure of program success.  However, as indicated earlier, the audit found that a significant portion of sampled clients who were retained in the program did not receive treatment on a regular basis.

Another important way DOHMH gauges the program’s effectiveness is through its comprehensive site reviews. Program specialists conduct these to determine whether providers are providing services according to the terms of their contract. DOHMH stated that it relies on these reviews to assess whether the IMT teams are successful in delivering treatment and services to clients and to track the progress of clients whose files have been reviewed.

However, auditors found that inconsistencies in the way the reviews were documented hindered aggregation of the results and thereby DOHMH’s ability to assess (1) the degree to which clients receive services, or (2) the overall progress of clients participating in the program. Additionally, DOHMH does not use the reviews to verify client-specific information reported monthly by providers.

Reviews Are Inconsistently Reported

Program specialists conduct two types of reviews: (1) an initial review within the first 90 days of the IMT team’s operation, and (2) a comprehensive review, performed every other year, which allows DOHMH to assess the program on an ongoing basis.[10]

The audit team reviewed 102 letters that were issued to 32 teams following a combination of 90-day and comprehensive reviews conducted between October 12, 2017, and September 19, 2023.

Nineteen of the letters were issued as a result of the 90-day initial review, and 83 letters were issued as a result of the comprehensive reviews.

90-day Initial Review:

According to DOHMH’s policies and procedures, during the 90-day review, program specialists assess the following areas: (1) whether the IMT team is staffed as required; (2) the safety of the program environment; (3) existence of program policies; (4) existence of the 24/7 call log (to ensure that an IMT member is reachable and on-call 24/7); (5) compliance with professional training for all team members; and (6) compliance with monthly updates in Maven. Program specialists also observe team discussions of clients’ status, and they select a sample of no fewer than 15 to 24 clients (minimum of three clients per team) per provider and review documentation and evidence of the provision of services. At the conclusion of the review, DOHMH program specialists provide written feedback to the IMT provider.

The 19 letters issued as a result of the initial 90-day review were missing information for at least one of the areas that should have been reviewed by the program specialist. Table I below illustrates the number of letters where the results of the areas reviewed during the initial review were referenced.

Table I: Number of Letters with References to Areas Reviewed During the Initial 90-Day Reviews

As shown in Table I, all 19 letters indicated that the reviews included checking the monthly updates in Maven. Conversely, only two letters indicated that the reviews included an examination of the call log. Overall, no letters made reference to all review areas and 12 letters made reference to seven of the areas that are to be assessed during the 90-day reviews.

Comprehensive Review:

In contrast to DOHMH’s policy pertaining to the 90-day reviews, the policies and procedures pertaining to the comprehensive reviews are not well defined. Instead, the policy only proposes what “may” be done during the review, with no clear requirement as to specific areas that must be reviewed.

According to DOHMH’s policies and procedures for comprehensive reviews, program specialists may collect or review the following: (1) key performances indicators for the previous months (procedures do not list the details of the key indicators); (2) staffing patterns for the previous 12 months; (3) recent incidents or grievances; (4) most recent audits (Office of Program Review audits and State Certification audits, if relevant [the procedures do not specify what is considered relevant]); (5) compliance reports of Assisted Outpatient Treatment (AOT), if relevant (the procedures do not specify what is considered relevant); and (6) verification of whether prior CAPs have been implemented.[11] The procedures also allow for a combination of any the following: review of charts, interview of clients, observation of the provision of services, and observation of the program environment (the procedures do not specify which of these areas or how many aspects must be reviewed).

Table II below presents an analysis of the areas referenced in the 83 sampled letters.

Table II: Number of Letters with References to Areas Reviewed During the Comprehensive Reviews

*Forty-six reviews pertained to providers that previously submitted CAPs. The percentage is based on the 46 reviews.

As shown in Table II, all 83 letters referred to an examination of client files and staffing levels. Conversely, none of the letters referred to examinations of recent incidents or of program review and state certification audits. Auditors were not able to determine whether the areas not referenced in the letters were due to program specialists finding no issues, the issue not being relevant to the review (e.g., no recent audits, incidents), or the areas were simply not examined.

In addition, the procedures call for a review of “key performance indicators,” but they do not define or specify the indicators, and the letters do not indicate what, if any, key performance indicators were examined.

Leaving the review methodology to the discretion of each program specialist allows for significant variations in the areas that may be covered during comprehensive reviews. The absence of guidelines for the program specialists to follow regarding the areas that must be covered during comprehensive reviews or what information must be contained in letters limits their usefulness and the agency’s ability to assess either the efficacy of the program or the providers’ overall performance in delivering services to clients. Moreover, inconsistencies in the reporting of results allows existing deficiencies in unreported areas to go undetected. It is also difficult to determine whether absence indicates that the area was simply not reviewed, or that no reportable issue was identified.

At the exit conference, DOHMH officials stated that the review letters generally only include the deficiencies identified, arguing that they would become unwieldy if every area examined were included. However, other than the review letters, DOHMH has identified no record detailing the results of the program specialists’ reviews. Establishing another recording tool (e.g., checklists signed by program specialists) would enable DOHMH to maintain a record of the areas covered during the review and help ensure that they are being conducted in a consistent manner and address all key areas.

Assessment of Services to Clients Not Tracked on an Aggregate Level

When conducting a review of a client’s file, program specialists review the following documents and assess the evidence for delivery of services: (1) risk and needs assessment; (2) crisis/safety plan and other treatment and recovery plans; (3) medical and mental health assessments; (4) diligent searches if person is missing; (5) progress notes documenting the progress of the case; (6) use of service dollars to engage the client; and (7) coordination with third party providers, family, and community.[12] As noted in the previous section, two of the 102 letters reviewed made no reference to the review of the client’s file. For the remaining 100 letters that did reference the review, the audit team found that 86 letters had one or both of the following inconsistencies:

• Program specialists did not consistently identify issues on a case-specific basis. Consequently, in 46 letters it is unclear whether the issue was found for one client or several.
• Letters did not consistently indicate the results for each of the areas listed above—63% were missing reference to service dollars utilized to engage clients, 19% were missing reference to diligent searches for missing clients, 18% were missing reference to crisis/safety and other treatment plans, and 15% were missing reference to any care coordination outside IMT.

Auditors did not find any of the above-mentioned inconsistencies in 14 letters.

DOHMH acknowledged that it does not aggregately track the extent to which clients receive various services. This means the agency’s ability to identify particular problem areas is limited, which in turn makes it more difficult to determine where increased monitoring is warranted.

DOHMH Does Not Assess Whether Client Data Submitted by Providers Is Supported

According to the City’s Procurement Policy Board Rules, agencies should develop performance indicators and monitor the vendor’s performance on an ongoing basis.[13] Part of that monitoring includes verifying that the performance data reported by vendors is adequately supported.

Although program specialists review progress notes in client case files during their comprehensive reviews, they do not reconcile, even on a sample basis, any of the information in the files with the client data reported to DOHMH. The monthly data reported by providers is therefore completely unverified and uncorroborated.

DOHMH officials and program specialists stated that they are in the process of reassessing the data collected from IMT providers and its own methods for evaluating IMT program and client progress. However, DOHMH had no timeline for when the process would be completed.

• The overall inadequacies with site visits and the letters used to document findings can be attributed in part to the ambiguities in the policy guidance developed for program specialists. For example, according to step #1 of the Program Policy Review Procedures, any combination of the following review procedures can be conducted: “review charts [files], interview consumers/tenants, observe service delivery, observe the program environment.” The same policy also states that the program specialist must include in the letter a “[s]ummary of findings, observations, strengths and weaknesses based upon step #1.” The risk of inconsistent approaches to reviews exists because the methodology used during the review is left to the interpretation and discretion of each program specialist.
• The absence of clear guidance on how reviews are to be conducted and documented limits DOHMH’s ability to assess whether program specialists conduct their site reviews in a satisfactory manner and may also undermine effective follow-up when issues are identified; this is especially relevant in instances when program specialists leave the program or are reassigned. DOHMH stated that it is in the process of standardizing the process but had no timeframe for when that would be completed.

DOHMH Needs to Improve Its Oversight Over Corrective Action Plans

If DOHMH program specialists identify egregious or persistent problems during a site review, they request a CAP from the IMT team,  which must be responded to by the team within 30 days. Per DOHMH’s policy, program specialists must also review whether the CAP was implemented during the subsequent visit.

During the five-year period covering October 12, 2017 through October 3, 2022, DOHMH program specialists performed 87 reviews (resulting in 87 letters), of which 56 reviews identified 137 issues serious enough to warrant a request for a CAP. These included such examples as an IMT position remaining vacant for more than 90 days, and client assessments that were repeatedly not completed.[14]

Auditors found that, on average, eight months passed before DOHMH visited the providers to assess CAP implementation. Days between the site visit relating to the CAP and DOHMH’s follow-up visit varied significantly, ranging from 41 days to 32 months. Auditors analyzed timelines of DOHMH’s follow up for 105 CAPs and found that the median amount of time that it took DOHMH to follow up on the CAPs after the initial site visit was six months.

Auditors found that although follow-up was conducted, DOHMH did not adequately review implementation of all issues noted in the CAPs.

The audit team reviewed the site visit letters for six providers, pertaining to 121 requested CAPs, to identify the follow-up efforts conducted by DOHMH in relation to those CAPs.[15] Auditors found the following:

• Based on the information noted in the letters issued upon the subsequent visits, program specialists reviewed the implementation status of CAPs for only 105 (87%) of the 121 issues cited. Subsequent letters made no mention of the implementation status of 16 issues noted in prior CAPs. For 7 of the 121 issues, providers failed to provide the requested CAPs.
• The review of the letters issued after subsequent visits indicate that IMT providers corrected only 53 (50%) of the 105 issues that had warranted CAPs during prior reviews. Though requested, DOHMH provided no information about what steps, if any, were taken regarding the implementation of the remaining 52 CAP items.

For example, DOHMH allowed one IMT team to continue operating even though the team lacked one of three required behavioral health specialists for almost three and a half years and had continuous issues with client assessments for close to four years, before the issue was resolved. Despite these issues, DOHMH allowed the provider to establish four additional teams. According to the reviews, three of the four new teams also had issues with staffing and assessments, indicating a persistent issue in these areas.

According to DOHMH, expansion of the program, as well as challenges due to COVID, led to a shortage of staff that prevented the agency from implementing a tracking system to monitor the issuance of CAPs—at the exit conference, officials stated that the agency currently has a staff vacancy rate in excess of 40%. As of December 1, 2023, the agency is in process of developing a system that will allow it to record CAPs requested from the IMT teams and track their progress and implementation. DOHMH did not have an estimated date for completion of the system.

Recommendations

The auditors recommend that DOHMH should:

1. Identify key treatment services provided to IMT clients and require providers to regularly submit to DOHMH information relating to the provision of those services.

DOHMH Response: DOHMH agreed with this recommendation.

2. Periodically (e.g., quarterly, annually) assess and report the degree to which the IMT initiative is effectively servicing clients in relation to all prescribed treatments; access to stable housing; and reduction in incarcerations.

DOHMH Response: DOHMH agreed with this recommendation.

3. Make all reasonable efforts to develop a less labor-intensive mechanism to obtain pre-program incarceration data for clients from the Department of Correction and use it in conjunction with the data submitted by providers to assess whether incarceration rates are reduced for clients in the program.

DOHMH Response: DOHMH agreed with this recommendation.

4. Develop reasonable targets for treatment provided to clients (e.g., percentage of treatment-related contacts held on a monthly basis) and establish protocols providers should follow when those targets are not met.

DOHMH Response: DOHMH generally agreed with this recommendation but stated that the “complex nature” of people referred to IMT made prescribing care requirements “unrealistic.” However, DOHMH said that it “expects providers to make minimum weekly contacts with their participants and/or weekly attempts to contact/locate those receiving services as a part of their IMT service provision.”

Auditor Comment: The auditors urge DOHMH to also consider the significance of establishing protocols for providers to follow when targets are not met and to establish protocols for providers to follow under those circumstances.

5. Establish performance measures that will allow the agency to assess and track the progress of the clients (e.g., extent to which clients’ treatment plans are followed and treatment goals are met) and identify areas that require improvement.

DOHMH Response: DOHMH agreed with this recommendation.

6. Make all reasonable efforts to reconcile, on a sample basis, key information contained in client case files with client data reported by providers.

DOHMH Response: DOHMH agreed with this recommendation.

7. Reassess the guidelines offered to program specialists in its Program Policy Review Procedures and ensure that it offers clear and unambiguous guidance about how comprehensive reviews should be conducted.

DOHMH Response: DOHMH disagreed with this recommendation, stating that “the Program Review Policy that applies to IMT also applies to the entire Bureau of Mental Health that monitors approximately 25 different program types across 500 or more unique programs”. DOHMH also stated that “program specialists receive regular supervision to discuss programs within their purview. In this supervision process, modeled on clinical supervision practices, program specialists receive additional guidance and clarification that is situation dependent”. DOHMH added that “additional formulaic specifications are not practical or advisable for the nature of this work, which must retain a level of flexibility to be truly centered around the unique needs of everyone served”.

Auditor Comment: The audit found that the absence of clear guidelines governing the conduct of the comprehensive reviews was a significant contributing factor for the inconsistencies identified. Those inconsistencies in turn limit the usefulness of the comprehensive reviews in assessing the efficacy of the program. Therefore, we urge DOHMH to reconsider implementing this recommendation.

8. Develop a uniform reporting template for program specialists to use so that their summary review reports clearly indicate the areas covered and note the severity of any deficiencies identified.

DOHMH Response: DOHMH partially agreed with this recommendation, stating that “policies and procedures are enacted to cover all bureau offices and service areas. In addition to policies and procedures, every unit has an identified outline to follow that guides their program reviews allowing for tailoring where necessary based on service type and clinical judgement.” Nevertheless, DOHMH agreed that it “will improve the consistency in documentation of program reviews.”

Auditor Comment: The auditors are pleased that DOHMH recognizes the significance of having consistent documentation of program reviews and urges DOHMH to consider fully implementing this recommendation so that there is a record of the areas covered and those where deficiencies are found are clearly identified, helping to facilitate effective follow-up.

9. Make all reasonable efforts to ensure that issues uncovered during the comprehensive reviews are rectified and develop a system for tracking the implementation status of all outstanding CAPs.

DOHMH Response: DOHMH agreed with this recommendation.

Scope and Methodology

We conducted this performance audit in accordance with Generally Accepted Government Auditing Standards (GAGAS). GAGAS requires that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions within the context of our audit objective(s). This audit was conducted in accordance with the audit responsibilities of the City Comptroller as set forth in Chapter 5, §93, of the New York City Charter.

The scope of this audit was January 1, 2016, through November 1, 2023.

To obtain an understanding of IMT’s operations, as well as DOHMH’s involvement in designing, maintaining, and overseeing IMT, auditors reviewed:

• Mayor’s Management Report
• IMT Program Concept Summary issued by DOHMH
• Contracts between NYC and IMT providers
• The Bureau of Mental Health Site Visit Policy & Procedures
• IMT Guidance Document
• Initial 90-Day Program Review Letters for IMT—Standardization

In addition, to further learn about IMT operations and to assess internal controls, auditors interviewed the following DOHMH officials and staff: Assistant Commissioner of the Bureau of Mental Health (BMH), Director of BMH, Deputy Director of BMH, Program Specialists, Director of Research and Evaluation, Deputy Director of Research and Evaluation, Comptroller of Fiscal Management, Acting Director of Close out Unit, Director of Contract and Compliance and Payments.

Auditors also observed four comprehensive reviews conducted by DOHMH program specialists as silent participants during the IMT teams’ meetings and watched program specialists conduct their reviews of the program.^[16]

To assess the information collected by DOHMH from the providers, auditors obtained a list of all clients enrolled in IMT between January 1, 2016, through March 3, 2023, the monthly update data for each of these clients, and the corresponding data dictionaries. To ensure that a correct interpretation of the data was applied, auditors met with the program specialists, as well as with the Director and Deputy Director of Research and Evaluation. Auditors conducted a walkthrough of Maven to understand how this data was collected, stored, and compiled into the reports that were used by DOHMH to produce quarterly reports for IMT clients.

Assessing the Extent to which IMT Met the Program Objectives

To assess the extent to which IMT met the program objectives, auditors reviewed the following pertaining to the clients: retention in the program, visits with the psychiatric care practitioner, adherence to prescribed psychiatric medication, incarcerations as reported by DOHMH, and housing trends. Each of these steps is described in detail below.

Retention

To determine the actual retention rate for all IMT clients, auditors obtained a list of 1,027 clients who were enrolled in the IMT program from January 1, 2016 through March 3, 2023. This data set contained, among other information, the client number, date of admission to IMT, date of discharge, if applicable, and date of transfer, if client was transferred to a different IMT team. Auditors considered all clients who were active or discharged and whose enrollment date had been on, or prior to, March 3, 2022. Auditors also excluded clients who were discharged due to death or transfer to a long-term psychiatric institution, assisted living, or jail. Applying this methodology, auditors were left with 602 clients and determined the retention rate for this set of clients using a 3-month, 6-month, 9-month and 12-month enrollment criteria. Auditors then compared the differences in rates to determine how the different time periods influenced the retention rates.

Treatment and Medication

To establish the population for testing treatment, medication and housing, auditors used a list of 1,027 clients who were enrolled in the IMT program from January 1, 2016 through March 3, 2023. To ensure that the data was meaningful, auditors excluded from the population any client who had been enrolled for less than six months. As a final result, the population consisted of 555 clients who were enrolled in IMT between September 1, 2020 and August 31, 2022 and who were still enrolled in the program as of March 3, 2023. Auditors chose a random sample of 56 of the 555 clients and analyzed the extent to which the clients received treatment, adhered with their medication schedule, and found housing. After a review of monthly updates, auditors excluded four clients for the purposes of testing treatment (three of the clients’ updates were missing for some months based on the data, auditors were not able to determine whether another client had been diagnosed with a mental illness). Twelve clients were excluded from the medication adherence test (three of the clients’ updates were missing for some months and nine were not prescribed medication).

For the remaining 52 clients, auditors reviewed the data to determine whether the clients were visited by an IMT psychiatrist and/or psychiatric nurse and calculated the number of months each client had been seen, as well as the percentage of the enrollment months that the client saw the mental health practitioner.

In addition, auditors analyzed the information provided in the monthly updates for 44 clients, who were prescribed medication, to determine whether clients were taking medication all of the time, most of the time, sometimes, or rarely.

Incarcerations

To test whether there was a reduction in the number of incarcerations for clients after entering the program, auditors reviewed two studies that had been conducted by DOHMH. The first study, completed in 2020, was based on data from January 2016 through March 2018, and the second study, completed in February 2022, and was based on data as of December 2018.

In addition, from the randomly selected sample of 56 clients, auditors excluded three clients because the monthly updates had been missing for some months. Monthly updates and adverse events data for the remaining 53 clients were reviewed to determine whether these clients had been imprisoned during their enrollment in the program.

Use and Verification of Monthly Update Data

To determine how DOHMH used the data pertaining to the monthly updates, and whether it independently verified the data, auditors met with representatives of BMH and Research and Evaluation, conducted walkthroughs of the Maven system, and participated in a meeting where a representative from the Research and Evaluation unit demonstrated how data was collected and translated to reports. Auditors tested the data pertaining to the clients and the monthly updates for completeness and accuracy to determine whether the data contained significant errors and requested an explanation from DOHMH in instances where the data had been missing and appeared to be inaccurate. DOHMH explained that it was aware that about five percent of the data reported by providers was either incomplete and/or inaccurate, but that DOHMH considered it to be a small and immaterial error rate and not consequential to the production of reports. Auditors found no errors with the completeness and accuracy of the client data and one percent error rate for the monthly updates were smaller than those DOHMH considered acceptable and therefore deemed the data reliable for testing.

Testing the Reviews

In October 2022, DOHMH provided auditors with 87 letters that were written at the completion of the reviews that DOHMH program specialists conducted from October 12, 2017 through October 3, 2022. In October 2023, an additional 15 letters were received. The audit team reviewed a total of 102 letters that were issued to 32 teams following a combination of 90-day and comprehensive reviews conducted between October 12, 2017 and September 19, 2023. Nineteen of the letters were issued as a result of the 90-day initial review, and 83 letters were issued as a result of the comprehensive reviews.

Auditors compared the content of the 19 letter from 90-day review to Initial 90-Day Program Review Letters for IMT—Standardization procedure; and 83 letters to DOHMH’s Program Site Visit Policy and Procedure, as well as to its Guidance Document to determine whether the letters covered all of the areas required to be reviewed during site visits and also compared the content of the letters to determine whether the letters were consistent in how findings were reported. Auditors also assessed whether the information, as reported, served as an adequate tool for the assessment of the program’s success and for the progress of the individual clients.

In addition, auditors reviewed the original 87 letters issued by DOHMH program specialists to determine whether issues found by the program specialist were egregious enough to warrant a request for a CAP. Auditors also reviewed the 51 responses from the providers to the request for CAPs to determine whether IMT providers responded with a CAP to address the issues.

Finally, to determine whether DOHMH program specialists followed up on the implementation of the CAPs and whether issues were corrected, auditors reviewed the letters issued for each subsequent review.^[17]

The results of the above tests, while not projectable to their respective populations, provided a reasonable basis for the auditors to evaluate and support their findings and conclusions regarding whether DOHMH has established measurable performance targets for the IMT initiative that align with its stated objectives, as well as the degree to which the initiative effectively services its clients.

Appendix

Table of 56 Sampled Clients and Their Outcomes for Housing, Treatment, and Incarcerations while Enrolled in IMT

Legend:

*              Samples excluded from housing review while clients temporarily housed in jail, mental health facility, hospital, or nursing home.
**             Samples excluded from treatment and medication adherence review due to incomplete updates or missing mental illness diagnosis.
***           Samples excluded from medication analysis due to missing information regarding adherence.

Addendum

Maura Hayes-Chaffe
Deputy Comptroller for Audit Office of the Comptroller
1 Centre Street Room 1100 New York, NY 10007-2341

Re: Response to Draft Audit Report on The Intensive Mobile Treatment (IMT) Initiative Carried Out by The Department of Health and Mental Hygiene
MG22-092A

Dear Maura Hayes-Chaffe:

The NYC Department of Health and Mental Hygiene (Health Department or DOHMH) reviewed the draft audit report on the Intensive Mobile Treatment (IMT) initiative carried out by the Health Department. The stated objectives of the audit were to determine whether the Health Department established performance targets for the IMT initiative that align with its stated objectives, as well as the degree to which the initiative effectively serves its clients.

The Health Department appreciates the auditors’ efforts to understand and assess the IMT initiative and identify improvement opportunities. The Health Department also thanks the auditors for the opportunity to respond to the draft report and for their courtesy and professionalism during the audit process.

Attached is the Health Department’s response to the draft audit report. If you have any questions or need further information, please contact Sara Packman, Assistant Commissioner, Audit Services, at spackman@health.nyc.gov or at (347) 396-6679.

Sincerely,
Ashwin Vasan, MD, PhD Commissioner

CC:
Emiko Otsubo, Chief Operating Officer/Executive Deputy Commissioner, DOHMH
Deepa Avula, Executive Deputy Commissioner, Division of Disease Control, DOHMH
Christina Chang, Chief Program Officer, DOHMH
Sara Packman, Assistant Commissioner, Bureau of Audit Services, DOHMH

DOHMH’S RESPONSE TO THE AUDITORS’ DRAFT AUDIT REPORT ON THE INTENSIVE MOBILE TREATMENT INITIATIVE CARRIED OUT BY THE DEPARTMENT OF HEALTH AND MENTAL HYGIENE
MG22-092A

The NYC Department of Health and Mental Hygiene (Health Department, DOHMH) reviewed the draft audit report on the Intensive Mobile Treatment (IMT) initiative carried out by the Health Department. The stated objectives of the audit were to determine whether the Health Department established performance targets for the IMT initiative that align with its stated objectives, as well as the degree to which the initiative effectively serves its clients.

DOHMH appreciates the auditors’ efforts to understand and assess IMT and identify improvement opportunities.

The IMT program began in 2016 and has grown to operate citywide with a capacity to serve 837 New Yorkers. The program was developed to serve people experiencing the most challenging and complex behavioral health and life circumstances, and to fill a critical gap in the behavioral healthcare, homeless services, and criminal justice systems. The IMT clients have been unsuccessfully served by traditional treatment models, models that are primarily direct care, brick and mortar sites with limited flexibility in terms of the service types they can offer and staff resources sufficient only to meet minimum expectations around frequency and duration of client contacts to obtain insurance reimbursement.

DOHMH developed the IMT program considering the complex needs of the individuals served, in turn creating a flexible program model to adapt to their needs; transient, often street homeless or moving between shelter locations, at risk of losing housing and with complex behavioral health and social needs. Through contracted providers, the IMT teams are dedicated to engaging and building trust with those who have had extensive trauma, and negative experiences with institutions and systems. The IMT services include:

• assertive outreach
• persistent engagement
• crisis intervention, including 24/7 crisis phone response
• peer support
• assistance meeting basic needs (safe and stable housing, access to nutritious food)
• individual counseling
• psychopharmacology and medication assisted treatment
• vocational and educational support
• individual and family psychoeducation
• wellness skills
• linkage to and follow up with primary medical care

A major focus of the IMT program is connecting with the participants, developing rapport, and addressing basic needs. Engagement attempts themselves hold a therapeutic value for participants, as do addressing the social determinants of health through securing food, toiletries, public benefits and ensuring a safe place to sleep. Finally, peer support sessions, and counseling sessions utilizing evidence-based practices such as motivational interviewing, and harm reduction, are all also important components of treatment that are included in DOHMH’s definition of IMT services. DOHMH is concerned that the auditors focused mainly on treatment services of prescriber meetings and medication adherence and did not consider the full array of IMT services. DOHMH’s response to the auditors’ recommendations follows.

Audit Recommendations and DOHMH’s Response

1. “Identify key treatment services provided to IMT clients and require providers to regularly submit to DOHMH information relating to the provision of those service.”
   
   DOHMH Response:
   DOHMH generally agrees with this recommendation although we consider key IMT services to include all the service specified above, which are provided by all members of the multi-disciplinary team whereas the auditors focused on services provided by the psychiatric care practitioner. IMT providers are required to submit monthly reports to DOHMH for each person served. Currently, services reported to DOHMH are limited. DOHMH is in the process of assessing the data collected to identify new indicators and will establish a level of service metric for monthly reporting starting in FY25.
2. “Periodically (e.g., quarterly, annually) assess and report the degree to which the IMT initiative is effective servicing clients in relation to all prescribed treatments; access to stable housing; and reduction in incarcerations. ”

   DOHMH Response:
   DOHMH generally agrees with this recommendation. Starting in FY25, DOHMH will annually measure clients’ progress in maintaining stable housing, receiving care and on clients’ decrease in the rate criminal justice involvement.
   The auditors focused on clients’ psychiatric care and adherence to prescribed medications. However, engaging clients in social emotional activities, assisting them to improve conditions in their environments, safe and stable housing, access to nutritious food, reduced exposure to violence, racism, and discrimination (collectively social determinants of health) are as valuable to an individual’s care as traditional treatment. As was told to the auditors, DOHMH is in the process of assessing data collected and will update its indicators to measure the additional impact of IMT services on its clients.

3.  “Make all reasonable efforts to develop a less labor-intensive mechanism to obtain pre- program incarceration data for clients from the Department of Correction and use it in conjunction with the data submitted by providers to assess whether incarceration rates are reduced for clients in the program.”

   DOHMH Response:
   DOHMH agrees with this recommendation. DOHMH will review the current process for comparing information from the Department of Corrections and work with DOC to identify possible means to reduce manual data matching where possible. DOHMH will aim to collect and analyze incarceration related data on an annual basis.

4. “Develop reasonable targets for treatment provided to clients (e.g., percentage of treatment -related contacts held on a monthly basis) and establish protocols providers should follow when those targets are not met.”

   DOHMH Response:
   DOHMH generally agrees with this recommendation. Due to the complex nature of the people referred to IMT services; often homeless, living on the street, unlocatable and transient, prescribing a requirement for care like other models that bill for services is not realistic. However, DOHMH expects providers to make minimum weekly contacts with their participants and/or weekly attempts to contact/locate those receiving services as a part of their IMT service provision.

5. “Establish performance measures that will allow the agency to assess and track the progress of the clients (e.g., extent to which clients’ treatment plans are followed and treatment goals are met) and identify areas that require improvement.”

   DOHMH Response:
   DOHMH agrees with this recommendation and is in the process of finalizing key performance indicators that will be used to measure, track and monitor clients’ progress on an annual basis starting in FY25.

6. “Make all reasonable efforts to reconcile, on a sample basis, key information contained in client case files with client data reported by providers.”

   DOHMH Response:
   DOHMH agrees with this recommendation and will identify the resources that will be required as we identify indicators that are important to the IMT service model and ways in which these can be reconciled via sample on a periodic basis.

7. “Reassess the guidelines offered to program specialists in its Program Policy Review Procedures and ensure that it offers clear and unambiguous guidance about how comprehensive reviews should be conducted.”

   DOHMH Response:
   DOHMH disagrees with this recommendation. The Program Review Policy that applies to IMT also applies to the entire Bureau of Mental Health that monitors approximately 25 different program types across 500 or more unique programs. IMT programs are within the purview of the Director of Treatment Services, who manages a team of experienced mental health program specialists who monitor several different types of mental health treatment programs. Program specialists receive regular supervision to discuss programs within their purview. In this supervision process, modeled on clinical supervision practices, program specialists receive additional guidance and clarification that is situation dependent. Additional formulaic specifications are not practical or advisable for the nature of this work, which must retain a level of flexibility to be truly centered around the unique needs of everyone served.

8. “Develop a uniform reporting template for program specialists to use so that their summary review reports clearly indicate the areas covered and note the severity of any deficiencies identified.”

   DOHMH Response:
   DOHMH partially agrees with this recommendation. DOHMH contracts with providers of multiple service types across the Division of Mental Hygiene’s bureaus. The IMT program sits within the Bureau of Mental Health that contracts for over 25 program types, totaling over 500 programs. Policies and procedures are enacted to cover all bureau offices and service areas. In addition to policies and procedures, every unit has an identified outline to follow that guides their program reviews allowing for tailoring where necessary based on service type and clinical judgement. DOHMH will improve the consistency in documentation of program reviews.

9. “Make all reasonable efforts to ensure that issues uncovered during the comprehensive reviews are rectified and develop a system for tracking the implementation status of all outstanding CAPS.”

   DOHMH Response:
   DOHMH agrees with this recommendation and will revise the CAP process. In FY24, DOHMH will review its options and select a mechanism for recording and monitoring the implementation of CAPS to issues uncovered during comprehensive reviews.

---

Endnotes

[1] As defined by DOHMH, a “traditional” treatment model includes mental health care where services are provided onsite at a physical office location. These services require regular participation, including expected frequency and duration of appointments. Traditional mental health services bill insurance and, in most instances, if a client doesn’t meet the requirements for active enrollment, they are discontinued from care.

[2] These details are not shared with DOHMH program specialists on a regular basis and are reviewed by DOHMH on a sample basis, during its comprehensive review.

[3] From its inception in 2016 until August 2019, the comprehensive review was conducted twice a year. From August 2019 until November 2021, the policy was revised, and comprehensive reviews were mandated to be completed once a year. The policy was revised again, and since November 15, 2021, comprehensive reviews are mandated biannually. Additionally, the number of charts selected for review has changed—at first, the policy required an annual review of five charts at minimum, but the policy as of November 2021 requires a review of three charts per each scheduled comprehensive review.

[4] DOHMH has not established a standard number of mental health visits for IMT clients. Each client is assessed by the provider, and treatment needs are established for each client individually. While DOHMH does require the provider to report monthly whether the client was visited by a psychiatric care practitioner, there is no requirement for providers to indicate the number of visits conducted. As a result, in its assessment of this area, the audit team disregarded the number of actual visits conducted each month and was only concerned with whether a visit was recorded at least once a month (gave credit for monthly visits).

[5] The standard procedure of DOHMH is to review a minimum of 3 clients per IMT team. The number of teams utilized by the 5 providers contracted by DOHMH ranges from 5 to 8 (there are currently 31 IMT teams in total). DOHMH’s current procedure is to conduct a comprehensive review of each IMT team once every other year. However, DOHMH strives to conduct these reviews more often. Auditors reviewed 36 letters for 24 IMT teams issued for reviews conducted since the inception of the comprehensive review on November 15, 2021, which were not the initial letters issued for the team, and determined that 35 of the reviews were in fact performed in less than 24 months since the prior review—27 of the 35 reviews were performed in 12 months or less.

[6] According to a DOHMH program specialist, the client records maintained by the providers contain the treatment plan and if applicable, the frequency of visits with psychiatric care practitioners.

[7] In its response to the audit report, DOHMH expressed concern that the auditors’ analysis focused mainly on treatment services of prescriber meetings and medication adherence and did not consider the full array of IMT services. As indicated in the report, the monthly data upon which the audit’s analysis is based does not contain meaningful statistics relating to the other services provided through the IMT program.

[8] DOHMH has not compiled any quarterly reports since September 2022, which is when the Mayor’s Office of Community Mental Health (OCMH) stopped requiring such reports.

[9] Maven, the source for the housing data reviewed, was brought into operation in July 2020.

[10] The comprehensive reviews had been performed more frequently prior to November 15, 2021.

[11] “Kendra’s Law” (§9.60 of the Mental Hygiene Law) mandates mental health services for a small number of individuals who have difficulty engaging in rehabilitation and can pose a risk to themselves or others in the community. The order is granted in civil court. The New York City Assisted Outpatient Treatment program is responsible for the implementation of Kendra’s Law in the five boroughs of New York City.

[12] The criteria of the client file review is listed in the procedure related to the 90-day review. The procedure for comprehensive review does not list the specific areas to be reviewed. It just instructs that a client file review may be done. Based on the review of the 83 comprehensive review letters, there is a practice for the program specialist to review the area as listed in the 90-day review procedure. Therefore, auditors applied the criteria to this analysis.

[13] Procurement Policy Board Rules, Chapter 4 (“Contract Administration”), Section 4-01, “Evaluation and Documentation of Vendor Performance.”

[14] The issues applied to all six providers and 26 of the 32 teams. One CAP is requested for each egregious issue; 28 reviews resulted in more than one CAP.

[15] Of the 137 requested CAPs, reviews connected with 16 of them were not due for a follow-up visit at the time of the audit test.

[16] This included a review of a sample of client files, as well as a review of the team’s staffing, training, Maven monthly updates, program policies, 24-hour call logs, and safety of the physical environment.

[17] For each letter that was the last one in the batch of 87 letters, and which had a CAP request, auditors asked DOHMH to provide a subsequent letter so they could assess whether a review had been conducted. In October 2023, auditors received 15 additional letters and used them to review the CAP follow-up and implementation.