Audit Report on the New York City Office of Technology and Innovation’s MyCity System
Audit Impact
Summary of Findings
The New York City Office of Technology and Innovation (OTI) developed the MyCity application to serve as a “one-stop shop” for users to apply for and receive City benefits and services. Although OTI created a Common Services platform and digitized an existing childcare application, the agency has very little else to show after spending more than four years and $100 million on the program. Users still cannot apply for City benefits and services through a single form or receive benefits and services via digital wallet. The applications process for low-income New Yorkers and City employees is not streamlined, as promised.
While MyCity provides access to information about benefits, business, and jobs services, it largely redirects users to pre-existing City websites that have been rebranded and redesigned as MyCity applications. In addition, the Childcare portal has limited and ineffective functionality, and the Artificial Intelligence (AI) Chatbot that was developed appears to be unable to provide accurate or consistent information. MyCity still cannot track City agency performance, and low website performance scores for the portal may impact user experiences.
Overall, based on the evidence provided by OTI, the audit concludes that MyCity was poorly managed from both a project management and contract oversight perspective. In addition to the $100 million spent so far, OTI has requested a further allocation of $81 million in the 2026 budget to maintain what has been created and to add additional functionalities.
Intended Benefits
The audit’s recommendations are intended to help OTI enhance MyCity workflows, strengthen project management and contract oversight, improve the delivery of services and benefits, and streamline access to City programs.
Introduction
Background
In January 2022, Mayor Adams signed Executive Order 3, a directive that consolidated the City’s various technology operations under a single agency: the New York City Office of Technology and Innovation (OTI). OTI was granted oversight of all Citywide information technology, information security, information privacy, and telecommunications matters, as well as the authority to develop and implement related projects.[1] That same month, the Mayor announced that the City, under OTI, was creating the MyCity portal, an online platform intended to serve as a “one-stop shop” for all City services and benefits, so that New Yorkers would not be required to visit different agency websites or offices to access and apply for services.
The portal was intended to help low-income New Yorkers access services for which they qualify but do not receive due to the complicated and time-consuming nature of applications processes. Although programs require much of the same information and documentation to support eligibility, applicants must complete multiple applications and, in some cases, appear for in-person interviews during the workweek, which may cause them to miss work and lose pay.
MyCity was intended to streamline this process by creating a universal application and verification process, so New Yorkers could qualify for multiple benefits at the same time. MyCity was expected to let users share their data with other City agencies, allowing them to receive proactive benefit recommendations and access benefits for which they might be eligible via a digital wallet. The portal was also intended to reduce the workload on City employees responsible for processing applications.
Childcare Portal
In March 2023, OTI launched phase one of the MyCity portal, which digitized the application for childcare assistance offered by the Administration for Children’s Services (ACS) and the Department of Education (DOE), allowing users to upload supporting documentation. Users were able to create a single MyCity account where they could apply for services in more than 10 languages and save their information and documentation for future applications.
Before the MyCity portal, residents applied for childcare assistance and submitted all required documentation through a manual paper process to ACS or DOE, depending on the type of childcare requested. Now, residents can self-screen for eligibility, apply for childcare vouchers and contracted care, and track application status online. In March 2024, OTI expanded childcare service functionality to allow families receiving benefits to submit their annual recertifications through MyCity.
In the Fiscal 2025 Mayor’s Management Report, OTI reported that as of June 2025, more than 120,000 applications and recertifications for childcare were submitted through the MyCity childcare portal since its launch in spring 2023.
Business Portal
Since its initial launch, OTI has added a business services portal to help users identify licenses, permits, and other requirements to operate a business, as well as an AI Chatbot to answer questions and provide information about owning and operating a business in the City. Later in March 2025, the Chatbot capabilities were expanded to include information on City government services.
Jobs Portal
In addition, OTI added a Jobs NYC portal to help people prepare for and enter the workforce by providing users with links to resources identifying available training programs, education, and job opportunities. The Jobs NYC portal also provides employers with links to resources for hiring employees or interns, and apprenticeship programs.
Benefits Portal
This portal redirects users to ACCESS NYC, a website maintained by the Mayor’s Office for Economic Opportunity that compiles information about benefit programs. The portal allows residents to check potential eligibility for local, state, and federal benefit programs, and offers information and guidance on how to apply for services, such as family support, food assistance, healthcare, and housing.
Organizational Structure
MyCity is overseen by three major divisions of OTI that work with vendors and stakeholders to plan, design, test, and implement the project. These three divisions are:
- Application Development Services, which is responsible for designing, building, and supporting enterprise-level products and services for the public and City agencies.
- The Office of Data Analytics, which develops data standards and governance to support Citywide data integration and enhances Citywide infrastructure and sharing.
- Design Services, which applies human-centered design and user research to shape the MyCity interface and notifications, guiding applicants through the enrollment process.
In addition, OTI contracted with vendors to design, develop, and oversee MyCity. Specifically, OTI contracted with Innovative Business Concepts Inc. (IBC) to assess current state and develop short and long-term plans for MyCity portal development, including a project plan, detailed functional requirements, and a roadmap, among other things.
OTI also contracted with Rangam Consultants Inc. to provide technical project management oversight.
OTI also entered into a Data Sharing Agreement (dated March 21, 2023) with various City agencies to streamline services that they administer. These agencies include ACS, DOE, the Department of Homeless Services (DHS), and the Human Resources Administration (HRA).
System Development Costs
MyCity was initially intended to be developed in-house, using OTI staff. However, OTI stated that due to a lack of staff resources, the office contracted with outside vendors for project planning, design, development, quality assurance testing, and system implementation and maintenance. OTI contracted with approximately 50 different vendors to perform various aspects of these services.
As of October 2025, OTI has spent more than $100 million through more than 120 agreements (including contracts, purchase orders, and delivery orders). According to OTI’s January 2026 budget plan, the office plans to spend an additional $74.3 million for capital needs related to the maintenance of existing services and future design. The plan also called for $6.6 million for Other than Personal Services (OTPS) expenses, including 24 new full-time staff to replace 16 consultant positions.
Objective
The objective of this audit was to determine whether MyCity streamlines the delivery of services and benefits and allows the City to track agency performance in real time.
Discussion of Audit Results with OTI
The matters covered in this report were discussed with OTI officials during and at the conclusion of this audit. An Exit Conference Summary was sent to OTI on November 21, 2025, and discussed with OTI officials at an Exit Conference held on December 1, 2025. On December 8, 2025, we submitted a Draft Report to OTI with a request for written comments. We received a written response from OTI on December 19, 2025. In its response, OTI disagreed with all seven recommendations and findings related to project management and contract oversight.
OTI stated that the report simplifies the concept of the “one-stop shop” and misstates the intent and functionality of the MyCity platform. OTI further stated that MyCity was never intended to replace individual agency website functionality. However, when MyCity was launched in March 2023, Mayor Adams stated in the introductory video on the portal that MyCity is a “one-stop shop” where New Yorkers can easily search and apply for City services and benefits, track application status, and securely store personal information all in one place.
In addition, OTI raised several arguments that were also raised at and following the Exit Conference and that were already addressed in the Draft Report, including assertions that the report wrongly assumes that the agency must use the waterfall system development model. OTI stated that MyCity was implemented using an agile methodology and the agency never intended for IBC to provide a final project plan, roadmap, timeline, or cost benchmarks for MyCity.
As detailed in this report, the OTI contract with IBC for MyCity portal design and development services states that IBC could use a hybrid system development methodology, employing both waterfall and agile methodologies. The purpose of this was to “retain the clarity and tracking system of the waterfall method” and to provide the flexibility of the agile method. Furthermore, although IBC contracts included deliverables for assessing the existing state, defining key functional requirements, creating milestone-based project plans with deadlines, and cost estimates, OTI failed to produce evidence that IBC delivered them.
OTI’s written response has been fully considered and, where relevant, changes and comments have been added to the report. The full text of OTI’s response is included as an addendum to this report.
Detailed Findings
Although OTI has created a Common Services platform and digitized an existing childcare application which allows users to perform related transactions online, OTI has very little else to show after the passage of four years and City expenditures that exceed $100 million.
OTI has not fulfilled its primary goal of creating a one-stop shop and users are still not able to apply for all City benefits and services through a single form, receive benefits and services via a digital wallet, or benefit from a streamlined applications process for low-income New Yorkers and City employees, as promised. MyCity is still not able to track City agency performance, as also promised, and “new” portals that have been established under the program add few new features or functionality.
The “new” Benefits portal within MyCity simply redirects users to the ACCESS NYC website, and the “new” Business Services and Jobs portals mainly consist of pre-existing websites that have been rebranded and redesigned as “MyCity applications.” Although OTI has also established an AI Chatbot, it appears to be unable to consistently provide accurate information to users. The Childcare portal—the most developed so far—has limited and ineffective functionality. In addition, low website performance scores for the portal may impact user experiences.
Overall, based on the evidence provided by OTI, the audit concludes that MyCity was poorly managed from both a project management and contract oversight perspective, and that it ultimately represents a poor return on investment. In addition to the $100 million spent so far, OTI has requested a further allocation of $81 million in the 2026 budget to maintain what has been created to date and to add additional (unspecified) functionalities.
OTI Lacked a Clear Vision and Detailed System Development Plan for MyCity
OTI contracted with Innovative Business Concepts, Inc. (IBC) to develop short- and long-term plans for MyCity portal development, including those related to the Business and Childcare workstreams. Each of the four contracts stated that IBC was responsible for producing deliverables, and OTI Project Managers were responsible for reviewing and accepting them.
According to the contract terms and conditions, IBC’s deliverables included identifying and defining MyCity workstreams and creating a project plan, defining key functional requirements and implementation timelines by workstream, and the provision of an overarching roadmap. The project plan provided by IBC was also to include cost estimates to ensure that the desired implementation scope was attainable within budget.
Despite OTI paying $2.5 million to IBC under the contracts, and despite repeated requests to OTI over many months, OTI has failed to produce any evidence that IBC delivered to the City a roadmap document, detailed project plans, or associated cost estimates. OTI has failed to produce a comprehensive system development plan for MyCity or any other documentation showing the “current state” as of the project start, or the anticipated “future state” upon project completion. OTI has also failed to produce time or project benchmarks that were to be met by IBC prior to payment being rendered or any documentation to show that OTI held IBC accountable for these.
All that OTI has produced to date is high-level documentation that does not constitute detailed project plans, a roadmap, project timeline or cost benchmarks. In the absence of this documentation, this audit concludes that OTI lacked a clear vision for the development of MyCity, lacked detailed project plans, and did not hold IBC accountable for deliverables it was intended to provide.
While MyCity provides access to information about benefits, business, and jobs services, OTI largely rebranded or redirected users to pre-existing City websites. Prior to MyCity, the City already had websites (such as NYC311) that compiled information about benefits and support, childcare, starting and operating businesses, and jobs, among other things, and directed users to City agency websites. Under MyCity, users must still complete multiple applications for benefits and services and apply for them through administering agency websites. This does not meaningfully streamline the application process.
These issues are discussed by workstream in greater detail below.
At the exit conference, OTI officials stated that the intent of the IBC contracts was for IBC to provide a “scan” of the current state and a high-level roadmap and was not expected to produce detailed system requirements. OTI officials simultaneously stated that the agency came up with its own final requirements, that MyCity was developed using an agile approach, and that the process is ongoing. They further stated that it is not good practice to be locked into a pre-destined path.
However, OTI’s assertions are contrary to the scope of services detailed in the IBC contract for design and development services for the MyCity portal. This contract state that IBC could use a hybrid methodology that would “retain the clarity and tracking system of the waterfall method” and provide the adaptability and flexibility of the agile method.[2] According to the National Institute for Standards and Technology (NIST), when the waterfall model is used, the requirements are usually well-defined so planning can be done up front. As also stated in NIST, this “model works well with complex projects where the requirements are well understood because the complexity can be dealt with in a structured way.”
According to the contracts, IBC was responsible for analyzing the as-is conditions (current state) of City service platforms to ensure a thorough understanding of existing service landscape, establishing the MyCity project workstreams and a draft of the future state architecture, and creating a milestone-based project plan for each workstream outlining key deliverables and deadlines as well as an overarching roadmap stitching together the various individual workstream timelines, among other things. Additionally, the contracts stated that IBC would present the proposed portal design and roadmap to the OTI Chief Technology Officer and executive sponsors for final approval. Although OTI paid IBC for MyCity portal design and development services in October 2022, OTI failed to produce any evidence that IBC delivered a roadmap document, detailed project plans, or associated cost estimates.
After the exit conference, OTI provided a limited number of work-in-process, quarterly roadmaps. However, these roadmaps were not prepared during the MyCity design and development phase in 2022. Rather, these roadmaps provide a high-level vision for the “Overall MyCity Solution” and a recap and updates to the MyCity Roadmap for the first, third, and fourth quarters of Calendar Year 2025.
MyCity Does Not Meaningfully Streamline the Process to Apply for City Benefits and Services
Common Services
MyCity’s Common Services includes systems integration technology that allows MyCity to save and unify user profile information, which allows the storage of personal information documents, tracking of application status, and receival of notifications. Users can create account profiles and save their personal information (such as name, date of birth, address, and Social Security number) or business information (such as business name, Employer Identification Number, or Social Security number), and upload documentation required for childcare program eligibility determinations. Users who create business profiles can also search for and save business transactions with other City agencies to a dashboard including licenses or permits, violations, or payments.
However, MyCity does not leverage this user information to make it easier to apply for benefits or services and does not evaluate this information to proactively recommend what services users are eligible for.
In addition, OTI did not implement a digital wallet that would allow users to access digital versions of City benefits and payment, as promised.
Benefits
MyCity’s Benefits workstream simply redirects the user to the ACCESS NYC website, to which OTI made no enhancements or functional improvements.
ACCESS NYC was designed and developed internally by the Mayor’s Office for Economic Opportunity in 2006 and relaunched in March 2017. This website has a screening tool that allows users to determine whether they are eligible for federal, New York State, and New York City health and human services benefit programs and provides users with program information including benefits offered, eligibility, and document requirements, and how to apply.
Jobs
When OTI initially launched the Jobs workstream, the office simply rebranded an existing City website, Jobs NYC, as MyCity. Like ACCESS NYC, this website was designed and developed internally by the Mayor’s Office for Economic Opportunity.
In addition, OTI added information for employers regarding hiring employees, internships, and apprenticeships. Users are provided with brief program descriptions and links to City agencies offering programs including HRA, the City University of New York (CUNY), the Department of Small Business Services (SBS), and the Department of Youth and Community Development (DYCD).
Since the initial launch, OTI has changed the website platform and added information from, or links to, existing New York City or New York State websites.[3] Users still cannot apply for jobs or employer services through MyCity; instead, they are redirected to applicable agency websites, as they were before.
Business Services
OTI redesigned the NYC Business website to provide information and resources for operating a business with the City, which became the MyCity Business portal. OTI created the “My Business Dashboard” (where users can view and save business transactions) and incorporated an AI Chatbot to answer business related questions. OTI later expanded the Chatbot’s purview to include government services and programs.
However, the MyCity Chatbot is still in the beta testing phase and does not consistently provide accurate responses, as detailed later in the report. Additionally, the portal lacks the capability to perform full transactions (such as applying for licenses and permits), as well as an intended feature for scheduling inspections.
Like the other workstreams, the Business portal merely redirects users to various City agency websites, where they can process transactions needed for their business operation.
Childcare
The only MyCity workstream that enhances user experience is the Childcare portal. MyCity digitized applications to streamline the manual paper application process for certain childcare assistance offered by ACS and DOE and allows users who create MyCity accounts to complete applications online and upload supporting documentation directly through the portal. In March 2024, some two years after the project started, OTI expanded Childcare functionality to allow families receiving benefits to submit their annual recertifications through MyCity. While users can apply for either ACS childcare vouchers or DOE-contracted care programs through the portal, agency notifications and eligibility determinations are still sent via mail or email.
In addition, MyCity offers an optional self-screening tool which provides users with guidance on ACS and DOE program eligibility based on their responses to questions about household income and children’s age, among other things.
However, these services are still limited and inefficient. OTI did not build into the online application process basic eligibility checks based on a child’s age and household income. If added, this functionality would immediately inform applicants upfront that they may not qualify for childcare services based on information they provided on applications and inform them about other services for which they may qualify.
Automating basic eligibility screening would streamline workflow for agencies by flagging applicants that may not qualify based on information provided on their applications and increase the efficiency of the eligibility and notification process. According to MyCity data, approximately half of the applications submitted through the portal were deemed ineligible. Since its launch, 140,760 applications were submitted through MyCity since inception, of which 65,369 (46%) were ineligible (as of October 2025). This high ineligibility rate may be due, in part, to the system’s lack of basic eligibility checks for household income and children’s ages.
Furthermore, while MyCity provides users with information about DOE Early Head Start and Head Start programs, users cannot apply for these programs through the portal. Users are advised to contact programs directly and are provided with a link to NYC Public Schools MySchools website.
At the exit conference, OTI officials stated that the New York State Office of Children and Family Services provided guidance regarding the use of computer systems to make benefit or program eligibility determinations or to assist agencies in making such determinations. According to OTI officials, this guidance prohibited OTI from using MyCity to determine eligibility or to assist agencies in determining eligibility for childcare programs based on children’s age or household income. After the exit conference, OTI cited the Codes, Rules and Regulations of the State of New York, Title 18, Section 404.2, which states that the “social services district shall be solely responsible for the determination of eligibility for services.”
While MyCity may not be used to prevent applicants from submitting applications or make final eligibility determinations, implementing basic logic checks for children’s age and household income could alert applicants and agency personnel of potential ineligibility and speed processing and notification timeframes. These actions are not prohibited.
In its formal response, OTI stated that digital wallet was a concept in MyCity’s early planning stages, and that after evaluating the concept they determined there was no validated business or users’ case to justify expending public funds. However, OTI included the digital wallet in its budget plans for FY 2025 and continued the request in the FY 2026 budget. The audit team is concerned that the agency has requested funding for a feature it has no plans to implement, raising serious questions about the justification for this expenditure.
Poor Project Management and Oversight
According to IBC’s recommendation, purchasing an existing commercial off-the-shelf (COTS) product instead of developing one in-house would reduce implementation time and costs, lower maintenance needs, and could reduce the procurement time through a single contract. However, OTI did not provide any “build-versus-buy” cost estimates for the MyCity project or disclose its final decision on this matter.
Instead, OTI spent approximately $100 million through more than 120 agreements (including contracts, purchase orders, and delivery orders) to plan, design, develop, and implement MyCity. These agreements included critical roles such as project management oversight, developers, and quality assurance. Further, OTI has requested an additional $80.9 million, which the office reported will primarily be used to maintain existing functionality and implement additional system integration functionality.
OTI used Azure DevOps (DevOps) as its project management tool to oversee the entire project lifecycle, including project scopes, deliverables, and timeframe. Within DevOps, project scopes, deliverables, task assignments, tests, and releases are tracked. To ensure effective project management oversight, the audit team requested the initial project planning and design documents for the main portal, childcare, and business workstreams from the vendor responsible for the development. These documents include the intended functional requirements, project deliverables, and the proposed timeframe.
Despite multiple requests over six months, the agency did not provide adequate documentation to support project oversight of system deliverables. Specifically, OTI did not provide the migration plan, findings and recommendations, finalized requirements package, or detailed implementation roadmap created by the vendor for the Business workstream.
Furthermore, OTI was unable to provide adequate documentation to demonstrate that appropriate analysis and planning was performed by the vendor for the MyCity portal. Specifically, OTI did not provide documentation supporting deliverables related to gap analysis, finalized project workstreams, or implementation timelines by workstream. In the absence of such documentation, the audit team concludes OTI had inadequate project management controls.
Additionally, the audit team requested clarification regarding certain features for MyCity that were planned but never implemented, such as the ability to schedule inspections through the Business portal, vaccination appointments in the Childcare portal, and displaying the number of site visitors in the portal. Although OTI engaged with the vendor to define the final requirements, these requirements were not implemented. OTI stated that such features were suggested in presentations, original proposals, or requirements documents, and did not require agency implementation. OTI, along with its business stakeholders, determined which suggested features warranted implementation. In the absence of this information, the audit concludes that OTI did not establish specifications.
OTI officials stated that they did not incorporate the vaccination appointment feature as originally planned since the Department of Health and Mental Hygiene (DOHMH) already had a scheduling tool on its website. OTI stated that it could potentially integrate this feature into MyCity in the future, but this is not currently on the roadmap. Furthermore, OTI officials stated that they could not implement the inspection scheduling feature due to policy changes required by other City agencies. This justification is inconsistent with the stated goal of providing a one-stop shop to consolidate and integrating individual agency website functionalities.
At the exit conference, OTI officials stated that the agency, not the vendor, had conducted a build-versus-buy analysis for the MyCity project. OTI stated that relying on a COTS solution and a single vendor posed a risk. The audit team understands that the choice of solution is determined by the agency; however, this decision should be based on a documented analysis. As a follow-up, the audit team requested that OTI provide supporting documentation for this analysis. To date, this has not been provided.
MyCity’s Low Performance Impacted User Experience
As a project intended to serve the people of New York City, user experience is a critical factor in determining the success of the MyCity portal. It influences user satisfaction, engagement, usability, and efficiency.
To assess user experience, the audit team conducted non-intrusive scans of the MyCity environment, including the Childcare, Business, and Jobs portals, and identified several performance issues related to these sites, including speed index issues related to page loading, high blocking times affecting input and page response time, and slow JavaScript running time. Consequently, these issues negatively affect user experience while navigating the portal.
The scans also revealed poor performance scores for the ACS childcare voucher and DOE-contracted care sites. The new mobile site for the childcare voucher scored 7 out of 100, while the desktop version scored 9 out of 100. Similarly, the contracted care site scored just 4 out of 100 for mobile and 9 out of 100 for desktop.[4]
The findings from the scan reports were forwarded to OTI officials for review on July 28, 2025. On September 11, 2025, OTI officials provided an estimated timeline for completing their analysis by December 31, 2025, with remediation expected by June 30, 2026. While the team is pleased that OTI is taking actions, the extended timeframe means that these issues persist.
OTI’s Citywide Policy for Performance Testing of Public-Facing Applications states that performance testing must be conducted and meet specified exit criteria prior to deployment. This includes meeting minimum performance standards for average page response and upload times and the application functioning as expected throughout the duration of the test, among other things. In addition, the performance test results must be approved by OTI’s QA Director and Business Project Managers. OTI officials stated that performance tests were conducted prior to each release. The audit team requested the test results and approvals for the most recent releases.
At the exit conference, OTI officials again stated that they conduct performance tests. However, OTI did not provide documentation to show compliance with required standards, such as meeting minimum performance benchmarks or satisfying specified criteria prior to deployment.
After the exit conference, OTI provided some recent performance scans for Childcare, Business, Jobs, and Chatbot which included the results of the stress, endurance, and stability tests. These tests identified certain response times that did not meet the specified minimum performance benchmarks, which require an average page response time under three seconds and a 90th percentile page response time under five seconds. OTI’s scans identified slow page loading and response times related to logins, viewing recommendations, and uploading documents, among other things. For example, OTI’s tests showed it took more than 24 seconds to load the “ChatNow” feature on the landing page and more than 11 seconds to load the “Business Recommendations” login page; the 90th percentile transaction response times for the Chatbot 311 questions are between 12.4 and 16.2 seconds and therefore do not meet the standard.
MyCity Chatbot Provides Inaccurate Information and Inconsistent Responses
As part of MyCity’s Business workstream, OTI developed an AI Chatbot to answer users’ questions about services and benefits for business owners. The Chatbot was launched in September 2023; in March 2025, it was expanded to include content from the 311 service.
However, the MyCity Chatbot has provided inconsistent responses to inquiries related to City businesses. In early 2024, users reported that Chatbot provided incorrect and inconsistent answers to identical questions.
The MyCity portal has two mechanisms to obtain user feedback on the AI ChatBot: a feedback form that allows users to rate the ChatBot on a scale of one to five stars and provide comments, and “thumbs up” and “thumbs down” buttons that also allow users to submit comments. According to OTI data, more than 2,200 questions were asked in July and August 2025, and 70 respondents provided “thumbs up” or “thumbs down” user feedback. Of these, 50 out of 70 respondents (71.4%) submitted negative feedback, indicating their dissatisfaction with the Chatbot’s answers. Furthermore, 19 out of 70 respondents indicated that the response was not helpful.
OTI officials stated that they review the Chatbot reports to evaluate and analyze user responses, including number of user prompts and questions asked, the accuracy of the responses, and areas for improvement. The July and August 2025 reports provided by OTI identified several high- and medium-priority issues that needed to be remediated, including language problems and inconsistent or inaccurate responses.
The Chatbot should have responded to questions about certain appropriate topics—such as trash collection schedules, business licenses, and cash assistance. Instead, these questions elicited the response: “I’m sorry, but I don’t have information on that. My knowledge is currently limited to New York City government topics.” These inaccurate and inconsistent responses have led to dissatisfaction and may result in misleading information being provided to users.
This weekly report also included OTI’s analysis of 48 user questions to which the Chatbot should have responded. It did not answer 23 questions related to government services and programs, five of which were related to childcare services. Specifically, the Chatbot did not provide a response to questions regarding the current NYC Mayor, ACS Commissioner, and childcare application process.
The audit team conducted independent reviews to assess whether previous issues had been resolved. However, the Chatbot still provided inconsistent responses to identical questions. For instance, when asked about business trash disposal, the Chatbot initially responded that “I’m sorry, but I don’t have information on that. My knowledge is currently limited to New York City government topics.” When asked the same question again, the Chatbot provided a detailed answer.[5]
User syntax and word choice also affected the outcome of responses. For example, the Chatbot did not generate an informative response when asked, “any summer camp park program for kid 5 year old” [sic]. However, the Chatbot provided a detailed response when the team rephrased the question (“any summer camp park program for 5 year old kid”). Similarly, the use of “NYC” versus “nyc” also affected the outcome of Chatbot responses. Subsequently, users with learning disabilities or those who speak English as a second language may be disproportionately impacted.
This issue was discussed with OTI officials, who acknowledged the inconsistencies and noted that, as a “generative AI,” the information produced may not always be consistent. While this may be true, according to the National Institute of Standards and Technology’s (NIST) Artificial Intelligence Risk Management Framework, AI systems should demonstrate reliability and consistency across various circumstances.
OTI reported that they had remediated some issues and updated the Chatbot site with a disclaimer that it is in the beta testing phase and may give inaccurate or incomplete information. According to OTI officials, there are no plans to move the Chatbot out of beta testing.
Based on the information provided regarding content governance and updates, OTI continues to update the backend data, which may cause the Chatbot’s responses to not always be consistent.
In its formal written response, OTI asserted that it shared data with the Comptroller demonstrating that, week-to-week, the Chatbot consistently provides over 95% accuracy in its generated responses, has nearly zero hallucinations, and negative feedback hovers around 2.25% of all responses.[6] However, OTI calculated the Chatbot accuracy rate based on the “Count of User Prompts Recorded (Q&R)” as opposed to the “Number of User Prompts (Questions Asked).” This resulted in higher accuracy rates. Had OTI calculated accuracy rates based on the number of questions asked, rates would have ranged from 84.8% to 92.7% for August 2025. OTI’s accuracy rate also does not include responses it categorized as having “Room for Improvement.”
In addition, we question the reliability of OTI’s Chatbot Weekly Production Report. The report provided for August 2025 did not record any inconsistent responses but as stated above, the audit team’s independent testing found that the Chatbot could not consistently provide information when asked the same question.
Regarding negative feedback, OTI’s own Chatbot Weekly Production Report calculates negative feedback based on the number of respondents who provided “thumbs up” or “thumbs down” user feedback. In its response, OTI appears to now be asserting that negative feedback should be calculated based on the total number of user prompts, which inappropriately assumes that users who don’t respond are satisfied with the Chatbot. As previously stated, of the users who submitted feedback, 71% expressed dissatisfaction with Chatbot responses.
MyCity Portal Does Not Track City Agencies’ Performance
On OTI’s website, MyCity is promoted as “[o]ne-stop for all city services and benefits,” and Mayor Eric Adams is quoted as saying, “I made a commitment to New Yorkers that we would leverage technology to improve the delivery of government services and track agency performance in real time, and we are doing just that.”
However, when the audit team requested information related to this aspect of OTI’s functionality, OTI officials stated that MyCity does not track agency performance. Although it was initially expected that agency performance would be available within MyCity, this no longer seems to be the case.
At the exit conference, OTI officials stated that MyCity was never intended to track agency performance in real time, which contradicts the information posted on the MyCity website.
Recommendations
To address the abovementioned findings, the auditors propose that OTI should:
- Conduct a comprehensive review of the project’s technical, economic, legal, and operational feasibility to determine whether to continue, modify project scope, or terminate the project to minimize resource waste.
OTI Response: OTI stated that it disagreed with this recommendation and the underlying findings. Nevertheless, OTI stated that the agency “welcomes the opportunity for a level-setting MyCity program review that would examine alignment with new administration priorities and ensure the resources necessary to deliver the vision to continue to make city services and benefits easier to access for New Yorkers.”
- Develop a clear and detailed project plan for MyCity’s end state, with detailed functional requirements and time and cost benchmarks.
OTI Response: OTI disagreed with this recommendation and stated the audit team is judging the program using a “rigid project management model.”
Auditor Comment: OTI has spent approximately $100 million so far and has requested an additional $81 million in the 2026 budget. To ensure effective project management oversight and accountability for City resources going forward, OTI should identify and document the incoming administration’s priorities and develop a clear and detailed project plan for MyCity with system requirements, implementation timelines, and cost estimates, and hold vendors accountable for associated deliverables.
- Establish a dedicated project management team responsible for ensuring that short and long-term project goals are met and completed on time and within budget.
OTI Response: OTI disagreed with this recommendation but acknowledged that “it would be beneficial to have a larger in-house technical team to build institutional knowledge and rely less on consultants” and that “there have been staffing challenges since the beginning of this project.” OTI inaccurately asserted that the auditors never requested a count of individuals working in project management related roles.
Auditor Comment: OTI was asked for but did not provide a count of the program’s project management related roles. The audit team requested a detailed list of project managers and their defined roles and responsibilities. This was not provided; OTI instead submitted only generic role descriptions.
The audit team also requested the initial planning, system design, project roadmap, and implementation timeline, all of which fall under IBC responsibilities.
According to OTI, generic project management’s responsibilities include, but are not limited to, leading planning, execution and delivery of programs adhering to the project timelines and budget. Additional responsibilities include coordinating, collaborating, and communicating with cross-functional teams and stakeholders. However, managing long-term project goals was not included among their assigned responsibilities.
- Hold vendors accountable for expected deliverables, timelines, budgets, and standards.
OTI Response: OTI disagreed with this recommendation, asserting that it has held vendors accountable.
Auditor Comment: As detailed in the report, the audit team repeatedly requested but was not provided with evidence that IBC produced key contract deliverables. At no point did OTI provide the requested information or evidence to demonstrate that changes in deliverables were authorized.
The Rules for the City of New York, Section 4-02 states that changes to contracts must be authorized and reflected in change orders, and changes in contract amounts due to authorized omitted work require appropriate price and cost analysis to determine reasonableness. Based on the audit team’s independent review of IBC contracts for design and development services, OTI submitted change orders for two IBC contracts for design and development services to increase the contract amounts for added deliverables. OTI did not at any point request the elimination of existing deliverables.
- Develop and implement eligibility criteria to alert or prevent submission of ineligible childcare applications.
OTI Response: OTI disagreed with this recommendation, stating that the New York State Social Services law limits eligibility determinations to social service agencies. Additionally, OTI stated that eligibility determinations cannot be sufficiently automated because there are New York State approved eligibility exceptions to age limits based on a child’s special needs and to income limits based on homelessness or other factors.
Auditor Comment: As stated in the report, while MyCity may not be used to prevent individuals from submitting applications based on children’s age or household income limits, implementing basic logic checks is not prohibited, and moreover, could alert applicants and agency personnel of potential factors impacting eligibility and New York State approved exceptions. This may reduce processing and notification timeframes and provide applicants with useful information.
- Ensure that MyCity meets minimum standards as specified in the Citywide Policy for Performance Testing of Public-Facing Applications.
OTI Response: OTI disagreed with the recommendation, arguing that “MyCity already meets/exceeds minimum standards.” In addition, OTI stated that the transactions from its performance scans are classified as complex transactions and therefore, fall in the category of 30 second average response time and 50 seconds on 90th percentile.
OTI further alleged stated that the tool used by the audit team to test performance was flawed, unreliable, and does not have the parameters necessary to accurately test performance.
Auditor Comment: OTI’s own performance scan reports showed multiple response-time scores outside the parameter criteria of 3 seconds on average and 5 seconds 90th percentile for simple transactions. Specifically, clicking the Chatbot feature on the landing page without submitting questions and accessing the login page are simple transactions. They do not include document uploads or multi-step backend processes that should take over 24 seconds of response time. There is a significant gap between the 3 to 5 second parameter specified in OTI’s Citywide policy for performance testing and the 30 to 50 second parameter OTI asserted should be used.
In addition, Google Lighthouse is a widely used and reliable tool for auditing web performance, accessibility, identifying issues, suggesting fixes, and providing insight or measurement. OTI is responsible for ensuring that the performance standards it sets are adhered to across agencies; it should at the very least ensure that OTI systems fully comply.
- Conduct structured testing exercises (also known as “AI red-teaming”) to probe the Chatbot to find flaws and vulnerabilities (such as inaccurate or inconsistent responses), improve data quality, and inform decision-making regarding future enhancements and use.
OTI Response: OTI disagreed with the recommendation, asserting that they already conduct AI red-teaming and other testing measures. OTI again argued that the MyCity Chatbot has an accuracy rate of 95-99%.
Auditor Comment: As part of its formal response, OTI provided a one-page summary for the test performed for the Chatbot in 2023; however, it lacks sufficient detail for the audit team to assess whether inaccurate or inconsistent responses were tested or how frequently testing occurred.
As stated in the report, the user experience feedback for July and August 2025 indicated very high rates of dissatisfaction with Chatbot’s answers and that responses were not helpful. The numbers reported were based solely on the provided user feedback and were neither inflated nor projected. The team’s independent review of the Chatbot responses also revealed inaccurate and inconsistent answers.
Recommendations Follow-up
Follow-up will be conducted periodically to determine the implementation status of each recommendation contained in this report. Agency reported status updates are included in the Audit Recommendations Tracker available here: https://comptroller.nyc.gov/services/for-the-public/audit/audit-recommendations-tracker/
Scope and Methodology
We conducted this performance audit in accordance with Generally Accepted Government Auditing Standards (GAGAS). GAGAS requires that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions within the context of our audit objective(s). This audit was conducted in accordance with the audit responsibilities of the City Comptroller as set forth in Chapter 5, §93, of the New York City Charter.
The scope of this audit covered the beginning implementation of MyCity Childcare in January 2022 to November 2025.
To understand OTI management structure and operations, the auditors requested the agency-wide and bureau-wide organizational charts, and viewed OTI’s Mayor’s Management Report for Fiscal Years 2022, 2023, 2024, and 2025.
To understand the project scope and contractor’s responsibilities, auditors reviewed contracts and amendments, data sharing agreements, functional matrixes, project status reports, and workflow diagrams for system development.
In addition, the audit team conducted system walk-throughs with OTI project managers to gain an understanding of how users of the MyCity application completed the process of applying for or obtained services related to childcare, job placement, small business resources, or other New York City benefits and common services. The team also met with OTI officials to review the MyCity Chatbot functions, creation, and processes.
Furthermore, the auditors also performed systems functional tests in the production and non-production environment to determine whether the system performed as intended and met the overall goals as stated in the system specifications and public briefings. The auditors performed independent performance assessments of the MyCity website to determine whether the agency monitors and considers website performance. Additionally, the auditors tested the accuracy, reliability, and consistency of responses provided by MyCity Chatbot in response to legitimate government procedural inquiries, previously documented problematic responses from media reports, and inappropriate content handling protocols. Furthermore, the auditors tested the accuracy, reliability, and consistency of responses provided by MyCity Chatbot in non-English languages.
The auditors reviewed NIST, GAO, and New York State frameworks: NIST AI RMF 1.0: Artificial Intelligence Risk Management Framework, NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST.SP.800-37 Rev.2: Risk Management Framework for Information Systems and Organizations – A System Life Cycle Approach for Security and Privacy, NIST SP 800-53 rev. 5 Security and Privacy Controls for Information Systems, GAO-21-519SP: Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities, and NYS-P24-001: Acceptable Use of Artificial Intelligence Technologies.
To determine whether MyCity streamlines the delivery of services and benefits and allows the City to track agency performance in real time, the auditors used the following policies as criteria for OTI’s compliance with the City requirements:
- OTI’s policies including: Citywide Cybersecurity for the Usage and Development of Artificial Intelligence (AI) Systems Policy, Citywide Policy for Performance Testing of Public-Facing Applications, Citywide Policy on Cloud, and The New York City Artificial Intelligence Action Plan.
- OTI NYC Cyber Command’s policies and standards including: Citywide Application Security Policy, Citywide Application Security Standard, and Citywide Cybersecurity Program (“Citywide CSP”).
Although the results of the above tests were not projectable to their respective populations, these results, together with the results of the audit procedures and tests, provided a reasonable basis for the audit team to evaluate and support the findings and conclusions within the context of the audit objective.
Endnotes
[1] OTI, formerly known as the Department of Information Technology and Telecommunication (DoITT), in conjunction with NYC Cyber Command, established policies and standards regarding the security, implementation, and maintenance of the City’s information infrastructure.
[2] The waterfall model is a linear, sequential System Development Life Cycle (SDLC) methodology that includes phases for requirements for gathering, design, implementation, testing, deployment, and maintenance. Agile projects define requirements both before and during execution and provide frequent and iterative deliveries of incremental products.
[3] OTI added a link to the Department of Citywide Administrative Services’ (DCAS) CityJobs website, which provides information about City civil service exams, jobs, internships and fellowships, and job training programs (August 2023) and added New York State Department of Labor job board content (December 2024).
[4] Based on the Google Lighthouse numeric values: 0 to 49 indicates poor performance, 50 to 89 needs improvement, and 90 to 100 represents good performance.
[5] The audit team asked “For my restaurant, I can dispose of my trash 5 ft away from my property line?”
[6] OTI defines a hallucination as the Chatbot providing a response that is false or fabricated by the Chatbot. This includes providing answers based on content outside of MyCity Business or NYC311 and ungrounding which OTI defines as instances in which the Chatbot provides a response that is detached from factual or contextual reality, leading to misleading or incorrect information.