Audit Report Evaluating the Response and Follow-up of the School Construction Authority to Its Customer Satisfaction Surveys
Executive Summary
The School Construction Authority (SCA) was established in 1988 by the New York State Legislature to manage the design, construction, and renovation of New York City public schools.
On March 27, 1997, the New York State Comptroller’s Office issued an audit, New York City School Construction Authority Improvements Needed in Construction Contracting Practices (Report 96-N-5). The report stated that the SCA did not consistently evaluate completed contractor work or measure customer satisfaction with completed contractor work.
The New York State Comptroller’s Office recommended that the SCA establish a system that provided for follow-up of completed projects and that it obtain feedback from its customers about the quality of SCA-supervised construction work. In response, the SCA created a Customer Satisfaction Survey questionnaire that is mailed every October to all principals of schools for which projects were substantially completed during the prior fiscal year.
The SCA assigns a color-coded rating that indicates the level of customer satisfaction to all survey responses received from the schools. A green code indicates a satisfactory job; a yellow code indicates contractor-related problems, such as lack of communication or poor planning; and a red code indicates problems that require follow-up action. In addition to the color codes, a “BOE” code is assigned to all survey responses with comments relating to work performed by the Board of Education and not the responsibility of the SCA. These survey responses are forwarded to the Board of Education.
The SCA’s Community Relations Division is responsible for mailing, receiving, and coding the responses to the Customer Satisfaction Survey. It forwards the coded survey responses to the Project Management Unit for follow-up. The Project Management Unit researches the problems described in the survey responses and sends a letter to school officials stating how the problem will be resolved. A copy of the letter and the survey response are returned to the Community Relations Division. Yellow-coded issues are also researched by the Project Management Unit, but written responses are not mandated. Green-coded responses require no follow-up action.
In October 2000, the SCA sent 1,063 Customer Satisfaction Surveys to school officials for all projects completed by the SCA during Fiscal Year 2000. Of the 1,063 surveys that were sent, 433 (41%) generated responses.
Our audit objectives were to determine:
- Whether the Customer Satisfaction Surveys were mailed to all principals of schools for which projects were completed during the fiscal year;
- Whether the SCA resolved the problems cited in the returned survey responses.
The scope of our audit was Fiscal Year 2000. To obtain an understanding of SCA operations, we interviewed the agency’s Executive Director, Senior Directors, Community Relations Division representatives, Project Officers, Project Managers, and the Special Counsel of Internal Audits. We also reviewed SCA policies and procedures for the Customer Satisfaction Survey, as well as the Customer Satisfaction Survey Tracking and Aging Reports for Fiscal Year 2000, and the New York State Comptroller’s Office audit on SCA construction contracting practices.
To verify whether the Customer Satisfaction Surveys were mailed to schools that the SCA recorded as not responding to the survey, we called a randomly selected sample of 114 school officials from the 630 schools that did not respond to the survey. As a result of the telephone interviews, we learned that 18 (16%) of the school officials who did not respond to the survey were also not satisfied with the work performed by the SCA.
We visited—with SCA officials—five of the 18 schools that did not respond to the survey and that were not satisfied with the work performed by the SCA to verify whether problems reported were related to the SCA contract work.
To determine whether the SCA resolved problems cited in the returned survey responses, we reviewed a randomly selected sample of 20 of 90 red-coded responses, including the attached Project Management Unit memos that outlined the resolution plan. We followed up with phone calls to school officials for these20 red-coded responses to find out whether the problem still existed and to determine whether school officials were satisfied with the follow-up work performed by the SCA. In addition, we visited—with SCA officials—10 of the 20 schools for which the survey responses were coded red and where the school officials were not satisfied with the SCA follow-up work, to discuss any remaining problems with the projects.
For a randomly selected sample of 130 survey responses (90 red-coded, 20 yellow-coded, 10 green-coded, and 10 BOE-coded) we verified whether: surveys were correctly coded; survey responses received by the Community Relations Division were forwarded to the Project Management Unit and subsequently returned to the Community Relations Division with a resolution plan; all applicable survey information was correctly entered in the Survey Tracking System; and red-coded responses were addressed within the required timeframe.
The Customer Satisfaction Survey is an important communication tool to help ensure that City public schools are safe and well maintained. We concluded that all surveys received by the SCA were correctly coded regarding the level of customer satisfaction. A resolution plan was developed for surveys for which follow-up action was needed. In addition, the Survey Tracking System was properly maintained. However:
- Surveys are often mailed out many months after the project has been substantially completed.
- SCA does not take follow-up action for Customer Satisfaction Surveys that receive no response.
- SCA does not ensure that schools are satisfied with the follow-up work done by the Project Management Unit, which is supposed to address problems reported by those schools on their Customer Satisfaction Survey responses.
- The Project Management Unit does not always submit follow-up action plans to the Community Relations Division within the required 30-day time period.
The Customer Satisfaction Surveys concerning Fiscal Year 2000 projects were mailed from 30 to 480 days after the projects were considered substantially complete.
There were only two surveys sent out within 100 days after the substantial
completion of a project; both received responses. When surveys were sent after more than 100 days had elapsed following the substantial completion of a project, the response rate dropped and remained fairly consistent, ranging from 38 percent to 44 percent.
Because of the small number (only two) of surveys sent out relatively early in the process, the above findings (linking survey timeliness to response rates) are inconclusive. However, our telephone interviews with the114 principals of schools in our sample who did not respond to the surveys disclosed that 22 (19%) were new to their school. The surveys for these 114 schools were sent from five to 15 months after the projects were substantially completed. The principals who were at these schools when the work was done were no longer there, and the new principals were not able to offer comments on the projects for the surveys in question.
In addition, when surveys are mailed so long after a project is substantially completed, it can become unclear whether problems cited in the surveys are attributable to poor contractor work, to normal building wear and tear, or to a lack of maintenance. Moreover, the SCA cannot hold contractors responsible for problems if it is not notified of the problems promptly and before the expiration of the contractor warranty.
By sending the surveys immediately aftera project’s substantial completion date, the SCA may encourage a higher survey response rate, enabling it to identify the causes of the problems and to resolve the problems more effectively, based on survey information that is timely and relevant to the completed project.
For projects completed during Fiscal Year 2000, the SCA mailed 1,063 Customer Satisfaction Surveys to school principals, of which 630 (59%) received no response.
SCA officials stated that if they do not receive a survey response from a school official, they consider the official to have been satisfied with the completed project. However, the results of our telephone calls to school principals indicate that this is not the case: 18 (17%) of the 109 principals we were able to contact were not satisfied with the work done and felt that further work was required.
SCA officials also stated that it is the schools’ responsibility to respond to the survey. However, more than half of the 109 principals we spoke with who had not responded to the survey said they had not received, or did not recall receiving, the surveys. In addition, 18 of the 109 principals we spoke with stated that they did in fact answer and return the survey. It seems obvious that in order to maximize the survey instrument’s usefulness, SCA should contact school officials who do not respond to the initial survey to ensure that the work at those schools was satisfactorily completed.
Of our sample of 20 schools whose responseswere coded red (problems requiring follow-up action), 10 (50%) were not satisfied with the follow-up work performed by the SCA.
We visited, accompanied by SCA officials, the 10 schools whose principals were not satisfied with the follow-up work performed by the SCA. As a result of our school visits, the SCA agreed to take corrective action at four schools. The SCA also agreed to look into the problems at two other schools to determine what work needs to be done. The problems at two other schools were considered to be the BOE’s responsibility, and the problems at the remaining two schools were resolved prior to our visit.
The SCA needs to do more to ensure that problems related to contracted work at the schools are addressed satisfactorily and in a timely manner, and then ensure that the corrective actions submitted by the Project Management Unit are in fact completed. When the reported problems are not covered under the project contract or are not the responsibility of the SCA, the agency should nonetheless follow up with schools officials, notifying them that the work cannot be handled by the SCA.
The Project Management Unit did not submit resolution plans to the Community Relations Division within the required 30-day time period for 28 (31%) of the 90 schools that required them. These late plans were submitted from one to 175 days after the end of the 30-day time frame specified in the SCA rules.
To ensure that construction problems at schools are promptly resolved, the Project Management Unit should respond in a timely manner to the concerns raised by school officials on their Customer Satisfaction Surveys.
This audit makes four recommendations to SCA officials, the most significant of which are:
- The SCA’s Community Relations Division should mail Customer Satisfaction Surveys immediately or soon aftereach project’s substantial completion.
- The SCA’s Community Relations Division should follow up with a phone call or letter to schools that do not respond to the surveys
- The SCA’s Project Management Unit should resolve problems cited in red-coded survey responses in a more timely manner, and then ensure that the corrective actions submitted by the Project Management Unit are in fact completed. When the reported problems are not covered under the project contract or are not the responsibility of the SCA, the agency should nonetheless follow up with schools officials, notifying them that the work cannot be handled by the SCA.
The matters covered in this report were discussed with SCA officials during and at the conclusion of this audit. A preliminary draft report was sent to SCA officials and discussed at an exit conference held on April 19, 2002. On April 25, 2002, we submitted a draft report to SCA officials with a request for comments. We received a written response from the SCA on May 16, 2002. SCA officials generally agreed with the audit’s findings and recommendations, stating:
“We hereby . . . thank the Comptroller’s office for the efforts expanded during this audit. The Authority always welcomes constructive recommendations that will assist it in carrying out its legislative mandate. . . .”
“While we are prepared to adopt three of the recommendations made, . . . we do not agree with the recommendation to issue Customer Satisfaction surveys on a ‘rolling,’ rather than the current (annual) basis.”