Letter Report on the Reliability and Accuracy of the Community Health Survey Administered by the New York City Department of Health and Mental Hygiene
EXECUTIVE SUMMARY
The Department of Health and Mental Hygiene (DoHMH) protects and promotes the health and mental well-being of all New Yorkers. On July 1, 2008, DoHMH entered a six year contract with Abt-SRBI to conduct the Community Health Survey (CHS) to provide information on health risk factors in the City. The cost of the initial contract was $8.3 million. Later DoHMH extended the contract to December 31, 2014, with an increase in total value to approximately $10.7 million. DoHMH uses the CHS data to assess the population’s health status, track prevalence of diseases, and monitor health trends.
Abt-SRBI uses a computer-assisted telephone interviewing (CATI) system to collect survey data. Surveys are recorded and monitored by Abt-SRBI. CHS is a multiple-choice survey that is conducted in English, Spanish, Russian, and Chinese. Survey questions are translated and reviewed for language accuracy by native speaker employees. Native speaker interviewers use the translated materials to conduct the surveys; the interviewees’ answers are input by the interviewer into CATI. Abt-SRBI supervisors monitor approximately 15 percent of the live phone interviews and oversee the data entries to ensure that the answers have been input correctly into CATI. Feedback to interviewers is provided on a continual basis to improve performance.
In 2008, Abt-SRBI conducted the CHS with 7,560 landline respondents and obtained a response rate of 33.3 percent. In 2009, DoHMH began including cell phone respondents in the survey in response to the increasing number of people using only cell phones. In 2013, the CHS survey identified 3,920 cell phone respondents and 4,580 landline respondents. The number of questions for the CHS varies each year. The questions cover a range of topics that include general health and mental health status; diabetes; asthma; smoking; sexual behavior; alcohol consumption; cancer screening; and other health topics.
In order to group and better represent survey results, DoHMH assigned neighborhoods for the CHS based on the zip codes and boroughs provided by respondents. DoHMH breaks down data by these neighborhoods to provide CHS statistical information and graphics on its website. On the DoHMH website, for instance, CHS data appears in bar charts and neighborhood maps.
Results
We found that contract deliverables for CHS data were achieved. Our review verified that the vendor Abt-SRBI conducted the required surveys and performed the necessary purification on the CHS data. In addition, we reviewed the system security plans and found that the CHS data exists in a secure environment. We also tested the password management controls and reviewed the information system contingency plan. DoHMH has adequate access controls and contingency planning policy over its CHS data.
We tested the 2010, 2011 and 2012 CHS data files as well for blank and invalid data to ascertain whether critical data elements were missing. We also reviewed the data cleaning and analyticreports to ensure DoHMH performs data cleaning and checks for duplicates. Based on our tests, the CHS data is generally accurate and sufficiently reliable for DoHMH to support its health program decisions. However, we found that DoHMH did not always use the proper neighborhood codes in the CHS. We found that when respondents’ neighborhoods are missing, DoHMH assigns neighborhood codes. Although the small percent of missing neighborhoods did not significantly affect the data analysis, greater accuracy in the CHS data will help better track public health throughout the City.
We recommend that DoHMH reviews the assignment of neighborhood codes to determine whether CHS data with missing UHFs can be assigned more accurately to increase reliability of CHS analysis.
DoHMH officials disagreed with our recommendation and stated that “DoHMH staff already regularly reviews the assignment of neighborhood codes and continue to work closely with the vendor to identify and develop new methods for assigning neighborhood, particularly for cell phone users who do not provide a valid New York City Zip.”
During our audit, we found that five percent of the cell phone survey had missing neighborhood codes. In those instances, either the respondent’s zip code was not a valid residential zip code, the respondent’s zip code was outside of New York City, or the respondents refused to provide their zip codes. Although DoHMH regularly reviews and develops methods for assigning neighborhood codes, DoHMH still could not ensure that the neighborhood codes assigned to those five percent are accurate. Greater accuracy in the CHS data will help better track public health throughout the City.