Audit Report on the Performance of the New York City Department of Education’s Achievement Reporting and Innovation System
AUDIT REPORT IN BRIEF
The New York City Department of Education (Department) provides primary and secondary education for more than one million students from pre-kindergarten through grade 12. The Department’s “Achievement Reporting and Innovation System” (ARIS) was developed under the Department’s “Children First Intensive” professional development program. In 2007, the City awarded an $81 million contract to the International Business Machines, Corp. (IBM) to develop and implement the ARIS system. ARIS would allow data analysis and collaboration tools to permit knowledge sharing across City schools, track student and school performance, and enable data integration and data quality assurance. Additionally, ARIS was intended to enable New York City educators to improve student performance by viewing student data, exploring instructional resources, sharing effective practices, and collaborating with colleagues within schools and City-wide. The system was placed in service in October 2008. Our audit focused on the utilization of ARIS by educators (principals and teachers) rather than parents.
Audit Findings and Conclusions
Despite spending more than $80 million on system design and development, the Department lacks effective measurements for gauging whether ARIS is an effective tool for enhancing and improving student performance. In addition, educators are not using ARIS to the extent for which it was intended. According to our survey of teachers and principals, many educators are not using the ARIS system to collaborate with other teachers as was intended, are using alternative computer systems to obtain information in place of, or in conjunction with, ARIS, and are not utilizing the system to its fullest extent. Therefore, we believe that the Department is not completely attaining all the benefits for which the ARIS system was intended.
Additionally, ARIS Usage Reports may not be reliable indicators because of discrepancies in the data. Furthermore, we determined that less than 50 percent of educators accessed ARIS from April 1, 2011, to June 30, 2011. The Department, however, has adequate internal controls to preclude unauthorized access to ARIS.
Audit Recommendations
This report makes a total of nine recommendations, including that the Department:
- Ensure that information in ARIS is always up-to-date.
- Provide additional training to users of the ARIS system.
- Ascertain why ARIS Connect is not being efficiently utilized by educators.
- Formulate measurements to assess whether ARIS is attaining its goal to improve and enhance student performance.
- Monitor the frequency and usage of ARIS by system users.
Department Response
In its response, the Department contended that “First, the Comptroller misunderstands ARIS’s goals and the Department’s measurements for gauging whether ARIS has attained those goals. Second, the Comptroller misunderstands the way that ARIS data is compiled and ARIS’s role in providing student information to educators and parents.”
We assert that there is no misunderstanding about the goals of the ARIS system. As described in the ARIS vision statement and stated in a 2010 study contracted by the Department (i.e., the American Institutes for Research, “Facilitating Collaborative Inquiry Using Data and Technology in New York City Schools”), the goals of ARIS were to “Understand students’ strengths and areas for improvement; Develop, discover, and organize instructional resources and gain access to professional development opportunities; Document, monitor, and discuss teacher practices and student progress; Share effective practices.”1 Nevertheless, the Department provided measurements for only one of the goals (i.e., understanding student’s strengths and areas for improvement) eight months after the information was requested. 2
The Department states that “To measure ARIS’s impact on increasing student performance, the Department looks at three questions: Does ARIS help educators and parents understand students’ strengths and weaknesses? . . . Are educators and parents using ARIS? . . . Is student performance increasing?” However, we question if the Department is asking the right questions as the key metrics provided are unrelated to ARIS’s stated goals, and the key data points provided do not provide any direct evidence regarding whether ARIS has had an impact on increasing student performance. In fact, a “key research report” relied on by the Department quotes a Department official as saying, “ . . . although teachers are getting steadily better at analyzing data, data analysis ‘is not yet leading to fundamental change in teacher practice or decision making.’”3 Department officials further acknowledged in a July 7, 2011, e-mail that “you can never directly tie changes in student achievement to any one program or initiative.”
Moreover, contrary to the Department’s contention that the data shows that both parents and educators are embracing ARIS, we believe the data shows signs that might indicate potential problems ahead. Specifically:
The Department states in its response that “Over 70% of principals who responded to the Department’s November 2010 Principal Satisfaction Survey . . . indicated that ARIS is very helpful or helpful for improving student outcomes.” (See Table 1.) What the Department neglects to mention is that the Spring 2011 survey shows that only 62 percent of principals who responded found ARIS to be very helpful or helpful for improving student outcomes. This is a decrease of 19 percent from a peak of 81 percent in the spring of 2009.4
1. American Institutes for Research, “Facilitating Collaborative Inquiry Using Data and Technology in New York City Schools: Year 2 Final Report on the Implementation of the Achievement Reporting and Innovation System (ARIS),” page 1.
2. Even for this one goal, the metric (i.e., the percentage of principals who responded that they found ARIS “helpful” or “very helpful” in the Principal Satisfaction Survey (survey)) does not appear to be truly applicable. It is unclear how measuring the percentage of principals who find ARIS “helpful” or “very helpful” for improving student outcomes pertains directly to whether or not teachers and parents understand a student’s strengths and weaknesses. In the survey, the Department asks the same questions of principals regarding progress reports and 65 percent responded “helpful” or “very helpful.” However, progress reports have no individual student level data. As this question also had a 65 percent positive rate, is the Department asserting that a progress report helps teachers and parents understand a student’s strengths and weaknesses? Further, the views of parents or teachers are not measured.
3. Bill Tucker, “Putting Data into Practice,” Education Sector Reports, page 13.
4. New York City Department of Education, “ARIS: Key Metrics,” dated December 12, 2011.