The Massachusetts Board of Elementary and Secondary Education
District Accountability Reviews - Process and Next Steps
Background
At the February 28 meeting of the Board of Elementary and Secondary Education, we will build on the reports the Board has received previously about District and School Accountability by focusing on the Level 3 and 4 district accountability review process. This month's report is designed to give Board members context about the district review process and ways the Department is using the review process to inform prevention work at Level 3 and intervention work at Level 4. No action is required at the February meeting.
At the March 22 Board meeting I will present the reviews completed this year of each of the four districts identified by the Board as "underperforming"/Level 4: Gill-Montague, Holyoke, Randolph and Southbridge. I will also present the semi-annual report by the monitor I appointed to oversee the Recovery Plan for the Fall River Public Schools, and a draft of the "road map" we are developing to guide Board and Department work when the Board identifies the Commonwealth's first Level 5 district(s) under the regulations adopted last spring. I will ask the Board at the April meeting to act on recommendations I will make for the future accountability level for each of these districts.
District Accountability Reviews
Mass. Gen. Laws Chapter 15, Section 55A, directs the Department to "review and report on the efforts of schools, charter schools and school districts, including regional school districts, to improve the academic achievement of their students." The law, which was enacted in 2008, transferred to the Department the district review responsibilities of the former Office of Educational Quality and Accountability (EQA) and Educational Management Audit Council (EMAC). Among other things, the statute directs the Department to "perform not less than 40 school district audits annually, not less than 75 per cent of which shall be in districts whose students achieve at low levels either in absolute terms or relative to districts that educate similar student populations." As we have discussed with the Board previously, the current appropriation for this purpose is not sufficient to accomplish what we need to do in this area, and we have identified it as a priority for funding in the FY2012 budget.
The Department's Center for School and District Accountability, under the direction of Eva Mitchell, leads the district review process. Consistent with our charge under the statute, we have conducted four types of accountability reviews:
- District Plans for School Improvement Reviews
The first reviews completed by the Center in 2009 were designed to ensure that the Department had current information about capacity in the Commonwealth's largest districts. These reviews focused on the ten largest urban districts' plans for intervening in their most struggling schools, and assessed their capacity to intervene effectively. - Promising Practices District Reviews
These are comprehensive reviews of district performance with a particular focus on how districts and schools are closing proficiency gaps for one of three subgroups: students with disabilities, English language learners, or students living in poverty. Districts are chosen when they have one or more schools in which MCAS scores over a two-year period reveal a narrowing of the proficiency gap for the identified subgroup. Promising Practices District Reviews have been conducted in Level 1, 2, 3 and 4 districts. - Level 3 District Accountability Reviews
These are comprehensive reviews of district performance that focus on Level 3 districts where student achievement trends suggest the districts are most at risk of requiring intervention as Level 4 districts. - Level 4 District Accountability Reviews
These are comprehensive reviews of district progress focusing on how the district has addressed the issues that placed it in Level 4 through the implementation of its turnaround plan; whether student achievement is on the rise; whether the district and schools have strong systems and practices in place; and whether the district has built the capacity to maintain continuous improvement on its own without continued DESE Targeted Assistance support and intervention.
All four types of reviews have been grounded in an examination of district performance on six standards, details of which can be found in Attachment #1:
- District Governance and Leadership
- Curriculum and Instruction
- Assessment
- Human Resources and Professional Development
- Student Support
- Finance and Asset Management
The District Review Process
District reviews are conducted by teams of independent educational experts who are selected and trained by the Center for District and School Accountability. The training is designed to deepen reviewers' understanding of the six District Standards, the methods of qualitative and quantitative analysis used during the course of reviews, the conduct of classroom observations using the Learning Walk criteria, and the guidelines for writing high-priority evidence-based findings and recommendations.
Each review begins with a two-day document and data review at ESE. The Office of Planning and Research to Close Proficiency Gaps (OPR) presents to each team a detailed data analysis for the district and schools under review. DESE data presented to the team (and also made available to districts) includes student demographic, behavioral, performance and proficiency gap trends, as well as staffing, financial, and educator data. The team reviews and discusses the data as well as documents provided by DESE and by the district, to begin learning about the district's systems and outcomes and to develop questions for district staff and stakeholders during the onsite visit.
The document review is followed by a four-day onsite visit to the district and its schools. The review team typically conducts at least 35 hours of interviews with a variety of stakeholders including: district and school leaders and staff, school committee members, teachers' union leadership, municipal officials, and parents. A typical visit will also consist of at least 50 classroom observations using the standardized observation protocol. The onsite visit concludes with the review team's presentation of emerging themes to district leaders, and if available, the Regional Assistance Director from the District and School Assistance Center (DSAC).
After the onsite visit, the review team drafts findings for each standard, which they carefully review with each other during a team meeting at DESE and use to develop recommendations. The draft recommendations are refined and reviewed during a final meeting at ESE. A technical writer ensures the draft reports meet standards for the quality of evidence, internal consistency, accuracy, and adherence to writing guidelines. Draft report findings are sent to the district for review and factual correction before the report and recommendations are finalized and made public. See Attachment #3 for an overview of the review process.
A specifically designed review protocol guides each type of review. See Attachment #4 for the key questions that guide each type of review. The review protocol is provided to districts in advance of the review to help them prepare and to understand the purposes and process for the review; the review protocol also provides clarity and transparency about the parameters of the review for the public as well as for the members of the review team. A review coordinator offers a pre-visit to each district to help the superintendent and staff further understand the review process, answer questions they may have, and address any concerns or needs that arise.
The Center has continued the rigor of the district review process formerly conducted under the auspices of the Educational Management Audit Council (EMAC). At the same time, the Center has added much-needed components, notably:
- More rigorous selection of and intensive training for reviewers;
- A more explicit focus on the impact of district systems on the district's schools;
- Sequenced and integrated findings that make systemic issues more apparent; and
- Recommendations that serve as useful starting points for district and Department planning.
To ensure that stakeholders in the community can gain perspective about the strengths and weaknesses of district performance, senior Department staff members have presented in person at open school committee meetings the findings and recommendations from the district reviews for Level 4 districts. Plans are being put in place to do the same for Level 3 district reports.
Findings from Level 3 District Reviews
The Center worked with an independent research analyst1 to examine the findings in each of eight Level 3 district reviews completed in 2010 to ascertain trends and help identify potential implications for policy and practice. The report, Level 3 Districts: key themes and challenges, identified three cross-cutting challenges that districts face:
- Building upon existing curriculum and assessment systems to implement an aligned and consistently well-taught curriculum.
- Developing district systems that support schools through an effective managerial and supervisory relationship.
- Maintaining capacity to meet students' academic and social-emotional needs in the face of reduced funding and rising expenses.
In addition, the analysis uncovered relevant findings about Level 3 district challenges in each of the six district standards. See Attachment #7 for the complete report.
Implications of the trends emerging in Level 3 Districts
Based on the cross-cutting trends and findings, the research analyst identified eight "questions to consider" as DESE establishes its priorities and strategies for accountability, intervention and assistance moving forward:
- How can the soon-to-be-developed DESE model educator evaluation system provide incentives for leaders and teachers to improve the consistency and quality of the taught curriculum?
- How can the regional DSACs support more districts to use the most effective ESE-developed tools and protocols to build instructional quality and improve the consistency of instruction, e.g., coaching models, learning walks?
- How can the DSACs help districts share their best practices for building instructional capacity in schools and classrooms?
- How can DESE help districts ensure that their systems and practices actively support schools in meeting student needs?
- How can DESE give districts feedback based on the question: "How, and for what purpose, is the district designing systems to meet such needs as communication, refining curriculum and assessments, identifying and sharing best practices and removing practices that are not working, teaming and professional learning, monitoring and supervision, or evaluating administrator and teacher performance?
- How might DESE make districts aware of systems of support that provide greater capacity to respond to funding challenges and how might DESE support districts to develop them?
- How can DESE and DSACs encourage and support districts to make more extensive use of regional approaches to secure and/or deliver services in order to a) identify and eliminate inefficiencies, b) achieve economies of scale and/or c) meet needs for which, alone, the district lacks sufficient capacity and/or technical expertise?
- Is DESE's district review process robust enough in the areas of student support and financial management to detect the impact of tightening resources on districts' capacities to respond systemically to non-academic barriers to learning?
These questions raised by the analyst have been helpful to Department staff engaged in accountability, intervention and assistance work. In some cases, the questions confirmed issues already identified and for which plans and/or activities were already underway to address them. In other cases, the questions are pushing staff members to consider altering or adopting new strategies.
Below is a sampling of work underway to address one or more of the questions posed.
Leadership and Governance
- The Center for Targeted Assistance is collaborating with the state superintendents and school committee associations to pilot district governance assistance teams to improve school committee practice and school-committee superintendent communication.
- The Department is incorporating the findings of this analysis of district governance issues into the curriculum for the three-year induction program for new superintendents launched in August 2010 by DESE and the Massachusetts Association of School Superintendents (M.A.S.S.).
Curriculum and Instruction
- Work is underway to adopt, disseminate, and support implementation of the new Massachusetts curriculum frameworks in English language arts/literacy and mathematics, incorporating the Common Core State Standards.
- The Department will ensure that the model educator evaluation system being developed pursuant to the Board's spring 2010 vote incorporates processes that will prompt school- and district-wide development of a shared vision of effective educator practice.
- The implementation plan for the new educator evaluation system is being designed to provide direction and support for districts to develop a shared vision of effective practice.
- The Department will disseminate and support the implementation of the model units and resources being developed with Race to the Top resources.
Assessment
- With Race to the Top funding, DSACs are strengthening their capacity to assist districts in more effective data analysis and use.
Human Resources and Professional Development
- The new framework and guidelines for educator evaluation are intended to address the widespread issue of ineffective supervision and evaluation practices.
Student Support
- The Center for School and District Accountability will revise the district review protocol to ensure sufficient focus and expertise in the area of Student Support with special focus on students with disabilities and English language learners.
Financial and Asset Management
- The Center for School and District Accountability will revise the district review protocol to more closely review districts' use of regional approaches to ensure economies of scale and address unmet needs in a cost-effective fashion.
- The Center will also complete a research-based Promising Practices document related to regional approaches to achieve economies of scale and address unmet needs.
The Accelerated Improvement Planning and Monitoring Process
The trends emerging in Level 3 districts along with the review of the 2009 District Plans for School Intervention and the Level 4 District Reviews being completed this month suggest that many districts lack capacity to generate, implement and/or monitor sound district improvement plans that hold promise for rapidly accelerating the improvement in teaching that is essential if student achievement is to increase dramatically. As a result, the Centers for District and School Accountability and Targeted Assistance have collaborated to design the Accelerated Improvement Plan process. Described in some detail in Attachment #8, the Plan is being implemented this winter and spring in all four Level 4 districts. The planning template will be made available next fall through the regional DSACs to Level 3 districts.
Conclusion
The Center for School and District Accountability has refined the district review process in ways that are making the findings more useful both to district leaders and Department staff. The superintendent of a Level 3 district recently described the district review for his district as "stunningly informative." Staff from several of the Department's centers, notably the Center for Targeted Assistance, the Center for Research and Evaluation, and the Center for Finance, are contributing to the analysis of district efforts. At the same time, they and others are beginning to make systematic use of findings from the district reviews to inform their on-going work with the districts. These are all much-needed and welcome developments for the Department as we intensify our accountability, assistance and intervention efforts.
Deputy Commissioner Karla Baehr and Eva Mitchell, Director of our Center for School and District Accountability, will be present to offer highlights of the work to date and respond to Board members' questions.
Enclosures:
1 Brett Lane, President of the Institute for Strategic Leadership and Learning (INSTLL), co-authored the Level 3 District Review Summary Report in Appendix #7.