The Massachusetts Board of Elementary and Secondary Education
Update on Next-Generation MCAS
In spring 2017, the first next-generation MCAS ELA and mathematics tests were administered to students in grades 3-8. About 250,000, or 60 percent, of students in grades 3-8 took the computer-based version of the test. For grades 4 and 8, where schools were required (with few exceptions) to administer the computer-based tests, more than 93 percent of students took the test on a computer.
Computer-based testing went very smoothly. Although we received reports of isolated issues or problems, the testing system worked well overall, and virtually all students who were assigned to take a computer-based test did so successfully.
Continued Transition to Computer-Based Testing
The transition to full computer-based testing will continue in spring 2018, with the expectation that all schools will administer the following tests using the computer-based format:
- Grade 4 ELA and Mathematics
- Grade 5 ELA, Mathematics, and legacy Science and Technology/Engineering (STE)
- Grade 7 ELA and Mathematics
- Grade 8 ELA, Mathematics, and legacy STE
For grades 3 and 6, schools may elect to administer either the computer-based or paper-based tests in spring 2018.
Paper-based tests will always be available as an accommodation if required by a student's individualized education program (IEP) and as needed for the small number of students who do not have IEPs but who are unable to take a computer-based test for other reasons. Similar to the 2016-2017 school year, this fall districts will have an opportunity to apply for a waiver from computer-based testing for one or more of their schools, provided the school is making a good-faith effort to comply and has a plan for transitioning to full computer-based testing in spring 2019.
Survey of Schools that Administered Computer-Based Tests
Following the spring 2017 test administration, the Department of Elementary and Secondary Education surveyed schools that administered computer-based tests, gathering information that will be useful as we work to improve the computer-based testing experience in Massachusetts. A few highlights from the survey are provided below:
- We received responses from 411 people, representing a range of roles at the school and district levels (primarily principals, assistant principals, district and school test coordinators, teachers, and test administrators).
- About half of the respondents had administered PARCC or another computer-based test prior to spring 2017.
- About half of the respondents reported that the recommended testing times were not sufficient, especially in ELA (although the test was untimed). We are carefully examining the issue of testing time, based on this feedback and other data, and will revise the recommended testing times for spring 2018, as needed.
- Sixty-five percent of respondents reported that the biggest challenge they face in moving to computer-based testing is the need for additional devices. Other factors identified by a large percentage of respondents included technology support, more days in the testing window, additional space, and staff training.
- More than 92 percent of respondents reported receiving a "timely and useful response" when they contacted either the MCAS Service Center (hosted by our contractor) or the Department's Office of Student Assessment.
Responses to the open-ended questions in the survey are still being analyzed. Overall we gathered valuable information in several key areas, including the usefulness of the various support materials and opportunities we provide (e.g., training sessions, tutorials, training modules, practice tests, manuals), as well as problems that schools experienced during tests (e.g., interruptions in connectivity, testing system interruptions) and how they resolved them.
Many respondents reported that after they received the training and gained some experience using the online systems, they had fewer issues and a much better sense of the expectations for staff and students.
August 2017 Standard Setting
From August 14-18, 2017, the Department conducted the standard setting for the next-generation 3-8 tests. As I reported to you in May, the Department convened experienced educators from across the Commonwealth to participate on panels to review test items and student responses, make judgments about student performance through a facilitated process, and ultimately make recommendations regarding the appropriate cut scores corresponding to each level of achievement (Exceeding Expectations, Meeting Expectations, Partially Meeting Expectations, Not Meeting Expectations).
The standard-setting process this summer provided an opportunity to identify and review the cut scores for all six tests at one time. During the last day of standard setting, a subset of the educators from each panel met to conduct a "vertical articulation" to ensure consistency across grades. The panels' recommendations were reviewed by staff in our Student Assessment Services office and at Measured Progress, with additional adjustments as needed to ensure proper scaling. I then gave my approval to the final recommendations.
We will go through a more extensive standard-setting process in 2019 following the first administration of the next-generation high school tests. The high school tests form the basis for the competency determination, which is a requirement for a Massachusetts high school diploma. The standards for the competency determination are set by the Board in its regulations (603 CMR 30), which we will also be discussing at the September meeting.
The standard-setting process for grades 3-8 resulted in a coherent system that relates student achievement from grade to grade. Staff will describe this in more detail at the September 26 Board meeting.
Equating of Computer- and Paper-Based Test Forms
During the transition to full computer-based testing, it is important to ensure that students' scaled scores represent equivalent achievement regardless of the test form (computer or paper). We plan to use test equating procedures to achieve that equivalence in the grades where schools had the choice to administer either paper-based or computer-based test forms in ELA and Mathematics (3, 5, 6 and 7).
The equating procedures are the same ones that we use to equate the difficulty of the tests from year to year. Student results on the parts of the test that are the same are used to estimate the difficulty of the parts of the test that differ from form to form. Only items that are written identically and that perform similarly with representative samples of students will be used as the equating items. Items that were worded exactly the same but performed substantially differently on paper will be calibrated as if they were different items.
The result will be that representative samples of students will have similar results whether their schools chose to administer paper or computer forms.
Schedule for Releasing Results from the 2017 Tests
We plan to release all official MCAS results publicly the week of October 17. This will include school, district, and state MCAS scores as well as grade 10 accountability results. Printed Parent/Guardian Reports will arrive in district offices on October 24. The Board will receive a full briefing on the results at the October Board meeting.
Parent/Guardian Reports and Other Resources for Parents
To help parents and students better understand the new achievement levels and scales, the Department has redesigned the report that is delivered by the contractor to districts and sent home to students' parents or guardians. A final version of that report is attached for your information. The redesigned report presents the student's test results (scores and achievement levels) and informs the parent/guardian about the areas in which the student might need additional assistance (results by reporting category and by individual test item).
The Department also produces translated shells of the reports in the 10 languages other than English that are most commonly spoken by students in the Commonwealth. In their shipment of parent reports, districts receive translated shells for students who need a translated version (as indicated in SIMS), and that shell is shipped to parents along with the English version containing the student's results. The translated report shells are also posted on our website.
The Department is preparing additional resources for parents, including an interactive guide about testing, one-pagers on how to interpret the next-generation MCAS results, and additional explanatory materials to go along with the reports. These will be posted on our website this fall.
Assessment Budget
The Department continues to pursue supplemental funding to fully fund the assessment program for fiscal year 2018. In the meantime, assessment staff are continuing the work to report results from 2017 and begin constructing the tests for spring 2018.
Rhode Island's Administration of MCAS Tests
As you learned last spring, Rhode Island plans to administer the MCAS ELA and mathematics tests to approximately 70,000 students in grades 3 through 8 in spring 2018. Students in Rhode Island will take the test during the same administration window as students in Massachusetts, beginning April 2.
Associate Commissioner Michol Stapel and Chief MCAS Analyst Bob Lee will join us for the discussion at the September 26 Board meeting.
Attachment:
