Questions about Impact and Attribution: Are Better Results Really Due To The New Curriculum?
It is difficult to construct and administer a single test or measure that can conclusively demonstrate superior student achievement using IMP and show that it was due to IMP and IMP alone. We have heard a variety of concerns regarding the impact of IMP and issues involving attribution, that is, related to cause and effect. For example, “maybe you got better results because…”
1. You “stacked the deck.” The IMP students were better students to start. They were brighter, or previously had more math courses in middle school, or were somehow different, which biased the results in favor of IMP.
2. You had the school’s better teachers teach the new curriculum.
3. IMP teachers graded their students more leniently. Students could get wrong answers and still pass the course because you used “alternate assessments,” which don’t test for right answers.
4. The gains in student achievement were due to increased professional development of the IMP teachers. If you give teachers who teach pre-standards curriculum the same amount of professional development you gave IMP teachers, how do you know you would not get similar results?
5. The effect is due to the use of collaborative learning groups, not the curriculum per se.
6. It could be the “Hawthorne effect.” You got better results because it was new and students and teachers were getting increased attention.
7. The IMP teachers have reductions in course load.
8. You selected only those test questions that favored the new curriculum.
9. You got the results because you tested only low achieving students. The higher achieving students don’t need IMP.
10. You got the results because you only tested IMP with the “better” students [delete ‘kids’]. Lower ability students would not be able to do this kind of math.
11. Colleges may not accept these new programs such as IMP, rather than “algebra.” And if they do, students will not be adequately prepared for a traditional college math course. Most colleges do not let students use graphing calculators so they will be at disadvantage.
12. Students who take these new curricula in high school are regularly placed in remedial math class in college because their basic skills are so low.
All of the above statements have plausible validity even if they have no basis in fact. They need to be taken seriously and addressed. In addition, different constituencies will value different indicators of success more than others. In short, the type of data collected has to be “psychologically real” for each constituent group to be accepted by that group. For example, a principal of a special admission school said he would only consider IMP successful if it could raise SAT scores by 30 points. Another principal expressed concern about attendance rates and preventing school dropouts, while another was concerned about scores on the Pennsylvania System of Student Assessment (PSSA).
In response to these methodological issues and the different values placed on different measures of success, our strategy has been to collect student achievement data from a variety of sources using a variety of indicators. At the same time, we have attempted to control or compensate for as many biasing factors as possible. Our goal was and is to determine whether there is a convergence of the data in any one direction that could be used to answer the bottom line question of the costs and benefits of using a standards-based curriculum, such as IMP .
We began collecting data during the first year of IMP’s implementation in Philadelphia in May 1994. Since then, we have used a variety of measures, which are listed in the Appendix under Student Achievement Measures There are several reasons to believe the superior results of NCTM standards-based curriculum and student centered instructional methods as shown with IMP in Philadelphia are actually understated.
1. All of the results documented herein were from pre-publication draft versions of IMP.
2. All of the IMP teachers were teaching the curriculum for the first time.
3. None of Philadelphia’s most able students were permitted to enroll in IMP at each school.
4. None of the 9th grade students had previous been exposed to standards-based k-8 materials.
5. Most of the results were based on students using IMP for only 2 and 2/3 years.
6. IMP teachers did not operate in a “reform teaching culture,” but were often isolated in their departments. The same was often true for IMP students.
7. Parents were unfamiliar with a standards-based curriculum and some did not know how to help their children.
Based on our experience, we hypothesize that significantly greater student achievement results would accrue if high school students who are taught using the IMP materials:
a) were first taught using a k-8 standards-based mathematics curriculum,
b) use the finished published version of the IMP text,
c) are taught by teachers who had several years of experience teaching the same level of IMP,
d) are enrolled in a school that had adopted a standards-based mathematics program for all its students.