Posted by Harry

http://crookedtimber.org/2010/02/09/how-can-schools-use-research/

http://crookedtimber.org/?p=14615

Madison School Board member Lucy Mathiak, with a lament that, presumably, all thinking school board officials in the US share:

For years, MMSD staff have advocated for their proposals and programming choices by arguing that they are research-based data driven best practices. At times, I have wondered whether the research selected has undergone critical review. That is, do the people selecting the research stop to ask whether the research is methodologically sound with verifiable results, much less whether it was conducted on populations or under conditions that are comparable to the Madison public school district.

I’ve also wondered at an understanding of research that ignores entire bodies of data or work that falls outside of the narrow educational research paradigm. (Prime examples of the latter case include the district’s unwillingness to consider the considerable body of research on how children learn to read that is carried out by cognitive psychologists, linguists, and communicative disorder researchers. But that’s another post.)

What follows is my longwinded response, which builds up to a plea for Districts (or groups of districts) and States to establish local versions of the Consortium on Chicago School Research.

Mathiak’s particular concern is that the only source concerning underrepresented minorities mentioned by name in a report on TAG developments is by Ruby Payne, who is not a researcher, and self-publishes. Whatever the merits of this particular instance of the worry, it is a shared worry for a reason. Educational research (broadly construed as it should be) is voluminous, to say the least, and even much of the best of it is not designed, or written, to be readily accessible to non-academics. Educational leaders, whether at the school or district level, are not trained in the consumption of educational research: in fact, they are not even presented with a great deal of it during their training, even for the purpose of learning what it says. Preparing them would be quite difficult, for a couple of reasons. First, education is beset by a culture of deference to ideological commitments, which makes it quite difficult to have some kinds of discussion in a way that is really sensitive to the evidence. Consider inclusion – the policy of including children with special educational needs in the regular classroom – which is, in some quarters, a matter of faith of such strength that evidence is really irrelevant. It is similarly difficult in some districts and schools to have an evidence-sensitive discussion of racial achievement gaps. When you do have the discussion, furthermore, it is not necessarily the discussion you think you are having! (The most unnerving conversation I had with a superintendent was one in which the superintendent told me that his district uses Ronald Ferguson’s work to design their policies around the racial achievement gap, which I would think was a pretty good idea had he not just told me, as truth, a whole bunch of claims that I had, the previous day, read a Ronald Ferguson essay disproving). Training leaders to conduct such discussions in these circumstances, in which some of them have, themselves, made the particular commitments of faith, is no easy task.

Second, as any reader of CT will know, peer review is no guarantee of quality. The best that peer review does is exclude the very worst and nuttiest research. Knowing the status ranking of peer review outlets helps a bit – my guess is that the average quality of research in very highly regarded journals is somewhat better than the average quality of research in moderately well regarded journals, but I doubt the difference is huge, and if what you are looking for is individual pieces of research you can easily go wrong.

Even without these problems, interpreting research is extremely difficult, and training practitioners to do it presents substantial challenges. What leaders want to know is what steps they should take in their circumstances to achieve the goals that they have set for themselves (or have been set for them). But the research never tells them the answer to that question. Studies using large datasets aggregate over many different schools and districts: knowing that some intervention has, on average, small positive or small negative effects, does not tell you whether it will have positive or negative effects in your circumstances. Studies that are more localized tell us something about what an intervention or change causes in those circumstances, which may or may not resemble one’s own. I spend a fair amount of time in the company of high quality educational researchers, many of whom have a lot of hand-on experience in schools, and they will tell you that they do not know exactly how to interpret their own research let alone that of others for practical purposes.

Let’s take a very specific example. Suppose your school has experienced demographic changes from 5% FRL population to a 35% FRL population, and is now trying to come to terms with the fact that your curriculum and habits of instruction do not serve the new population well. Suppose, further, that the feeder middle school which serves half of the FRL students is a disaster, so that they come in even less well-prepared than they might. Should you move toward a curriculum which quasi-tracks students, providing classes specifically designed for less well prepared students. Or should you direct your teachers to “differentiate within the classroom”: that is, expect students of all different levels of preparedness in the classroom, but to vary instruction depending on the level of the preparation each student has?

Well, the literature consists of a large body of studies, most of which are not experimental in nature. In other words, they compare tracking schools with non-tracking schools, rather than looking at the effects of the kind of change you are considering. Many studies do not look at the presence, or otherwise, of special education students. Reading the literature, it is impossible to conclude that tracking is generally better, or generally worse, for low income students than non-tracking. Even if it were possible, that would not decide what you should do. First, the effects on non-low-income students matter, both intrinsically, and because a healthy school and school district needs to retain the support of their parents. Second, even if differentiation generally benefitted low income students that would not mean t hat you should do it. Maybe tracking generally harms low income students because the lower tracks are assigned lower quality and less experienced teachers, but you have the dynamism and ability as a leader to assign stronger and more experienced teachers to those tracks. Maybe your school has an unusual demographic profile that makes differentiation harder than it generally is to do well. Maybe you lack the leadership skills or the professional development resources to get your teachers to differentiate effectively. These are judgments that even much more unambiguous set of findings than we in fact have would still call forth from the leaders.

None of this is to say that we should not be training our educational leaders better to interpret research – we absolutely should, and I applaud Mathiak for raising the issue in the sharp way that she has done. But it’s a tall order, and meeting it takes considerable institutional resources. I know of two models. The first, which people who had experience of it (I’m much too young) praise was the research department of the now-deceased Inner London Education Authority. I don’t know how it was established, but its mission was to do school based research and communicate with the schools about the results. I can never tell whether its champions are victims of some sort of golden age-ism, but some of the best UK researchers in education worked there at some point, and it was associated with the landmark 15,000 Hours study, which gave some account of what was actually happening while children attended school.

The second model is contemporary, and at last I get to the point of the post. The Consortium on Chicago School Research has been working closely with CPS for 20 years, doing research in and about the schools with a specific focus on improvement, and with a mandate to communicate very closely with the district and the schools about its findings and their policy relevance. This remarkable document explains the way the Consortium works, independent of, but in a close institutional relationship with, CPS (and they’ve just published Organizing Schools for Improvement: Lessons from Chicago which everyone interested school reform will have to read).

Two things to note about the Consortium. First, it requires researchers who are high quality researchers but who are also willing to develop and exercise the very different skill-set of communicating that research in ways that enable policymakers and practitioners to make use of it. The incentives in academia are not to do the latter; it is more prestigious to be presenting at big academic conferences than to small groups of teachers and principals, and career success is easier if you make the geographical moves that make it especially difficult to develop the local knowledge and become well enough trusted in local networks that enables one to communicate effectively. Second, it maintains complete independence from the district, which enables it to be critical, but also, oddly, helps it to maintain the trust of principals and teachers who, if it were too close to the district, might regard it with the suspicion that naturally attaches to district initiatives and proclamations. The ILEA research department was embedded in ILEA, but my guess is that that worked only because education was, at the time, a less political issue than it is now, and because until the 1990s most English LEAs had much closer and more trusting relationships with their schools than most do now, and than most large-ish American districts do.

So, the plea. Get your school board members to look at what the Consortium does, and how it does it, and whether it represents a model that your district or, if your district is small, it and neighboring districts, could replicate.

[Not sure whether this counts as disclosure or name-dropping, but since I started writing this post (in early January), the Consortium has announced my friend and colleague Paul Goren as its new director.]