If Diverse Scholars Produce Better Research, Judging Research on its Merits Will Enhance Diversity

An Open Letter to a Professional Society

This is a guest post by Chris Rhoads, an associate professor in the Educational Psychology Department at the University of Connecticut.  His Ph.D. is in the field of statistics, and he has spent most of his career working on quantitative evaluations of educational policies and programs.

He was asked to serve as a peer reviewer for the Research Methods section of the (recently completed) 2023 annual conference of the Society for Research on Educational Effectiveness (SREE).  SREE was founded in 2005 as an outgrowth of the movement to promote scientifically based research in education. As of the 2023 conference it decided to institute a “Diversity and Equity” criterion that was used as part of the review process for all proposals submitted.  Below is Chris’s response upon being asked to review for the conference. 

Images below did not appear in Chris’s letter. I added them. LJ.

---

Dear [Name Omitted, Chair of the Research Methods section of the 2023 SREE conference],

You have asked me to review proposals for the Research Methods section of the 2023 SREE conference.  I have reviewed proposals for SREE’s annual conference most years that the society has been in existence and every time I have been asked.  Despite this I must decline your invitation.  The reason is the following review criterion that has been added for proposals to the conference.

[Lee’s sidebar: Chris makes some important substantive points below. But part of me wishes he would have written instead something like, “In order to advance SREE’s Diversity, Equity, and Inclusion goals, I must step aside so that SREE can have greater opportunity to solicit reviewers from marginalized groups.”].

Diversity and Equity Criterion

To what extent do (1) the authors or session participants reflect various dimensions of diversity (e.g., diversity of race, ethnicity, gender, career stage, institutional affiliation, and role, including policy-maker and practitioner roles), and (2) the methods and/or content of this submission embed equity principles and practices.

You may be surprised at my objections to the new criterion.  You might believe it is obvious that research is better if it “reflects dimensions of diversity” and “embeds equity principles”.  Why shouldn’t we use these criteria to evaluate proposals to the conference?  In answering this question, I will draw on the arguments presented here.

I can illustrate the problem with the diversity portion of the review criterion as follows. Suppose the following two proposals are submitted to the conference. Proposal A concerns the problem of using small area estimation in conjunction with propensity score stratification to improve generalizations from randomized experiments.  It purports to have discovered that a particular Bayesian implementation of the approach can substantially reduce bias compared to other estimators that have been suggested in the literature.  The research seems solid.  It seems worthy of being accepted for presentation at the conference.

Proposal B is from a different research team but is the exact same proposal as proposal A.  It makes the same arguments, uses the same research methods, and reaches the same conclusions (it can happen, after all, Leibniz and Newton independently invented calculus).  Proposal A was submitted by a research team of all white males from the same institution, all of whom are full professors in an educational policy program.  Proposal B was submitted by a research team of individuals from different racial backgrounds and different genders.  The team contains both full and assistant professors from different institutions.  It even has a practitioner and a policy-maker on the author team!

The SREE program committee would have me believe that proposal B is more worthy of inclusion in the conference due to the enhanced diversity of the authors.  This claim is absurd.  The scientific content of the proposals is identical.  There is no rational basis for preferring one to the other.

If the quality of the research being proposed was enhanced by the diversity of the author team that produced it, then the value of that diversity will be captured by other review criteria.  If the quality of the research was not enhanced by the diversity of the author team, then the program committee must be mistaken in its presumption that diversity is of such overarching importance that every proposal submitted to the conference must be evaluated relative to this criterion.  Either way, the diversity portion of the criterion is not needed.

Image by Deddy Rachman

As regards the equity portion of the review criterion, I cannot imagine accounting for it in the review process, because I do not know what it means to “embed equity principles and practices.”  I also believe that there is no general agreement within SREE, the larger educational research community, or society at large, as to what this phrase means.  SREE has been an important force in helping to produce high-quality quantitative research, and this is extremely poor measurement practice from such an organization.  You can’t have a bunch of reviewers reviewing proposals from a bunch of authors when neither the reviewers nor the authors are operating with the same understanding of the review criterion.

Additionally, including the diversity and equity criterion distracts researchers submitting proposals from what should be their true focus: Accurately describing the scientific and practical value of their research contribution.  And it distracts reviewers from what should be their true focus: Evaluating the scientific and practical value of the research described in the proposal.

Finally, let’s say the quiet part out loud.  By adopting this review criterion, SREE is taking sides in a highly politicized fight regarding the extent to which the education system should address itself to topics of “social justice”. Three years ago, SREE was a well-respected scientific society that any well-meaning educational researcher, regardless of his/her ideological or political commitments, could feel comfortable supporting.  That has changed with the themes of the last few conferences, culminating in the new Diversity and Equity Review Criterion, which many researchers will rightly interpret as an ideological purity test.

Shift from Science to Politics

Prior to the 2021 conference, themes and messages from program chairs focused on topics such as “adoption and implementation of effective research-based practice” (2018), “tensions and tradeoffs” (2019) or “practical significance” (2020).  However, starting in 2021 we begin to see program chairs making overtly political statements unsupported by empirical evidence.

For instance, in 2021 the program chairs asserted “gaps in achievement and opportunity are further exacerbated by social injustices, racism, and explicit or implicit biases that continue to plague our country”.  So much for SREE’s reputation for urging caution before asserting knowledge about the causes of complex phenomena.  Apparently the 2021 program chairs were able to magically conjure up the following knowledge. Our country continues to be plagued by racism, social injustice and bias AND these features of our society are at least partially responsible for achievement gaps.  Why even bother with the conference when insight into complex social problems is so easily discernable?

In 2022 the number of highly debatable and unsupported claims made by the program chairs are too many to list. I will provide only a few.

There is the claim that a “second pandemic was triggered by the police killings of Breonna Taylor and George Floyd”. To which I ask, “What is the nature of this pandemic?  What is the justification for referring to it (whatever it is) as a pandemic?”. 

There is the claim that “we must engage in reconciliation with communities of color”. Who is the “we” in this sentence?  The membership of SREE?  And who shall be in charge of negotiating this reconciliation on behalf of both SREE and “communities of color”?  What does this have to do with a conference about educational effectiveness?

There is the claim that “We must honor ways of knowing that have not been traditionally valued in academia”  Why?  SREE is an academic society.  Much of its membership thought it worthwhile to get a PhD by engaging in “ways of knowing traditionally valued in academia.”  These ways of knowing have formed the basis of the research presented at SREE ever since its founding.  What is wrong? What needs to change? 

Now we are in 2023 and there is a new diversity and equity criterion and the suggestion throughout the message from the program chairs of the need to “embed equity” into all that we do.

Trust in public education has been decreasing for some time.  It has recently become highly politicized. Look at the graph entitled “Americans’ confidence in U.S. Public Schools, by Party ID” available here [LJ: I tracked it down, its shown above]. In 2019 roughly 30% of both Democrats and Republicans had a great deal of confidence in U.S. Public Schools (down from over 50% in both parties in the 1970s).  Starting in 2020 that diverges such that now 43% of Democrats and 19% of Republications have a great deal of confidence in U.S. Public Schools. If SREE continues down this path it will be exacerbating this sort of political polarization.  It will be contributing to, not helping to solve, “pressing challenges in education.”

In short, I will not participate in this year’s SREE conference, either as a reviewer or in any other fashion.

Sincerely,

Chris

P.S.—The situation SREE finds itself in is reminiscent of a similar situation involving the Society for Personality and Social Psychology.  Some of the particulars are different.  However, in both cases a society has decided to promote ideological goals rather than remaining true to its core mission.  I would encourage you to read the following article from Lee Jussim regarding the Society for Personality and Social Psychology situation (and this update).

P.P.S-I have also sent this letter to the 2023 conference co-chairs and to select members of the Board of Directors of SREE.