Adversarial Collaborations Usually Produce Findings That Require Walking Back Social Justice Narratives

My Response to a Question at the SPSP Panel on "Is SPSP a Healthy Organization?"

This post presents my response to a question our panel received at the SPSP symposium on whether SPSP is a healthy organization. For the full set of talks in video format, go here. For text versions of the talks, go here, here, here, and here.

The question was something like this:

If I want to do research on political topics as free from political biases as possible, what should I do?

My Answer: An Adversarial Collaboration

Preliminaries

This was not part of my response but is presented here to give you some sense of what adversarial collaborations are, and why they are one avenue for limiting political biases. Also, there are plenty of things one can do besides adversarial collaborations, but this post focuses on my answer to the question. And if there is any one thing one can do to limit their political biases, the single best is probably to find a political adversary with whom to collaborate.

A semi-deep dive into adversarial collaborations can be found here:

It links to our recentish paper which will provide the full deep dive. The Very Short Version is captured in this model, found both in the Substack post and paper:

The key elements are these:

• adversaries are those who have staked out opposing positions on some theoretical or political social science topic. Only the political topics are relevant to the question that led off this post.

• they have the potential (though there is no guarantee) to improve rigor because adversaries will be highly motivated to seriously skeptically vet one another’s claims. By not allowing each other to get away with confirmation bias, soft tests of one’s pet hypotheses, and weak methods, the resulting research is likely much more rigorous than much of what is done on political/politicized topics by those with agendas.

However, because the left skew of most social science fields in general and social psychology in particular, it is becoming progressively harder and harder to engage in an adversarial collaboration because there are too few non-progressives in the field. Nonetheless, some can be found, and more can often be found in bona fide research positions outside academia (think tanks, freestanding research institutes, etc.).

Back to My Response

First, I recommended that SPSP ban adversarial collaborations. This is because: 1. It is clearly committed to social justice; and 2. Almost every adversarial collaboration of which I am aware on social justice-related topics has required walking back major social justicey scholarly claims. (This of course was a bit tongue-in-cheek, and mainly served as a springboard to launch into my tour-de-adversarial collaborations in social psychology on social justicey topics).

For example, this large team published the paper shown below in 2005, finding no accuracy in national character (personality) stereotypes:

A few years later, the senior author, McCrae, contacted me inquiring if I would be willing to collaborate with them on future papers assessing the accuracy of stereotypes. McCrae did so explicitly because he knew that I had previously staked out the position that many stereotypes are at least moderately, and sometimes highly, accurate. I know this because he said so (well, it was by email, so strictly speaking, he wrote so). From memory, I had the sense that he was not expecting me to take him up on his offer. But I loved the idea and did.

This was, I think, even before the term “adversarial collaboration” was coined. It was certainly before the term came into common use. But that’s what it was. and here are two of the papers we produced:

The empirical answers to the titular questions in both papers was a clear “yes.” Now, in fairness, we also had a third paper that replicated their original finding (more rigorously than the original, in my view, because I had both some skeptical views of the methods that produced their original findings AND ALSO a proposal to address that skepticism — which, when we did so, STILL produced their original finding):

Nonetheless, the package included two of three papers finding highly accurate stereotypes, and I’d argue that sex stereotypes especially were more “social justicey” than national character stereotypes.

This did not appear in my response to the question at SPSP

Then there is this:

A Brief Detour About the Red Team Approach

This paper was in my response, but this detour was not in my response (not enough time) but I present it here so that you can understand what the Red Team approach is, and how and why it is similar enough to an adversarial collaboration for me to have included it in my response.

This paper was not exactly an adversarial collaboration, but a close cousin. The “red team” approach is an innovation pioneered by science reformer Daniel Lakens and his colleagues. It involves enlisting a team of outsiders to intensely skeptically vet the proposed research.

Here is how they wrote about it in their paper (citations deleted for flow):

The prevalence of gender bias in hiring and other forms of group-based discrimination are among the most controversial issues in the social sciences . Concerns about potential researcher ideological and intellectual commitment biases on both sides are common in this space

To further optimize our methods, we employed the innovative new “red team” approach. A red team is a designated team of scientific experts external to the core author group (the “blue team”). Two coordinators recruited an independent team of experts on statistics, meta-analysis, and gender research, as well as a librarian, to critique all aspects of our meta-analysis plan, point out potential issues, and suggest improvements. The goal of the red team approach was to improve the quality of the research project by identifying flaws and challenging dominant assumptions in our work, incorporate different viewpoints, and invite early feedback from international experts.

Such an approach allows for an exchange between researchers and a “devil’s advocate” that aims to produce a higher quality research plan before submission to a journal by identifying oversights, soliciting feedback from experts, and preventing groupthink. A red team is also similar in some respects to an adversarial collaboration, where researchers with directly opposing predictions work to design a study together, except that red team members are recruited for expertise alone rather than their intellectual committments.

Back to My Response to the Question

This study had three key findings: 1. Their meta-analysis found almost no evidence of discrimination against women in hiring going back about 30 years; 2. Over the last 15 years or so, there was more evidence of discrimination against men than women (though that bias was small); and 3. Their forecasting survey involved academics predicting the results of the meta-analysis, and they wildly overestimated the amount of discrimination against women. “Social justice” narratives emphasizing biases against women can be seen in full retreat, even if academics do not realize it.

I summarized this study here:

No Evidence for Workplace Sex Discrimination

Then there is this adversarial collaboration, which found no bias against women in four of six domains and, in the two where there were biases, they were quite small, much smaller than much of the “scholarly” literature often claimed.

And this one:

We found biases against men, not against women. The short version (with a link to the full paper) can be found here:

REVERSAL: Science Faculty's "Subtle" Gender Biases Against Men

This next paper was a review, not an empirical study. In short, it found almost no evidence supporting the net benefit of university DEI programming, not because the programs were found to be ineffective, but because there were almost no scientific studies even attempting to evaluate them. And, in fairness, for the same reason (lack of evaluation studies), there was almost no evidence for most of the criticisms of DEI programs. Still, I’d argue that requires a major walking back of the once-pervasive in academia testaments to the value of DEI programs.

And then there is this douzy (at the time of my response, it was only available as a preprint, but it has subsequently been published). Key findings are shown below in bullet points:

Conclusions

1. If you want to maximally limit your potential for political biases to lead your research to make unjustified claims, find an adversary, and collaborate with them on the topic.

2. Nearly every adversarial collaboration on a social justice-related topic has produced findings requiring the walking back of social justice narratives, producing:

   1. strong evidence of accuracy in sex and age stereotypes (though replicating prior findings of no accuracy in national personality stereotypes)

   2. more evidence of biases against men than women in academia, and that even when there are biases against women, they tend to be quite small.

   3. more evidence of biases against men than women in hiring generally

   4. little evidence to support the value of DEI programming (or that of its critics)

   5. no racial discrimination, borderline trivial incremental validity (for predicting discrimination) of implicit measures over explicit measures of prejudice, far stronger relations of explicit prejudice than of implicit measures to discrimination, and scores of 0 on the implicit association test correspond to anti-White bias, not to eglatiarianism.

---

Commenting

Before commenting, please review my commenting guidelines. They will prevent your comments from being deleted. Here are the core ideas:

1. Don’t attack or insult the author or other commenters.

2. Stay relevant to the post.

3. Keep it short.

4. Do not dominate a comment thread.

5. Do not mindread, its a loser’s game.

6. Don’t tell me how to run Unsafe Science or what to post. (Guest essays are welcome and inquiries about doing one should be submitted by email).