Nathan Honeycutt is Manager of Polling and Analytics and a Research Fellow at FIRE. Nate studies higher education, which often involves surveys or experiments with university faculty on topics including expression, political bias, and the evaluation of DEI statements.
Note: SPSP refers to the Society for Personality and Social Psychology. This is the text and slides of one of the talks presented at the panel on “Is SPSP a Healthy Organization?” If you prefer video, you can find videos of all four talks here.
Nate’s Talk
When considering, “is SPSP a healthy scientific organization,” I’m going to be going at this from a slightly different angle, looking to see if we can quantify the health of SPSP.
Some of you reading might have been at SPSP last year when it was in Denver. I was not in attendance, but I did hear a lot about the conference after. In particular, Salty Psych took time to post a summary of last year’s conference, which I found absolutely fascinating, very illuminating, and quite helpful to read through. There was also a lot of discussion and debate online in response to her summary.
In her summary, Salty Psych observed out how psychology isn’t letting go of woke. In fact, she said the field appeared to be doubling down on woke, and from her perspective, about a fifth of the talks or posters or presentations that she saw or attended at SPSP were explicitly woke. Now, again, I found this absolutely fascinating. I’m thankful to Salty Psych for the time she took to post this summary.
For me, this led to a number of questions. In particular, I wondered, ‘are these observations that Salty Psych is offering quantifiable?’ ‘What proportion of SPSP talks have a conservative tilt, or have a liberal or progressive tilt?’ ‘How, if at all, has this potentially changed or morphed over time?’ And ‘what could these data points indicate about the health of SPSP?’
And more broadly, what makes a scientific society healthy? I would argue, that what a scientific society deems worthy of including within their programming, what they choose to accept for symposia and have presented as symposia at a conference might offer us some really helpful insight to the intellectual priorities, the health, and the blind spots of the organization.
If we look at this from a signaling framework, these accepted symposia sessions are a signal. They are a signal conveying information about the priorities of SPSP, the values of SPSP, the interests of SPSP, and the direction of SPSP. Or at least of the priorities, values, interests, and direction of the organization’s gatekeepers: the reviewers, perhaps the program selection committee, and others. Because, again, the symposia are competitive, they’re curated, and in some ways, that could be consequential.
So this led to the present study. I wondered, could we take information that we have about known talks that were given at SPSP, and use this to quantify the health of the organization, the health of SPSP?
So, with the assistance of my collaborators and a number of RAs, we gathered all available SPSP convention symposia abstracts. Many of these are in conference programs that are online. Others, we had to obtain with the assistance of SPSP. This led to a collection of a population of nearly 7,500 symposium abstracts from the past nearly 2.5 decades--so 7,469 symposia abstracts in total. And again, why symposia abstracts? Compared to, for example, posters, symposia are more competitive, they reflect not only member interests, but also institutional and member judgments about what merits scientific muster and should merit attention at the conference.
After we collected all of the abstracts, we then submitted them to our abstract analyzer tool. This is a command line tool developed for this project was used to analyze the conference abstracts. In essence, it sent all the conference abstracts through multiple large language models via Open Router, collected the ratings, and then gave us the output. I’ll have a link to this at the end if anyone’s interested in taking a look and tinkering around with the tool.
Specifically, we had three independent large language model raters that (these specific three chosen in part because they are optimized for speed and cost). Each of these three raters rated all 7,469 abstracts on a number of outcome variables. The two I’m going to talk about today are political valence, and a binary activism orientation classification.
All three raters rated all the abstracts on these, and then the scores were averaged across the raters.
So, what did we find?
I’m going to start with political valence first. Political valence was on a 5-point scale from negative 2, strongly conservative, up to positive 2, strongly liberal or progressive, and then zero was right in the middle as neutral or balanced.
To give a sense of what corresponds to the two ends of the scale, here are a couple examples. First, a strong conservative example one was: “[SPSP] has allowed itself to become a participant in the sectarian corruption of science.. [and] has become a club for leftists and an engine for far left/progressive activism.”1
This is an excerpt from Lee’s abstract for his talk as a part of this panel. And interestingly, four of the five abstracts rated as having a strong conservative-lean were talks that were critiquing the ideological health of SPSP, which apparently is a conservative position!
Then a strong liberal/progressive abstract: “Addressing racism and promoting racial justice requires identifying barriers to systemic change and developing tools to overcome these barriers. Given that White people often occupy ‘gatekeeping’ positions of power, one key barrier is rooted in White people’s emotional and emotion-regulatory responses when considering their own role in racism.” So clearly leaning into very progressive topics.
Now, to the results. What did we find? To orient you to the first figure, we have average political valence of SPSP symposia abstracts (y-axis) over time (x-axis).
What did we find over time? What was the trend that emerged in the ratings for political valence?
We find that over time—over the last two and a half decades—SPSP Symposia abstracts have always leaned liberal or progressive. The mean hovered around 0.2 up until around 2016, when it took off. And we do see an interesting dip in the last few years. There are number of reasons we might speculate for why that’s occurring. But overall we see SPSP symposia abstracts have consistently liberal-progressive, on average, over time and have become increasingly so within the last 10 years.
If we look at this from a slightly different angle—percentages by year—we see an illuminating trend emerge. The three bins are conservative, neutral, and liberal/progressive.
Up until 2014, over 70% of SPSP symposia abstracts were rated as neutral or balanced, and on average around 20-25% were rated as leaning liberal/progressive lean,
Then in 2016, we start to see a sharp increase for liberal/progressive. And then between 2020 and 2021a flip takes place, with symposia rated as liberal/progressive overtaking neutral for the majority of symposia abstracts each year. In the last 6 years, now, the majority of SPSP symposia abstracts have had a liberal or progressive lean.
Additionally at the very bottom of the figure you can see a red line that just barely hovers above zero. It’s not that there weren’t any conservative-leaning abstracts, but it was pretty close to zero, about 1%--85 out of 7,469 abstracts in the last 2.5 decades were rated as leaning conservative. And there really hasn’t been any movement on that over time. It’s been low, and consistently low.2
Next, activist orientation. This variable was a binary classification, 0 or 1. An abstract was rated 0 if it was explicitly framed as explanation or prediction. An abstract was rated 1 if it was framed as trying to produce some sort of change. Social change, political change, institutional change.
As an aside, it’s important to note that activist orientation could apply to any research. For example, research that has a liberal lean could be a 0 or a 1 here. Conservative could have a 0 or a 1. Neutral could have a 0 or 1. That is, this is operating independent of political valence.
As an example, an abstract rated as 1—explicitly framed as trying to produce some sort of change: “The Marley Hypothesis states that those who are ignorant of the injustices of the past are less likely to recognize the injustices of the present, and are therefore bound to perpetuate them… Results support the notion that interactive historical education has the potential to increase … support for policy reform to address systemic racism.”
This abstract is directly calling for some sort of change.
So, what do we find for the average activist orientation in these abstracts over time? Similar to political valence, the proportion of SPSP symposia abstracts coded as have an activist orientation has steadily increased over time, most substantially in the last 10 years.
As with political valence, it’s helpful to also look at the percentages. Up until 2014, 90% or more of SPSP symposia abstracts were rated as not activist (rated “0”). That is, up until 2014, 90% or more of the symposia abstracts were explicitly framed as explanation or prediction. And then we see a change in the last 10 years with a steady rise in abstracts having activist orientation, hitting a peak of around 40%.
To recap, what did we find looking at the last 2.5 decades of SPSP symposia abstracts?
Digging into these SPSP symposia abstracts we find that over the last two and a half decades, SPSP’s symposium programming appears to have shifted from a politically mixed or neutral framing toward a more consistently progressive framing. These abstracts have, on average, always leaned liberal/progressive, but they become increasingly so. Additionally, a greater percentage are now framed or aimed at producing some sort of change.
The pattern itself appears to no longer be anecdotal.
How does this fit?
If you’ll indulge me just briefly, let’s consider a few major events related to the ideological health of psychology or social and personality psychology in particular. (There are many more important milestones than this, but I just want to briefly highlight a few)
The field has grappled with issues related to its ideological health for a very long time. We had Phil Tetlock’s paper in the early 1990s on politicized psychology. That was addressing the whole field of psychology, not social and personality psychology in particular. But still relevant for our field.
Then Haidt’s keynote in 2011 was a pivotal moment for SPSP. Haidt claimed that social and personality psychology had become a politically homogenous “tribal moral community.” And back in 2011, Haidt challenged the field to set a goal of becoming 10% conservative by 2020. Though instead of accepting Haidt’s challenge, we see from data produced by Inbar and Lammers, and also data from SPSP’s 2019 climate survey, that SPSP has actually gone in the opposite direction. The field has, over time become even less ideologically diverse.
If we consider some of these milestones and map them onto what we’re seeing with the political valence ratings, perhaps… for example, Haidt’s talk in 2011. Interestingly, we see a slight dip in average political valence for the abstracts the year following Haidt’s talk. So maybe his talk had an impact on the immediate horizon. But then, in the years following that, we see the steady increase in average political valence over time, with greater proportions of abstracts having a liberal or progressive framing.
So… so what? Just a few takeaways here as I’m getting close to the end.
I think we see from these data that the pattern is no longer anecdotal. We see in a quantifiable sense that political valence has shifted over the last 2.5 decades, and quite dramatically in the last 10 years. SPSP’s programming, arguably, could be shaping what graduate students see as respectable. Grad students and junior faculty show up, they’re seeing what research is being actively conducted in the field, what’s being presented, how it’s being discussed—what topics are active and up for debate and discussion versus settled. What is the field not talking about, or what is taboo?
And these programming trends, arguably, may be functioning as norm-setting mechanisms. Or, to return back to the signaling framework, functioning as signals.
To harken back to Haidt’s question or challenge for the field: is SPSP a moral community, or is it a polaristic scientific one?
I think these data could stand as a diagnostic, they’re scalable, and I think these data show that the intellectual culture within SPSP is measurable—it is quantifiable.
So, a question I want to close with is: what is the trajectory for SPSP? In this talk I didn’t get in to why viewpoint to intellectual diversity may or may not matter, because the results I’ve presented are diagnostic and descriptive. They can’t really tell us what the cause is. Though that’s an important question for future research.
But a broad takeaway here is that if symposia programming reflects what SPSP—what our field—considers worthy of attention, what does the decline of neutral framing and the rise of liberal and progressive framing signal about SPSP’s intellectual trajectory?
I think this is a really important question for the field and for researchers within it to grapple with as we look ahead to what is the trajectory for SPSP and for research and researchers within this field.
I’d like to thank a handful of people. First, I’d like to thank Michelle Jester at SPSP. We are very grateful to her for her assistance helping us collect abstracts that weren’t available in conference programs online. Thank you, Michelle.
I would also like to thank my collaborators, who are from many different institutions and backgrounds and brought all sorts of unique and helpful skills to the table for this project. I’m very thankful for their assistance and for being willing to collaborate. Thank you to Aaron, Marius, Luke, JP, and Mitch.
Commenting
Before commenting, please review my commenting guidelines. They will prevent your comments from being deleted. Here are the core ideas:
Don’t attack or insult the author or other commenters.
Stay relevant to the post.
Keep it short.
Do not dominate a comment thread.
Do not mindread, its a loser’s game.
Don’t tell me how to run Unsafe Science or what to post. (Guest essays are welcome and inquiries about doing one should be submitted by email).
Footnotes
Lee here. That quote about SPSP becoming an engine for far left political activism? I am not raising concerns about Nate’s LLM-based coding system. However, in my personal politics (who I vote for, campaigns I have supported, etc.) I am definitely not conservative (not that there is anything wrong with that, in fact, for SPSP’s health, and academia’s health more broadly, I think there is quite a lot right about adding more conservative faculty). I am not emitting a “dogmaflare” (from The Orwelexicon) to curry favor with my more progressive colleagues — who I definitely think have been mostly either enthusiastic participants in the political corruption of social psychology (and beyond) OR passive bystanders who have permitted that corruption with (for the most part) nary a peep of pushback.
At this point, given how corrupt and/or craven so many have been, I’d rather garner their ire than their approval. More important, I think the statement Nate quoted is simply epistemically true. And I have never had any problem agreeing or working with anyone, including but not restricted to conservatives, when I think they are right or working towards goals I endorse.
Lee here again. This paper conducted a similar, but far broader analysis, covering all available social science articles in major journals going back to the 1960s, a total of about 600,000 publications:
It found results quite similar to Nate’s":
About 30% of the 600k articles had a political lean.
90% of those leaned left.
Could we include the gender of those speaking? If 80% are women and 100% of their talks are "woke", then we can't expect any change, no matter what anyone does.
This is an issue of grammar, not specifically related to the subject matter: The term should be "symposium abstracts", not "symposia abstracts". "Symposium" comes from Greek. The plural of "symposium" is "symposia". Saying "symposia abstracts" is like saying "conferences abstracts" instead of "conference abstracts". While a general population might not know this, I think that Ph.D. scientists should.