It's hard to make heads or tails out of the results without seeing the data.
For example, Fig. 2b shows "16% difference" in the fraction of students who thought the admissions officer should issue a public apology. Is that 58-42? Or 20-4? The difference is important.
For example, Fig. 2a shows a "% difference b/t treatment and control" of 35 for (e.g.) "how many microaggressions did the applicant experience?". Does that mean that on average people in the treatment group conjured 35% more microaggressions? Or that 35% more people conjured microaggressions? I presume it's the former because I see "t-test" in the footnote. But is the difference because more people were "seeing" microaggressions or was it that the ones who did "see" microaggressions saw more?
That they had written articles about the study but then canceled them just before publication?
Or that they had made some commitment to publish articles about it but decided not to?
Or that there was an expectation that they would publish articles about them because they had historically published articles about "similar" studies (i.e., similar in the sense of "hard-hitting, uncomfortable research on hot-button issues")?
“The New York Times, which has cited NCRI’s work in nearly 20 previous articles, suddenly demanded that this particular research undergo peer review”. Why wouldn’t you welcome peer review to bolster this study and your argument?
"The New York Times, which has cited NCRI’s work in nearly 20 previous articles, suddenly demanded that this particular research undergo peer review—a requirement that had never been imposed on the institute’s earlier findings, even on similarly sensitive topics like extremism or online hate. At Bloomberg, the story was quashed outright by an editor known for public support of DEI initiatives."
Did we get a link as to why this was shelved? The explanation from the Bloomberg editor, or the peer review? I'd be curious about their reasoning and if it's well-meaning-yet-flawed, or full of shite.
There’s a feature that allows you to listen to the article. Click on the link below. When the article comes up, to listen, click on this symbol ⧁, right above the title.
Although the goals of DEI are laudable, the way it is practiced and implemented simply doesn’t work. It actually amplifies division and hostility between groups. In many US states, diversity is increasing to the point where DEI is no longer necessary. I’m in Texas, and we would have a diverse group of students without even having to try.
As to trust in Bloomberg and NYT, they lost my trust years ago. And they keep doing things to be sure I don't regret my decision to mentally label them as cheap propaganda.
I think maybe thou dost protest a wee bit too much. The NCRI study wasn't withdrawn. It was published. And 2 out of how many major outlets held it up? And only one of those actually spiked it. And for the life of me I do not see why you would oppose the NYT's referral for peer review. If it comes back from peer review with solid acceptance, that makes the study arguments and conclusions even stronger. Isn't that the desired result? BTW, major new media outlets spike or turn down research stories all the time. As for the research itself, it seems like it's based on some big assumptions regarding the meaning of the outcomes. Of course the DEI training is likely to increase study subject sensitivity to claims of racism. That's the reason for DEI. But we don't live under Jim Crow racism anymore. We can debate the magnitude of racism in America, but insofar as it exists, it is going to exist in subtle forms, subject to much interpretation. I am a DEI skeptic. Personally I think DEI and its big spokespersons (Kendi et al) have got it all wrong. I am just not sure this study and the NYT's treatment of it amount to quite the big deal you seem to think they do. Still, I welcome this info. If it isn't the barrage that blows a whole in DEI's defenses, it's still a worthy contribution to the debate.
Yes, I agree. The study seems to show that DEI training does EXACTLY what it's supposed to do, ie "increases awareness" of various inequities and manifestations of opression and potential prejudice at work. The assumption is that those inequities exist and people don't condemn them because they consider them normal and acceptable. I'd assume this effect (sensitising to potential discrimination) is an explicit aim of Kendi etc. One can (and imo should) argue with the aim, of course.
Although it was both NYT and Bloomberg, I actually mostly agree with your analysis (relevant side note: I often post guest posts without endorsing every claim in them). In this case, I am not at all sure that it is fair to characterize either NYT or Bloomberg as "suppressing" the story.
I assume DEI persists so stubbornly—at least in corporate spaces—because companies keep convincing themselves (with ample encouragement from consultants conveniently armed with pricey diversity programs) that there's a compelling "business case" for diversity. Yet, the evidence shows, at best, a tepid link between demographic diversity and performance. Worse, these justifications are not only scientifically questionable but also morally fraught—and likely counterproductive (just like the programs themselves, as you so aptly demonstrate with this work).
Heh. So you have no AUTHENTIC scientific, ANTI-RACIST expertise at all. Hey, its (supposedly) a free country (even though it's the granddaddy of all white-supremacist-based nation-states on the globe). I agree --- "That's why used car dealers think they know vax's cause autism." Like you, they believe they know what anti-racism is, and how it's related to science. I have absolutely no idea why you apparently believe that I want to know more about you (to the extent and/or degree that I would search). If you knew anything at all about anti-racism, and how it's related to the scientific method --- we wouldn't even be having this conversation. Indeed, the likes of you have absolutely nothing to teach me --- relative to so-called "instructions" concerning authentic research, as it relates to ANTI-RACISM (nothing at all) --- zilch.
There’s a feature that allows you to listen to the article. Click on the link below. When the article comes up, to listen, click on this symbol ⧁, right above the title.
Dr Jussim, your paper seems to refer to figures that are out of order or aren’t in the paper. Perhaps the Appendix might be from a previous version of the paper?
Thanks for flagging this, I sincerely appreciate it. We've since found several other minor errors or omissions (nothing that changes the overall patterns, but I would have preferred NCRI to have waited to be sure it was more polished, but so be it) and will upload an improved report over the next week or two.
@Lee Lee there has been a lot of ineffective organizational initiatives done over many years.
What’s baffling to me with this research is that people who consider themselves serious researchers use the concepts of two shallow, reductionist, infantilizing, and just bad popular books that met the zeitgeist with perfect storm timing as the core texts for their experimentation.
Were there no other controls or alternate approaches to DEI/Diversity Management/Inclusion that were used?
Or was it simply group identity-focused texts used to confirm a fixed hypothesis?
If the purpose is to make DEI in all of its breadth “wrong” or bad, this study reinforces it.
If it is to open up a heterodox view beyond reacting with the easiest point of make-wrong, it doesn’t occur as scholarship or a “study”to me. It occurs as politics.
For reasons that are not clear to me, Substack is glitching when I try to post my response, so I am going to try to break it up into two replies here. Not sure how many it will be, it seems to be glitching when I paste in more than a paragraph.
Hmmm. First, let's discuss your scientific criticisms, and leave your questions about our motives for the end of this reply.
1. Why did we use Kendi & DiAngelo? Because their ideas/rhetoric are all over DEI trainings and interventions. The topic network found just this as reported early in the paper.
2. As such, we think our findings are potentially vastly more informative and general than had we used the specific trainings used in a specific place. Had we, say, used the DEI materials used by Milwaukee's HR Dept, our studies would be justifiably dismissed as hopelessly trivial and uninformative about anything except Milwaukee's HR dept.
3. Alternative types of DEI. Most DEI training materials are not made publicly available. If you have access to any that are substantially different than the rhetoric used by K&D, please contact me. I would definitely consider using them in followup studies.
4. "Confirming a fixed hypothesis." Well, to test any hypothesis, you need to "fix it" -- that is, make it clear, specific, and empirically testable. So, sure, the hypothesis was fixed. What was not fixed were the results.
5. We definitely suspected that the rhetoric used by K & D, which are all over many DEI materials, demonizes "oppressors." That was the hypothesis. What we did not know was whether that would actually happen. That's why we conducted the studies -- to determine if such demonization would occur. Put differently, the hypothesis was "fixed" but the results were not knowable in advance.
6. It is, fortunately, still a (mostly) free country. Anyone is welcome to believe or state anything they like about our studies, including that they think the studies are crap and that crap studies should not be used to dismantle DEI programs. Actually, these are a set of preliminary studies, and even I do not think they imply any particular course of action other than ... they should inspire additional research on the upsides and downsides of different types of DEI programs/interventions.
However, if one's view is something like "we don't dismantle programs based on crappy research" I say, great! This implies that you think real social programs should be based on sound research. If so, I agree wholeheartedly. The thing is, to be coherent, you would then need to argue for dismantling DEI programs -- but not because of our studies. You would need to argue for dismantling DEI programs because they have been widely implemented based on **no scientific research at all" and, instead, based on the ideology and beliefs of advocates.
Much of what your paper cites is true. Not news but to let you know I read it thoroughly.
The priming in the experiments and conclusions based on it are expected.
And, they are likely to be replicable across larger cohorts if done with the same methods.
I have worked in the field for 20+ years and criticize it perhaps to the extent that one might label me a DEI naysayer.
Kendi and DiAngelo have become popular and equating them with DEI is easy because of their popularity. What they purport isn’t the work I know.
And, unfortunately without considering the tradeoffs associated with driving their ideological fixations forward, many organizations jumped onto this inevitably divisive and ineffective slow motion train wreck. A train wreck planted by many academic institutions, not by people who have been practicing in companies for any period of time.
Have there been people in the practice of advocating on behalf of some group identities for a while?Yes. They can have some success but not in isolation. If the advocacy leads to greater intergroup contact than the status quo, the network effects I’ve observed are largely positive.
As well, there are companies who have been building capabilities that help individuals and their organizations thrive and make their best contributions to the companies’ missions.
The thriving outcomes are not based on one or two shot training interventions. And, few are shared and likely won’t be shared externally. It’s not worth the distraction from the responses.
A lot of Kendi and DiAngelo-like trainings were a reaction to incomplete data about cops disproportionately killing black people . Also untrue, but perception trumps data when the data is loosely disseminated. The assumptions are what get us.
Of course, all have the right to publish anything they’d like. Your research is conformational (thus my using the language of “fixed hypothesis) in that as you cited Kalev and Dobbin, who have repeatedly documented most diversity interventions (especially mandatory training and grievance-based approaches) don’t work for the outcome of representation or promotion of less represented group identities.
Inclusion and diversity skills are not about getting more black or other group identities jobs and promotions. That might be one outcome to create more effective systems in the candidate/employee lifecycle but that isn’t the point—at least not in my work.
If you cannot manage the tensions and complexity of differences and similarities (aka diversity) managing complexity is at best difficult, at best many things will be delayed. And this goes well beyond race and gender as Roosevelt Thomas informed us on in 1991.
Bringing people together to solve complex challenges that require a diversity or expertise that can be accentuated by visible diversity, works well.
The last five years of training-by -fire short-term interventions (like Kendi-DiAngelo content) are suspect and should be stopped.
To make their work representative of the last 50 years of diversity and inclusion programming and impact, is not helpful.
Your thoughtful work and that of others who are questioning (or dunking on) DEI will also benefit from timing, similarly to that of Kendi and DiAngelo just with a different context.
Amri, thank you for your thoughtful comments and constructive criticism. No one should come away believing that our studies addressed all of DEI, let alone other programs for improving quality of work life in orgs, or combating real biases. We are currently in process of revising the report for all sorts of reasons, but I will strongly suggest we add, to the conclusion, a brief statement to the effect that we only studied certain types of social justice rhetoric and is irrelevant to any program that eschews such rhetoric.
It's hard to make heads or tails out of the results without seeing the data.
For example, Fig. 2b shows "16% difference" in the fraction of students who thought the admissions officer should issue a public apology. Is that 58-42? Or 20-4? The difference is important.
For example, Fig. 2a shows a "% difference b/t treatment and control" of 35 for (e.g.) "how many microaggressions did the applicant experience?". Does that mean that on average people in the treatment group conjured 35% more microaggressions? Or that 35% more people conjured microaggressions? I presume it's the former because I see "t-test" in the footnote. But is the difference because more people were "seeing" microaggressions or was it that the ones who did "see" microaggressions saw more?
Are the data available anywhere?
What does "editorial reversals" mean?
That they had written articles about the study but then canceled them just before publication?
Or that they had made some commitment to publish articles about it but decided not to?
Or that there was an expectation that they would publish articles about them because they had historically published articles about "similar" studies (i.e., similar in the sense of "hard-hitting, uncomfortable research on hot-button issues")?
“The New York Times, which has cited NCRI’s work in nearly 20 previous articles, suddenly demanded that this particular research undergo peer review”. Why wouldn’t you welcome peer review to bolster this study and your argument?
"The New York Times, which has cited NCRI’s work in nearly 20 previous articles, suddenly demanded that this particular research undergo peer review—a requirement that had never been imposed on the institute’s earlier findings, even on similarly sensitive topics like extremism or online hate. At Bloomberg, the story was quashed outright by an editor known for public support of DEI initiatives."
Did we get a link as to why this was shelved? The explanation from the Bloomberg editor, or the peer review? I'd be curious about their reasoning and if it's well-meaning-yet-flawed, or full of shite.
I did a deep dive into the study itself, including the supplemental material: https://borncurious.blog/p/the-study-on-dei-causing-hostile
Like a lot of psych studies, there are multiple issues. Of course, this is preaching to the choir at Unsafe Science.
There’s a feature that allows you to listen to the article. Click on the link below. When the article comes up, to listen, click on this symbol ⧁, right above the title.
https://medium.com/@howardjeagle/research-b551399e889a
Although the goals of DEI are laudable, the way it is practiced and implemented simply doesn’t work. It actually amplifies division and hostility between groups. In many US states, diversity is increasing to the point where DEI is no longer necessary. I’m in Texas, and we would have a diverse group of students without even having to try.
Thank you for writing this.
As to trust in Bloomberg and NYT, they lost my trust years ago. And they keep doing things to be sure I don't regret my decision to mentally label them as cheap propaganda.
I linked to this excellent post in a post of my own which goes some way to explaining why the study was ignored.
https://www.lorenzofromoz.net/p/modelling-coordination-in-an-activist
I think maybe thou dost protest a wee bit too much. The NCRI study wasn't withdrawn. It was published. And 2 out of how many major outlets held it up? And only one of those actually spiked it. And for the life of me I do not see why you would oppose the NYT's referral for peer review. If it comes back from peer review with solid acceptance, that makes the study arguments and conclusions even stronger. Isn't that the desired result? BTW, major new media outlets spike or turn down research stories all the time. As for the research itself, it seems like it's based on some big assumptions regarding the meaning of the outcomes. Of course the DEI training is likely to increase study subject sensitivity to claims of racism. That's the reason for DEI. But we don't live under Jim Crow racism anymore. We can debate the magnitude of racism in America, but insofar as it exists, it is going to exist in subtle forms, subject to much interpretation. I am a DEI skeptic. Personally I think DEI and its big spokespersons (Kendi et al) have got it all wrong. I am just not sure this study and the NYT's treatment of it amount to quite the big deal you seem to think they do. Still, I welcome this info. If it isn't the barrage that blows a whole in DEI's defenses, it's still a worthy contribution to the debate.
Yes, I agree. The study seems to show that DEI training does EXACTLY what it's supposed to do, ie "increases awareness" of various inequities and manifestations of opression and potential prejudice at work. The assumption is that those inequities exist and people don't condemn them because they consider them normal and acceptable. I'd assume this effect (sensitising to potential discrimination) is an explicit aim of Kendi etc. One can (and imo should) argue with the aim, of course.
Although it was both NYT and Bloomberg, I actually mostly agree with your analysis (relevant side note: I often post guest posts without endorsing every claim in them). In this case, I am not at all sure that it is fair to characterize either NYT or Bloomberg as "suppressing" the story.
And thanks for the compliments on the studies!
I assume DEI persists so stubbornly—at least in corporate spaces—because companies keep convincing themselves (with ample encouragement from consultants conveniently armed with pricey diversity programs) that there's a compelling "business case" for diversity. Yet, the evidence shows, at best, a tepid link between demographic diversity and performance. Worse, these justifications are not only scientifically questionable but also morally fraught—and likely counterproductive (just like the programs themselves, as you so aptly demonstrate with this work).
Eric Hoffman, "All causes start as a movement, become a business, and then degenerate into a racket". In this case a multi-billion-dollar industry
Heh. So you have no AUTHENTIC scientific, ANTI-RACIST expertise at all. Hey, its (supposedly) a free country (even though it's the granddaddy of all white-supremacist-based nation-states on the globe). I agree --- "That's why used car dealers think they know vax's cause autism." Like you, they believe they know what anti-racism is, and how it's related to science. I have absolutely no idea why you apparently believe that I want to know more about you (to the extent and/or degree that I would search). If you knew anything at all about anti-racism, and how it's related to the scientific method --- we wouldn't even be having this conversation. Indeed, the likes of you have absolutely nothing to teach me --- relative to so-called "instructions" concerning authentic research, as it relates to ANTI-RACISM (nothing at all) --- zilch.
Dear Dr. Jussim & All,
I have something to add:
There’s a feature that allows you to listen to the article. Click on the link below. When the article comes up, to listen, click on this symbol ⧁, right above the title.
https://medium.com/@howardjeagle/research-b551399e889a
Dr Jussim, your paper seems to refer to figures that are out of order or aren’t in the paper. Perhaps the Appendix might be from a previous version of the paper?
https://x.com/CuroRelics/status/1861450054525194325
Thanks for flagging this, I sincerely appreciate it. We've since found several other minor errors or omissions (nothing that changes the overall patterns, but I would have preferred NCRI to have waited to be sure it was more polished, but so be it) and will upload an improved report over the next week or two.
@Lee Lee there has been a lot of ineffective organizational initiatives done over many years.
What’s baffling to me with this research is that people who consider themselves serious researchers use the concepts of two shallow, reductionist, infantilizing, and just bad popular books that met the zeitgeist with perfect storm timing as the core texts for their experimentation.
Were there no other controls or alternate approaches to DEI/Diversity Management/Inclusion that were used?
Or was it simply group identity-focused texts used to confirm a fixed hypothesis?
If the purpose is to make DEI in all of its breadth “wrong” or bad, this study reinforces it.
If it is to open up a heterodox view beyond reacting with the easiest point of make-wrong, it doesn’t occur as scholarship or a “study”to me. It occurs as politics.
For reasons that are not clear to me, Substack is glitching when I try to post my response, so I am going to try to break it up into two replies here. Not sure how many it will be, it seems to be glitching when I paste in more than a paragraph.
Hmmm. First, let's discuss your scientific criticisms, and leave your questions about our motives for the end of this reply.
1. Why did we use Kendi & DiAngelo? Because their ideas/rhetoric are all over DEI trainings and interventions. The topic network found just this as reported early in the paper.
2. As such, we think our findings are potentially vastly more informative and general than had we used the specific trainings used in a specific place. Had we, say, used the DEI materials used by Milwaukee's HR Dept, our studies would be justifiably dismissed as hopelessly trivial and uninformative about anything except Milwaukee's HR dept.
3. Alternative types of DEI. Most DEI training materials are not made publicly available. If you have access to any that are substantially different than the rhetoric used by K&D, please contact me. I would definitely consider using them in followup studies.
4. "Confirming a fixed hypothesis." Well, to test any hypothesis, you need to "fix it" -- that is, make it clear, specific, and empirically testable. So, sure, the hypothesis was fixed. What was not fixed were the results.
5. We definitely suspected that the rhetoric used by K & D, which are all over many DEI materials, demonizes "oppressors." That was the hypothesis. What we did not know was whether that would actually happen. That's why we conducted the studies -- to determine if such demonization would occur. Put differently, the hypothesis was "fixed" but the results were not knowable in advance.
6. It is, fortunately, still a (mostly) free country. Anyone is welcome to believe or state anything they like about our studies, including that they think the studies are crap and that crap studies should not be used to dismantle DEI programs. Actually, these are a set of preliminary studies, and even I do not think they imply any particular course of action other than ... they should inspire additional research on the upsides and downsides of different types of DEI programs/interventions.
However, if one's view is something like "we don't dismantle programs based on crappy research" I say, great! This implies that you think real social programs should be based on sound research. If so, I agree wholeheartedly. The thing is, to be coherent, you would then need to argue for dismantling DEI programs -- but not because of our studies. You would need to argue for dismantling DEI programs because they have been widely implemented based on **no scientific research at all" and, instead, based on the ideology and beliefs of advocates.
Much of what your paper cites is true. Not news but to let you know I read it thoroughly.
The priming in the experiments and conclusions based on it are expected.
And, they are likely to be replicable across larger cohorts if done with the same methods.
I have worked in the field for 20+ years and criticize it perhaps to the extent that one might label me a DEI naysayer.
Kendi and DiAngelo have become popular and equating them with DEI is easy because of their popularity. What they purport isn’t the work I know.
And, unfortunately without considering the tradeoffs associated with driving their ideological fixations forward, many organizations jumped onto this inevitably divisive and ineffective slow motion train wreck. A train wreck planted by many academic institutions, not by people who have been practicing in companies for any period of time.
Have there been people in the practice of advocating on behalf of some group identities for a while?Yes. They can have some success but not in isolation. If the advocacy leads to greater intergroup contact than the status quo, the network effects I’ve observed are largely positive.
As well, there are companies who have been building capabilities that help individuals and their organizations thrive and make their best contributions to the companies’ missions.
The thriving outcomes are not based on one or two shot training interventions. And, few are shared and likely won’t be shared externally. It’s not worth the distraction from the responses.
A lot of Kendi and DiAngelo-like trainings were a reaction to incomplete data about cops disproportionately killing black people . Also untrue, but perception trumps data when the data is loosely disseminated. The assumptions are what get us.
Of course, all have the right to publish anything they’d like. Your research is conformational (thus my using the language of “fixed hypothesis) in that as you cited Kalev and Dobbin, who have repeatedly documented most diversity interventions (especially mandatory training and grievance-based approaches) don’t work for the outcome of representation or promotion of less represented group identities.
Inclusion and diversity skills are not about getting more black or other group identities jobs and promotions. That might be one outcome to create more effective systems in the candidate/employee lifecycle but that isn’t the point—at least not in my work.
If you cannot manage the tensions and complexity of differences and similarities (aka diversity) managing complexity is at best difficult, at best many things will be delayed. And this goes well beyond race and gender as Roosevelt Thomas informed us on in 1991.
Bringing people together to solve complex challenges that require a diversity or expertise that can be accentuated by visible diversity, works well.
The last five years of training-by -fire short-term interventions (like Kendi-DiAngelo content) are suspect and should be stopped.
To make their work representative of the last 50 years of diversity and inclusion programming and impact, is not helpful.
Your thoughtful work and that of others who are questioning (or dunking on) DEI will also benefit from timing, similarly to that of Kendi and DiAngelo just with a different context.
Amri, thank you for your thoughtful comments and constructive criticism. No one should come away believing that our studies addressed all of DEI, let alone other programs for improving quality of work life in orgs, or combating real biases. We are currently in process of revising the report for all sorts of reasons, but I will strongly suggest we add, to the conclusion, a brief statement to the effect that we only studied certain types of social justice rhetoric and is irrelevant to any program that eschews such rhetoric.
Indeed, they definitely had "a fixed hypothesis," and indeed, this IS obviously about right wing politricks, rather than authetic scholarship.