---
title: "Psychology Studies Managers Quote: Only 36% Replicated"
description: When 100 psychology studies were rerun, significant results fell from 97% to 36%. Where Stanford Prison and the marshmallow test stand, and how HR should check.
image: https://nobusato.jp/hubfs/management-psych-experiments-replication-en.png
---

[Nobu Sato](https://nobusato.jp/en?hsLang=en)

[Home](https://nobusato.jp/en?hsLang=en) [Profile](https://nobusato.jp/en/profile?hsLang=en) [Articles](https://nobusato.jp/en/blog?hsLang=en) [Contact](https://nobusato.jp/en/contact?hsLang=en)

[日本語](https://nobusato.jp/ja/blog/management-psych-experiments-replication)[English](https://nobusato.jp/en/blog/management-psych-experiments-replication)[中文](https://nobusato.jp/zh/blog/management-psych-experiments-replication)

[Japan by the Numbers](https://nobusato.jp/en/blog/tag/japan-by-the-numbers)

# Psychology Studies Managers Quote: Only 36% Replicated

[Nobu Sato](https://nobusato.jp/en/profile?hsLang=en)

![Bar chart: significant results fell from 97% in original psychology studies to 36% in replications](https://nobusato.jp/hs-fs/hubfs/management-psych-experiments-replication-en.png?width=1200&height=630&name=management-psych-experiments-replication-en.png)

When 100 published psychology studies were run again by independent teams, only 36% produced a significant result. The originals had claimed 97%. Before a study ends up in your hiring rubric or your performance review form, that gap is worth knowing.

I bring this up because the studies managers love most are often the shakiest. Global training decks still open with the Stanford Prison Experiment. Assessment vendors in Tokyo and elsewhere still say "research shows." And I use classic theories on this blog too, so I'm putting my own posts under the same light at the end.

## What did the big replication project find?

The reference point is the Open Science Collaboration's "Estimating the reproducibility of psychological science," published in Science in 2015. It was a large coordinated replication: teams selected 100 experimental and correlational studies from three leading psychology journals' 2008 issues, used the original materials where they could, and ran them with more statistical power. The project ran from November 2011 to December 2014.

97% of the original studies reported significant results (p \< .05). 36% of the replications did. Only 47% of original effect sizes fell inside the replication's 95% confidence interval, and the teams judged 39% of effects to have replicated. The average effect size, converted to a correlation r, fell from 0.403 to 0.197. Roughly half.

Share of studies with a significant result (%, 0 to 100) Original studies 97% Replications 36% Mean effect size r (0 to 0.5) Original studies 0.403 Replications 0.197 Top panel scaled 0 to 100%, bottom panel r 0 to 0.5. Dotted line marks each panel's maximum. Source: Open Science Collaboration (2015), Science 349, aac4716

**Fig.1**In the Open Science Collaboration's 2015 replication of 100 psychology studies, the share of significant results fell from 97% to 36%, and the mean effect size fell from r = 0.403 to r = 0.197.

Two cautions on reading this. The 100 studies weren't a random sample of psychology. They came from three journals in one year, picked partly because they were feasible to rerun. The field mattered too. In the paper's own breakdown, 50% of cognitive psychology studies (21 of 42) replicated with a significant result, against 25% of social psychology studies (14 of 55). So "64% of psychology is false" is the wrong takeaway. The authors found that replication success was predicted more by the strength of the original evidence than by who ran either study. A single significant result was simply trusted too much.

## Why do effects shrink on the second try?

Mostly because journals favor significant results. A small study that gets a big effect by luck gets published. The ones that found nothing stay in the drawer. Run the study again and the luck washes out, so the effect looks smaller.

If you hire, you've seen this pattern in people. The candidate who shone in one interview often doesn't shine the same way in the second. One strong signal usually comes back weaker the next time.

## Where do the famous management stories stand now?

### The Stanford Prison Experiment

In 1971, 24 male students were split into guards and prisoners in a mock prison in a Stanford basement. The study was planned for two weeks and stopped after six days. It became the go-to story for "give ordinary people power and they turn cruel."

Thibault Le Texier, a French researcher, went through the archives and interviewed 15 participants. His 2018 book, Histoire d'un mensonge, and his 2019 paper "Debunking the Stanford Prison Experiment" in American Psychologist report that the guards received precise instructions on how to treat prisoners. That looks much more like people doing what the researchers wanted than cruelty emerging on its own. Zimbardo and colleagues have disputed the critique. Still, as evidence of what people do when left alone with power, the study is badly weakened. The broader idea that situations shape behavior is a separate claim and survives.

### The marshmallow test

The popular version: four-year-olds who wait for the second treat do better in school years later. The follow-up behind it, Shoda, Mischel and Peake (1990, Developmental Psychology), tracked children from a preschool on the Stanford campus. Of 653 children, 185 could be followed up, and the SAT correlations in the key condition rested on about 35 kids.

Watts, Duncan and Quan (2018, Psychological Science) ran a conceptual replication: same question, different and much larger data, 918 children from the US NICHD Study of Early Child Care and Youth Development. The link between waiting time and achievement at 15 was about half the size of the original. Controlling for family background, early cognitive ability and the home environment cut it by another two thirds. A lot of what looked like willpower was the household showing through.

### Ego depletion

"Willpower is a battery, so make big decisions in the morning" traces back to Baumeister and colleagues' experiments in 1998. Hagger and colleagues ran a preregistered multi-lab replication (2016, Perspectives on Psychological Science) across 23 labs and 2,141 participants. The effect was d = 0.04, with a 95% confidence interval from −0.07 to 0.15. That's essentially zero. Some researchers argue the task they used was too weak, and the debate isn't settled. It still isn't strong enough to build a policy on.

## A filter before "a study says" becomes policy

I run any claim through four questions before it gets near a selection process or a review form.

Who was studied, and how many? Students, children and working adults are different populations, and 35 people is different from 3,000.

Has an independent team replicated it, ideally with a preregistered design?

How big is the effect once other factors are controlled? Significance alone tells you little.

Was it tested on people at work? None of the three studies above were.

All four answered well, and it can inform a system. Two or three, and it's a hypothesis: try it small and check against your own data. None, and it's a story for a workshop opener. When a vendor tells your Japan team a tool is "scientifically proven," ask for the paper's name before anything else. The same habit applies to statistics that look official: [Japan's two overtime numbers](https://nobusato.jp/en/blog/japan-overtime-two-numbers?hsLang=en) differ by eight hours depending on who reports and who gets counted.

## Apply it to my posts too

This blog leans on classic theories, and some of them are contested. [Herzberg's two-factor theory](https://nobusato.jp/en/blog/herzberg-pay-removes-complaints?hsLang=en) comes from 1959 interviews with 203 engineers and accountants around Pittsburgh, who recalled times they felt especially good or bad about work. Critics since King in Psychological Bulletin (1970) have argued that the split between the two factors may come from that interview method itself, a point that post also raises.

Dunbar's number is estimated from primate brain size, and my [Dunbar post](https://nobusato.jp/en/blog/dunbar-numbers-redraw-the-org-chart?hsLang=en) notes the academic dispute. The Allen curve comes from observing R&D engineers in the 1970s, long before Slack. Ringelmann's rope-pulling data were published in 1913, and a later experiment by Ingham and colleagues (1974) suggested most of the lost effort was motivation rather than coordination.

I keep using them as drawing aids for org design. I don't treat them as laws, and I wouldn't put any of their numbers straight into how someone is paid or rated. Read my posts with the same four questions. If one of them fails, I'd rather hear it.

Sources

- [Open Science Collaboration (2015), Estimating the reproducibility of psychological science, Science 349, aac4716: significant results 97% to 36%, mean r 0.403 to 0.197](https://doi.org/10.1126/science.aac4716)
- [University of Edinburgh repository: full text of the same paper (47%, 39%, cognitive 50% vs social 25%)](https://www.pure.ed.ac.uk/ws/files/21474290/RPP.pdf)
- [Le Texier, T. (2019), Debunking the Stanford Prison Experiment, American Psychologist 74(7), 823–839: archives and interviews with 15 participants](https://doi.org/10.1037/amp0000401)
- [Haney, Banks & Zimbardo (1972), Interpersonal Dynamics in a Simulated Prison, technical report for the Office of Naval Research: 24 selected from 75 volunteers, ended after six days](https://apps.dtic.mil/sti/pdfs/AD0751041.pdf)
- [Watts, Duncan & Quan (2018), Revisiting the Marshmallow Test, Psychological Science: 918 children, about half the original correlation, two thirds smaller with controls](https://doi.org/10.1177/0956797618761661)
- [Hagger et al. (2016), A multilab preregistered replication of the ego-depletion effect, Perspectives on Psychological Science 11(4): 23 labs, 2,141 participants, d = 0.04](https://doi.org/10.1177/1745691616652873)
- [King, N. (1970), Clarification and evaluation of the two-factor theory of job satisfaction, Psychological Bulletin 74(1), 18–31](https://doi.org/10.1037/h0029444)
- [Britannica, Two-factor theory (secondary): Herzberg 1959 sample of 203 engineers and accountants](https://www.britannica.com/topic/two-factor-theory)
- [Ingham, Levinger, Graves & Peckham (1974), The Ringelmann effect: Studies of group size and group performance, Journal of Experimental Social Psychology 10(4), 371–384](https://doi.org/10.1016/0022-1031(74)90033-X)
- [Kravitz & Martin (1986), Ringelmann rediscovered: The original article, Journal of Personality and Social Psychology 50(5): on Ringelmann's 1913 paper](https://doi.org/10.1037/0022-3514.50.5.936)

---

Shorter notes that don't make it into a full article go on LinkedIn. [Follow me there](https://www.linkedin.com/in/realnobusato/).

## More in this column

- ### [Japan by the Numbers: Check Who Was Counted](https://nobusato.jp/en/blog/guide-japan-by-the-numbers?hsLang=en)
  
  Most quoted Japan statistics are correct but count a different group than you assume. A guide to labor, pay, customer ...
- ### [10.23 Million Want to Change Jobs in Japan. 3.3 Million Did.](https://nobusato.jp/en/blog/willing-vs-actual-job-changers-japan?hsLang=en)
  
  Japan's Labour Force Survey: 10.23M people wanted to change jobs in 2025, a record since 2013, yet only 3.3M moved and ...
- ### [Terracotta Army fingerprints: who did the work?](https://nobusato.jp/en/blog/terracotta-army-fingerprints?hsLang=en)
  
  A 2026 study of 79 fingerprints on 36 Terracotta Army figures estimated 45% adult men, 33% women, 22% children. What it ...

[Column guide: Japan by the Numbers](https://nobusato.jp/en/blog/guide-japan-by-the-numbers?hsLang=en) · [Glossary](https://nobusato.jp/en/blog/glossary?hsLang=en)

[← All articles](https://nobusato.jp/en/blog?hsLang=en)

## Writing about launching in Japan, in three languages.

[Profile](https://nobusato.jp/en/profile?hsLang=en) [Get in touch](https://nobusato.jp/en/contact?hsLang=en)

Nobu Sato

Writing about launching in Japan, in three languages.

[info@nobusato.jp](mailto:info@nobusato.jp) [LinkedIn](https://www.linkedin.com/in/realnobusato/) [X](https://x.com/nobusato5888)

© 2026 Nobu Sato. All rights reserved.

```json
{
  "@context" : "https://schema.org",
  "@type" : "BreadcrumbList",
  "itemListElement" : [ {
    "@type" : "ListItem",
    "item" : "https://nobusato.jp/en",
    "name" : "Home",
    "position" : 1
  }, {
    "@type" : "ListItem",
    "item" : "https://nobusato.jp/en/blog",
    "name" : "Articles",
    "position" : 2
  }, {
    "@type" : "ListItem",
    "item" : "https://nobusato.jp/en/blog/management-psych-experiments-replication",
    "name" : "Psychology Studies Managers Quote: Only 36% Replicated",
    "position" : 3
  } ]
}
```

```json
{
  "@context" : "https://schema.org",
  "@id" : "https://nobusato.jp/en/blog/management-psych-experiments-replication",
  "@type" : "WebPage",
  "author" : {
    "@id" : "https://nobusato.jp/#person"
  },
  "inLanguage" : "en",
  "isPartOf" : {
    "@id" : "https://nobusato.jp/en/blog",
    "@type" : "Blog",
    "name" : "Articles"
  },
  "name" : "Psychology Studies Managers Quote: Only 36% Replicated",
  "url" : "https://nobusato.jp/en/blog/management-psych-experiments-replication"
}
```

```json
{
  "@context" : "https://schema.org",
  "@id" : "https://nobusato.jp/#person",
  "@type" : "Person",
  "name" : "Nobu Sato",
  "sameAs" : [ "https://www.linkedin.com/in/realnobusato/", "https://x.com/nobusato5888" ],
  "url" : "https://nobusato.jp/en/profile"
}
```

```json
{
  "@context" : "https://schema.org",
  "@type" : "BlogPosting",
  "author" : {
    "@type" : "Person",
    "name" : "Nobu Sato",
    "url" : "https://nobusato.jp/en/blog/author/nobu-sato"
  },
  "dateModified" : "2026-10-24T10:50:15.292Z",
  "datePublished" : "2026-10-24T10:50:15.000Z",
  "headline" : "Psychology Studies Managers Quote: Only 36% Replicated",
  "image" : [ "https://nobusato.jp/hubfs/management-psych-experiments-replication-en.png" ],
  "mainEntityOfPage" : {
    "@id" : "https://nobusato.jp/en/blog/management-psych-experiments-replication",
    "@type" : "WebPage"
  },
  "publisher" : {
    "@type" : "Organization",
    "logo" : {
      "@type" : "ImageObject",
      "url" : "https://nobusato.jp/hubfs/nobusatojp-1.jpg"
    },
    "name" : "nobusato.jp"
  }
}
```