---
title: "Psychology Studies Managers Quote: Only 36% Replicated"
description: When 100 psychology studies were rerun, significant results fell from 97% to 36%. Where Stanford Prison and the marshmallow test stand, and how HR should check.
image: https://nobusato.jp/hubfs/covers-2026-11/management-psych-experiments-replication-cover-en-2026-10-m.png
---

[Nobu Sato](https://nobusato.jp/en?hsLang=en)

[Profile](https://nobusato.jp/en/profile?hsLang=en) [Method](https://nobusato.jp/en/blog/method?hsLang=en) [Articles](https://nobusato.jp/en/blog?hsLang=en) [Glossary](https://nobusato.jp/en/blog/glossary?hsLang=en) [Contact](https://nobusato.jp/en/contact?hsLang=en)

[日本語](https://nobusato.jp/ja/blog/management-psych-experiments-replication)[English](https://nobusato.jp/en/blog/management-psych-experiments-replication)[中文](https://nobusato.jp/zh/blog/management-psych-experiments-replication)

Japan by the Numbers

In this article

1. [What did the big replication project find?](https://nobusato.jp/en/blog/management-psych-experiments-replication#s1)
2. [Why do effects shrink on the second try?](https://nobusato.jp/en/blog/management-psych-experiments-replication#s2)
3. [Where do the famous management stories stand now?](https://nobusato.jp/en/blog/management-psych-experiments-replication#s3)
4. [A filter before "a study says" becomes policy](https://nobusato.jp/en/blog/management-psych-experiments-replication#s4)
5. [Now apply it to my posts](https://nobusato.jp/en/blog/management-psych-experiments-replication#s5)

1. [Home](https://nobusato.jp/en?hsLang=en)
2. [Articles](https://nobusato.jp/en/blog?hsLang=en)
3. [Japan by the Numbers](https://nobusato.jp/en/blog/tag/japan-by-the-numbers)

# Psychology Studies Managers Quote: Only 36% Replicated

Rerun 100 studies and a third stay significant, with effects half the size

[Nobu Sato](https://nobusato.jp/en/profile?hsLang=en)8 min read

Contents5

1. [What did the big replication project find?](https://nobusato.jp/en/blog/management-psych-experiments-replication#s1)
2. [Why do effects shrink on the second try?](https://nobusato.jp/en/blog/management-psych-experiments-replication#s2)
3. [Where do the famous management stories stand now?](https://nobusato.jp/en/blog/management-psych-experiments-replication#s3)
4. [A filter before "a study says" becomes policy](https://nobusato.jp/en/blog/management-psych-experiments-replication#s4)
5. [Now apply it to my posts](https://nobusato.jp/en/blog/management-psych-experiments-replication#s5)

Many of the psychology studies managers quote don't hold up. When 100 published psychology studies were run again by independent teams, only 36% produced a significant result. The originals had claimed 97%. If a study is about to end up in your hiring rubric or your performance review form, you should know about that gap.

I bring this up because the studies managers love most are often the shakiest. Global training decks still open with the Stanford Prison Experiment. Assessment vendors in Tokyo and elsewhere still say "research shows." I lean on classic theories on this blog too, so at the end I put my own posts under the same light.

## What did the big replication project find?

The reference point is the Open Science Collaboration's "Estimating the reproducibility of psychological science," published in Science in 2015. It was a large, coordinated replication. Teams picked 100 experimental and correlational studies from the 2008 issues of three leading psychology journals, used the original materials where they could, and reran them with more statistical power. The project ran from November 2011 to December 2014.

97% of the original studies reported significant results (p \< .05). 36% of the replications did. Only 47% of original effect sizes fell inside the replication's 95% confidence interval, and the teams judged 39% of effects to have replicated. The average effect size, converted to a correlation r, fell from 0.403 to 0.197. Roughly half.

Rerun 100 psychology studies and a third stay significant, with effects half the size

Open Science Collaboration (2015), Science 349. 100 studies from three leading psychology journals, 2008 volumes, run again

**Share with a significant result**　p \< .05, 0 to 100%

- Original

  97%
- Replication

  36%

**Mean effect size r**　scale 0 to 0.5

- Original

  0.403
- Replication

  0.197

We trusted single significant results too much. Before a finding goes into policy, check whether it replicated.

Figure 1. When the Open Science Collaboration (2015) reran 100 studies from three psychology journals, the share of significant results fell from 97% to 36% and the mean effect size r from 0.403 to 0.197, so one significant result is a weak basis for HR policy.

Two cautions before you read too much into this. The 100 studies weren't a random sample of psychology. They came from three journals in one year, picked partly because they were feasible to rerun. The field mattered too. In the paper's own breakdown, 50% of cognitive psychology studies (21 of 42) replicated with a significant result, against 25% of social psychology studies (14 of 55). So "64% of psychology is false" is the wrong takeaway. The authors found that the strength of the original evidence predicted replication success better than who ran either study. People had simply trusted a single significant result too much.

## Why do effects shrink on the second try?

The main culprit is that journals favor significant results. A small study that gets a big effect by luck gets published. The ones that found nothing stay in the drawer. Run the study again and the luck washes out, so the effect looks smaller.

If you hire, you've seen this in people. The candidate who shone in one interview often doesn't shine the same way in the second. One strong signal usually comes back weaker.

## Where do the famous management stories stand now?

### The Stanford Prison Experiment

In 1971, 24 male students were split into guards and prisoners in a mock prison in a Stanford basement. The study was planned for two weeks and stopped after six days. It became the go-to story for "give ordinary people power and they turn cruel."

Thibault Le Texier, a French researcher, went through the archives and interviewed 15 participants. His 2018 book, Histoire d'un mensonge, and his 2019 paper "Debunking the Stanford Prison Experiment" in American Psychologist report that the guards were given precise instructions on how to treat the prisoners. That looks a lot more like people doing what the researchers wanted than like cruelty emerging on its own. Zimbardo and colleagues have disputed the critique. Still, as evidence of what people do when left alone with power, the study is badly weakened. The broader idea that situations shape behavior is a separate claim, and it survives.

### The marshmallow test

The popular version: four-year-olds who wait for the second treat do better in school years later. The follow-up behind it, Shoda, Mischel and Peake (1990, Developmental Psychology), tracked children from a preschool on the Stanford campus. Of 653 children, 185 could be followed up, and the SAT correlations in the key condition rested on about 35 kids.

Watts, Duncan and Quan (2018, Psychological Science) ran a conceptual replication. Same question, different and much larger data: 918 children from the US NICHD Study of Early Child Care and Youth Development. The link between waiting time and achievement at 15 was about half the size of the original. Controlling for family background, early cognitive ability and the home environment cut it by another two thirds. A lot of what looked like willpower was the household showing through.

### Ego depletion

"Willpower is a battery, so make big decisions in the morning" goes back to experiments by Baumeister and colleagues in 1998. Hagger and colleagues ran a preregistered multi-lab replication (2016, Perspectives on Psychological Science) across 23 labs and 2,141 participants. The effect was d = 0.04, with a 95% confidence interval from −0.07 to 0.15. Essentially zero. Some researchers argue the task they used was too weak, and the debate isn't settled. Either way, I wouldn't build a policy on it.

## A filter before "a study says" becomes policy

Before any claim gets near a selection process or a review form, I ask four questions.

Who was studied, and how many? Students, children and working adults are different populations, and 35 people is different from 3,000.

Has an independent team replicated it, ideally with a preregistered design?

How big is the effect once other factors are controlled? Significance alone tells you little.

Was it tested on people at work? None of the three studies above were.

If all four come back well, it can inform a system. Two or three, and it's a hypothesis: try it small and check it against your own data. None, and it's a story for a workshop opener. When a vendor tells your Japan team a tool is "scientifically proven," ask for the name of the paper first. The same habit applies to statistics that look official. [Japan's two overtime numbers](https://nobusato.jp/en/blog/japan-overtime-two-numbers?hsLang=en) differ by eight hours depending on who reports and who gets counted.

## Now apply it to my posts

This blog leans on classic theories, and some of them are contested. [Herzberg's two-factor theory](https://nobusato.jp/en/blog/herzberg-pay-removes-complaints?hsLang=en) comes from 1959 interviews with 203 engineers and accountants around Pittsburgh, who recalled times they felt especially good or bad about work. Critics since King in Psychological Bulletin (1970) have argued that the split between the two factors may come from that interview method itself, a point that post also raises.

Dunbar's number is estimated from primate brain size, and my [Dunbar post](https://nobusato.jp/en/blog/dunbar-numbers-redraw-the-org-chart?hsLang=en) notes the academic dispute. The Allen curve comes from observing R&D engineers in the 1970s, long before Slack. Ringelmann's rope-pulling data were published in 1913, and a later experiment by Ingham and colleagues (1974) suggested most of the lost effort was motivation rather than coordination.

I keep using them as drawing aids for org design. I don't treat them as laws, and I wouldn't feed any of their numbers straight into how someone is paid or rated. Read my posts with the same four questions. If one of them fails, I'd like to hear about it.

Sources

- [Open Science Collaboration (2015), Estimating the reproducibility of psychological science, Science 349, aac4716: significant results 97% to 36%, mean r 0.403 to 0.197](https://doi.org/10.1126/science.aac4716)
- [University of Edinburgh repository: full text of the same paper (47%, 39%, cognitive 50% vs social 25%)](https://www.pure.ed.ac.uk/ws/files/21474290/RPP.pdf)
- [Le Texier, T. (2019), Debunking the Stanford Prison Experiment, American Psychologist 74(7), 823–839: archives and interviews with 15 participants](https://doi.org/10.1037/amp0000401)
- [Haney, Banks & Zimbardo (1972), Interpersonal Dynamics in a Simulated Prison, technical report for the Office of Naval Research: 24 selected from 75 volunteers, ended after six days](https://apps.dtic.mil/sti/pdfs/AD0751041.pdf)
- [Watts, Duncan & Quan (2018), Revisiting the Marshmallow Test, Psychological Science: 918 children, about half the original correlation, two thirds smaller with controls](https://doi.org/10.1177/0956797618761661)
- [Hagger et al. (2016), A multilab preregistered replication of the ego-depletion effect, Perspectives on Psychological Science 11(4): 23 labs, 2,141 participants, d = 0.04](https://doi.org/10.1177/1745691616652873)
- [King, N. (1970), Clarification and evaluation of the two-factor theory of job satisfaction, Psychological Bulletin 74(1), 18–31](https://doi.org/10.1037/h0029444)
- [Britannica, Two-factor theory (secondary): Herzberg 1959 sample of 203 engineers and accountants](https://www.britannica.com/topic/two-factor-theory)
- [Ingham, Levinger, Graves & Peckham (1974), The Ringelmann effect: Studies of group size and group performance, Journal of Experimental Social Psychology 10(4), 371–384](https://doi.org/10.1016/0022-1031(74)90033-X)
- [Kravitz & Martin (1986), Ringelmann rediscovered: The original article, Journal of Personality and Social Psychology 50(5): on Ringelmann's 1913 paper](https://doi.org/10.1037/0022-3514.50.5.936)

For what survived the 2018 replication of the marshmallow test, and what it means for judging a candidate on one trait, see [The Marshmallow Test Failed. So Does Hiring on One Trait](https://nobusato.jp/en/blog/marshmallow-test-one-trait-hiring?hsLang=en).

Author

[Nobu Sato](https://nobusato.jp/en/profile?hsLang=en)

Launching in Japan, starting with people and systems. I write about building organizations, sales teams and technology organizations in Japan, in English, Japanese and Chinese.

[Profile](https://nobusato.jp/en/profile?hsLang=en)[Contact](https://nobusato.jp/en/contact?hsLang=en)

## Read next

1. 01[Does Monitoring Raise Performance? Data on 23,461 People](https://nobusato.jp/en/blog/monitoring-and-performance?hsLang=en)6 min
   
   Ravid et al. (2023) pooled 94 samples and 23,461 people: monitoring and performance correlate at about zero, while ...
2. 02[Muir's Super-Chicken Study: What the Papers Actually Say](https://nobusato.jp/en/blog/super-chickens-team-performance?hsLang=en)7 min
   
   Selecting hens on whole-cage results cut annual mortality from 68% to 8.8% (Muir 1996). What the papers show, where the ...
3. 03[Terracotta Army fingerprints: who did the work?](https://nobusato.jp/en/blog/terracotta-army-fingerprints?hsLang=en)7 min
   
   A 2026 study of 79 fingerprints on 36 Terracotta Army figures estimated 45% adult men, 33% women, 22% children. What it ...

[Column guide: Japan by the Numbers](https://nobusato.jp/en/blog/guide-japan-by-the-numbers?hsLang=en) · [Glossary](https://nobusato.jp/en/blog/glossary?hsLang=en) · [The method behind the articles](https://nobusato.jp/en/blog/method?hsLang=en)

[← All articles](https://nobusato.jp/en/blog?hsLang=en)

## Writing about launching in Japan, in three languages.

[Profile](https://nobusato.jp/en/profile?hsLang=en) [Get in touch](https://nobusato.jp/en/contact?hsLang=en)

Nobu Sato

Writing about launching in Japan, in three languages.

[info@nobusato.jp](mailto:info@nobusato.jp) [Substack](https://nobusato.substack.com)[Newsletter](https://www.linkedin.com/newsletters/the-japan-entry-memo-7519750783555518464/)[Medium](https://medium.com/@nobusato5888)[LinkedIn](https://www.linkedin.com/in/realnobusato/)[X](https://x.com/nobusato5888) [RSS](https://nobusato.jp/en/blog/rss.xml?hsLang=en)

[Profile](https://nobusato.jp/en/profile?hsLang=en)[Method](https://nobusato.jp/en/blog/method?hsLang=en)[Articles](https://nobusato.jp/en/blog?hsLang=en)[Glossary](https://nobusato.jp/en/blog/glossary?hsLang=en)[Contact](https://nobusato.jp/en/contact?hsLang=en)

Views here are my own and don’t represent any organization I work or have worked for.

© 2026 Nobu Sato. All rights reserved.·[Privacy and disclaimer](https://nobusato.jp/en/privacy?hsLang=en)

```json
{
  "@context" : "https://schema.org",
  "@type" : "BreadcrumbList",
  "itemListElement" : [ {
    "@type" : "ListItem",
    "item" : "https://nobusato.jp/en",
    "name" : "Home",
    "position" : 1
  }, {
    "@type" : "ListItem",
    "item" : "https://nobusato.jp/en/blog",
    "name" : "Articles",
    "position" : 2
  }, {
    "@type" : "ListItem",
    "item" : "https://nobusato.jp/en/blog/tag/japan-by-the-numbers",
    "name" : "Japan by the Numbers",
    "position" : 3
  }, {
    "@type" : "ListItem",
    "item" : "https://nobusato.jp/en/blog/management-psych-experiments-replication",
    "name" : "Psychology Studies Managers Quote: Only 36% Replicated",
    "position" : 4
  } ]
}
```

```json
{
  "@context" : "https://schema.org",
  "@id" : "https://nobusato.jp/en/blog/management-psych-experiments-replication",
  "@type" : "WebPage",
  "about" : {
    "@type" : "Thing",
    "name" : "Japan by the Numbers"
  },
  "author" : {
    "@id" : "https://nobusato.jp/#person"
  },
  "description" : "When 100 psychology studies were rerun, significant results fell from 97% to 36%. Where Stanford Prison and the marshmallow test stand, and how HR should check.",
  "inLanguage" : "en",
  "isPartOf" : {
    "@id" : "https://nobusato.jp/en/blog",
    "@type" : "Blog",
    "name" : "Articles"
  },
  "keywords" : "Japan by the Numbers",
  "name" : "Psychology Studies Managers Quote: Only 36% Replicated",
  "url" : "https://nobusato.jp/en/blog/management-psych-experiments-replication"
}
```

```json
{
  "@context" : "https://schema.org",
  "@id" : "https://nobusato.jp/#person",
  "@type" : "Person",
  "alternateName" : [ "佐藤伸明", "Nobu Sato", "左滕" ],
  "description" : "Japan market launches: organization, sales and technology",
  "knowsAbout" : [ "Japan market entry", "Organization design", "Talent acquisition", "Sales organization building", "Technology organization building", "Japanese labor law" ],
  "name" : "Nobu Sato",
  "sameAs" : [ "https://www.linkedin.com/in/realnobusato/", "https://x.com/nobusato5888" ],
  "url" : "https://nobusato.jp/en/profile"
}
```

```json
{
  "@context" : "https://schema.org",
  "@type" : "BlogPosting",
  "author" : {
    "@type" : "Person",
    "name" : "Nobu Sato",
    "url" : "https://nobusato.jp/en/blog/author/nobu-sato"
  },
  "dateModified" : "2026-10-24T14:23:28.384Z",
  "datePublished" : "2026-10-24T10:50:15.000Z",
  "headline" : "Psychology Studies Managers Quote: Only 36% Replicated",
  "image" : [ "https://nobusato.jp/hubfs/covers-2026-11/management-psych-experiments-replication-cover-en-2026-10-m.png" ],
  "mainEntityOfPage" : {
    "@id" : "https://nobusato.jp/en/blog/management-psych-experiments-replication",
    "@type" : "WebPage"
  },
  "publisher" : {
    "@type" : "Organization",
    "logo" : {
      "@type" : "ImageObject",
      "url" : "https://nobusato.jp/hubfs/seo/logo-nobusato-2026-10.png"
    },
    "name" : "nobusato.jp"
  }
}
```