Preamble

I blame/credit (depends on the beholder) ChatGPT4 for the title

More details here: https://chat.openai.com/c/c9c1fc78-e4d1-46d2-9138-abf27db0f560

The facial feedback hypothesis

• Facial feedback hypothesis (FFH): feedback from facial expression affects emotional experience and behavior (Buck 1980)
• Strack et al (Strack, Martin, and Stepper 1988) devised a study to overcome possible demand characteristics
• They found a “significant” difference (t(89) = 1.85, p = .03, one-tailed)

The facial feedback hypothesis failure to replicate

Wagenmakers et al. (2016) conducted a pre-registered replication across 17 labs using a similar procedure as Strack et al. (1988).

Meta-analysis from Coles et al.

• Coles, Larsen, and Lench (2019) Analyzed 286 ESs from 138 papers

• Found that the FFH effect was small, but significant

• No evidence of publication bias (at least for emotional experience)

  

The Many Smiles collaboration

• The replication from Wagenmakers et al. (2016) did not rule out if the theory is falsified, or just an operationalization

  • Some authors suggest that FFH requires the display of genuine exhibits of emotions

• The meta-analysis from Coles, Larsen, and Lench (2019) might

  • have been under-powered for detecting publication bias

  • have been biased by low-quality studies

Adversarial collaboration led by Nicholas Coles aimed to :

Specify when effect should most reliably emerge. Determine best way to test those beliefs. Design and execute an international multi-lab experiment

The Many Smiles study: how did I get there?

Things (often) happen by chance but networking increases chance (tip for ECRs!)

• When discussing the study I proposed the use of linear models, and running a power analysis by simulation

  • I thus condemned myself to cross a valley of teaRs :-)

  

The Many Smiles study: research design 1

Can happy facial poses initiate or only modulate feelings of happiness?

The Many Smiles study: research design 2

Do facial poses only influence happiness if they resemble a natural expression?

The Many Smiles study: research design 3

Do facial poses only influence happiness if they resemble a natural expression?

The Many Smiles study: research design 4

Do facial poses only influence happiness if they resemble a natural expression?

The Many Smiles study: procedure

• The DV consisted of averaging the responses to the four happiness-related items on a 7-point Likert-type format.

The Many Smiles study: power analysis assumptions

• Power: 1 -\(\beta\) = .95

• Simulation based on Gelman and Hill (2006), Chapter 20.

  Assumptions:

• Fixed effects Effects, ESs the two pilot studies (N = 206), but only for the Emotion (Happy > Neutral), Presence of the Positive stimulus (Present > Absent), and their interaction

• Random Effects

  • participant-level intercepts from the pilot study
  • lab-level slopes from the \(\tau^2\) observed in Wagenmakers et al. (2016)

Results:

• N = 1,584, increased to N = 2,281 considering an exclusion rate = 44%

The Many Smiles study: power analysis in R 1

#---ESTIMATING PARAMETERS FROM THE PILOT STUDY
df <- read.csv("ManySmiles_clean_long.csv", stringsAsFactors = T) 
df$ID <- as.factor(df$exper_ssid_var) #ID as factor
df$z_happiness <- scale(df$happiness) #standardize the dependent variable
df$trial <- relevel(df$trial, ref="neutr") #neutral as reference
options('contrasts'=c("contr.sum", "contr.poly")) # set contrast sum
m1 <- lmer(z_happiness ~ condition*trial*study + (1|ID), data = df) # fit the model

bPose.pilot <- -fixef(m1)['trial1']*2 #contr sum happy > neutr
bEmoStim.pilot <- - fixef(m1)['study1']*2 #contr present > absent
bPoseEmoStim.pilot <- -fixef(m1)['trial1:study1']*4 # (happy_present > neutr_present) > (happy_absent > neutr_absent)

The Many Smiles study: power analysis in R 2

#---ESTIMATE RANDOM INTERCEPTS FOR THE HYPOTHESIZED EFFECTS FROM THE META-ANALYSIS

VarCorr(m1) #vcov is for the fixed, VarCorr for the Random

##  Groups   Name        Std.Dev.
##  ID       (Intercept) 0.69751 
##  Residual             0.60283

sigma.alphaSub.pilot <- as.data.frame(VarCorr(m1))[1, 'sdcor']
sigma.y.pilot <- as.data.frame(VarCorr(m1))[2, 'sdcor']

#---ESTIMATE RANDOM SLOPES FOR THE HYPOTHESIZED EFFECTS FROM THE META-ANALYSIS
se_meta = .068 #from Wagenmakers et al. 2016
sd_meta = se_meta*sqrt(17)

Data analysis 1

Frequentist analyses, Led by Nick Coles:

• Encountered convergence issues in lme4 because, eventually, the slopes varied very little across labs

# fit model
models[["primary"]][["prereg"]] <-
  lmer(happiness ~ trial * condition * image +
         (1 | lab) + (1 | ResponseId) + 
         (0 + trial | lab) +
         (0 + condition | lab)  +
         (0 + image | lab) +
         (0 + trial : image | lab) +
         (0 + trial : condition | lab)  + 
         (0 + condition : image | lab) + 
         (0 + trial : condition : image | lab),
       data = DF.l.inc)

Data analysis 2

Bayesian Analyses, led by MTL and Marco Marozzi (Ca’ Foscari University of Venice)

• Set prior and iterations as medium (\(r scale = \frac{1}{2}\)), wide (\(r scale = \frac{\sqrt(2)}{2}\)), or ultra-wide (\(r scale = 1\))).

pr <- "medium"
#pr <- "wide"
#pr <- "ultra-wide"
p.1 <- 
  lmBF(happiness ~ trial +
         ResponseId + lab + lab:trial, 
       whichRandom = c("ResponseId", "lab", "lab:trial"),
       rscaleFixed = pr,
       data = DF.l.inc,
       iterations = it)


p.0 <- 
  lmBF(happiness ~ 1 +
         ResponseId + lab + lab:trial, 
       whichRandom = c("ResponseId", "lab", "lab:trial"), 
       rscaleFixed = pr,
       data = DF.l.inc,
       iterations = it)

p <- p.1 / p.0 # 102.6253 ±9.38%

Results

Very strong support for the main effect of FFH \(BFs\_{10} > 61\) Very strong support for interaction with the task \(BFs_{10} > 34\) The effect of the FFH in the mimicry and voluntary action conditions: \(BFs_{10} > 25.2\) There was moderate support for \(H_0\) in the pen-in-the mouth \(BFs_{01}> 9\)

Post publication (peer) review

Comment on a FB page:

“As a statistics student myself, I was surprised to see that multilevel regression was used (requiring a continuous scale DV) while ordinal multilevel regression should have been used, but it is Nature so statistical standard are low (one of the winning journals regarding publication of inflated effects).”

• Even though the DV was somewhat skewed, we decided for linear models because analyzing the pilot with non parametric approaches led to the very same results

• It is true that Liddell and Kruschke (2018) provided this suggestion, but it would not be very feasible to estimate ordinal models having the sum of four 7-point Likert-type items (meaning 23 thresholds to be estimated)

• “All models are wrong, but some are useful” George Box

The meaning of hypothesis testing 1

• Large-scale pre-registered replication is a good Popperian way to put a theory under severe test, but “it is not the theory alone that is subject to empirical test, but the theory in conjunction with all background assumptions [including assumptions about the operationalizations and that measurements] are required for the deduction and interpretation of a given observation ((Duhem 1908), (Quine 1953), cited in (Gawronski and Bodenhausen 2015), Chapter 1)”

  

The meaning of hypothesis testing 2

• Popper demarcation problem is a valuable guidance, but a simplistic version of falsificationism might be untenable (even for Popper, actually).

• Imre Lakatos, Methodology of Scientific Research Programmes: degenerating vs. progressive programs.

• Feyerabend has some point in questioning that is the criterion under which we should call a research program as degenerating vs. progressive

  

Conclusions

• Multi-lab studies are hard, but they are worth the hassle!

• A priori power analyses are not obvious when you want to take random structures into account

• Beware of the priors

• Not sure if post-publication peer review is appealing, but can complement traditional peer review

• There are so many ways to analyze the data, be pragmatic and/or go for multi-verse analyses

• Replication is not an on/off issue

  • be skeptic of the superkeptics

Thanks!

Nicholas Coles
Marco Marozzi
Fernando Marmolejo-Ramos