RESUMEN

Multiverse analyses-the systematic examination of the effects of decisions that researchers can take over the course of a research project-became more common in recent psychophysiological research. However, multiverse analyses in psychophysiology almost exclusively focus on methodological and statistical decisions that can have a considerable impact on the findings. The role of the conceptual multiverse regarding theory-related research decisions is largely ignored. We argue that the choice of a theory that guides hypotheses, study design, measurement methods, and statistical analyses is the first plane of the psychophysiological multiverse. Depending on the chosen theoretical framework, researchers will choose different methods, and statistical analyses will emphasize specific aspects. We illustrate this process with a research example studying the effects of task difficulty manipulations on cardiovascular effects reflecting effort. We argue in favor of an approach that explicitly acknowledges the various theoretical accounts that can constitute the background of a study and demonstrate how a comparative analytical approach can provide a comprehensive multiverse without increasing type I error due to mere exploration.

RESUMEN

OBJECTIVE: Serving as a channel for communication with locked-in patients or control of prostheses, sensorimotor brain-computer interfaces (BCIs) decode imaginary movements from the recorded activity of the user's brain. However, many individuals remain unable to control the BCI, and the underlying mechanisms are unclear. The user's BCI performance was previously shown to correlate with the resting-state signal-to-noise ratio (SNR) of the mu rhythm and the phase synchronization (PS) of the mu rhythm between sensorimotor areas. Yet, these predictors of performance were primarily evaluated in a single BCI session, while the longitudinal aspect remains rather uninvestigated. In addition, different analysis pipelines were used to estimate PS in source space, potentially hindering the reproducibility of the results. APPROACH: To systematically address these issues, we performed an extensive validation of the relationship between pre-stimulus SNR, PS, and session-wise BCI performance using a publicly available dataset of 62 human participants performing up to 11 sessions of BCI training. We performed the analysis in sensor space using the surface Laplacian and in source space by combining 24 processing pipelines in a multiverse analysis. This way, we could investigate how robust the observed effects were to the selection of the pipeline. MAIN RESULTS: Our results show that SNR had both between- and within-subject effects on BCI performance for the majority of the pipelines. In contrast, the effect of PS on BCI performance was less robust to the selection of the pipeline and became non-significant after controlling for SNR. SIGNIFICANCE: Taken together, our results demonstrate that changes in neuronal connectivity within the sensorimotor system are not critical for learning to control a BCI, and interventions that increase the SNR of the mu rhythm might lead to improvements in the user's BCI performance.

RESUMEN

The large number of different analytical choices used by researchers is partly responsible for the challenge of replication in neuroimaging studies. For an exhaustive robustness analysis, knowledge of the full space of analytical options is essential. We conducted a systematic literature review to identify the analytical decisions in functional neuroimaging data preprocessing and analysis in the emerging field of cognitive network neuroscience. We found 61 different steps, with 17 of them having debatable parameter choices. Scrubbing, global signal regression, and spatial smoothing are among the controversial steps. There is no standardized order in which different steps are applied, and the parameter settings within several steps vary widely across studies. By aggregating the pipelines across studies, we propose three taxonomic levels to categorize analytical choices: 1) inclusion or exclusion of specific steps, 2) parameter tuning within steps, and 3) distinct sequencing of steps. We have developed a decision support application with high educational value called METEOR to facilitate access to the data in order to design well-informed robustness (multiverse) analysis.

Asunto(s)

RESUMEN

Performance monitoring has been widely studied during different forced-choice response tasks. Participants typically show longer response times (RTs) and increased accuracy following errors, but there are inconsistencies regarding the connection between error-related event-related brain potentials (ERPs) and behavior, such as RT and accuracy. The specific task in any given study could contribute to these inconsistencies, as different tasks may require distinct cognitive processes that impact ERP-behavior relationships. The present study sought to determine whether task moderates ERP-behavior relationships and whether these relationships are robustly observed when tasks and stimuli are treated as random effects. ERPs and behavioral indices (RTs and accuracy) recorded during flanker, Stroop, and Go/Nogo tasks from 180 people demonstrated a task-specific effect on ERP-behavior relationships, such that larger previous-trial error-related negativity (ERN) predicted longer RTs and greater likelihood of a correct response on subsequent trials during flanker and Stroop tasks but not during Go/Nogo task. Additionally, larger previous-trial error positivity (Pe) predicted faster RTs and smaller variances of RTs on subsequent trials for Stroop and Go/Nogo tasks but not for flanker task. When tasks and stimuli were treated as random effects, ERP-behavior relationships were not observed. These findings support the need to consider the task used for recording performance monitoring measures when interpreting results across studies.

Asunto(s)

RESUMEN

This study tackles the Garden of Forking Paths, as a challenge for replicability and reproducibility of ERP studies. Here, we applied a multiverse analysis to a sample ERP N400 dataset, donated by an independent research team. We analyzed this dataset using 14 pipelines selected to showcase the full range of methodological variability found in the N400 literature using systematic review approach. The selected pipelines were compared in depth by looking into statistical test outcomes, descriptive statistics, effect size, data quality, and statistical power. In this way we provide a worked example of how analytic flexibility can impact results in research fields with high dimensionality such as ERP, when analyzed using standard null-hypothesis significance testing. Out of the methodological decisions that were varied, high-pass filter cut-off, artifact removal method, baseline duration, reference, measurement latency and locations, and amplitude measure (peak vs. mean) were all shown to affect at least some of the study outcome measures. Low-pass filtering was the only step which did not notably influence any of these measures. This study shows that even some of the seemingly minor procedural deviations can influence the conclusions of an ERP study. We demonstrate the power of multiverse analysis in both identifying the most reliable effects in a given study, and for providing insights into consequences of methodological decisions.

RESUMEN

Even when guided by strong theories and sound methods, researchers must often choose a singular course of action from multiple viable alternatives. Regardless of the choice, it, along with all other choices made during the research process, individually and collectively affects study results, often in unpredictable ways. The inability to disentangle how much of an observed effect is attributable to the phenomenon of interest, and how much is attributable to what have come to be known as researcher degrees of freedom (RDF), slows theoretical progress and stymies practical implementation. However, if one could examine the results from a particular set of RDF (known as a universe) against a systematically and comprehensively determined background of alternative viable universes (known as a multiverse), then the effects of RDF can be directly examined to provide greater context and clarity to future researchers, and greater confidence in the recommendations to practitioners. This tutorial demonstrates a means to map result variability directly and efficiently, and empirically investigate RDF impact on conclusions via multiverse analysis. Using the R package multiverse, we outline best practices in planning, executing and interpreting of multiverse analyses.

RESUMEN

When analyzing data, researchers make some choices that are either arbitrary, based on subjective beliefs about the data-generating process, or for which equally justifiable alternative choices could have been made. This wide range of data-analytic choices can be abused and has been one of the underlying causes of the replication crisis in several fields. Recently, the introduction of multiverse analysis provides researchers with a method to evaluate the stability of the results across reasonable choices that could be made when analyzing data. Multiverse analysis is confined to a descriptive role, lacking a proper and comprehensive inferential procedure. Recently, specification curve analysis adds an inferential procedure to multiverse analysis, but this approach is limited to simple cases related to the linear model, and only allows researchers to infer whether at least one specification rejects the null hypothesis, but not which specifications should be selected. In this paper, we present a Post-selection Inference approach to Multiverse Analysis (PIMA) which is a flexible and general inferential approach that considers for all possible models, i.e., the multiverse of reasonable analyses. The approach allows for a wide range of data specifications (i.e., preprocessing) and any generalized linear model; it allows testing the null hypothesis that a given predictor is not associated with the outcome, by combining information from all reasonable models of multiverse analysis, and provides strong control of the family-wise error rate allowing researchers to claim that the null hypothesis can be rejected for any specification that shows a significant effect. The inferential proposal is based on a conditional resampling procedure. We formally prove that the Type I error rate is controlled, and compute the statistical power of the test through a simulation study. Finally, we apply the PIMA procedure to the analysis of a real dataset on the self-reported hesitancy for the COronaVIrus Disease 2019 (COVID-19) vaccine before and after the 2020 lockdown in Italy. We conclude with practical recommendations to be considered when implementing the proposed procedure.

Asunto(s)

RESUMEN

OBJECTIVES: To present an application of specification curve analysis-a novel analytic method that involves defining and implementing all plausible and valid analytic approaches for addressing a research question-to nutritional epidemiology. STUDY DESIGN AND SETTING: We reviewed all observational studies addressing the effect of red meat on all-cause mortality, sourced from a published systematic review, and documented variations in analytic methods (eg, choice of model, covariates, etc.). We enumerated all defensible combinations of analytic choices to produce a comprehensive list of all the ways in which the data may reasonably be analyzed. We applied specification curve analysis to data from National Health and Nutrition Examination Survey 2007 to 2014 to investigate the effect of unprocessed red meat on all-cause mortality. The specification curve analysis used a random sample of all reasonable analytic specifications we sourced from primary studies. RESULTS: Among 15 publications reporting on 24 cohorts included in the systematic review on red meat and all-cause mortality, we identified 70 unique analytic methods, each including different analytic models, covariates, and operationalizations of red meat (eg, continuous vs quantiles). We applied specification curve analysis to National Health and Nutrition Examination Survey, including 10,661 participants. Our specification curve analysis included 1208 unique analytic specifications, of which 435 (36.0%) yielded a hazard ratio equal to or more than 1 for the effect of red meat on all-cause mortality and 773 (64.0%) less than 1. The specification curve analysis yielded a median hazard ratio of 0.94 (interquartile range: 0.83-1.05). Forty-eight specifications (3.97%) were statistically significant, 40 of which indicated unprocessed red meat to reduce all-cause mortality and eight of which indicated red meat to increase mortality. CONCLUSION: We show that the application of specification curve analysis to nutritional epidemiology is feasible and presents an innovative solution to analytic flexibility.

Asunto(s)

RESUMEN

Why is the empirical evidence for birth-order effects on human psychology so inconsistent? In contrast to the influential view that competitive dynamics among siblings permanently shape a person's personality, we find evidence that these effects are limited to the family environment. We tested this context-specific learning hypothesis in the domain of risk taking, using two large survey datasets from Germany (SOEP, n = 19,994) and the United States (NLSCYA, n = 29,627) to examine birth-order effects on risk-taking propensity across a wide age range. Specification-curve analyses of a sample of 49,621 observations showed that birth-order effects are prevalent in children aged 10-13 years, but that they decline with age and disappear by middle adulthood. The methodological approach shows the effect is robust. We thus replicate and extend previous work in which we showed no birth-order effects on adult risk taking. We conclude that family dynamics cause birth-order effects on risk taking but that these effects fade as siblings transition out of the home.

Asunto(s)

RESUMEN

It is well established that P3 latencies increase with age. Investigating these age-related differences requires numerous methodological decisions, resulting in pipelines of great variation. The aim of the present work was to investigate the effects of different analytical pipelines on the age-related differences in P3 latencies in real data. Therefore, we conducted an explorative multiverse study and varied the low-pass filter (4 Hz, 8 Hz, 16 Hz, 32 Hz, and no filter), the latency type (area vs. peak), the level of event-related potential analysis (single participant vs. jackknifing), and the extraction method (manual vs. automated). Thirty young (18-21 years) and 30 old (50-60 years) participants completed three tasks (Nback task, Switching task, Flanker task), while an EEG was recorded. The results show that different analysis strategies can have a tremendous impact on the detection and magnitude of the age effect, with effect sizes ranging from 0% to 88% explained variance. Likewise, regarding the psychometric properties of P3 latencies, we found that the reliabilities fluctuated between rtt = .20 and 1.00, while the homogeneities ranged from rh = -.12 to .90. Based on predefined criteria, we found that the most effective pipelines relied on a manual extraction based on a single participant's data. For peak latencies, manual extraction performed well for all filters except for 4 Hz, while for area latencies, filters above 8 Hz produced desirable results. Furthermore, our findings add to the evidence that jackknifing combined with peak latencies can lead to inconclusive results.

Asunto(s)

RESUMEN

Reaction time (RT) data are often pre-processed before analysis by rejecting outliers and errors and aggregating the data. In stimulus-response compatibility paradigms such as the approach-avoidance task (AAT), researchers often decide how to pre-process the data without an empirical basis, leading to the use of methods that may harm data quality. To provide this empirical basis, we investigated how different pre-processing methods affect the reliability and validity of the AAT. Our literature review revealed 108 unique pre-processing pipelines among 163 examined studies. Using empirical datasets, we found that validity and reliability were negatively affected by retaining error trials, by replacing error RTs with the mean RT plus a penalty, and by retaining outliers. In the relevant-feature AAT, bias scores were more reliable and valid if computed with D-scores; medians were less reliable and more unpredictable, while means were also less valid. Simulations revealed bias scores were likely to be less accurate if computed by contrasting a single aggregate of all compatible conditions with that of all incompatible conditions, rather than by contrasting separate averages per condition. We also found that multilevel model random effects were less reliable, valid, and stable, arguing against their use as bias scores. We call upon the field to drop these suboptimal practices to improve the psychometric properties of the AAT. We also call for similar investigations in related RT-based bias measures such as the implicit association task, as their commonly accepted pre-processing practices involve many of the aforementioned discouraged methods. HIGHLIGHTS: • Rejecting RTs deviating more than 2 or 3 SD from the mean gives more reliable and valid results than other outlier rejection methods in empirical data • Removing error trials gives more reliable and valid results than retaining them or replacing them with the block mean and an added penalty • Double-difference scores are more reliable than compatibility scores under most circumstances • More reliable and valid results are obtained both in simulated and real data by using double-difference D-scores, which are obtained by dividing a participant's double mean difference score by the SD of their RTs.

Asunto(s)

RESUMEN

In behavioral, cognitive, and social sciences, reaction time measures are an important source of information. However, analyses on reaction time data are affected by researchers' analytical choices and the order in which these choices are applied. The results of a systematic literature review, presented in this paper, revealed that the justification for and order in which analytical choices are conducted are rarely reported, leading to difficulty in reproducing results and interpreting mixed findings. To address this methodological shortcoming, we created a checklist on reporting reaction time pre-processing to make these decisions more explicit, improve transparency, and thus, promote best practices within the field. The importance of the pre-processing checklist was additionally supported by an expert consensus survey and a multiverse analysis. Consequently, we appeal for maximal transparency on all methods applied and offer a checklist to improve replicability and reproducibility of studies that use reaction time measures.

Asunto(s)

RESUMEN

In recent years, there has been a growing interest in utilizing symptom-network models to study psychopathology and relevant risk factors, such as cognitive and physical health. Various methodological approaches can be employed by researchers analyzing cross-sectional and panel data (i.e., several time points over an extended period). This paper provides an overview of some commonly used analytical tools, including moderated network models, network comparison tests, cross-lagged network analysis, and panel graphical vector-autoregression (VAR) models. Using an easily accessible dataset (easySHARE), this study demonstrates the use of different analytical approaches when investigating (a) the association between mental health and cognitive functioning, and (b) the role of chronic disease in mediating or moderating this association. This multiverse analysis showcases both converging and diverging evidence from different analytical avenues. These findings underscore the importance of multiverse investigations to increase transparency and communicate the extent to which conclusions depend on analytical choices.

RESUMEN

What factors underlie immigrants' social distance towards natives? Previous studies found that immigrants who perceive themselves as rejected by natives express more negative intergroup attitudes towards natives. Another line of research found that contingent on their origin country, immigrants face different degrees of social distance from natives. In this study, we employ an intergroup threat approach to integrate these separate research strands. The theoretical model we develop predicts that immigrants from groups that receive greater social distance from natives will perceive more personal discrimination, which, as a mediating mechanism, will be associated with greater social distance towards natives. Empirically, we draw on a cross-sectional probability sample of 1789 immigrants from 38 origin countries living in Germany (i.e., a comparative origin design). The results of multilevel mediation analyses prove consistent with our theoretical expectations, which points to the benefits of examining social distance among immigrants and natives in conjunction.

Asunto(s)

RESUMEN

Manhood is a precarious state that men seek to prove through the performance of masculine behaviors-including, at times, acts of aggression. Although correlational work has demonstrated a link between chronic masculine insecurity and political aggression (i.e., support for policies and candidates that communicate toughness and strength), experimental work on the topic is sparse. Existing studies also provide little insight into which men-liberal or conservative-are most likely to display increased political aggression after threats to their masculinity. The present work thus examines the effects of masculinity threat on liberal and conservative men's tendency toward political aggression. We exposed liberal and conservative men to various masculinity threats, providing them with feminine feedback about their personality traits (Experiment 1), having them paint their nails (Experiment 2), and leading them to believe that they were physically weak (Experiment 3). Across experiments, and contrary to our initial expectations, threat increased liberal-but not conservative-men's preference for a wide range of aggressive political policies and behaviors (e.g., the death penalty, bombing an enemy country). Integrative data analysis (IDA) reveals significant heterogeneity in the influence of different threats on liberal men's political aggression, the most effective of which was intimations of physical weakness. A multiverse analysis suggests that these findings are robust across a range of reasonable data-treatment and modeling choices. Possible sources of liberal men's heightened responsiveness to manhood threats are discussed. Supplementary Information: The online version contains supplementary material available at 10.1007/s11199-023-01349-x.

RESUMEN

Persistent fear is a cardinal feature of posttraumatic stress disorder (PTSD), and deficient fear extinction retention is a proposed illness mechanism and target of exposure-based therapy. However, evidence for deficient fear extinction in PTSD has been mixed using laboratory paradigms, which may relate to underidentified methodological variation across studies. We reviewed the literature to identify parameters that differ across studies of fear extinction retention in PTSD. We then performed Multiverse Analysis in a new sample, to quantify the impact of those methodological parameters on statistical findings. In 25 PTSD patients (15 female) and 36 trauma-exposed non-PTSD controls (TENC) (20 female), we recorded skin conductance response (SCR) during fear acquisition and extinction learning (day 1) and extinction recall (day 2). A first Multiverse Analysis examined the effects of methodological parameters identified by the literature review on comparisons of SCR-based fear extinction retention in PTSD versus TENC. A second Multiverse Analysis examined the effects of those methodological parameters on comparisons of SCR to a danger cue (CS+) versus safety cue (CS-) during fear acquisition. Both the literature review and the Multiverse Analysis yielded inconsistent findings for fear extinction retention in PTSD versus TENC, and most analyses found no statistically significant group difference. By contrast, significantly elevated SCR to CS+ versus CS- was consistently found across all analyses in the literature review and the Multiverse Analysis of new data. We discuss methodological parameters that may most contribute to inconsistent findings of fear extinction retention deficit in PTSD and implications for future clinical research.

Asunto(s)

RESUMEN

Rapidly evaluating our environment's beneficial and detrimental features is critical for our successful functioning. A classic paradigm used to investigate such fast and automatic evaluations is the affective priming (AP) paradigm, where participants classify valenced target stimuli (e.g., words) as good or bad while ignoring the valenced primes (e.g., words). We investigate the differential impact that verbs and adjectives used as primes and targets have on the AP paradigm. Based on earlier work on the Linguistic Category Model, we expect AP effect to be modulated by non-evaluative properties of the word stimuli, such as the linguistic category (e.g., if the prime is an adjective and the target is a verb versus the reverse). A reduction in the magnitude of the priming effect was predicted for adjective-verb prime-target pairs compared to verb-adjective prime-target pairs. Moreover, we implemented a modified crowdsourcing of statistical analyses implementing independently three different statistical approaches. Deriving our conclusions on the converging/diverging evidence provided by the different approaches, we show a clear deductive/inductive asymmetry in AP paradigm (exp. 1), that this asymmetry does not require a focus on the evaluative dimension to emerge (exp. 2) and that the semantic-based asymmetry weakly extends to valence (exp. 3).

Asunto(s)

RESUMEN

Although research interest in leader narcissism has been on the rise over the past few years, prior literature has predominantly discussed leader narcissism from a leader-centric perspective. In this article, we provide a relational-based perspective of leader narcissism by examining the interaction between follower personality traits and leader narcissism on follower engagement in an online context. We combine a machine learning (ML) approach and multiverse analysis to predict the personality traits of a large sample of leaders and engaged followers across 18 created multiverses and analyze hypothesized interactions using multilevel regressions, also accounting for leader gender moderation effects. We find that the interaction between leader narcissism and follower agreeableness and follower neuroticism positively predicts follower engagement, whereas the interaction between leader narcissism and follower openness negatively predicts follower engagement. In addition, we find that leader gender plays an important moderating role. Limitations and implications are discussed.

Asunto(s)

RESUMEN

Combining datasets in an integrative data analysis (IDA) requires researchers to make a number of decisions about how best to harmonize item responses across datasets. This entails two sets of steps: logical harmonization, which involves combining items which appear similar across datasets, and analytic harmonization, which involves using psychometric models to find and account for cross-study differences in measurement. Embedded in logical and analytic harmonization are many decisions, from deciding whether items can be combined prima facie to how best to find covariate effects on specific items. Researchers may not have specific hypotheses about these decisions, and each individual choice may seem arbitrary, but the cumulative effects of these decisions are unknown. In the current study, we conducted an IDA of the relationship between alcohol use and delinquency using three datasets (total N = 2245). For analytic harmonization, we used moderated nonlinear factor analysis (MNLFA) to generate factor scores for delinquency. We conducted both logical and analytic harmonization 72 times, each time making a different set of decisions. We assessed the cumulative influence of these decisions on MNLFA parameter estimates, factor scores, and estimates of the relationship between delinquency and alcohol use. There were differences across paths in MNLFA parameter estimates, but fewer differences in estimates of factor scores and regression parameters linking delinquency to alcohol use. These results suggest that factor scores may be relatively robust to subtly different decisions in data harmonization, and measurement model parameters are less so.

Asunto(s)

RESUMEN

In neuropsychological research, there are a near limitless number of different approaches researchers can choose when designing studies. Here we showcase the multiverse/specification curve technique to establish the robustness of analytical pathways choices within classic psychometric test validation in an example test of executive function. We examined the impact of choices regarding sample groups, sample sizes, test metrics, and covariate inclusions on convergent validation correlations between tests of executive function. Data were available for 87 neurologically healthy adults and 117 stroke survivors, and a total of 2,220 different analyses were run in a multiverse analysis. We found that the type of sample group, sample size, and test metric used for analyses affected validation outcomes. Covariate inclusion choices did not affect the observed coefficients in our analyses. The present analysis demonstrates the importance of carefully justifying every aspect of a psychometric test validation study a priori with theoretical and statistical factors in mind. It is essential to thoroughly consider the purpose and use of a new tool when designing validation studies.

Asunto(s)