RESUMEN

We conducted a systematic review of conference papers in social psychology at two large psychology conferences in Japan: the Japanese Psychological Association and the Japanese Society for Social Psychology. The conference papers were effectively not subjected to peer review; hence, they were suitable for testing if psychologists selectively reported statistically significant findings without pressure from journal editors and reviewers. We investigated the distributions of z-values converted from the p-values reported in the articles presented at the 2013 and 2018 conferences. The z-curve analyses suggest the existence of selective reporting by the authors in 2013. The expected discovery rate (EDR) was much lower than the observed discovery rate (ODR; 7% vs. 76%, respectively), and the 95% confidence interval (CI) did not include the ODR. However, this does not mean that the set of studies completely lacked evidential value. The expected replication rate (ERR) was 31%; this is significantly higher than 5%, which was expected under the null hypothesis of no effect. Changes were observed between 2013 and 2018. The ERR increased (31% to 44%), and the EDR almost doubled (7% to 13%). However, the estimation of the maximum false discovery rate (FDR; 68% in 2013 and 35% in 2018) suggested that a substantial proportion of the reported findings were false positives. Overall, while social psychologists in Japan engaged in selective reporting, this does not mean that the entire field was covered with false positives. In addition, slight signs of improvement were observed in how they reported their findings. Still, the evidential value of the target studies was weak, even in 2018, allowing for no optimism.

Asunto(s)

RESUMEN

In many research fields, the widespread use of questionable research practices has jeopardized the credibility of scientific results. One of the most prominent questionable research practices is p-hacking. Typically, p-hacking is defined as a compound of strategies targeted at rendering non-significant hypothesis testing results significant. However, a comprehensive overview of these p-hacking strategies is missing, and current meta-scientific research often ignores the heterogeneity of strategies. Here, we compile a list of 12 p-hacking strategies based on an extensive literature review, identify factors that control their level of severity, and demonstrate their impact on false-positive rates using simulation studies. We also use our simulation results to evaluate several approaches that have been proposed to mitigate the influence of questionable research practices. Our results show that investigating p-hacking at the level of strategies can provide a better understanding of the process of p-hacking, as well as a broader basis for developing effective countermeasures. By making our analyses available through a Shiny app and R package, we facilitate future meta-scientific research aimed at investigating the ramifications of p-hacking across multiple strategies, and we hope to start a broader discussion about different manifestations of p-hacking in practice.

RESUMEN

OBJECTIVE: A recent meta-review of attentional bias research in eating disorders suggests that meta-analyses and systematic reviews include many low-quality and underpowered studies (Stott et al., 2021). As such, we examined whether published research examining the link between attentional bias, using the emotional Stroop task, and eating disorders among women with eating disorders has evidential value (ruling out selective reporting of a statistically significant effect) using a p-curve analysis. A p-curve analysis plots statisticall significant p-values onto a curve from .01 to .05 to examine its distribution. We hypothesized that the p-curve would be flat, indicating no true effect. METHOD: The hypothesis, database search strategy, and data analytic approach were pre-registered. The inclusion criteria were reports that compared control and eating disorder groups, reported inferential statistics, and that used body shape/weight or general threat target words. RESULTS: Fifty published reports were included in the p-curve analyses. Unexpectedly, the half and full p-curves were significantly right-skewed, indicating evidential value. However, the results were not robust to the exclusion of the seven lowest p-values and on average, reports were underpowered. There were also 18 reports with null results (they had a p-value greater than .05), which precluded their inclusion in the p-curve analyses. DISCUSSION: The findings suggest that most of the evidence from research examining attentional biases using the emotional Stroop task among women with an eating disorder or with elevated eating disorder symptoms is underpowered and so should be interpreted with considerable caution. PUBLIC SIGNIFICANCE: Concerns have been raised about the low quality of research examining attentional biases among women with eating disorders using the emotional Stroop task. In the current research, we observed that the evidential value of primary research reporting differences between women with and without eating disorders was equivocal and had low statistical power. These results can guide researchers towards conducting more rigorous research on attentional biases among people with eating disorders.

OBJETIVO: Una meta-revisión reciente de la investigación del sesgo atencional en los trastornos de la conducta alimentaria sugiere que los metaanálisis y las revisiones sistemáticas incluyen muchos estudios de baja calidad y con poco poder estadístico (Stott et al., 2021). Como tal, examinamos si la investigación publicada que examina el vínculo entre el sesgo atencional, el uso de la tarea de Stroop emocional y los trastornos de la conducta alimentaria entre mujeres con trastornos alimentarios tiene valor de evidencia (descartando el informe selectivo de un efecto estadísticamente significativo) mediante un análisis de curva-p. Un análisis de curva-p traza p valores significativos en una curva de .01 a .05 para examinar su distribución. Planteamos la hipótesis de que la curva-p sería plana, lo que indica que no hay un efecto verdadero. MÉTODO: La hipótesis, la estrategia de búsqueda en la base de datos y el enfoque analítico de datos fueron pre-registrados. Los criterios de inclusión fueron informes que compararon grupos control y de trastornos de la conducta alimentaria, informaron estadísticas inferenciales y que utilizaron palabras clave de figura/peso corporal o de amenaza general. RESULTADOS: Cincuenta estudios publicados fueron incluidos en los análisis de la curva-p. Inesperadamente, las curvas-p medias y completas estaban significativamente sesgadas a la derecha, lo que indica un valor de evidencia. Sin embargo, los resultados no fueron robustos para la exclusión de los siete p valores más bajos y, en promedio, los reportes tuvieron poca potencia. También hubo 18 reportes con resultados nulos (tenían un p valor mayor que .05), lo que impidió su inclusión en los análisis de la curva-p. DISCUSIÓN: Los hallazgos sugieren que la mayor parte de la evidencia de la investigación que examina los sesgos atencionales utilizando la tarea de Stroop emocional entre las mujeres con un trastorno de la conducta alimentaria o con síntomas elevados del trastorno alimentario tiene poco poder y, por lo tanto, debe interpretarse con mucha precaución.

Asunto(s)

RESUMEN

According to the embodied cognition view, comprehending action-related language requires the participation of sensorimotor processes. A now sizeable literature has tested this proposal by stimulating (with TMS or tDCS) motor brain areas during the comprehension of action language. To assess the evidential value of this body of research, we exhaustively searched the literature and submitted the relevant studies (N = 43) to p-curve analysis. While most published studies concluded in support of the embodiment hypothesis, our results suggest that we cannot yet assert beyond reasonable doubt that they explore real effects. We also found that these studies are quite underpowered (estimated power < 30%), which means that a large percentage of them would not replicate if repeated identically. Additional tests for excess significance show signs of publication bias within this literature. In sum, extant brain stimulation studies testing the grounding of action language in the motor cortex do not stand on solid ground. We provide recommendations that will be important for future research on this topic.

Asunto(s)

RESUMEN

Systemic efforts have been employed to improve the reproducibility of published findings in psychology. To date, little research has been conducted evaluating how well these efforts work. In an effort to bridge this gap, the current study looked at journal submission requirements intended to encourage authors to engage in best practices for facilitating reproducible science and offers preliminary evidence for their potential efficacy. We calculated reproducibility indices (p-curves) for three randomly selected empirical studies published in each of 23 psychology journals in 2019 and correlated quantitative results from those analyzes with the number of submission requirements for each journal that intended to ensure compliance with best reporting practices. Results indicated a greater number of submission requirements at a given outlet was associated with indices indicating greater likelihood of reproducibility of findings. We frame findings as impetus for future, more extensive, research to identify causal links between submission requirements and reproducibility.

Asunto(s)

RESUMEN

Cognitive bias modification (CBM), which retrains implicit biases towards unhealthy foods, has been proposed as a promising adjunct to improve the efficacy of weight loss interventions. We conducted a systematic review of research on three CBM approaches (i.e., cue-specific inhibitory control, approach bias modification, and attentional bias modification) for reducing unhealthy eating biases and behavior. We performed a p-curve analysis to determine the evidential value of this research; this method is optimally suited to clarify whether published results reflect true effects or false positives due to publication and reporting biases. When considering all CBM approaches, our results suggested that the findings of CBM trials targeting unhealthy eating are unlikely to be false positives. However, only research on attentional bias modification reached acceptable levels of power. These results suggest that CBM interventions may be an effective strategy to enhance the efficacy of weight loss interventions. However, there is room for improvement in the methodological standards of this area of research, especially increasing the statistical power can help to fully clarify the clinical potential of CBM, and determine the role of potential moderators.

Asunto(s)

RESUMEN

Although there is an abundance of evidence linking the function of the hypothalamic-pituitary-adrenal (HPA) axis to adverse early-life experiences, the precise nature of the association remains unclear. Some evidence suggests early-life adversity leads to cortisol hyper-reactivity, while other evidence suggests adversity leads to cortisol hypo-reactivity. Here, we distinguish between trauma and adversity, and use p-curves to interrogate the conflicting literature. In Study 1, trauma was operationalized according to DSM-5 criteria; the p-curve analysis included 68 articles and revealed that the literature reporting associations between trauma and blunted cortisol reactivity contains evidential value. Study 2 examined the relationship between adversity and cortisol reactivity. Thirty articles were included in the analysis, and p-curve demonstrated that adversity is related to heightened cortisol reactivity. These results support an inverted U-shaped function relating severity of adversity and cortisol reactivity, and underscore the importance of distinguishing between "trauma" and "adversity".

Asunto(s)

RESUMEN

Patients' presurgical psychological profiles have been posited to predict pain and function following arthroplastic surgery of the hip and knee. Nevertheless, findings are conflicting, and this may be rooted in biased reporting that makes the determination of evidential value difficult. This ambiguity may have negative consequences for researchers and governmental agencies, as these rely on findings to accurately reflect reality. P-Curve analyses were used to establish the presence of evidential value and selective reporting in a sample of studies examining the effect of presurgical psychological predictors on outcomes following knee and hip arthroplastic surgery. A systematic search of the literature revealed 26 relevant studies. The examined sets of studies indicate that there is evidential value for the effect of depression on pain intensity and function, anxiety on pain intensity and function, pain catastrophizing on pain intensity, as well as the combined effects of all psychological predictors on pain intensity and function. The presence of evidential value was inconclusive for the effect of optimism on pain intensity. There were no signs that any results were influenced by biased reporting. The results highlight the importance of patients' psychological profiles in predicting surgical outcomes, which represent a promising avenue for future treatment approaches. PERSPECTIVE: The effects of P-hacking are difficult to detect and might be at the root of conflicting findings pertaining to the predictive properties of presurgical psychological variables on postsurgical outcomes. P-Curve analysis allows the determination of evidential value underlying these relationships and detection of P-hacking to ensure that findings are not the result of selective reporting.

Asunto(s)

RESUMEN

The overall reliability or evidentiary value of any body of literature is established in part by ruling out publication bias for any observed effects. Questionable research practices have potentially undermined the evidentiary value of commonly used research paradigms in psychological science. Subsequently, the evidentiary value of these common methodologies remains uncertain. To quantify the severity of these issues in the literature, we selected the Taylor Aggression Paradigm (TAP) as a case study and submitted 170 hypothesis tests spanning over 50 years of research to a preregistered p-curve analysis. The TAP literature (N = 24,685) demonstrated significant evidentiary value but yielded a small average effect size (d = 0.29) and inadequate power (38%). The main effects demonstrated greater evidentiary value, power, and effect sizes than interactions. Studies that tested the effects of measured traits did not differ in evidentiary value or power to those that tested the effects of experimentally manipulated states. Exploratory analyses revealed that evidentiary value, statistical power, and effect sizes have improved over time. We provide recommendations for researchers who seek to maximize the evidentiary value of their psychological measures.

Asunto(s)

RESUMEN

The concept of home numeracy has been defined as parent-child interactions with numerical content. This concept started to receive increasing attention since the last decade. Most of the studies indicated that the more parents and their children engage in numerical experiences, the better children perform in mathematical tasks. However, there are also contrasting results indicating that home numeracy does not play a role or that there is a negative association between the parent-child interactions and children's mathematics performance. To shed light on these discrepancies, a systematic review searching for available articles examining the relationship between home numeracy and mathematical skills was conducted. Thirty-seven articles were retained and a p-curve analysis showed a true positive association between home numeracy and children's mathematical skills. A more qualitative investigation of the articles revealed five common findings: (1) Advanced home numeracy interactions but not basic ones are associated with children's mathematical skills. (2) Most participants in the studies were mothers, however, when both parents participated and were compared, only mothers' reports of formal home numeracy activities (i.e., explicit numeracy teaching) were linked to children's mathematical skill. (3) Formal home numeracy activities have been investigated more commonly than informal home numeracy activities (i.e., implicit numeracy teaching). (4) The number of studies that have used questionnaires to assess home numeracy is larger compared with the ones that have used observations. (5) The majority of the studies measured children's mathematical skills with comprehensive tests that index mathematical ability with one composite score rather than with specific numerical tasks. These five common findings might explain the contradictory results regarding the relationship between home numeracy and mathematical skills. Therefore, more research is necessary to draw quantitative conclusions about these five points.

RESUMEN

Performance-monitoring event-related brain potentials (ERPs), such as the error-related negativity (ERN) and reward positivity (RewP), are advocated as biomarkers of depression symptoms and risk. However, a recent meta-analysis indicated effect size heterogeneity in the ERN and RewP literatures. Hence, advocating these ERPs as biomarkers of depression might be premature or possibly misguided due to the selective reporting of significant analyses on the part of researchers (e.g., p-hacking or omission of non-significant findings). The present study quantified the degree of selective reporting and the evidential value for a true relationship between depression and ERN and RewP using a p-curve analysis. We predicted that the ERN and RewP literatures would fail to show evidential value for a relationship between each ERP and depression. Contrary to expectations, both literatures showed evidential value, albeit weak. The statistical power of the included ERN studies was between 20% and 25%, and the statistical power of the RewP was around 27%. Taken together, these findings provide support for a relationship between these ERPs and depression, which strengthens claims that these ERPs represent candidate biomarkers of depression symptoms and risk. In light of the evidence for these relationships being weak, some recommendations moving forward include conducting a priori power analyses, increasing sample sizes to improve statistical power, assessing the internal consistency of ERP scores, and carefully planning statistical approaches to maximize power.

Asunto(s)

RESUMEN

Whether fear conditioning can take place without contingency awareness is a topic of continuing debate and conflicting findings have been reported in the literature. This systematic review provides a critical assessment of the available evidence. Specifically, a search was conducted to identify articles reporting fear conditioning studies in which the contingency between conditioned stimuli (CS) and the unconditioned stimulus (US) was masked, and in which CS-US contingency awareness was assessed. A systematic assessment of the methodological quality of the included studies (k = 41) indicated that most studies suffered from methodological limitations (i.e., poor masking procedures, poor awareness measures, researcher degrees of freedom, and trial-order effects), and that higher quality predicted lower odds of studies concluding in favor of contingency unaware fear conditioning. Furthermore, meta-analytic moderation analyses indicated no evidence for a specific set of conditions under which contingency unaware fear conditioning can be observed. Finally, funnel plot asymmetry and p-curve analysis indicated evidence for publication bias. We conclude that there is no convincing evidence for contingency unaware fear conditioning.

Asunto(s)

RESUMEN

Research demonstrating that infants discriminate between small (e.g., 1 vs. 3 dots) and large numerosities (e.g., 8 vs. 16 dots) is central to theories concerning the origins of human numerical abilities. To date, there has been no quantitative meta-analysis of the infant numerical competency data. Here, we quantitatively synthesize the evidential value of the available literature on infant numerosity discrimination using a meta-analytic tool called p-curve. In p-curve the distribution of available p-values is analyzed to determine whether the published literature examining particular hypotheses contains evidential value. p-curves demonstrated evidential value for the hypotheses that infants can discriminate between both small and large unimodal and cross-modal numerosities. However, the analyses also revealed that the published data on infants' ability to discriminate between large numerosities is less robust and statistically powered than the data on their ability to discriminate small numerosities. We argue there is a need for adequately powered replication studies to enable stronger inferences in order to use infant data to ground theories concerning the ontogenesis of numerical cognition.

Asunto(s)

RESUMEN

Psychologists have debated whether the right fusiform face area's (FFA) responses are domain specific to faces, or domain general for certain object categories that we have visual expertise with. This latter domain general expertise account has been criticised for basing its assumptions upon studies that suffer from small participant numbers, small effects, and statistically significant p-values that are close to .05. An additional criticism is that these findings are difficult to replicate. A modern reader familiar with the replication crisis may therefore question whether the FFA's expertise effect is real. The p-curve is a relatively new form of meta-analysis that enables researchers to identify whether there is evidential value for any given effect in the literature. We put the literature to the test by running p-curve analyses on all published expertise studies. Contrary to aforementioned criticisms, our meta-analyses confirm the right FFA's expertise effect is based upon evidential value. We therefore review the broader literature to address additional criticisms of the expertise account and propose ways to improve replicability.

Asunto(s)

RESUMEN

Selective reporting (i.e., only reporting significant findings as opposed to all analyses or results) is a questionable research practice that undermines the integrity of published research. Psychophysiology research may be susceptible to selective reporting, given the high number of decision points and methodological complexity in analyses of psychophysiology data. We aimed to assess the presence of selective reporting and evidential value (i.e., that significant results are due to true underlying effects) in recent and past psychophysiological research by utilizing p-curve analyses. Study protocols and methods were pre-registered on the Open Science Framework (OSF). P-values and the associated test statistics were extracted from articles in the most recent issue (as of January 2018) and 10-year previous counterpart issue of three major psychophysiology journals: Psychophysiology, International Journal of Psychophysiology, and Journal of Psychophysiology. Using the p-curve application, 10 primary p-curves were conducted: all recent articles, all past articles, recent articles split by journal, past articles split by journal, recent cognitive electrophysiology articles, and past cognitive electrophysiology articles. Evidential value and generally adequate average power (≥78% average power) were present in all p-curves, except those that only included articles from the Journal of Psychophysiology because of the small number of articles published in the journal. Findings provide some positive news and indicate that, generally, results were not selectively reported, and selective reporting may not be a primary issue for this sample of psychophysiological research. Future p-curve analyses examining sub-disciplines of psychophysiology are recommended.

Asunto(s)

RESUMEN

The Violation-of-Expectation paradigm is a widespread paradigm in infancy research that relies on looking time as an index of surprise. This methodological review aims to increase the reliability of future VoE studies by proposing to standardize reporting practices in this literature. I review 15 VoE studies on false-belief reasoning, which used a variety of experimental parameters. An analysis of the distribution of p-values across experiments suggests an absence of p-hacking. However, there are potential concerns with the accuracy of their measures of infants' attention, as well as with the lack of a consensus on the parameters that should be used to set up VoE studies. I propose that (i) future VoE studies ought to report not only looking times (as a measure of attention) but also looking-away times (as an equally important measure of distraction); (ii) VoE studies must offer theoretical justification for the parameters they use, and (iii) when parameters are selected through piloting, pilot data must be reported in order to understand how parameters were selected. Future VoE studies ought to maximize the accuracy of their measures of infants' attention since the reliability of their results and the validity of their conclusions both depend on the accuracy of their measures.

Asunto(s)

RESUMEN

BACKGROUND: P-curve has been proposed as a statistical test of evidential value. The distributions of sets of statistically significant p-values are tested for skewness. P-curves of true effects are right-skewed, with greater density at lower p-values than higher p-values. Analyses of null effects result in a flat or left-skewed distribution. The accuracy of p-curve has not been tested using published research analyses of a null effect. We examined whether p-curve accurately rejects a set of significant p-values obtained for a nonexistent effect. METHODS: Homeopathic ultramolecular dilutions are medicinal preparations with active substances diluted beyond Avogadro's number. Such dilute mixtures are unlikely to contain a single molecule of an active substance. We tested whether p-curve accurately rejects the evidential value of significant results obtained in placebo-controlled clinical trials of homeopathic ultramolecular dilutions. RESULTS: P-curve accurately rejected the evidential value of significant results obtained in placebo-controlled clinical trials of ultramolecular dilutions. Robustness testing using alternate p-values yielded similar results. CONCLUSION: Our results suggest that p-curve can accurately detect when sets of statistically significant results lack evidential value.

RESUMEN

In line with the goal of limiting health risk behaviors in adolescence, a growing literature investigates whether individual differences in functional brain responses can be related to vulnerability to engage in risky decision-making. We review this body of work, investigate when and in what way findings converge, and provide best practice recommendations. We identified 23 studies that examined individual differences in brain responsivity and adolescent risk taking. Findings varied widely in terms of the neural regions identified as relating to risky behavior. This heterogeneity is likely due to the abundance of approaches used to assess risk taking, and to the disparity of fMRI tasks. Indeed, brain-behavior correlations were typically found in regions showing a main effect of task. However, results from a test of publication bias suggested that region of interest approaches lacked evidential value. The findings suggest that neural factors differentiating riskier teens are not localized to a single region. Therefore, approaches that utilize data from the entire brain, particularly in predictive analyses, may yield more reliable and applicable results. We discuss several decision points that researchers should consider when designing a study, and emphasize the importance of precise research questions that move beyond a general desire to address adolescent risk taking.

Asunto(s)

RESUMEN

Alarm pheromones are widely used in the animal kingdom. Notably, there are 26 published studies (N = 1652) highlighting a human capacity to communicate fear, stress, and anxiety via body odor from one person (66% males) to another (69% females). The question is whether the findings of this literature reflect a true effect, and what the average effect size is. These questions were answered by combining traditional meta-analysis with novel meta-analytical tools, p-curve analysis and p-uniform-techniques that could indicate whether findings are likely to reflect a true effect based on the distribution of P-values. A traditional random-effects meta-analysis yielded a small-to-moderate effect size (Hedges' g: 0.36, 95% CI: 0.31-0.41), p-curve analysis showed evidence diagnostic of a true effect (ps < 0.0001), and there was no evidence for publication bias. This meta-analysis did not assess the internal validity of the current studies; yet, the combined results illustrate the statistical robustness of a field in human olfaction dealing with the human capacity to communicate certain emotions (fear, stress, anxiety) via body odor.

Asunto(s)

RESUMEN

A substantial number of studies have been published over the last decade, claiming that transcranial direct current stimulation (tDCS) can influence performance on cognitive tasks. However, there is some skepticism regarding the efficacy of tDCS, and evidence from meta-analyses are mixed. One major weakness of these meta-analyses is that they only examine outcomes in published studies. Given biases towards publishing positive results in the scientific literature, there may be a substantial "file-drawer" of unpublished negative results in the tDCS literature. Furthermore, multiple researcher degrees of freedom can also inflate published p-values. Recently, Simonsohn, Nelson and Simmons (2014) created a novel meta-analytic tool that examines the distribution of significant p-values in a literature, and compares it to expected distributions with different effect sizes. Using this tool, one can assess whether the selected studies have evidential value. Therefore, we examined a random selection of studies that used tDCS to alter performance on cognitive tasks, and tDCS studies on working memory in a recently published meta-analysis (Mancuso et al., 2016). Using a p-curve analysis, we found no evidence that the tDCS studies had evidential value (33% power or greater), with the estimate of statistical power of these studies being approximately 14% for the cognitive studies, and 5% (what would be expected from randomly generated data) for the working memory studies. It is likely that previous tDCS studies are substantially underpowered, and we provide suggestions for future research to increase the evidential value of future tDCS studies.

Asunto(s)