RESUMEN

Even when guided by strong theories and sound methods, researchers must often choose a singular course of action from multiple viable alternatives. Regardless of the choice, it, along with all other choices made during the research process, individually and collectively affects study results, often in unpredictable ways. The inability to disentangle how much of an observed effect is attributable to the phenomenon of interest, and how much is attributable to what have come to be known as researcher degrees of freedom (RDF), slows theoretical progress and stymies practical implementation. However, if one could examine the results from a particular set of RDF (known as a universe) against a systematically and comprehensively determined background of alternative viable universes (known as a multiverse), then the effects of RDF can be directly examined to provide greater context and clarity to future researchers, and greater confidence in the recommendations to practitioners. This tutorial demonstrates a means to map result variability directly and efficiently, and empirically investigate RDF impact on conclusions via multiverse analysis. Using the R package multiverse, we outline best practices in planning, executing and interpreting of multiverse analyses.

RESUMEN

BACKGROUND: In 2020, the Lancet Commission identified 12 risk factors as priorities for prevention of dementia, and other studies identified APOE e4/e4 genotype and family history of Alzheimer's disease strongly associated with dementia outcomes; however, it is unclear how robust these relationships are across dementia subtypes and analytic scenarios. Specification curve analysis (SCA) is a new tool to probe how plausible analytical scenarios influence outcomes. METHODS: We evaluated the heterogeneity of odds ratios for 12 risk factors reported from the Lancet 2020 report and two additional strong associated non-modifiable factors (APOE e4/e4 genotype and family history of Alzheimer's disease) with dementia outcomes across 450,707 UK Biobank participants using SCA with 5357 specifications across dementia subtypes (outcomes) and analytic models (e.g., standard demographic covariates such as age or sex and/or 14 correlated risk factors). RESULTS: SCA revealed variable dementia risks by subtype and age, with associations for TBI and APOE e4/e4 robust to model specification; in contrast, diabetes showed fluctuating links with dementia subtypes. We found that unattributed dementia participants had similar risk factor profiles to participants with defined subtypes. CONCLUSIONS: We observed heterogeneity in the risk of dementia, and estimates of risk were influenced by the inclusion of a combination of other modifiable risk factors; non-modifiable demographic factors had a minimal role in analytic heterogeneity. Future studies should report multiple plausible analytic scenarios to test the robustness of their association. Considering these combinations of risk factors could be advantageous for the clinical development and evaluation of novel screening models for different types of dementia.

Asunto(s)

RESUMEN

OBJECTIVES: Sleep is associated with cognitive function in older adults. In the current study, we examined this relationship from subjective and objective perspectives, and determined the robustness and dimensional specificity of the associations using a comprehensive modelling approach. METHODS: Multiple dimensions of subjective (sleep quality and daytime sleepiness) and objective sleep (sleep stages, sleep parameters, sleep spindles, and slow oscillations), as well as subjectively reported and objectively measured cognitive function were collected from 55 older adults. Specification curve analysis was used to examine the robustness of correlations for the effects of sleep on cognitive function. RESULTS: Robust associations were found between sleep and objectively measured cognitive function, but not with subjective cognitive complaints. In addition, subjective sleep showed robust and consistent associations with global cognitive function, whereas objective sleep showed a more domain-specific association with episodic memory. Specifically, subjective sleep quality and daytime sleepiness correlated with global cognitive function, and objective sleep parameters correlated with episodic memory. CONCLUSIONS: Overall, associations between sleep and cognitive function in older adults depend on how they are measured and which specific dimensions of sleep and domains of cognitive function are considered. It highlights the importance of focusing on specific associations to ameliorate the detrimental effects of sleep disturbance on cognitive function in later life.

Asunto(s)

RESUMEN

The alcohol Stroop is a widely used task in addiction science to measure the theoretical concept of attentional bias (a selective attention to alcohol-related cues in the environment), which is thought to be associated with clinical outcomes (craving and consumption). However, recent research suggests findings from this task can be equivocal. This may be because the task has many different potential analysis pipelines, which increase researcher degrees of freedom when analysing data and reporting results. These analysis pipelines largely come from how outlying reaction times on the task are identified and handled (e.g. individual reaction times > 3 standard deviations from the mean are removed from the distribution; removal of all participant data if > 25% errors are made). We used specification curve analysis across two alcohol Stroop datasets using alcohol-related stimuli (one published and one novel) to examine the robustness of the alcohol Stroop effect to different analytical decisions. We used a prior review of this research area to identify 27 unique analysis pipelines. Across both data sets, the pattern of results was similar. The alcohol Stroop effect was present and largely robust to different analysis pipelines. Increased variability in the Stroop effect was observed when implementing outlier cut-offs for individual reaction times, rather than the removal of participants. Stricter outlier thresholds tended to reduce the size of the Stroop interference effect. These specification curve analyses are the first to examine the robustness of the alcohol Stroop to different analysis strategies, and we encourage researchers to adopt such analytical methods to increase confidence in their inferences across cognitive and addiction science.

Asunto(s)

RESUMEN

OBJECTIVES: To present an application of specification curve analysis-a novel analytic method that involves defining and implementing all plausible and valid analytic approaches for addressing a research question-to nutritional epidemiology. STUDY DESIGN AND SETTING: We reviewed all observational studies addressing the effect of red meat on all-cause mortality, sourced from a published systematic review, and documented variations in analytic methods (eg, choice of model, covariates, etc.). We enumerated all defensible combinations of analytic choices to produce a comprehensive list of all the ways in which the data may reasonably be analyzed. We applied specification curve analysis to data from National Health and Nutrition Examination Survey 2007 to 2014 to investigate the effect of unprocessed red meat on all-cause mortality. The specification curve analysis used a random sample of all reasonable analytic specifications we sourced from primary studies. RESULTS: Among 15 publications reporting on 24 cohorts included in the systematic review on red meat and all-cause mortality, we identified 70 unique analytic methods, each including different analytic models, covariates, and operationalizations of red meat (eg, continuous vs quantiles). We applied specification curve analysis to National Health and Nutrition Examination Survey, including 10,661 participants. Our specification curve analysis included 1208 unique analytic specifications, of which 435 (36.0%) yielded a hazard ratio equal to or more than 1 for the effect of red meat on all-cause mortality and 773 (64.0%) less than 1. The specification curve analysis yielded a median hazard ratio of 0.94 (interquartile range: 0.83-1.05). Forty-eight specifications (3.97%) were statistically significant, 40 of which indicated unprocessed red meat to reduce all-cause mortality and eight of which indicated red meat to increase mortality. CONCLUSION: We show that the application of specification curve analysis to nutritional epidemiology is feasible and presents an innovative solution to analytic flexibility.

Asunto(s)

RESUMEN

Why is the empirical evidence for birth-order effects on human psychology so inconsistent? In contrast to the influential view that competitive dynamics among siblings permanently shape a person's personality, we find evidence that these effects are limited to the family environment. We tested this context-specific learning hypothesis in the domain of risk taking, using two large survey datasets from Germany (SOEP, n = 19,994) and the United States (NLSCYA, n = 29,627) to examine birth-order effects on risk-taking propensity across a wide age range. Specification-curve analyses of a sample of 49,621 observations showed that birth-order effects are prevalent in children aged 10-13 years, but that they decline with age and disappear by middle adulthood. The methodological approach shows the effect is robust. We thus replicate and extend previous work in which we showed no birth-order effects on adult risk taking. We conclude that family dynamics cause birth-order effects on risk taking but that these effects fade as siblings transition out of the home.

Asunto(s)

RESUMEN

Purpose: Previous studies focusing on the relationship between adolescents' screen time and mental health have uncovered contradictory results. By focusing on smartphone use content (SUC), this study uses specification curve analysis to explore the different effects of SUCs on mental health-based on longitudinal data. Methods: A total of 2552 adolescents were surveyed in the first (July 2020) and second year (April 2021). A total of 2049 eligible participants (average age = 14.39 ± 2.27, female = 1062) are included in the analysis. Participants reported 20 types of content used by them during smartphone use and their mental health (depression, anxiety, and somatization). Specification curve analysis was used to examine the longitudinal relationship between SUCs and their mental health. Results: Smartphone use for listening to music (median ß = 0.18, p < 0.001, NSRPD = 25/27, p < 0.05), chatting online (median ß = 0.15, p < 0.001, NSRPD = 24/27, p < 0.05), watching TV (median ß = 0.14, p < 0.001, NSRPD = 24/27, p < 0.05), and playing games (median ß = 0.09, p < 0.001, NSRPD = 19/27, p < 0.05) produce high to medium negative effects on subsequent mental health. Only using smartphones for online courses exerts no effect on their subsequent mental health (median ß = 0.01, p > 0.05, NSRPD = 0/27, p > 0.05). The left 15 types of smartphone content showed unstable effects on future mental health. Depending on the types of content used, these effects ranged from high, medium, and small to none. The relatively descending order of effect on mental health is listening to music, chatting online, watching TV, playing games, and types of content (e.g., browsing social media, making payments, reading online novels) with high but unstable effects, types of content with medium (e.g., browsing news and posting/sharing) but unstable effects, types of content (e.g., using the camera, obtaining life information, and making calls) with small but unstable effects, such as finishing homework and taking online courses. Conclusions: This study enlightens researchers and policymakers to update their understanding of adolescents' technology use, especially to adopt a differentiated attitude towards different media use content. As nutritionists often do, a "nutritionally balanced" digital diet for young people should be recommended to the public, rather than just suggesting limits on the amount of time they can spend using digital media.

RESUMEN

The relationship between self-reported sleep and cognitive function is complex; it is unclear whether self-reported sleep is a robust correlate of people's cognitive function. We address this gap by using a comprehensive large-scale dataset (N = 1054) coupled with a novel modeling approach, specification curve analysis (SCA), to test the association between self-reported sleep and cognitive function. The results of the SCA showed robust correlations between self-reported sleep and cognitive function, with poorer sleep associated with worse cognitive function. Furthermore, the correlations between sleep components and cognitive function were heterogeneous, with differences emerging across cognitive measures and domains. Specifically, daytime dysfunction was associated with the strongest effect on subjective cognitive function, whereas sleep duration and sleep efficiency had the strongest effect on objective cognitive function. Therefore, the relationship between self-reported sleep and cognition depends largely on what and how cognitive function is measured. Our findings guide measurement and domain selection for future research on the role of sleep in cognitive function.

Asunto(s)

RESUMEN

In neuropsychological research, there are a near limitless number of different approaches researchers can choose when designing studies. Here we showcase the multiverse/specification curve technique to establish the robustness of analytical pathways choices within classic psychometric test validation in an example test of executive function. We examined the impact of choices regarding sample groups, sample sizes, test metrics, and covariate inclusions on convergent validation correlations between tests of executive function. Data were available for 87 neurologically healthy adults and 117 stroke survivors, and a total of 2,220 different analyses were run in a multiverse analysis. We found that the type of sample group, sample size, and test metric used for analyses affected validation outcomes. Covariate inclusion choices did not affect the observed coefficients in our analyses. The present analysis demonstrates the importance of carefully justifying every aspect of a psychometric test validation study a priori with theoretical and statistical factors in mind. It is essential to thoroughly consider the purpose and use of a new tool when designing validation studies.

Asunto(s)

RESUMEN

This manuscript contributes to a future definition of objectivity by bringing together recent statements in epistemology and methodology. It outlines how improved objectivity can be achieved by systematically incorporating multiple perspectives, thereby improving the validity of science. The more result-biasing perspectives are known, the more a phenomenon of interest can be disentangled from these perspectives. Approaches that call for the integration of perspective into objectivity at the epistemological level or that systematically incorporate different perspectives at the statistical level already exist and are brought together in the manuscript. Recent developments in research methodology, such as transparency, reproducibility of research processes, pre-registration of studies, or free access to raw data, analysis strategies, and syntax, promote the explication of perspectives because they make the entire research process visible. How the explication of perspectives can be done practically is outlined in the manuscript. As a result, future research programs can be organized in such a way that meta-analyses and meta-meta-analyses can be conducted not only backward but forward and prospectively as a regular and thus well-prepared part of objectification and validation processes.

RESUMEN

In educational large-scale assessment (LSA) studies such as PISA, item response theory (IRT) scaling models summarize students' performance on cognitive test items across countries. This article investigates the impact of different factors in model specifications for the PISA 2018 mathematics study. The diverse options of the model specification also firm under the labels multiverse analysis or specification curve analysis in the social sciences. In this article, we investigate the following five factors of model specification in the PISA scaling model for obtaining the two country distribution parameters; country means and country standard deviations: (1) the choice of the functional form of the IRT model, (2) the treatment of differential item functioning at the country level, (3) the treatment of missing item responses, (4) the impact of item selection in the PISA test, and (5) the impact of test position effects. In our multiverse analysis, it turned out that model uncertainty had almost the same impact on variability in the country means as sampling errors due to the sampling of students. Model uncertainty had an even larger impact than standard errors for country standard deviations. Overall, each of the five specification factors in the multiverse analysis had at least a moderate effect on either country means or standard deviations. In the discussion section, we critically evaluate the current practice of model specification decisions in LSA studies. It is argued that we would either prefer reporting the variability in model uncertainty or choosing a particular model specification that might provide the strategy that is most valid. It is emphasized that model fit should not play a role in selecting a scaling strategy for LSA applications.

RESUMEN

The past two decades have witnessed controversy over whether the use of digital technology has damaged or enhanced adolescents' social relationships, which influences their development. In this study, we addressed this debate by specifying the effect of different types of smartphone use content on social relationships, rather than simply relying on screen time spent on digital media. To avoid selective analysis and report of different variables, we used specification curve analysis (SCA) in a large dataset (N = 46,018) to explore the correlations between 20 types of smartphone use content and adolescents' social relationships (parent-child, peer, and teacher-student). The types of smartphone use content were measured by the revised version of Mobile Phone Use Pattern Scale, the Parent-Child Relationship Scale, the Peer Relationship Scale, and the Teacher-Student Relationship Scale assessed three different social relationships, respectively. Of the 20 types of smartphone use content, only playing games (negatively explaining 1% of the variation), taking online courses (positively explaining 1.6% of the variation), using search engines (positively explaining 1.2% of the variation), using a dictionary (positively explaining 1.3% of the variation), and obtaining life information (positively explaining 1.5% of the variation) showed a significant effect size. The association between smartphone use and adolescents' social relationships depends on the various types of content with which adolescents engage during smartphone use. The various effects of different types of smartphone use content deserve the attention of both the public and policy-makers.

Asunto(s)

RESUMEN

Brain size and IQ are positively correlated. However, multiple meta-analyses have led to considerable differences in summary effect estimations, thus failing to provide a plausible effect estimate. Here we aim at resolving this issue by providing the largest meta-analysis and systematic review so far of the brain volume and IQ association (86 studies; 454 effect sizes from k = 194 independent samples; N = 26 000+) in three cognitive ability domains (full-scale, verbal, performance IQ). By means of competing meta-analytical approaches as well as combinatorial and specification curve analyses, we show that most reasonable estimates for the brain size and IQ link yield r-values in the mid-0.20s, with the most extreme specifications yielding rs of 0.10 and 0.37. Summary effects appeared to be somewhat inflated due to selective reporting, and cross-temporally decreasing effect sizes indicated a confounding decline effect, with three quarters of the summary effect estimations according to any reasonable specification not exceeding r = 0.26, thus contrasting effect sizes were observed in some prior related, but individual, meta-analytical specifications. Brain size and IQ associations yielded r = 0.24, with the strongest effects observed for more g-loaded tests and in healthy samples that generalize across participant sex and age bands.

RESUMEN

BACKGROUND: Cluster analyses have become popular tools for data-driven classification in biological psychiatric research. However, these analyses are known to be sensitive to the chosen methods and/or modelling options, which may hamper generalizability and replicability of findings. To gain more insight into this problem, we used Specification-Curve Analysis (SCA) to investigate the influence of methodological variation on biomarker-based cluster-analysis results. METHODS: Proteomics data (31 biomarkers) were used from patients (n = 688) and healthy controls (n = 426) in the Netherlands Study of Depression and Anxiety. In SCAs, consistency of results was evaluated across 1200 k-means and hierarchical clustering analyses, each with a unique combination of the clustering algorithm, fit-index, and distance metric. Next, SCAs were run in simulated datasets with varying cluster numbers and noise/outlier levels to evaluate the effect of data properties on SCA outcomes. RESULTS: The real data SCA showed no robust patterns of biological clustering in either the MDD or a combined MDD/healthy dataset. The simulation results showed that the correct number of clusters could be identified quite consistently across the 1200 model specifications, but that correct cluster identification became harder when the number of clusters and noise levels increased. CONCLUSION: SCA can provide useful insights into the presence of clusters in biomarker data. However, SCA is likely to show inconsistent results in real-world biomarker datasets that are complex and contain considerable levels of noise. Here, the number and nature of the observed clusters may depend strongly on the chosen model-specification, precluding conclusions about the existence of biological clusters among psychiatric patients.

Asunto(s)

RESUMEN

Gaming disorder (also known as dysregulated gaming) has received significant research and policy attention based on concerns that certain patterns of play are associated with decreased mental well-being and/or functional impairment. In this study, we use specification curve analysis to examine analytical flexibility and the strength of the relationship between dysregulated gaming and well-being in the form of general mental health, depressive mood and life satisfaction. Dutch and Flemish gamers (n = 424) completed an online survey containing five unique dysregulated gaming measures (covering nine scale variants) and three well-being measures. We find a consistent negative relationship; across 972 justifiable regression models, the median standardized regression coefficient was -0.39 (min: -0.54, max: -0.19). Data show that the majority of dysregulated gaming operationalizations converge upon highly similar estimates of well-being. However, variance is introduced by the choice of well-being measure; results indicate that dysregulated gaming is more strongly associated with depressive mood than with life satisfaction. Weekly game time accounted for little to no unique variance in well-being in the sample. We argue that research on this topic should compare a broad range of psychosocial well-being outcomes and explore possible simplifications of the DSM-5 gaming disorder criteria. Given somewhat minute differences between dysregulated gaming scales when used in survey-based studies and largely equivalent relationships with mental health indicators, harmonization of measurement should be a priority.

RESUMEN

In vertebrates, large body size is often a key diagnostic feature of species threatened with extinction. However, in amphibians the link between body size and extinction risk is highly uncertain, with previous studies suggesting positive, negative, u-shaped, or no relationship. Part of the reason for this uncertainty is 'researcher degrees of freedom': the subjectivity and selectivity in choices associated with specifying and fitting models. Here, I clarify the size-threat association in amphibians using Specification Curve Analysis, an analytical approach from the social sciences that attempts to minimize this problem by complete mapping of model space. I find strong support for prevailing negative associations between body size and threat status, the opposite of patterns typical in other vertebrates. This pattern is largely explained by smaller species having smaller geographic ranges, but smaller amphibian species also appear to lack some of the life-history advantages (e.g. higher reproductive output) that are often assumed to 'protect' small species in other taxa. These results highlight the need for a renewed conservation focus on the smallest species of the world's most threatened class of vertebrates, as aquatic habitats become increasingly degraded by human activity.

Asunto(s)

RESUMEN

BACKGROUND: Throughout the developed world, adolescents are growing up with increased access to and engagement with a range of screen-based technologies, allowing them to encounter ideas and people on a global scale from the intimacy of their bedroom. The concerns about digital technologies negatively influencing sleep are therefore especially noteworthy, as sleep has been proven to greatly affect both cognitive and emotional well-being. The associations between digital engagement and adolescent sleep should therefore be carefully investigated in research adhering to the highest methodological standards. This understood, studies published to date have not often done so and have instead focused mainly on data derived from general retrospective self-report questionnaires. The value of this work has been called into question by recent research showing that retrospective questionnaires might fail to accurately measure these variables of interest. Novel and diverse approaches to measurement are therefore necessary for academic study to progress. METHODS: This study analyses data from 11,884 adolescents included in the UK Millennium Cohort Study to examine the association between digital engagement and adolescent sleep, comparing the relative effects of retrospective self-report vs. time-use diary measures of technology use. By doing so, it provides an empirical lens to understand the effects of digital engagement both throughout the day and before bedtime and adds nuance to a research area primarily relying on retrospective self-report. RESULTS: The study finds that there is a small negative association relating digital engagement to adolescent sleep both on weekdays and weekend days (median standardized association ßweekday = -0.06 and ßweekend = -0.03). There is a more negative association between digital engagement and total sleep time on weekdays compared to weekend days (median standardized ßweekday = -0.08, median standardized ßweekend = -0.02), while there is no such difference when examining adolescents' bedtime. Surprisingly, and contrary to our expectations, digital technology use before bedtime is not substantively associated with the amount of sleep and the tardiness of bedtime in adolescents. CONCLUSIONS: Results derived from the use of transparent Specification Curve Analysis methods show that the negative associations in evidence are mainly driven by retrospective technology use measures and measures of total time spent on digital devices during the day. The effects are overall very small: for example, an additional hour of digital screen time per day was only related to a 9 min decrease in total time spent sleeping on weekdays and a 3 min decrease on weekends. Using digital screens 30 min before bed led to a 1 min decrease in total time spent sleeping on weekdays and weekends. The study shows that more work should be done examining how to measure digital screen time before interventions are designed.

RESUMEN

The notion that digital-screen engagement decreases adolescent well-being has become a recurring feature in public, political, and scientific conversation. The current level of psychological evidence, however, is far removed from the certainty voiced by many commentators. There is little clear-cut evidence that screen time decreases adolescent well-being, and most psychological results are based on single-country, exploratory studies that rely on inaccurate but popular self-report measures of digital-screen engagement. In this study, which encompassed three nationally representative large-scale data sets from Ireland, the United States, and the United Kingdom ( N = 17,247 after data exclusions) and included time-use-diary measures of digital-screen engagement, we used both exploratory and confirmatory study designs to introduce methodological and analytical improvements to a growing psychological research area. We found little evidence for substantial negative associations between digital-screen engagement-measured throughout the day or particularly before bedtime-and adolescent well-being.

Asunto(s)

RESUMEN

The idea that birth-order position has a lasting impact on personality has been discussed for the past 100 years. Recent large-scale studies have indicated that birth-order effects on the Big Five personality traits are negligible. In the current study, we examined a variety of more narrow personality traits in a large representative sample ( n = 6,500-10,500 in between-family analyses; n = 900-1,200 in within-family analyses). We used specification-curve analysis to assess evidence for birth-order effects across a range of models implementing defensible yet arbitrary analytical decisions (e.g., whether to control for age effects or to exclude participants on the basis of sibling spacing). Although specification-curve analysis clearly confirmed the previously reported birth-order effect on intellect, we found no meaningful effects on life satisfaction, locus of control, interpersonal trust, reciprocity, risk taking, patience, impulsivity, or political orientation. The lack of meaningful birth-order effects on self-reports of personality was not limited to broad traits but also held for more narrowly defined characteristics.

Asunto(s)