RESUMEN
An intervention's effectiveness is judged by whether it produces positive outcomes for participants, with the randomized experiment being the gold standard for determining intervention effects. However, the intervention-as-implemented in an experiment frequently differs from the intervention-as-designed, making it unclear whether unfavorable results are due to an ineffective intervention model or the failure to implement the model fully. It is therefore vital to accurately and systematically assess intervention fidelity and, where possible, incorporate fidelity data in the analysis of outcomes. This paper elaborates a five-step procedure for systematically assessing intervention fidelity in the context of randomized controlled trials (RCTs), describes the advantages of assessing fidelity with this approach, and uses examples to illustrate how this procedure can be applied.
Asunto(s)
Guías como Asunto , Evaluación de Procesos y Resultados en Atención de Salud/métodos , Psicoterapia/métodos , Ensayos Clínicos Controlados Aleatorios como Asunto , Práctica Clínica Basada en la Evidencia , Humanos , Evaluación de Programas y Proyectos de Salud , Reproducibilidad de los ResultadosRESUMEN
The Career Award in Biomedical Sciences (CABS) was launched by the Burroughs Wellcome Fund to assist talented scientists to cross the "bridge" that connects postdoctoral training with independent investigator status. This study examines the causal effects of CABS, using both conventional methods and a newer method (propensity score analysis) in attempting to account for selection biases (e.g., selecting the "best and the brightest") in estimating the award's impact. The results indicated that grantees outperformed unsuccessful applicants in obtaining a faculty position, competing for an R01 grant, and publishing in prestigious journals. However, it was nearly impossible to remove all the effects of selection bias from estimates of program effects. Because selection bias is a pervasive problem in assessing these programs, recommendations for improving evaluations of similar merit-based programs are offered.
Asunto(s)
Distinciones y Premios , Investigación Biomédica/normas , Movilidad Laboral , Adulto , Selección de Profesión , Femenino , Humanos , MasculinoRESUMEN
The authors used a pretest-posttest control group design with random assignment to evaluate whether early reading failure decreases children's motivation to practice reading. First, they investigated whether 60 first-grade children would report substantially different levels of interest in reading as a function of their relative success or failure in learning to read. Second, they evaluated whether increasing the word reading ability of 15 at-risk children would lead to gains in their motivation to read. Multivariate analyses of variance suggest marked differences in both motivation and reading practice between skilled and unskilled readers. However, bolstering at-risk children's word reading ability did not yield evidence of a causal relationship between early reading failure and decreased motivation to engage in reading activities. Instead, hierarchical regression analyses indicate a covarying relationship among early reading failure, poor motivation, and avoidance of reading.
Asunto(s)
Dislexia/psicología , Motivación , Niño , Dislexia/diagnóstico , Dislexia/prevención & control , Intervención Educativa Precoz , Femenino , Humanos , Masculino , Índice de Severidad de la EnfermedadRESUMEN
This article summaries some of the project-level assessment and evaluation activities that have been used to assess the value added for innovations that incorporate learning science principles and learning technology underlying the HPL model proposed by Bransford et al. [1]. Based on a counterfactual model of causal analysis, VaNTH investigators have been encouraged to use experimental and quasi-experimental research designs to estimate the "value added" for their innovations. Although some studies have been undertaken with little difficulty, practical constraints have resulted in the need to allow quasi-experimental designs to emerge over time and institutions. By applying the logic, principles, and criteria of a counterfactual causal model, as opposed to a "cookbook" application of designs and statistical procedures, VaNTH investigators have begun to develop a firm knowledge base about the relative effectiveness of their HPL-inspired innovations. The overall effects and differences among studies are being investigated through the use of a statistical technique known as meta-analysis. This article demonstrates that it is possible to assess and evaluate, in a quantitative way, the relative effects of educational innovations in engineering courses. It also demonstrates, albeit in a preliminary fashion, that a broader assessment of the HPL model underlying VaNTH can be undertaken by systematically looking across studies within VaNTH. By implication, the knowledge gained about engineering education from A&E efforts within VaNTH should be much greater than the sum of its parts.