RESUMEN
OBJECTIVE: Test the automation transparency design principle using a full-scope nuclear power plant simulator. BACKGROUND: Automation transparency is a long-held human factors design principle espousing that the responsibilities, capabilities, goals, activities, and/or effects of automation should be directly observable in the human-system interface. The anticipated benefits of transparency include more effective reliance, more appropriate trust, better understanding, and greater user satisfaction. Transparency has enjoyed a recent upsurge in use in the context of human interaction with agent-oriented automation. METHOD: Three full-scope nuclear power plant simulator studies were conducted with licensed operating crews. In the first two experiments, transparency was implemented for interlocks, controllers, limitations, protections, and automatic programs that operate at the local component level of the plant. In the third experiment, procedure automation assumed control of plant operations and was represented in dedicated agent displays. RESULTS: Results from Experiments 1 and 2 appear to validate the human performance benefits of automation transparency for automation at the component level. However, Experiment 3 failed to replicate these findings for automation that assumed control for executing procedural actions. CONCLUSION: Automation transparency appears to yield expected benefits for component-level automation, but caution is warranted in generalizing the design principle to agent-oriented automation. APPLICATION: The automation transparency design principle may offer a powerful means of compensating for the detrimental impacts of hidden automation influence at the component level of complex systems. However, system developers should exercise caution in assuming that the principle extends to agent-oriented automation.
Asunto(s)
Sistemas Hombre-Máquina , Análisis y Desempeño de Tareas , Automatización , Humanos , Plantas de Energía Nuclear , ConfianzaRESUMEN
OBJECTIVE: This article is a response to Wickens et al.'s (2019) critique of Jamieson and Skraaning (2019). BACKGROUND: Wickens et al. (2019) offer a five-point critique of Jamieson and Skraaning (2019) that they claim tempers the strength of our conclusions. APPROACH: We first correct a misrepresentation in the critique and then respond to each of the criticisms. RESULTS: We preserve the strength of our skeptical conclusions about the applicability of the lumberjack model to complex work settings. APPLICATIONS: We continue to caution system designers about the lack of evidence supporting the lumberjack model in the context of complex work systems.
Asunto(s)
Automatización , Concienciación , Sistemas Hombre-Máquina , Lugar de Trabajo , Predicción , Humanos , Análisis y Desempeño de Tareas , Carga de TrabajoRESUMEN
OBJECTIVE: The objective of this study was to test the predictions of the routine-failure trade-off (or lumberjack) model in a full-scope simulator study with expert operators performing realistic control tasks. BACKGROUND: A meta-study of degree of automation (DOA) studies concluded that DOA predicts task performance under both routine and automation failure conditions, workload, and situation awareness. Empirical support for this conclusion appears to be weak for complex work situations. METHOD: A full-scope nuclear power plant simulator experiment was conducted in which licensed operating crews completed realistic procedure execution tasks. Dependent measures selected from the lumberjack model were collected and analyzed for systematic effects. RESULTS: Situation awareness increased with increasing DOA, which contradicts the lumberjack model. Anticipated workload and failure task performance effects were not observed. CONCLUSION: The experimental results add further evidence challenging the applicability of the lumberjack model to complex work situations. APPLICATION: Practitioners should use caution when extending the predictions of the lumberjack model based on data from simple work situations to complex work situations. Researchers should invest more resources in testing the predictive power of the lumberjack model in complex work situations.
Asunto(s)
Automatización , Sistemas Hombre-Máquina , Lugar de Trabajo , Adulto , Humanos , Persona de Mediana Edad , Plantas de Energía Nuclear , Suecia , Análisis y Desempeño de Tareas , Carga de TrabajoRESUMEN
The Process Overview Measure is a query-based measure developed to assess operator situation awareness (SA) from monitoring process plants. A companion paper describes how the measure has been developed according to process plant properties and operator cognitive work. The Process Overview Measure demonstrated practicality, sensitivity, validity and reliability in two full-scope simulator experiments investigating dramatically different operational concepts. Practicality was assessed based on qualitative feedback of participants and researchers. The Process Overview Measure demonstrated sensitivity and validity by revealing significant effects of experimental manipulations that corroborated with other empirical results. The measure also demonstrated adequate inter-rater reliability and practicality for measuring SA in full-scope simulator settings based on data collected on process experts. Thus, full-scope simulator studies can employ the Process Overview Measure to reveal the impact of new control room technology and operational concepts on monitoring process plants. Practitioner Summary: The Process Overview Measure is a query-based measure that demonstrated practicality, sensitivity, validity and reliability for assessing operator situation awareness (SA) from monitoring process plants in representative settings.
Asunto(s)
Concienciación , Plantas de Energía Nuclear , Análisis y Desempeño de Tareas , Humanos , Investigación Operativa , Reproducibilidad de los ResultadosRESUMEN
We introduce Process Overview, a situation awareness characterisation of the knowledge derived from monitoring process plants. Process Overview is based on observational studies of process control work in the literature. The characterisation is applied to develop a query-based measure called the Process Overview Measure. The goal of the measure is to improve coupling between situation and awareness according to process plant properties and operator cognitive work. A companion article presents the empirical evaluation of the Process Overview Measure in a realistic process control setting. The Process Overview Measure demonstrated sensitivity and validity by revealing significant effects of experimental manipulations that corroborated with other empirical results. The measure also demonstrated adequate inter-rater reliability and practicality for measuring SA based on data collected by process experts. Practitioner Summary: The Process Overview Measure is a query-based measure for assessing operator situation awareness from monitoring process plants in representative settings.
Asunto(s)
Concienciación , Investigación Operativa , Conducta , Toma de Decisiones , Humanos , Reproducibilidad de los Resultados , Análisis y Desempeño de TareasRESUMEN
Query- or probe-based situation awareness (SA) measures sometimes rely on process experts to evaluate operator actions and system states when used in representative settings. This introduces variability of human judgement into the measurements that require inter-rater reliability assessment. However, the literature neglects inter-rater reliability of query/probe-based SA measures. We recruited process experts to provide reference keys to SA queries in trials of a full-scope nuclear power plant simulator experiment to investigate the inter-rater reliability of a query-based SA measure. The query-based SA measure demonstrated only 'moderate' inter-rater reliability even though the queries were seemingly direct. The level of agreement was significantly different across pairs of experts who had different levels of exposure to the experiment. The results caution that inter-rater reliability of query/probe-based techniques for measuring SA cannot be assumed in representative settings. Knowledge about the experiment as well as the domain is critical to forming reliable expert judgements. PRACTITIONER SUMMARY: When the responses of domain experts are treated as the correct answers to the queries or probes of SA measures used in representative or industrial settings, practitioners should take caution in assuming (or otherwise assess) inter-rater reliability of the situation awareness measures.
Asunto(s)
Concienciación , Simulación por Computador , Variaciones Dependientes del Observador , Análisis y Desempeño de Tareas , Investigación Empírica , Humanos , Plantas de Energía Nuclear/normas , Psicometría/métodos , Reproducibilidad de los ResultadosRESUMEN
OBJECTIVE: We determine whether an ecological interface display for nuclear power plant operations supports improved situation awareness over traditional and user-centered displays in a realistic environment. BACKGROUND: Ecological interface design (EID) has not yet been fully evaluated with real operators facing realistic scenarios. METHOD: Ecological displays were evaluated alongside traditional and user-centered "advanced" displays in a full-scope nuclear power plant simulation. Licensed plant operators used the displays in realistic scenarios that either had procedural support or did not have procedural support. All three displays were evaluated for their ability to support operator situation awareness. RESULTS: A significant three-way interaction effect was observed on two independent measures of situation awareness. For both measures, ecological displays improved situation awareness in scenarios that did not have procedural support, primarily in the detection phases of those scenarios. No other pronounced effects appeared across both measures. CONCLUSIONS: The observed improvement was sufficiently large to suggest that EID could improve situation awareness in situations where procedures are unavailable. However, the EID displays did not lead to improved situation awareness in the other conditions of the evaluation, and participants using these displays occasionally underperformed on single measures of situation awareness. This suggests that the approach requires further development, particularly in integrating EID with procedural support. APPLICATION: This research has important findings for the ongoing development of the EID approach, the design of industrial operator displays, and design to support situation awareness.